A method and system for large-scale diffusion of Transformer elastic inference for volatile GPU resources
By dynamically adjusting pipeline parallelism and rebuilding communication groups, the unreliable topology switching problem of the diffusion Transformer inference system in a volatile GPU resource environment is solved, achieving low-overhead model parameter migration and time step granularity recovery, thus improving the system's stability and resource utilization efficiency.
Patent Information
- Application Number
- CN202610950217.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-29
- Publication Date
- 2026-11-17
AI Technical Summary
Existing diffusion Transformer inference systems struggle to dynamically adjust pipelined parallel topology, migrate model parameters with low overhead, rebuild communication groups, and support time-step granular recovery in volatile GPU resource environments. This results in cold start recovery, high overhead for parameter loading input and output, unreliable communication topology switching, and scheduling strategies that fail to balance global utility and service stability.
By acquiring model state and GPU resource events, the pipeline parallelism is dynamically adjusted, a target parallel configuration is generated, a global layer identifier is assigned, a distributed communication group is rebuilt, parameter hot migration and topology consistency verification are performed, scaling up and down processing is achieved, and time step granularity recovery is supported.
It avoids a full restart of the inference cluster, reduces overhead from remote storage or disk I/O, lowers the risk of inference errors and service interruptions caused by topology switching, and improves the marginal utility of resource allocation and service stability.
Smart Images

Figure CN123114640A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence inference services, distributed model parallelism, cloud computing resource scheduling, and elastic fault tolerance, and particularly to a large-scale diffusion Transformer elastic inference method and system for volatile GPU resources. Background Technology
[0002] Diffusion Transformer models have become an important model architecture in multimedia generation tasks such as image generation and video generation. These models typically divide the input noise latent variables into multiple image patches or spatiotemporal markers and perform iterative denoising through a multi-layer Transformer structure. The diffusion Transformer can include image diffusion Transformer, video diffusion Transformer, or other hierarchical Transformer generative models with multi-step denoising processes. As the input resolution, video frame rate, model parameter size, and sampling steps increase, a single inference request requires significant GPU computing power and video memory resources. Online service systems typically need to meet throughput and latency requirements through pipelined parallelism, sequence parallelism, tensor parallelism, or combinations thereof.
[0003] Existing parallel inference schemes for diffusion models mostly employ static parallel topologies. Tensor parallelism requires partitioning the weight matrix within operators and performing frequent synchronization communication within Transformer blocks, which can easily introduce communication bottlenecks. Pipeline parallelism can partition the model hierarchically, reducing operator-level synchronization overhead, but traditional solutions often fix the model partitioning method, the number of pipeline stages, and the communication group topology during the service initialization phase. When the number of GPU resources changes, the system typically needs to destroy the original service instance, reload the model parameters from disk or remote storage, and reinitialize the distributed communication environment, resulting in high cold start latency and long service interruption times.
[0004] In cloud-native environments, spot instances, preemptible instances, shared GPU resource pools, container orchestration platform node eviction, and GPU failure recovery can all lead to the volatility of GPU resources. The reclamation of a single GPU node can disrupt the distributed pipeline, while new GPUs may appear in a short period of time. If the inference system cannot adjust the pipeline parallelism at runtime, it is difficult to simultaneously obtain low-cost resources and a stable service level.
[0005] In addition, multiple models are usually deployed simultaneously in multi-tenant diffusion Transformer inference clusters. The throughput improvement of different models under different GPU numbers, batch sizes, resolutions and accuracies may exhibit non-linear, non-convex, step-like or diminishing marginal returns. Simple random scheduling, longest queue priority or single-step greedy strategies are difficult to obtain a globally optimal resource allocation scheme in the event of continuous GPU arrival or continuous GPU failure.
[0006] Traditional checkpointing mechanisms typically save the complete model state or runtime state periodically, which is costly. Diffusion Transformer inference has a multi-time-step denoising structure. If the latent state, sampler state, and conditional context can be saved at the time-step boundaries, recalculation from the initial noise can be avoided. However, existing static parallel systems struggle to coordinate this time-step granularity recovery mechanism with pipeline topology changes.
[0007] Therefore, a resilient inference scheme is needed that can dynamically adjust the parallel topology of the diffusion Transformer pipeline, migrate model parameters with low overhead, rebuild communication groups, verify topology consistency, and support time-step granular recovery in volatile GPU resource environments. Summary of the Invention
[0008] To address the problems of existing diffusion Transformer inference systems, such as the need for cold start recovery during GPU resource preemption or release, high overhead for parameter loading input and output, unreliable communication topology switching, and difficulty in balancing global utility and service stability in scheduling strategies, this invention provides a large-scale diffusion Transformer elastic inference method and system for volatile GPU resources.
[0009] In a first aspect, the present invention provides a large-scale diffusion Transformer elastic inference method for volatile GPU resources, comprising the following steps:
[0010] Obtain model status, GPU resource events, and online runtime metrics for diffusion-oriented Transformer inference models;
[0011] In response to the GPU resource event, the target pipeline parallelism is determined based on the model state and the online runtime metrics;
[0012] A target parallel configuration is derived based on the target pipeline parallelism.
[0013] Based on the diffusion Transformer inference model, a global layer identifier is assigned to each Transformer block, and the corresponding target model hierarchy partitioning, target parameter fragmentation layout, and target communication topology are generated according to the target pipeline parallelism.
[0014] Compare the current parameter fragment layout with the target parameter fragment layout, generate an enhanced parameter migration plan, and determine the set of reusable parameters, the set of parameters to be migrated, the set of parameters to be loaded, and the set of parameters to be released;
[0015] Based on the set of parameters to be migrated, hot migration of parameters is performed on the parameter fragments that have been loaded into the existing GPU memory, so as to redistribute the parameter fragments to the target GPU set.
[0016] Based on the numerical relationship between the target pipeline parallelism and the current pipeline parallelism, perform expansion or reduction processing.
[0017] The distributed communication group is reconstructed based on the target communication topology, and the inference service of the inference model is restored or extended based on the reconstructed distributed communication group after the topology consistency check and parameter integrity check pass.
[0018] Secondly, the present invention provides a large-scale diffusion Transformer elastic inference system for volatile GPU resources, comprising:
[0019] The status acquisition module is used to acquire the model status, GPU resource events, and online running metrics of the diffusion Transformer inference model.
[0020] The elastic scheduling module is used to respond to the GPU resource event and determine the target pipeline parallelism based on the model state and the online running metrics;
[0021] The topology planning module is used to derive the target parallel configuration based on the target pipeline parallelism, and at the same time, based on the diffusion Transformer inference model, assign global layer identifiers to each Transformer block, and generate the corresponding target model hierarchy partitioning, target parameter fragmentation layout and target communication topology according to the target pipeline parallelism.
[0022] The remote execution node agent module is used to uniformly schedule the initialization, model loading, parameter migration, inference execution, status query and destruction of remote GPU worker nodes;
[0023] The parameter hot migration module is used to generate a parameter migration plan based on the current parameter piece layout and the target parameter piece layout, and to perform parameter tensor dictionary transfer, missing layer construction, parameter loading and source unloading through the GPU communication group.
[0024] The communication reconstruction module is used to set up computing barriers, destroy failed communicators, rebuild communication groups, issue new distributed initialization endpoints, and reinitialize the runtime state when the topology changes.
[0025] The version verification module is used to maintain the topology version number and service status, and to perform runtime status consistency verification, communication connectivity verification, and parameter integrity verification.
[0026] An interruptible inference module is used to save the current time step state of the inference task when a volatile GPU instance termination notification is detected, and to resume inference on a new node or a new topology.
[0027] The request routing module is used to route inference requests to serviceable model instances based on model service status, topology version, and queue status.
[0028] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method for large-scale diffusion of Transformer elastic inference for volatile GPU resources.
[0029] The present invention has the following beneficial effects:
[0030] First, this invention avoids a full restart of the inference cluster when GPU resources are preempted or released by dynamically adjusting the pipeline parallelism at runtime.
[0031] Second, the present invention reuses parameter fragments already loaded in GPU memory by hot migration of parameters based on the difference between the current layout and the target layout, thereby reducing the overhead of remote storage or disk input / output.
[0032] Third, this invention reduces the risk of resource deadlock and resource waiting by using incremental resource requests, eliminating the need to re-apply for complete cluster resources when expanding capacity;
[0033] Fourth, this invention reduces the risk of inference errors and service interruptions caused by topology switching through service state machines, topology version numbers, communication group reconstruction and verification mechanisms;
[0034] Fifth, this invention uses asymmetric marginal utility scheduling based on health status perception without offline profiles to protect vulnerable models by utilizing resource saturation barriers during scaling down, and to improve resource allocation benefits based on queue pressure, service level gap, load trend, and unit resource load during scaling up.
[0035] Sixth, this invention reduces task recalculation overhead by using time-step interruptible inference to save the minimum necessary state when a volatile GPU instance is about to terminate. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings are briefly described below.
[0037] Figure 1 A flowchart of a large-scale diffusion Transformer elastic inference method provided in an embodiment of the present invention;
[0038] Figure 2 A schematic diagram of a flexible inference system architecture provided in an embodiment of the present invention;
[0039] Figure 3 A parameter thermal migration timing diagram provided in an embodiment of the present invention;
[0040] Figure 4 A flowchart of asymmetric marginal utility scheduling is provided for an embodiment of the present invention;
[0041] Figure 5 A schematic diagram of a topology switching state machine provided in an embodiment of the present invention;
[0042] Figure 6 This invention provides a flowchart for interruptible reasoning recovery.
[0043] Figure 7 A schematic diagram of an electronic device structure provided in an embodiment of the present invention;
[0044] Figure 8 A Ray expansion execution flowchart is provided as an embodiment of the present invention;
[0045] Figure 9 This is a schematic diagram of a multi-model, multi-round logical iterative scheduling method provided in an embodiment of the present invention. Detailed Implementation
[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. The described embodiments are only some embodiments of the present invention, and not all embodiments.
[0047] Example 1: Method Example
[0048] like Figure 1 As shown, this embodiment provides a method for large-scale diffusion of Transformer elastic inference for volatile GPU resources. This method can be applied to diffusion of Transformer online inference services deployed in spot instances, preemptible instances, shared GPU pools, container orchestration platform GPU nodes, or other dynamic GPU resource pools, and specifically includes the following steps.
[0049] S110. Obtain the inference cluster status
[0050] In this embodiment, the system periodically or event-triggeredly collects the model state of each inference model, which can be represented as:
[0051]
[0052] in, For model identification, This represents the number of GPUs or pipeline stages currently allocated to the i-th model. To smooth out load intensity, For resource saturation, This is the length of the backlog queue. This refers to the current throughput capacity or actual throughput. As a delay indicator, Based on recent load trends, This is the topology version number. In service status.
[0053] Furthermore, in this embodiment, the smoothed load intensity is updated using an exponential moving average (EMA), where the EMA is used to smooth the request arrival rate within the observation window to reduce the impact of instantaneous traffic fluctuations on scheduling decisions.
[0054]
[0055] in, The request arrival rate within the current observation window. This is the attenuation factor.
[0056] Furthermore, in this embodiment, resource saturation can be defined as:
[0057]
[0058] in, This refers to the current throughput capacity or actual throughput. To meet the target throughput requirements; when When this occurs, it indicates that the model has a risk of defaulting on service level objectives.
[0059] S120, Monitor GPU resource events
[0060] In this embodiment, GPU resource events include GPU preemption events, GPU release events, GPU addition events, GPU failure events, and GPU recovery events.
[0061] GPU preemption events can originate from cloud platform auction instance termination notifications, container orchestration platform node eviction notifications, abnormal GPU health checks, or recycling instructions from the underlying resource management system; GPU release or addition events can originate from successful application for a new auction instance, GPU node recovery, or resource pool expansion.
[0062] S130. Determine the target pipeline parallelism.
[0063] In some embodiments, the system checks whether the target pipeline parallelism is valid before scaling up or down. For example, the target pipeline parallelism should be greater than 0 and not exceed the number of available GPUs or the number of GPUs that can be requested in the current cluster. When the target pipeline parallelism is the same as the current pipeline parallelism, the system can directly return to the current configuration without performing physical topology changes.
[0064] Furthermore, such as Figure 4 As shown, when a GPU resource event is a preemption event or a failure event, the system performs scaling down scheduling. The goal of scaling down scheduling is to prioritize reducing the overall service level default risk when it is necessary to reduce the number of GPUs. The system performs multiple rounds of logical iterations on the number of GPUs reduced due to GPU preemption events or GPU failure events. In each round of logical iteration, the system selects the model with the lowest scaling down cost based on the online scaling down marginal loss proxy and the resource saturation barrier term, performs virtual scaling down, and updates the virtual state.
[0065] The cost of reducing capacity is calculated using the following formula:
[0066]
[0067] in, The normalized load heat of the i-th inference model. This is a proxy for the online scaling-down marginal loss calculated based on online operating metrics. For resource saturation, To prevent the lower limit of division by zero, , and Configurable weights.
[0068] Furthermore, when a GPU resource event is a release, addition, or recovery event, the system performs expansion scheduling. The goal of expansion scheduling is to prioritize the allocation of new resources to models with high marginal benefits from online expansion, severe queue backlogs, rising load trends, or unmet service level targets, when the number of new GPUs is limited.
[0069] The expansion revenue is calculated using the following formula:
[0070]
[0071] in, For the online expansion marginal revenue agent calculated based on online operation indicators, To normalize the backlog queue length, For resource saturation, , , and Configurable weights.
[0072] Furthermore, in this embodiment, the unit resource load can be expressed as:
[0073]
[0074] in, Used to indicate the first The request load borne by a single GPU or a single pipeline stage in an inference model.
[0075] Furthermore, the online scaling-down marginal loss agent can be expressed as:
[0076]
[0077] Furthermore, the marginal revenue agent for online scaling can be represented as:
[0078]
[0079] in, To normalize the queue length, To normalize the delay pressure, To normalize the load trend, to and to This allows for configurable weights. This method eliminates the need for the system to pre-build an offline lookup table representing the number of GPUs to the theoretical throughput.
[0080] S140, Parallel Configuration of Derivative Targets
[0081] In this embodiment, the system generates a new parallel configuration based on the target pipeline parallelism. The parallel configuration includes pipeline parallelism, the number of pipeline slices, the number of global processes, and the scale of diffusion Transformer parallelism.
[0082] Furthermore, in this embodiment, the target pipeline parallelism can be set to 3. The system updates the pipeline parallelism configuration to this target value, sets the number of pipeline slices to 3, and recalculates the global process count based on the product of tensor parallelism, sequence parallelism, classifier free-guided parallelism, data parallelism, and target pipeline parallelism. Then, the diffusion Transformer parallelism scale is set to the global process count.
[0083] Furthermore, the system in this embodiment also synchronously updates the diffusion Transformer parallel scale in the pipeline parallel, data parallel, sequence parallel and tensor parallel sub-configurations to ensure that all working nodes have a consistent configuration view of the target topology.
[0084] Furthermore, the system in this embodiment can generate the target parallel configuration by deep copying the current parallel configuration, modifying only the fields related to the target pipeline parallelism, thereby preserving the original tensor parallelism, sequence parallelism, data parallelism, classifier free-guided parallelism, runtime configuration, and model configuration.
[0085] S150, Generation of target model hierarchical division
[0086] In this embodiment, the diffusion Transformer model typically includes multiple sequentially stacked Transformer blocks. The system assigns a global layer identifier to each Transformer block and divides the global layer identifier into multiple consecutive intervals according to the target pipeline parallelism; the basic implementation can divide it evenly according to the number of layers.
[0087] Furthermore, the system can divide the total number of global layers by the target pipeline parallelism to obtain the base layer number of each worker node, and then take the remainder based on the total number of global layers divided by the target pipeline parallelism. The number of layers allocated to the j-th pipeline stage is either the base layer number plus one or the base layer number, and the remainder is preferentially allocated to smaller stage numbers.
[0088] Furthermore, in this embodiment, the system can also utilize parameter size, activation size, or communication overhead for load balancing. For example, when some Transformer layers have higher parameter sizes or higher activation communication overhead, the corresponding stage can hold fewer layers to reduce pipeline bubbles.
[0089] S160, Generate parameter migration plan
[0090] like Figure 3 As shown, in this embodiment, the system obtains the global layer identifier set that each current working node is responsible for, and forms the current parameter sharding layout.
[0091] In one implementation, the worker node scans the model state dictionary for parameter keys containing global layer identifiers, such as Transformer block parameter keys carrying global layer identifiers or Transformer block parameter keys with module prefixes and carrying global layer identifiers, extracts the set of global layer identifiers, and returns the current layer assignment in the form of a mapping between pipeline stage numbers and the set of global layer identifiers.
[0092] In this embodiment, the current layer allocation can be represented as: Phase 0 holds layers 0 to a, and Phase 1 holds layers a+1 to b.
[0093] In this embodiment, the target layer allocation can be represented as follows: Phase 0 holds layers 0 to c, Phase 1 holds layers c+1 to d, and Phase 2 holds layers d+1 to b. Then, a mapping from global layer identifiers to the current phase and a mapping index from global layer identifiers to target phases are established. For each global layer, if the target phase number differs from the current phase number, the corresponding global layer set is aggregated using the target phase and the current phase as keys, and a migration task is generated.
[0094] Furthermore, the migration task may include the receiver stage number, the sender stage number, and a set of global layer identifiers to be migrated.
[0095] Through this step, the system in this embodiment only generates migration tasks for layers where the target layout is missing and the current layout already exists, retaining reusable layers and avoiding full reload.
[0096] S170, Execution Parameter Hot Migration
[0097] like Figure 3 As shown, for each transfer task, the sender filters parameter tensors from the model state dictionary based on the global layer identifier. Parameter keys can include Transformer block parameter keys carrying the global layer identifier, Transformer block parameter keys with a module prefix and carrying the global layer identifier, or equivalent prefixes. The sender packages the parameter tensors in the same transfer task into a tensor dictionary and sends it to the receiver through the GPU communication group participating in the transfer.
[0098] In one embodiment, the sender calls the point-to-point tensor dictionary sending interface to send the parameter tensor dictionary, and the receiver calls the corresponding receiving interface to receive the parameter tensor dictionary. This transmission path does not go through the file system disk transfer.
[0099] Further, the receiver parses the global layer identifier in the tensor key; if the corresponding Transformer block does not exist locally, it constructs the missing block according to the model configuration and loads the received parameter tensor into it. For example, for a stable diffusion Transformer of the third generation model type, the receiver can construct the missing Transformer block based on the hidden dimension, number of attention heads, attention head dimension, normalization method, set of dual attention layers, and total number of layers in the model configuration, and determine whether the layer is the last layer and whether dual attention is enabled based on the global layer identifier. Subsequently, the receiver moves the received parameter tensor to the target device and writes it into the target model in a partial loading manner.
[0100] After the migration is complete, the sender can delete the Transformer blocks that it is no longer responsible for locally and release the GPU memory cache. Migration tasks are processed in the same order according to their sequence numbers, and a communication barrier is executed after each task to reduce the risk of deadlock in point-to-point communication.
[0101] In some embodiments, migration tasks are executed serially, with barrier synchronization performed after each migration task is completed, and GPU synchronization performed after all migration tasks are completed. For GPU topologies with uneven communication costs, the system can calculate the total migration cost based on the inter-GPU communication cost matrix and use the Hungarian algorithm, dynamic programming, or heuristic sorting to reduce the total migration cost.
[0102] In some embodiments, after loading the parameters, the receiver checks whether the sent parameter keys have actually been loaded into the model. If the sent parameter keys are still reported as missing, or the global layer specified by the migration task does not exist on the receiver, the migration is deemed to have failed. The system can also collect parameter summaries for each global layer before and after the topology change, such as parameter sums, maximum absolute values, number of parameters, device and data type, and compare the differences in the global layer set, number of parameters, device, data type and parameter summaries after the topology change to determine whether the target topology is serviceable.
[0103] S180, Capacity Expansion Processing
[0104] Furthermore, when the target pipeline parallelism is greater than the current pipeline parallelism, the system calculates the number of new diffusion Transformer worker nodes, requests incremental GPU resources only for the new nodes, creates new remote execution nodes, assigns them global stage sequence numbers and GPU resources, and adds them to the worker node list.
[0105] Furthermore, during the expansion process, the system can divide the migration plan into two categories: tasks where the receiver is an existing node, which performs hot migration of parameters between existing nodes; and tasks where the receiver is a new node, where the new node loads the model shards it is responsible for based on the target parallel configuration, while the source node unloads layers it is no longer responsible for. Among these, migration between existing nodes, unloading at the source node, and loading of model shards by the new node can be executed asynchronously and in parallel.
[0106] In one Ray implementation, during scaling, the system creates incremental resource groups based on the number of newly added Diffusion Transformer worker nodes, with each new worker node requesting one GPU resource unit. New worker nodes are created by a remote execution node proxy and bound to the Diffusion Transformer worker node class and a global stage number.
[0107] For migration tasks where the recipient is an existing worker node, the system creates a temporary communication group on the old worker node's stage sequence set and performs hot parameter migration. For tasks where the recipient is a newly added worker node, the system does not require the new worker node to join the old communication group. Instead, the new worker node calls the model loading interface and loads the target shard from the model storage based on the target parallel configuration. At the same time, the source worker node unloads the global layer it is no longer responsible for. Migration between old worker nodes, unloading at the source node, and shard loading by the new worker node are started asynchronously and converge before the communication environment is rebuilt and verified.
[0108] Furthermore, such as Figure 8As shown, Ray's scaling process begins with the pipeline parallelism expansion interface and sequentially completes the following steps: target parallel configuration derivation, parameter summary collection, migration plan calculation, incremental resource group creation, new worker node generation, initial endpoint update, hot migration of old worker node parameters, source node unloading, new worker node shard loading, old worker node distributed environment reconstruction, and target topology verification. The implementation path of this process clearly demonstrates the control logic of "planning first, then parallel execution, and finally unified verification" during scaling.
[0109] S190, Volume Reduction Processing
[0110] Furthermore, when the target pipeline parallelism is less than the current pipeline parallelism, the system first generates a migration plan and a set of GPUs to be released based on the target layout. For layers in the GPUs to be released that are still needed by the target layout, the system migrates them to reserved GPUs before releasing resources. After the migration is complete, the system destroys the distributed environment of the worker nodes to be released, terminates the corresponding remote execution nodes, and removes them from the list of diffused Transformer worker nodes and the full list of worker nodes.
[0111] S200, rebuild the communication environment and switch services
[0112] Furthermore, in this embodiment, the system generates new distributed initialization endpoints for the target topology, such as new master node addresses and ports, and distributes them to all target worker nodes. Subsequently, the system destroys the old model parallel communicator and distributed environment, and reinitializes the distributed environment, pipeline communication group, sequence parallel communication group, and runtime state. When rebuilding the runtime state, the system cleans up old topology-related caches, such as encoder hidden state cache or pipeline receive cache, to prevent subsequent inference from referencing expired topology states.
[0113] In one implementation, the control plane suspends subsequent inference calls during physical topology changes; after parameter migration, source unloading, new node shard loading, distributed environment reconstruction, runtime state reconstruction, and consistency verification are completed, subsequent inference calls are then initiated based on the target topology. This implementation does not require the participation of online request routers, while in the online service implementation, new requests can be drained and routed using the topology version number and service state machine.
[0114] Example 2: Topology Switching and Verification Example
[0115] like Figure 5 As shown, this embodiment provides a topology switching state machine, where each model instance maintains a topology version number and service status.
[0116] Furthermore, the service status can be switched in the following order: serviceable status, empty status, migration status, communication reconstruction status, verification status, and serviceable status; when verification fails or reconstruction fails, the service status switches to the failure rollback status.
[0117] Furthermore, in the serviceable state, the requesting router can send new requests to the model instance. Upon receiving the GPU resource event, the model enters a drain state, and the requesting router stops sending new requests to the old topology. Requests already in the inference process can complete the current denoising time step or trigger time step state saving. The system waits for the current denoising time step to complete, prevents the old topology from entering the next denoising time step, and continues execution after the target topology becomes serviceable. Subsequently, the model enters the migration state, performing parameter hot migration, source unloading, and new node shard loading. After migration, the model enters the communication reconstruction state, destroys the old communicator, and creates the target communication group. After the communication group is successfully created, the model enters the verification state.
[0118] Furthermore, the verification includes runtime state consistency verification, communication connectivity verification, and parameter integrity verification.
[0119] The runtime state consistency check requires that the pipeline parallelism, number of global processes, and number of pipeline slices of all target worker nodes be consistent and conform to the target configuration. The parameter integrity check requires that the global layer set be consistent before and after the topology change, that the number of parameters in each global layer be non-zero, that the parameters reside on the target device, that the parameter data type conform to the target precision, and that the difference between the parameter digest and the digest before migration be less than a threshold. The parameter digest may include the parameter sum, maximum absolute value, number of parameters, device, data type, L2 norm, checksum, or version number.
[0120] In one embodiment, the system compares the global layer set before and after the topology change, checks whether the number of parameters in each layer is zero, whether the parameters are located in the graphics processor runtime device, and determines whether the parameters and differences exceed a preset threshold, such as 1e-3.
[0121] Only after successful verification will the system mark the target topology version as the currently serviceable version. If verification fails, the system enters a fallback state, which can restore the old topology, re-execute the migration plan, resend the missing layer from the existing GPUs, or trigger a cold start loading if recovery is impossible. If the receiver loading fails, the receiver layer is missing, the runtime state is inconsistent, or the parameter digest verification fails, the system can directly determine that this scaling up / down operation has failed and keep the target topology unserviceable.
[0122] Example 3: Parameter Redistribution
[0123] Furthermore, this embodiment uses a diffusion Transformer model containing 24 Transformer blocks as an example. The initial pipeline parallelism is 2, and the two worker nodes hold the following global layers respectively:
[0124] For example, stage 0: {0, 1, …, 11}; stage 1: {12, 13, …, 23}. When the system detects the addition of one available GPU and sets the target pipeline parallelism to 3, the target hierarchy is divided as follows: stage 0: {0, 1, …, 7}; stage 1: {8, 9, …, 15}; stage 2: {16, 17, …, 23}. Further comparison of the current layout and the target layout reveals that: stage 0 can reuse {0, …, 7}, and {8, …, 11} is to be released; stage 1 can reuse {12, …, 15}, and {16, …, 23} is to be migrated out or unloaded; stage 2 requires {16, …, 23}. If the newly added node has not yet joined the old communication group, the system allows Phase 2 to load {16, …, 23} from the model storage according to the target topology, while Phase 1 unloads these layers. For {8, …, 11}, if the target receiver is an existing Phase 1, a hot parameter migration from Phase 0 to Phase 1 is performed. In one implementation, this migration is performed within a temporary communication group corresponding to the set of phase numbers of the old working nodes. Phase 0 sends the parameter tensor dictionary corresponding to layer {8, …, 11} to Phase 1, Phase 1 constructs the missing layer and partially loads the parameters, and then Phase 0 deletes the migrated-out layer. After the migration, unloading, and loading of the new node are completed, the system reconstructs the communication environment for the three phase numbers and performs verification.
[0125] Example 4: Interruptible Reasoning Example
[0126] like Figure 6 As shown, the diffusion Transformer inference process in this embodiment consists of multiple denoised time steps. Each time step mainly depends on the current latent state, time step identifier, conditional context, sampler state, and random number state. Compared with checkpoints that require saving the complete model state, saving the state at the time step granularity has lower overhead.
[0127] Furthermore, the system in this embodiment listens for termination notifications of volatile GPU instances. When it detects that an instance will terminate after a grace period, the system waits for the current denoising time step boundary and saves the latent state z_t, time step t, sampler state, random seed, input prompt word or conditional context, model identifier, parameter version number, and topology version number. For synchronous pipelines, the system saves the latent tensor and text embedding of the current time step. For asynchronous pipelines, the system saves the pipeline slice index, the latent tensor of each slice, the latent tensor of the slice of the previous time step, and the asynchronous reception state; for single-step samplers, the sampler state may include the current time step index and scheduler step sequence; for multi-step samplers, the sampler state may also include historical latent states, derivative estimates, or sampler internal caches; for classifier-guided inference, the input conditional context may include positive conditional embeddings and negative conditional embeddings.
[0128] Furthermore, the current time-step state of the inference task includes the latent state z_t, time step t, sampler state, random seed, input prompt or conditional context, model identifier, parameter version number, and topology version number. This current time-step state can be serialized into a tensor file and a metadata file, and written to shared storage, distributed cache, or object storage. During recovery, the scheduler selects a target node with the corresponding model parameters or whose parameters can be obtained through hot migration, loads the saved current time-step state of the inference task, and continues denoising inference from time step t until the final image or video result is generated. When the diffusion Transformer is a video diffusion Transformer, the latent state can include a spatiotemporal latent tensor with temporal, spatial, and channel dimensions, and can further include video frame indexes, temporal slice indexes, or pipeline slice indexes.
[0129] In some embodiments, the system saves the time-step state only when a preemption notification, node eviction notification, or GPU health anomaly is triggered, rather than periodically, thereby reducing additional overhead in the normal inference path. Furthermore, the system can be configured with a low-frequency backup strategy to handle failures without notification.
[0130] Example 5: Scheduling Method Example
[0131] This embodiment provides a model-level GPU resource selection method that does not rely on offline performance sampling. Instead of pre-building a lookup table to calculate the number of GPUs to the theoretical throughput, the system maintains the online state of each inference model in the control plane and calculates the cost of scaling down and the benefits of scaling up based on the online state.
[0132] In some embodiments, online status may include model identifier, current number of GPUs or pipeline stages, exponential moving average of request arrival rate, resource saturation, request queue length, current throughput or actual throughput, latency pressure metric, recent load trend, current topology version, and current service status.
[0133] Furthermore, the system calculates the unit resource load based on the online status and normalizes the queue length, latency pressure, and load trend to obtain... , and The normalized queue length, normalized latency pressure, normalized load trend, and unit resource load are used as online state-derived indicators, which are substituted into the corresponding formulas for online scaling-down marginal loss proxy and online scaling-up marginal benefit proxy, respectively, to complete the calculation. When the GPU resource event is a preemption or failure event, the system completes the calculation according to the online scaling-down marginal loss proxy formula and selects the logical scaling-down object for this round based on the scaling-down cost function. The scaling-down cost function contains... As a barrier term, this term increases rapidly when the model's resource saturation is low, thereby increasing the cost of recovering resources from the model. When GPU resource events are addition, release, or recovery events, the system calculates according to the online expansion marginal benefit proxy formula and selects the logical expansion target for this round based on the expansion benefit function. This expansion benefit function evaluates the return on resource investment through service level gap, queue backlog, load trend, and unit resource load, avoiding simply allocating resources to the model with the worst performance in a particular metric.
[0134] Furthermore, during multiple rounds of logical iteration, each round only updates the number of virtual GPUs and the virtual online state in the control plane, without immediately triggering physical topology changes. For models that are logically expanded, the system increases their number of virtual GPUs and updates the virtual resource saturation, unit resource load, and queue pressure accordingly; for models that are logically scaled down, the system decreases their number of virtual GPUs and recalculates the corresponding service level gap and barrier penalty. The next iteration recalculates the benefits or costs based on the updated virtual state, thereby avoiding single-step greedy bias under continuous resource events.
[0135] Furthermore, for a resource event involving multiple GPUs, the system executes multiple rounds of logical iterations. After these iterations, the system aggregates the final target pipeline parallelism of each model and performs a physical topology reconfiguration, parameter hot migration, and communication group reconstruction. This avoids triggering pipeline reconstruction in every decision round, reducing reconfiguration overhead.
[0136] Example 6: Scheduling Case
[0137] like Figure 9As shown in the example, this embodiment will illustrate how the system performs multi-round logical expansion planning for a single resource event involving multiple newly added GPUs without offline performance profiling.
[0138] Furthermore, assume that the control plane observes the following online states of the three diffusion Transformer models: Model A currently has 2 GPUs, an exponential moving average of request arrival rate of 0.80, resource saturation of 0.70, queue length of 20, latency pressure of 1.30, and a load trend of increasing; Model B currently has 3 GPUs, an exponential moving average of request arrival rate of 0.50, resource saturation of 1.10, queue length of 4, latency pressure of 0.80, and a load trend of stable; Model C currently has 1 GPU, an exponential moving average of request arrival rate of 0.65, resource saturation of 0.85, queue length of 15, latency pressure of 1.10, and a load trend of increasing.
[0139] Furthermore, in the first round of logical expansion iteration, the system calculates the benefit of each model based on the marginal benefit of online expansion. Model A has a low resource saturation, a long queue, and an increasing load trend, so it has the highest expansion benefit. The system assigns a new GPU logic to model A and only updates the number of virtual GPUs and virtual resource saturation of model A in the control plane, without immediately performing physical topology changes.
[0140] Furthermore, in the second round of logical expansion iteration, the system recalculates the expansion benefits based on the updated virtual state. Since Model A has already acquired a virtual GPU, its service level gap and unit resource load have decreased, while Model C may receive higher expansion benefits due to queue backlog and insufficient resources. Therefore, the system logically allocates the second newly added GPU to Model C.
[0141] After two rounds of logical iterations, the system aggregates the target pipeline parallelism for models A and C, and performs a physical topology reconfiguration for each. For each model requiring scaling, the system generates the target parallel configuration, target hierarchical partitioning, and parameter migration plan, and performs incremental resource allocation, hot parameter migration between existing nodes, shard loading for new nodes, communication environment reconstruction, and verification. In this way, the system avoids immediately rebuilding the pipeline topology every time a GPU is allocated.
[0142] like Figure 9 As shown, the logical expansion process of the three models and two newly added GPUs does not involve directly distributing resources to the physical topology all at once. Instead, it first completes multiple rounds of logical iterations in the control plane, and then aggregates the final virtual allocation results into the target pipeline parallelism. The state tables of models A, B, and C, the expansion benefit proxy, and the results of the two rounds of virtual allocation in the figure are consistent with the numerical description in this embodiment, and can be used to intuitively present the scheduling method of "logical decision first, physical reconfiguration later".
[0143] Example 7: Experiment and Implementation
[0144] Furthermore, this embodiment provides a reproducible experimental setup for verifying the feasibility of capacity expansion and reconfiguration, parameter hot migration, and verification mechanisms.
[0145] Furthermore, the entry script sets the model type to an elastic diffusion Transformer model, the initial number of global processes under the Ray framework is 2, and the initial pipeline parallelism is 2, indicating that GPUs include 0, 1, and 2. The system first creates two diffusion Transformer worker nodes and loads the model, then calls the elasticity manager to adjust the pipeline parallelism to 3.
[0146] In this experiment, the system recorded the time consumption of the following stages: generating target parallel configuration, computing migration plan, applying for incremental GPU resource groups, creating new remote execution nodes, hot migration of parameters between existing nodes, source layer unloading, new node model shard loading, distributed initialization endpoint distribution, old worker node communication environment reconstruction, runtime state reconstruction, and eventual consistency verification.
[0147] The experimental verification items include: the pipeline parallelism, number of global processes, and number of pipeline slices of all target worker nodes are consistent; the global layer set before and after the topology change is consistent; the number of parameters in each global layer is non-zero; the parameters are located on the target GPU device; the parameter data type meets the target precision; and the parameters and differences before and after the topology change are less than the preset threshold.
[0148] For example, the parameters and difference threshold can be set to 1e-3. After the expansion verification passes, the system uses fixed prompt words, fixed random seeds, fixed input size, and fixed inference steps to perform subsequent inference to verify that the target topology can continue to provide inference services.
[0149] As an exemplary implementation, the solution of this application can be implemented in the following engineering form, but this example does not constitute a limitation on the scope of protection of the claims.
[0150] In one embodiment, the entry script sets the initial pipeline parallelism to 2 and creates 2 diffusion Transformer worker nodes via Ray; during scaling, the elastic manager calls the pipeline parallelism extension interface to generate the target parallel configuration, updates the pipeline parallelism and pipeline slice count to 3, and recalculates the global process count and diffusion Transformer parallel scale.
[0151] In one embodiment, the elastic manager checks whether the target pipeline parallelism is greater than 0 and does not exceed the number of available GPUs in the Ray cluster before scaling. When the target pipeline parallelism is the same as the current pipeline parallelism, the elastic manager returns success directly; when the target pipeline parallelism is greater than the current value, the scaling-up path is executed; when the target pipeline parallelism is less than the current value, the scaling-down path is executed.
[0152] In one embodiment, a remote execution node proxy encapsulates and diffuses Transformer worker nodes, providing a unified method distribution interface. The worker nodes support initializing the distributed environment, loading pre-trained models, updating parallel configurations, performing parameter migrations, unloading specified layers, querying current layer allocations, querying runtime status, and querying parameter summaries.
[0153] In one embodiment, the current layer allocation is obtained by reading the Transformer block parameter key carrying the global layer identifier or the Transformer block parameter key with a module prefix and carrying the global layer identifier from the model state dictionary. The target layer allocation is obtained by dividing the target pipeline parallelism into consecutive layers. The target layer allocation is generated evenly according to the number of layers, with the remainder preferentially allocated to smaller stage numbers. The migration plan is obtained by comparing the current layer allocation and the target layer allocation, and is in the format of a list of receiver stage number, sender stage number, and global layer identifier set.
[0154] In one embodiment, during parameter migration, the sender filters parameter tensors belonging to a specified global layer into a dictionary and sends it through the tensor dictionary sending interface of the communication group; the receiver receives the parameter tensor dictionary through the tensor dictionary receiving interface, constructs a missing Transformer block based on the global layer identifier, and executes a partial parameter loading interface. If a sent parameter key is reported as not loaded, or the receiver lacks the global layer specified by the migration task, the migration is considered to have failed. The sender deletes the migrated layer and clears the GPU memory cache. After each migration task is completed, a communication barrier is executed. After all migrations are completed, the GPU is synchronized, and the Transformer is repackaged so that subsequent inference logic is executed according to the target topology.
[0155] In one embodiment, during expansion, only the incremental resource groups required for the new worker nodes are created. The system splits tasks according to the receiving stage sequence number of the migration task. Tasks receiving existing worker nodes perform hot parameter migration within the old worker node communication group. Tasks receiving new worker nodes are transformed into source unloading tasks, and the new worker nodes load the target shards from the model storage. Migration between old nodes, source unloading, and shard loading for new nodes are initiated asynchronously. Subsequently, new distributed initialization endpoints are issued to all worker nodes, old nodes are destroyed and the distributed environment is rebuilt, and new nodes enter the verification process together after completing loading.
[0156] In one embodiment, the verification process collects the pipeline parallelism, global process count, and pipeline slice count of all worker nodes to ensure consistency with the target configuration. Simultaneously, it collects the parameter sums, maximum absolute values, parameter counts, device and data types for each global layer, comparing the layer set and parameter summary before and after the topology change to confirm the target topology is serviceable. Parameter sums and differences can be set with preset thresholds, such as 1e-3. If the verification fails, the system returns a scaling failure message and does not consider the target topology as a serviceable topology.
[0157] Example 8: System Example
[0158] like Figure 2 As shown, this embodiment provides a large-scale diffusion Transformer elastic inference system for volatile GPU resources. The system includes a state acquisition module, an elastic scheduling module, a topology planning module, a remote execution node agent module, a parameter hot migration module, a communication reconstruction module, a version verification module, an interruptible inference module, and a request routing module, wherein:
[0159] The status acquisition module is used to collect the request arrival rate, current throughput, target throughput, backlog queue length, GPU health status, GPU resource events, topology version number and service status of the inference model.
[0160] The elastic scheduling module is used to perform health-aware asymmetric marginal utility scheduling that does not depend on offline performance profiles based on GPU preemption or release events, and outputs the target number of GPUs and the target pipeline parallelism for each model.
[0161] The topology planning module is used to generate the target parallel configuration, model hierarchy partitioning, parameter sharding layout, and communication topology based on the target pipeline parallelism.
[0162] The remote execution node proxy module is used to uniformly schedule the initialization, model loading, parameter migration, inference execution, status query and destruction of remote GPU worker nodes. This module can encapsulate specific worker node classes and call worker node methods through a unified method dispatch interface.
[0163] The parameter hot migration module is used to generate a parameter migration plan based on the current parameter piece layout and the target parameter piece layout, and to perform parameter tensor dictionary transfer, missing layer construction, parameter loading and source unloading through the GPU communication group.
[0164] The communication reconstruction module is used to set up computing barriers, destroy failed communicators, create new communication groups, issue new distributed initialization endpoints, and reinitialize the runtime state when the topology changes.
[0165] The version verification module is used to maintain the topology version number and service status, and to perform runtime status consistency verification, communication connectivity verification, and parameter integrity verification.
[0166] An interruptible inference module is used to save the current time step state of the inference task when a volatile GPU instance termination notification is detected, and to resume inference on a new node or a new topology.
[0167] The request routing module is used to route inference requests to serviceable model instances based on model service status, topology version, and queue status.
[0168] Example 9: Electronic Device Example
[0169] like Figure 7 As shown, this embodiment provides an electronic device. The electronic device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor.
[0170] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Without affecting the desired effects of the technical solutions of the present invention, the steps described in the present invention can be executed sequentially, in parallel, or in a reordered manner, and some steps can be added, deleted, or replaced as needed for implementation.
[0171] Those skilled in the art can modify, combine, sub-combine, substitute, or improve the above embodiments according to design requirements and actual application scenarios. All modifications, substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for large-scale diffusion of Transformer elastic inference for volatile GPU resources, characterized in that, Includes the following steps: Obtain model status, GPU resource events, and online runtime metrics for diffusion-oriented Transformer inference models; In response to the GPU resource event, the target pipeline parallelism is determined based on the model state and the online runtime metrics; A target parallel configuration is derived based on the target pipeline parallelism. Based on the diffusion Transformer inference model, a global layer identifier is assigned to each Transformer block, and the corresponding target model hierarchy partitioning, target parameter fragmentation layout, and target communication topology are generated according to the target pipeline parallelism. Compare the current parameter fragment layout with the target parameter fragment layout, generate an enhanced parameter migration plan, and determine the set of reusable parameters, the set of parameters to be migrated, the set of parameters to be loaded, and the set of parameters to be released; Based on the set of parameters to be migrated, hot migration of parameters is performed on the parameter fragments that have been loaded into the existing GPU memory, so as to redistribute the parameter fragments to the target GPU set. Based on the numerical relationship between the target pipeline parallelism and the current pipeline parallelism, perform expansion or reduction processing. The distributed communication group is reconstructed based on the target communication topology, and the inference service of the inference model is restored or extended based on the reconstructed distributed communication group after the topology consistency check and parameter integrity check pass.
2. The method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 1, characterized in that, The GPU resource events include GPU preemption events, GPU release events, GPU addition events, GPU failure events, and GPU recovery events.
3. The method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 2, characterized in that, When the GPU resource event is preemption or failure, a scaling-down scheduling is performed. Multiple rounds of iteration are performed on the reduced number of GPUs. In each round, the model with the lowest scaling-down cost is selected based on the online scaling-down marginal loss proxy and the resource saturation barrier term for virtual scaling-down, and the virtual state is updated. When the GPU resource event is release, addition, or recovery, expansion scheduling is performed. When the number of new GPUs is limited, priority is given to models with high marginal benefits of online expansion, severe queue backlog, rising load trend, or failure to meet service level objectives.
4. The method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 3, characterized in that, The parallel configuration includes pipeline parallelism, number of pipeline slices, number of global processes, and number of diffusion Transformer worker nodes.
5. A method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 1 or 4, characterized in that, When generating a parameter migration task and performing the parameter hot migration, the following steps are performed: Get the set of global layer identifiers held by each pipeline stage at present; Calculate the set of global layer identifiers that each pipeline stage of the target should hold based on the target pipeline parallelism; Establish a mapping from global layer identifiers to the current stage and an index for mapping global layer identifiers to the target stage; Generate migration tasks for global layer identifiers that are missing in the target stage but already exist in the source GPU; For each migration task, the sender selects the parameter tensors to be migrated from the model state dictionary based on the global layer identifier; The sender packages the parameter tensors in the same migration task into a parameter tensor dictionary and sends it to the receiver through the GPU communication group participating in the migration. The receiver constructs a missing Transformer block based on the global layer identifier in the tensor dictionary and loads the received parameter tensor into the Transformer block. After the transmission is complete, the sender deletes the migrated Transformer block and releases the video memory cache; Each phase involved in the migration executes a communication barrier after each migration task.
6. The method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 5, characterized in that, When performing the expansion or reduction process, the following are included: When the target pipeline parallelism is greater than the current value, expansion processing is performed. Incremental GPU resources are requested only for new nodes, the number of new diffusion Transformer worker nodes is calculated, new remote execution nodes are created and allocated with global stage sequence number and GPU resources, and added to the worker node list. When the target pipeline parallelism is less than the current value, a scaling-down process is performed. First, a migration plan and a set of GPUs to be released are generated. Then, the layers that the target layout still needs in the GPUs to be released are migrated to the reserved GPUs. Finally, the distributed environment of the nodes to be released is destroyed, the remote execution nodes are terminated, and the nodes are removed from the list of worker nodes.
7. The method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 6, characterized in that, When performing the aforementioned expansion process, the migration task is split into two categories, supporting asynchronous parallel execution, including: The receiver performs hot migration of parameters between existing nodes; The receiver loads the corresponding model shards for the newly added node according to the target configuration, while the source node unloads the layers it is no longer responsible for.
8. The method for large-scale diffusion of Transformer elastic inference for volatile GPU resources according to claim 1, characterized in that, When rebuilding the distributed communication group, the following steps are performed: Generate a new distributed initialization endpoint for the target topology and distribute it to all target worker nodes; Destroy the old model's parallel communicator and distributed environment, reinitialize the distributed environment, pipelined communication group, sequential parallel communication group, and runtime state, and synchronously clean up the old topology-related caches during reconstruction.
9. A large-scale diffusion Transformer elastic inference system for volatile GPU resources, characterized in that, include: The status acquisition module is used to acquire the model status, GPU resource events, and online running metrics of the diffusion Transformer inference model. The elastic scheduling module is used to respond to the GPU resource event and determine the target pipeline parallelism based on the model state and the online running metrics; The topology planning module is used to derive the target parallel configuration based on the target pipeline parallelism, and at the same time, based on the diffusion Transformer inference model, assign global layer identifiers to each Transformer block, and generate the corresponding target model hierarchy partitioning, target parameter sharding layout and target communication topology according to the target pipeline parallelism. The remote execution node agent module is used to uniformly schedule the initialization, model loading, parameter migration, inference execution, status query and destruction of remote GPU worker nodes; The parameter hot migration module is used to generate a parameter migration plan based on the current parameter piece layout and the target parameter piece layout, and to perform parameter tensor dictionary transfer, missing layer construction, parameter loading and source unloading through the GPU communication group. The communication reconstruction module is used to set up computing barriers, destroy failed communicators, rebuild communication groups, issue new distributed initialization endpoints, and reinitialize the runtime state when the topology changes. The version verification module is used to maintain the topology version number and service status, and to perform runtime status consistency verification, communication connectivity verification, and parameter integrity verification. An interruptible inference module is used to save the current time step state of the inference task when a volatile GPU instance termination notification is detected, and to resume inference on a new node or a new topology. The request routing module is used to route inference requests to serviceable model instances based on model service status, topology version, and queue status.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
