Resource processing method and device for multi-tenant oriented large model training platform

Abstract

The embodiment of the application provides a resource processing method and device of a multi-tenant-oriented large model training platform, innovatively designs a resource portrait system, effectively expresses demands through stage division and state space, constructs a multi-level isolation mechanism, combines conflict analysis and parameter optimization to establish reliable resource protection, introduces intelligent scheduling, ensures the rationality of allocation through priority evaluation and scheme generation, and effectively solves the deficiencies of traditional technologies in resource portrait, isolation mechanism and scheduling optimization, thereby providing technical support for large model training.

Description

Resource processing method and apparatus for large-scale model training platform for multi-tenancy Technical Field

[0001] This application relates to the field of data processing, specifically to a resource processing method and apparatus for a large-scale model training platform for multi-tenants. Background Technology

[0002] Existing resource handling methods for large model training platforms have significant shortcomings. Traditional systems perform poorly in resource profiling and boundary delineation, failing to effectively adapt to multi-tenant scenarios and impacting training efficiency.

[0003] Furthermore, existing technologies suffer from bottlenecks in resource isolation and conflict resolution. Most systems lack robust isolation mechanisms and competition analysis strategies, leading to inefficient resource utilization.

[0004] Existing systems have technical shortcomings in scheduling optimization. The lack of in-depth analysis of training features makes it difficult to achieve efficient resource allocation through policy adjustments, thus impacting platform performance. Solving these problems is crucial for improving the training capabilities of large models. Summary of the Invention

[0005] To address the problems in the existing technology, this application provides a resource processing method and apparatus for a multi-tenant large model training platform, which can effectively solve the shortcomings of traditional technologies in resource profiling, isolation mechanisms and scheduling optimization, and provide technical support for large model training.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] Firstly, this application provides a resource processing method for a multi-tenant large model training platform, including:

[0008] Collect resource usage data for training tasks, and divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage, and model saving stage according to model training characteristics. Normalize the resource requirement data of each stage to generate a resource profile matrix. Based on the resource profile matrix, construct a state space that includes memory usage, computing load, and communication bandwidth. Dynamically divide the resource boundaries of the tenant training tasks according to the state space.

[0009] Multi-level resource isolation is implemented, training processes are isolated through container namespaces, computing cores are isolated using time-slice round-robin, gradient synchronization communication is isolated based on virtual switches, training instructions are mapped to independent queues according to tenant identifiers, memory curves and computation latency detection methods are used to identify competition relationships in training tasks, a conflict graph is constructed with training nodes as vertices and competition relationships as edges, conflict features are extracted from the conflict graph, and isolation parameters including batch size, computation quota, and synchronization bandwidth are generated.

[0010] Training tasks are scheduled based on a candidate scheme generation and layer-by-layer screening method. The task priority score is calculated according to the isolation parameter. The priority score and the conflict feature are input into the scheduler to generate a training resource allocation scheme. Training node expansion and task migration are performed. Graphics card usage data and training events are recorded to the auditing system. Billing and settlement are carried out according to training duration, model size and communication volume.

[0011] Furthermore, it also includes: collecting the memory usage, computing core utilization, and network bandwidth consumption of the training task based on the resource monitoring module, aligning and labeling the monitoring data according to the time series, constructing an original dataset containing timestamps, resource types, and numerical attributes, and resampling and standardizing the original dataset;

[0012] Based on the training progress, the standardized data is divided into parameter loading stage, gradient calculation stage, parameter update stage, and model saving stage. The average and fluctuation range of resource requirements for each stage are calculated, and the calculation results are organized into a resource profile matrix.

[0013] Furthermore, it also includes: calculating the resource consumption feature vectors of the training task at different stages based on the resource profile matrix, mapping the feature vectors to a three-dimensional state space containing the memory dimension, computing power dimension, and bandwidth dimension, and dividing the state space into grids to generate resource configuration units;

[0014] Calculate the resource requirement trajectory of each tenant's training task, project the requirement trajectory into the grid cells of the state space, determine the resource boundaries between tenants based on the occupancy of the grid cells, and generate a boundary parameter matrix that includes the upper limit of video memory, computing power quota, and bandwidth limit.

[0015] Furthermore, it also includes: deploying the tenant training process to run in an independent container, implementing process isolation based on the container namespace, allocating core computing resources using a time-slice round-robin scheduling mechanism, dividing independent communication queues in the virtual switch to achieve gradient synchronization traffic isolation, and generating isolation configuration parameters that include process identifier, time slice ratio, and bandwidth quota;

[0016] The resource competition relationship between training tasks is identified by using memory usage curve analysis and computation latency monitoring methods. Based on the competition relationship, a task dependency graph is constructed, and the dependency graph is used to guide the dynamic adjustment of isolation configuration parameters.

[0017] Furthermore, it also includes: using training nodes as graph vertices, establishing connection edges between nodes based on resource competition relationships, calculating the weight coefficients of the edges, normalizing the weight coefficients, and constructing a conflict graph model that reflects the intensity of resource competition in the training task.

[0018] The conflict graph is divided into communities and its centrality is calculated. The degree of competition between nodes and the intensity of competition between edges are extracted as conflict features. Based on the conflict features, the upper limit of batch size, the proportion of computing resource quota, and the gradient synchronization bandwidth threshold of the training task are calculated to generate an isolation parameter matrix.

[0019] Furthermore, it also includes: generating multiple candidate resource configuration schemes for each training task based on the task type and training stage, calculating the matching degree between the candidate schemes and the isolation parameters, calculating the priority coefficient based on the matching degree score and the task service level, and filtering and sorting the candidate schemes according to the priority coefficient;

[0020] Priority scores and conflict features are combined to form the scheduling decision input. A multi-objective optimization algorithm is used to search for the optimal configuration combination in the candidate scheme space to generate a training resource scheduling scheme that includes node allocation, resource ratio, and task order.

[0021] Furthermore, it also includes: based on the computation node expansion requirements of the training resource scheduling scheme, allocating computation resources and network identifiers to the nodes to be expanded, migrating training tasks to the target nodes in the order of dependencies, and generating audit data containing resource allocation records, task migration logs, and operation timestamps;

[0022] Extract the GPU usage time, model parameter size, and total gradient communication amount of the training task from the audit data, calculate the training cost based on resource usage, and generate a billing list that includes basic training costs, resource consumption costs, and communication transmission costs.

[0023] Secondly, this application provides a resource processing device for a multi-tenant large model training platform, comprising:

[0024] The resource partitioning module is used to collect resource usage data of training tasks, and divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage and model saving stage according to model training characteristics. The resource requirement data of each stage is normalized to generate a resource profile matrix. Based on the resource profile matrix, a state space including memory usage, computing load and communication bandwidth is constructed. The tenant training task is dynamically partitioned according to the state space.

[0025] The resource isolation module is used to perform multi-level resource isolation. It isolates the training process through container namespaces, isolates the computing core using time-slice round-robin, isolates gradient synchronization communication based on virtual switches, maps training instructions to independent queues according to tenant identifiers, identifies the competition relationship of training tasks using memory curves and computation latency detection methods, constructs a conflict graph with training nodes as vertices and competition relationships as edges, extracts conflict features from the conflict graph, and generates isolation parameters including batch size, computation quota, and synchronization bandwidth.

[0026] The resource scheduling module is used to schedule training tasks based on candidate scheme generation and layer-by-layer screening methods. It calculates task priority scores according to the isolation parameters, inputs the priority scores and conflict features into the scheduler to generate training resource allocation schemes, performs training node expansion and task migration, records graphics card usage data and training events to the auditing system, and performs billing and settlement according to training duration, model size, and communication volume.

[0027] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the resource processing method for a multi-tenant large model training platform.

[0028] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the resource processing method for a multi-tenant large model training platform.

[0029] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the resource processing method for a multi-tenant large model training platform.

[0030] As can be seen from the above technical solution, this application provides a resource processing method and apparatus for a multi-tenant large-scale model training platform. Through an innovative design of a resource profiling system, and by using stage division and state space, it achieves effective expression of requirements. A multi-level isolation mechanism is constructed, combined with conflict analysis and parameter optimization, to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively solves the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a flowchart illustrating the resource processing method for a multi-tenant large model training platform in an embodiment of this application.

[0033] Figure 2 is a structural diagram of the resource processing device of the multi-tenant large model training platform in the embodiment of this application;

[0034] Figure 3 is a schematic diagram of the structure of the electronic device in the embodiment of this application.

[0035] Figure label:

[0036] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The acquisition, storage, use, and processing of data in this application all comply with relevant laws and regulations.

[0039] To address the problems existing in current technologies, this application provides a resource processing method and apparatus for a multi-tenant large-scale model training platform. It innovatively designs a resource profiling system, achieving effective expression of requirements through stage division and state space. A multi-level isolation mechanism is constructed, combined with conflict analysis and parameter optimization, to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively solves the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training.

[0040] To effectively address the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, and to provide technical support for large-scale model training, this application provides an embodiment of a resource processing method for a multi-tenant large-scale model training platform. Referring to Figure 1, the resource processing method for a multi-tenant large-scale model training platform specifically includes the following:

[0041] Step S101: Collect resource usage data of training tasks, divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage and model saving stage according to model training characteristics, normalize the resource demand data of each stage to generate a resource profile matrix, construct a state space containing video memory usage, computing power load and communication bandwidth based on the resource profile matrix, and dynamically divide the tenant training tasks according to the state space.

[0042] This embodiment focuses on step S101 and is geared towards the online operating environment of a multi-tenant large model training platform.

[0043] First, the training jobs of each tenant in the platform are deployed in the form of containers on a shared GPU cluster. The training process includes stages such as parameter loading, gradient calculation, parameter updating, and model saving. Resource requirements show obvious segmentation and fluctuations on the timeline. This embodiment first solves the problem of "how to reliably distinguish stages and quantify resource characteristics", and then solves the problem of "how to abstract the stage characteristics into a unified state space and thereby generate dynamic resource boundaries". The two parts are tightly coupled, and the isolation and scheduling of subsequent steps directly depend on the parameter set output here.

[0044] In this embodiment, resource monitoring probes are deployed on the data acquisition side. The raw metrics collected include: memory usage (unit MiB, per tenant container and per device), computing core utilization (SM utilization %, core clock normalized load), and communication bandwidth (divided into parameter synchronization uplink / downlink rates, unit Gbps). Training metadata such as training iteration count, step time, and mini-batch size are also included.

[0045] To eliminate sampling phase differences between different nodes, all metrics are aligned with the global training step as the master clock. The sampling window extracts two types of values ​​at a fixed phase point in each step: the average value and the peak value within the step. The original sequence may contain jitter and missing measurements. In this embodiment, resampling (e.g., interpolation at fixed step intervals) and standardization (subtracting the median within the step and scaling by the interquartile range) are performed first to normalize the dimensions of different tenants and different models, providing a usable scale for subsequent comparisons.

[0046] The phase division is not simply based on fixed time intervals, but rather on "resource morphology characteristics." In this embodiment, the criteria are: the parameter loading phase is characterized by a positive first-order difference in memory growth with a growth rate exceeding a set threshold, low communication bandwidth, and unstable SM usage; the gradient calculation phase is characterized by high and stable SM usage, near-zero memory changes, and periodic communication pulses; the parameter update phase is characterized by a superposition of short-term communication peaks and a decrease in SM usage; and the model saving phase is characterized by a continuous increase in I / O or network bandwidth when SM is low.

[0047] If the boundaries of individual tasks become blurred due to asynchronous strategies, this embodiment adopts a rollback rule: Gradient calculation intervals are first defined using stable, high-occupancy segments of the SM (Synchronous Mode) framework. Loading and updates are located before and after this interval based on memory increases and communication peaks. Residual intervals that cannot meet the rule are categorized as "hybrid segments," and their data still enters the profile, but with reduced weights to avoid link breakage. The rollback rule is emphasized to ensure that downstream algorithms requiring "full-time resource profiling" have input and are not interrupted by a few abnormal steps.

[0048] After completing the stage segmentation, this embodiment statistically analyzes representative quantities of resource requirements for each stage: mean, peak, and fluctuation of GPU memory usage; mean and stability of computing power load (the reciprocal of variance is used as a stability indicator); and mean, peak, and pulse period of communication bandwidth. To compress dimensionality and retain key differences, a resource profile matrix R is constructed, with rows corresponding to stages and columns arranged sequentially as features such as "GPU memory - mean / peak / fluctuation," "computing power - mean / stable," and "bandwidth - mean / peak / period." After standardization, a numerically stable representation is formed. Matrix R reflects the resource requirement profile of the same task at different stages and is suitable for horizontal comparison between different tenants and different models. Considering the differences between the warm-up and stable phases of large model training, this embodiment further splits the row items in the gradient calculation stage by the first k epochs and subsequent epochs, thereby covering the slight over-provisioning phenomenon during the warm-up period.

[0049] Based on the resource profiling matrix, this embodiment constructs a three-dimensional state space S, with its axes representing memory usage M, computing power load C, and communication bandwidth B, respectively. The three-dimensional space is then divided into grids, with the grid granularity set according to the cluster resource scale. For example, memory usage is incremented by GiB, computing power by SM percentage, and bandwidth by Gbps. The feature vectors of each stage in R are mapped to S to obtain the "resource trajectory" of the task within a training cycle. Since different tenants run concurrently, multiple trajectories in S may overlap in some grid cells. This embodiment treats the occupancy of each grid cell as a resource allocation unit, recording the proportion of time it is occupied by different tenants and the peak overlap amount to measure potential conflicts.

[0050] The delineation of dynamic resource boundaries depends on the grid occupancy statistics mentioned above.

[0051] In this embodiment, the set of projections of each tenant's trajectory onto the grid, G_tenant, is calculated within each training scheduling cycle. If the stacking demand of a certain unit exceeds the available capacity of the node, the boundary parameters need to be changed. The logic for the change is to prioritize finding the space of "variable parameters": for example, in the gradient calculation stage, there is a substitution relationship between batch size and computing power quota, and a coupling relationship between communication bandwidth limit and synchronization interval.

[0052] This embodiment reads the fluctuation and stability of the current stage from the resource profile. If the stability is high, it tends to cut the boundary in the computing power quota dimension (allocating a stricter SM ratio). If the fluctuation is large, it prioritizes adjusting the bandwidth limit to smooth out peaks. For the memory dimension, the boundary is determined by "peak value + safety margin". The safety margin is estimated from historical fluctuation statistics to avoid instantaneous OOM. The above decisions are written as a boundary parameter matrix Bnd. The entries of the matrix are the [M_max, C_quota, B_limit] triples for each tenant and each stage. Subsequent isolation and scheduling are performed according to this.

[0053] In real-world scenarios, there are issues with overlapping trajectories and asynchronous phases among multiple tenants, especially when the AllReduce intensive phase overlaps with the checkpoint of another tenant, both of which require bandwidth.

[0054] This embodiment introduces a "time duty cycle" in the state space, defining the conflict intensity of each grid cell as the superposition demand multiplied by the time overlap ratio, so as not to overreact to brief overlaps when dividing the boundary. In addition, if a tenant occupies a position close to the upper limit for a long time and causes others to frequently roll back, this embodiment will increase the weight of its bandwidth and computing power characteristics in the next round of profile matrix updates, thereby suppressing high-interference trajectories and outputting a tighter B_limit and a lower C_quota in subsequent cycles.

[0055] This embodiment also considers the impact of stage determination errors on the overall process. If stage features are misclassified due to asynchronous pipelines or hybrid parallelism, "cross-row leakage" will occur in the resource profile. The solution is to perform consistency checks on R: check whether the SM mean of the gradient calculation row is higher than that of the update row, and check whether the memory increase of the loading row exceeds the threshold; if not, trigger backtracking correction, use a sliding window to re-estimate boundary points, and reassign some samples to stages that better match the features to ensure that the trajectory in S does not undergo non-physical jumps. Windows that still cannot be classified are marked as "hybrid segments," and a conservative resource limit is allocated in Bnd to avoid jitter caused by overreaching.

[0056] The scale of the state space varies across different model sizes. To avoid large models naturally consuming more resources and thus squeezing out the fairness of smaller models, this embodiment introduces a scale correction based on the number of model parameters P. The feature vectors are lightly normalized along the M and B axes to ensure that resource requirements of equal proportions are comparable in S. However, the computing power axis C remains unchanged because the differences in parallelizability of different models need to be preserved. This correction does not affect the final physical quota conversion and is only used for fairness measurement in boundary judgment.

[0057] To provide a measurable trigger threshold for boundary delineation, this embodiment can define a simple congestion criterion:

[0058] Z = wM·(M_d / C_m) + wC·C_d + wB·(B_d / B_cap),

[0059] Where M_d represents the cumulative memory requirement of a certain grid cell, C_m is the available memory capacity of that node, C_d is the cumulative computing power requirement ratio, B_d is the cumulative bandwidth requirement, B_cap is the available bandwidth of the link, and wM, wC, and wB are weighting coefficients. When Z exceeds the threshold T, the Bnd of the tenants involved in the corresponding cell is tightened at this stage; when Z is much lower than T, Bnd allows the reclamation of some constraints to allow the scheduler to improve throughput. This expression is consistent with the actual resource bottleneck, with the upper limits of memory and bandwidth constrained by the hardware upper bound, while the computing power dimension reflects the shared characteristics of SM.

[0060] Through the above process, the core problem solved by this embodiment is that the resource requirements of training tasks are not stationary in both time and tenant dimensions. Traditional resource allocation based on a single average indicator can lead to inter-stage mismatch and cross-tenant interference. Here, a chain of staged acquisition—profile matrix—3D state space—grid occupancy—boundary matrix is ​​used to compress the complex time-varying requirements into an operable parameter set, which is both interpretable and can drive subsequent isolation and scheduling. Two typical scenarios illustrate this:

[0061] Firstly, tenant A's large model experiences rapid memory growth and weak communication in the first few epochs, so Bnd assigns a higher M_max during the loading phase while maintaining a conservative B_limit.

[0062] Secondly, tenant B experiences strong bandwidth spikes during the synchronously intensive AllReduce phase, leading to high congestion of B units in the state space. The system accordingly tightens B_limit and staggers the synchronization of B with other tenants in the next scheduling step, thereby reducing mutual interference.

[0063] The Bnd and R outputs from the entire S101 will be directly referenced in subsequent steps, completing the closed loop from monitoring to control.

[0064] Step S102: Perform multi-level resource isolation, isolate the training process through container namespaces, isolate the computing core using time-slice round-robin, isolate gradient synchronization communication based on virtual switches, map training instructions to independent queues according to tenant identifiers, identify the competition relationship of training tasks using memory curves and computation latency detection methods, construct a conflict graph with training nodes as vertices and competition relationships as edges, extract conflict features from the conflict graph, and generate isolation parameters including batch size, computation quota, and synchronization bandwidth;

[0065] This embodiment focuses on step S102, where multiple tenants concurrently conduct distributed training on the same GPU cluster. In step S101, the boundary parameter matrix and resource profile for each tenant at each stage have already been output. The goal of this embodiment is to translate abstract boundary constraints into executable multi-level isolation configurations, and to fine-tune these configurations at runtime based on conflict detection results. This ensures that the training process has its own boundaries in terms of computing power, memory, and communication, avoiding cross-tenant interference.

[0066] This embodiment first implements process boundaries on the container side. The platform generates an independent container instance for each tenant task, opens namespaces such as PID, IPC, Mount, and Network, mounts independent driver and library directories, and writes the initial budget (GPU passthrough / slice label, CPU quota, memory limit, IO weight) into cgroups v2. To match the staged constraints obtained from S101, the training process running within the container is injected with probes by a daemon process. The probes listen for the global step and stage label, and switch cgroup parameters according to the stage. For example, during the parameter loading stage, the file system read cache and IO weight are increased, and during the gradient calculation stage, the GPU device visible list is locked and the upper limit of the number of available streams is fixed to avoid resource narrowing caused by out-of-bounds access. The necessity of namespace isolation is to separate system calls and kernel resource views, so that even if the upper-layer scheduling temporarily fails, the hard threshold of the container boundary still exists.

[0067] The sharing of computing cores uses time-slice rotation. GPU-side isolation is achieved through a combination of two methods:

[0068] One approach is to use GPU slicing / multi-instance or vGPU quota marking when available to form space shares;

[0069] Secondly, when slicing is not possible or fine-grained control is required, the driver layer time slice scheduler is enabled to map the tenant's CUDA streams to scheduling entities and rotate kernel launches according to quotas.

[0070] The round-robin parameters are defined by a triplet of "time slice length, maximum concurrent kernel count, and stream priority mapping table". The selection of the time slice length requires a trade-off between context switching overhead and the ability to suppress starvation for long tasks. In this embodiment, based on the computing power stability in S101, longer slices are used in stages with high stability to reduce switching overhead, while shorter slices are used in stages with large fluctuations to limit bursty occupancy. The GPU driver plugin maintains an instruction queue for each tenant, controls the kernel submission rate through a token bucket, implements preemption points on the queue side, and adds fence callbacks to large kernel functions to prevent long-term blocking during inter-slice switching.

[0071] Gradient synchronization communication is isolated at the virtual switch level. Each training node's container veth is connected to the virtual switch, tagged with VLAN / TC class based on tenant identifier, and configured with HTB hierarchical queues and priorities. Synchronization traffic such as AllReduce is matched to high-priority channels, with its bandwidth limit calculated by S101's B_limit and node link margin. Model saving and data loading traffic enters low-priority queues to avoid congestion during the synchronization window. For cross-node communication, the control plane injects DSCP tags into the RDMA or TCP stack layer, accessing the queue scheduling rules on the switching device. If a tenant experiences a sudden anomaly, the policy engine modifies its queue bandwidth parameters at the millisecond level, dynamically shaping the queue without affecting the throughput of other tenants.

[0072] To ensure equitable queuing, this embodiment maps training instructions to independent submission queues per tenant. This mapping occurs within the container by intercepting critical calls to the CUDA Runtime or communication library, extracting the tenant ID, and appending it to the kernel and communication request metadata. The driver maintains hash dispatch, placing requests from the same tenant into the same queue, preventing head-of-line blocking between tenants. Combined with time slices, this mechanism is equivalent to adding a fairness gate before the hardware, preventing long-running kernel functions from occupying the device's front-end scheduling port for extended periods and affecting others.

[0073] This embodiment uses a dual indicator of "memory curve + computation latency" to identify competition. The memory curve is derived from container-level memory usage records, and the computation latency is derived from the kernel execution time within a training step and the total step latency. If, within an adjacent time window, tenant A's memory usage increases and stabilizes at a high level, while tenant B's kernel queuing time lengthens or step latency increases, then resource contention is determined to exist. If, at the same time, network bandwidth statistics show that tenant A's synchronization peaks overlap and tenant B's effective bandwidth decreases, then the competition is further confirmed at the communication level.

[0074] The identification results are organized into a conflict graph, where the vertices are training nodes (or the projections of tenants onto nodes), and edges represent competitive relationships. Edge weights are weighted sums of waiting latency increments, memory stacking ratios, and bandwidth yield ratios, normalized to fall within the range [0,1]. Community partitioning is performed on the graph to identify interfering clusters; vertex centrality and betweenness are calculated to identify "high-influence" nodes. Conflict feature vectors are then extracted and used as input for subsequent isolation parameter adjustments. If a tenant exhibits high centrality across multiple nodes, it indicates a global suppressive effect on its training instructions, requiring tightening at both the computation and communication ends. If edge weights are high only on individual links, it's likely a local link or storage bottleneck; in this case, only the queue and rate limiting of that link need to be adjusted.

[0075] The generation of isolation parameters follows a coupling rule of "batch size - computing power quota - synchronization bandwidth". Batch size directly affects memory and computing intensity, computing power quota limits kernel concurrency and time slice ratio, and bandwidth limit constrains the synchronization window width.

[0076] This embodiment merges the conflict features with the boundary matrix of S101 to provide parameter triples for each tenant at each stage. If the conflict graph shows that the tenant causes high latency to adjacent nodes during the gradient calculation stage, its time slice ratio and the number of kernels that can be submitted per step are reduced first. If the conflict is concentrated in the synchronization stage, the gradient accumulation steps are appropriately increased or the synchronization interval is widened without disrupting training convergence, making bandwidth usage smoother, and the peak bandwidth threshold of its queue is lowered. The batch size adjustment follows the memory boundary, prioritizing tenants with redundancy to avoid causing OOM. After the parameter update is written back to the container and switch configuration, the changes in the conflict graph in the next cycle are observed to form a closed loop.

[0077] Consider two specific scenarios. First, multiple tenants share an 8-GPU server. Tenant A occupies a large amount of bandwidth in the AllReduce segment, and the edge weights in the conflict graph are concentrated between AB and AC, exhibiting high bandwidth backoff characteristics. This embodiment lowers the synchronization queue limit for A and raises the synchronization priority of B and C by one level. In the next cycle, the edge weights decrease, and the system maintains its convergence speed. Second, tenant D's memory curve shows a step increase in the later stages of training and remains unreleased for a long time. Tenant E's kernel waiting time continuously increases, and the edge weights increase in the computational dimension. The system shortens D's time slice, limits the number of concurrent kernels to a smaller upper limit, and enables a threshold alarm for memory release and reclamation for its containers, waiting for its training script to trigger gradient checkpoints to reduce memory usage.

[0078] To formalize the decision-making process, this embodiment sets up a simple conflict score for threshold determination:

[0079] S = a·ΔT + b·R_mem + c·R_bw,

[0080] Where ΔT represents the average latency increment on the adjacent edge of the target tenant (physical meaning: the queuing lengthened due to contention), R_mem is the memory overlay ratio (physical meaning: the proportion of the tenant's memory exceeding the boundary on the shared node), R_bw is the bandwidth yield ratio (physical meaning: the ratio of the available bandwidth of the link where the tenant is located to that of others), and a, b, and c are weights (physical meaning: the relative importance of the three types of resources in the isolation decision). When S exceeds the threshold, the tightening strategy of the parameter triplet is triggered; when S falls back to the low threshold, the limit is gradually withdrawn to avoid long-term bias.

[0081] Through the above process, the technical problem solved by this embodiment is: in a multi-tenant, phase-asynchronous, and three-dimensionally coupled resource environment, how to implement soft policies to verifiable hard boundaries and achieve rapid convergence when interference occurs. The implementation effect is manifested in clear boundaries of the training process at the container, computing core, and network levels; interference can be identified and suppressed at specific points; isolation parameters can be dynamically adjusted according to phase and conflict states; and the environment received by the subsequent scheduler is smoother and more controllable.

[0082] Step S103: Schedule training tasks based on candidate scheme generation and layer-by-layer screening method, calculate task priority score according to the isolation parameter, input the priority score and conflict feature into the scheduler to generate training resource allocation scheme, perform training node expansion and task migration, record graphics card usage data and training events to the audit system, and perform billing and settlement according to training duration, model size and communication volume.

[0083] This embodiment revolves around step S103, with the background being multi-tenant parallel training of a large model on a shared GPU cluster. In step S101, a resource profile matrix and state space boundaries were output. In step S102, the boundaries were degraded to three layers of isolation: containers, computing cores, and the network, and a conflict graph and isolation parameter triples [M_max, C_quota, B_limit] were generated in real time. Based on this, the task of the current step is to transform the "executable resource boundaries" into "implementable scheduling actions." Through candidate scheme generation, layer-by-layer screening, and execution closed loop, an integrated process of scaling / migration, resource reallocation, and auditing and billing is achieved.

[0084] In this embodiment, multiple candidate schemes are generated for each active training task in each scheduling cycle T. Each scheme consists of three types of decision variables: a node allocation set N (allowing the addition / release of nodes or cross-rack migration), a resource allocation vector Q (the number of GPU slices, time slice ratio, CPU / memory quota, and bandwidth limit per node), and training level parameters A (batch size, gradient accumulation steps, and synchronization interval). Candidate generation is not blind enumeration but rather constrained by the resource trajectory of S101 and the isolation parameters of S102, eliminating choices that conflict with the boundaries. For example, if B_limit is under tight constraint, the generator does not propose a scheme to increase synchronization bandwidth but instead provides an equivalent combination that increases gradient accumulation to reduce peak bandwidth. For distributed training, the selection of N must satisfy topological constraints (communication domains of the same task should preferably fall within the same exchange domain); otherwise, communication costs are penalized in the scoring.

[0085] To compare heterogeneous candidates, this embodiment defines a task priority score P, which comprehensively considers the Service Level Agreement (SLA) weight w_s, task urgency w_d (e.g., jobs nearing the assessment deadline), resource utilization benefit w_u (the absorption effect of idle resources), and conflict mitigation contribution w_c. The underlying quantities for the score are all observable in previous steps, avoiding the introduction of unverifiable subjective terms. In the specific calculation, the feasibility of each candidate is first scored using isolation parameter triples (avoiding memory limits, exceeding computing power quotas, and exceeding bandwidth limits). Then, the impact of the scheme on adjacent tasks is evaluated using conflict feature vectors (conflict graph centrality and edge weight summaries from S102). Candidates that can reduce high edge weights receive higher scores. The logic here is to simultaneously incorporate "feasibility within resource boundaries" and "controllability of cross-tenant interference," avoiding prioritizing single-task throughput at the expense of overall stability.

[0086] The candidate solution selection adopts a "layer-by-layer screening" structure. The first layer is a constraint feasibility screening: eliminating solutions that violate hard boundaries and node capacity; the second layer is a consistency screening: checking for global resource conflicts between solutions, such as two tasks competing for the same GPU slice or the bandwidth peak of the same switch port; the third layer is a multi-objective scoring screening, which sorts the solutions that pass the first two layers using a multi-objective function J. For ease of implementation and explanation, this embodiment provides a linearized scoring form:

[0087] J = α·Throughput_gain + β·Fairness_gain − γ·Migration_cost − δ·Risk,

[0088] Where Throughput_gain represents the gain on global task throughput (physical meaning: the increase in the number of effective training steps expected to be completed in the current cycle), Fairness_gain represents the improvement in tenant fairness (physical meaning: the degree to which the difference between resource allocation and SLA weights is reduced), Migration_cost is the cost of migration and scaling (physical meaning: model state migration time and cache invalidation loss), Risk is the risk term (physical meaning: the probability and magnitude of encountering high-conflict edges), and α, β, γ, and δ are tradeoff coefficients (physical meaning: the priority of scheduling objectives). Each component in this form can be estimated from profiles, conflict graphs, and audit history, following the natural relationship between resources and performance: the solution with greater benefits, smaller costs, and lower risks is prioritized.

[0089] Once the selected solution for each task is determined, the execution phase begins. The scaling operation first reserves GPU quotas and network identifiers on the target node, initializes the container image and necessary data cache; the migration operation follows a "cold before hot" order: first, the static model weights and dependency libraries are copied, then the weights are reconnected through the checkpoint of the distributed training framework or the parameter server, and the training steps are resumed after a short pause.

[0090] To reduce communication interruptions, this embodiment pre-configures the destination container's queue and DSCP on the virtual switch, switching the routing label instantly upon migration to shorten the black hole period. If the candidate scheme involves batch size and gradient accumulation adjustments, the executor injects configuration change events within the training task to ensure they take effect at the step boundary, avoiding abnormal loss curves caused by mid-process changes.

[0091] The effectiveness and traceability of scheduling need to be quantified and recorded. This embodiment generates audit entries at the execution layer for each resource adjustment, including: tenant identifier, task ID, node / device list, GPU usage share change, bandwidth limit change, reason for change (e.g., "conflict graph edge weight threshold exceeded"), timestamp, and operator (policy engine). GPU usage data during training is sampled and stored in the database as a time series, with fields including GPU duration, SM activity rate, memory allocation, and communication bytes. Training events such as scaling up, scaling down, migration, and failure rollback are all generated into structured logs. This data provides factual basis for candidate generation in the next cycle and raw data for the billing list.

[0092] On the billing side, this embodiment calculates cost items based on audit data, mapping GPU usage time to basic training costs, model parameter size to computing power percentage costs, and communication volume to transmission costs. Parameter size is derived from task metadata and static analysis, while communication volume is accumulated from the byte count of the network acquisition layer. In multi-node tasks, the costs of each node are aggregated to the task according to usage proportions and then merged by tenant account. To avoid conflicts with scheduling objectives, billing does not directly affect real-time decisions but serves as input to long-term strategy weights, reflecting the natural decrease in priority of "high-consumption, low-SLA" tasks in subsequent cycles.

[0093] Consider two typical implementation scenarios.

[0094] First, tenant A enters the AllReduce peak, increasing the edge weights of A and its multiple adjacencies on the conflict graph. The candidate generator proposes two solutions: reduce the synchronization bandwidth threshold of A and increase gradient accumulation, or migrate A to a rack with more bandwidth. The multi-objective scoring gives the first solution a lower migration cost and faster convergence. The executor adjusts the bandwidth and training parameters while keeping the nodes unchanged. In the next cycle, the edge weights decrease, and the audit records the change "reason = communication conflict exceeding the threshold".

[0095] Secondly, tenant D's job is close to its SLA deadline, so the Fairness_gain item is weighted higher, and the screening process tends to give it an extra GPU slice. However, since the C_quota of E and F on the same node is close to the upper limit, the first-level feasibility screening excludes "same-machine expansion". In the end, cross-node expansion and a small migration of task E are chosen. The migration cost increases, but the overall multi-objective value is still better, so it is adopted.

[0096] To prevent frequent oscillations, this embodiment sets a fallback threshold and cooldown time in the policy engine: if J increases insufficiently for two consecutive cycles and the conflict score remains high, a more "hard" action (migration or batch size rollback) is triggered; conversely, if the conflict score falls back to a low threshold, the previously tightened quota is gradually released to avoid long-term suppression. Combined with the real-time conflict detection in preceding S102 and subsequent auditing and billing, this scheduling closed loop translates the phased resource profile into an executable resource allocation scheme, enabling multi-tenant training to maintain an interpretable balance between interference and fairness.

[0097] As described above, the resource processing method for a multi-tenant large-scale model training platform provided in this application can effectively express requirements through an innovatively designed resource profiling system, using stage division and state space. It constructs a multi-level isolation mechanism, combining conflict analysis and parameter optimization to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively solves the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training.

[0098] In one embodiment of the resource processing method for a multi-tenant large model training platform of this application, the method may further include the following:

[0099] Step S201: Based on the resource monitoring module, collect the memory usage, computing core utilization, and network bandwidth consumption of the training task, align and label the monitoring data according to the time series, construct an original dataset containing timestamps, resource types, and numerical attributes, and resample and standardize the original dataset.

[0100] Step S202: Divide the standardized data into parameter loading stage, gradient calculation stage, parameter update stage, and model saving stage according to the training progress. Calculate the average and fluctuation range of resource requirements for each stage and organize the calculation results into a resource profile matrix.

[0101] This embodiment focuses on steps S201 to S202, running in a multi-tenant shared distributed training platform where GPUs, networks, and storage are reused across multiple nodes. The goal is to transform the original resource monitoring stream into a computable and comparable staged resource profile matrix, providing a consistent data foundation for the state space of the preceding S101 and the isolation of the subsequent S102 and scheduling of S103.

[0102] In this embodiment, the resource monitoring module's acquisition topology is first set up in S201. A lightweight probe is loaded into each training container to read the GPU driver layer's memory usage and SM activity rate, as well as the byte count and latency samples of the communication library (such as NCCL / RDMA), supplementing the CPU and I / O context information. The node-side daemon process aggregates the container's reports, forming three main metrics: memory usage Mem(t), computing core utilization Core(t), and network bandwidth Bw(t), all carrying tenant ID, task ID, and node and device identifiers. To avoid clock skew between different nodes, sampling is doubly annotated with the training step global step and the local clock, and alignment is performed according to a "step priority, timestamp backup" strategy when storing data.

[0103] The alignment rule is as follows: within the same task, samples from multiple devices are merged into one record first, based on the step. If the step cannot be obtained, time window alignment is used (e.g., aggregation with a 50ms window). This process ensures that the comparison dimensions for subsequent stage divisions are consistent with the training semantics.

[0104] Raw data is prone to missing measurements, glitches, and inconsistencies in scale due to different units. In this embodiment, resampling and standardization are performed after the data is stored.

[0105] Resampling addresses non-uniform records by generating a target sequence on the timeline with a fixed step size (by step or second), employing proximity preservation or linear interpolation for missing points. To avoid introducing spurious peaks, GPU memory is backfilled, as memory is a monotonically or step-changing resource. SMA smoothing is used for bandwidth to suppress microsecond-level jitter. Standardization employs a piecewise robust metric: for each task and each metric, the median (Median) and interquartile range (IQR) are calculated within a rolling window, and the value z = (x − Median) / IQR, eliminating dimensional differences and reducing the impact of outliers. For distributed multi-GPU tasks, this embodiment performs device aggregation before resampling: Mem is summed, Cores are weighted averages (weighted by the percentage of active time for each GPU), and Bw is summed or maximized according to parallel communication mode (aggregate communication peaks are based on bottleneck links). The final constructed original dataset is stored in triplet form.<timestamp / step, metric_type, value> Include metadata to preserve traceability.

[0106] Entering S202, the standardized time series needs to be divided into four stages based on the training progress. This division does not rely on external manual annotation but uses a hybrid rule of "index morphology + training semantics." The parameter loading stage is characterized by a positive first-order difference in the memory sequence that exceeds the threshold θm within a continuous window, while the Core is still not high and Bw is low. The gradient calculation stage is characterized by high Core and low variance, Mem basically plateauing, and Bw exhibiting periodic pulses. The parameter update stage is characterized by a superposition of Bw peaks and Core declines within a short window, with minimal Mem fluctuations. The model saving stage is characterized by low Core, a continuous increase in Bw or I / O rate, and a duration significantly longer than single-step synchronization. Considering that pipeline parallelism or mixed precision can blur boundaries, this embodiment introduces a backtracking strategy: first, the "stable high Core" backbone segment is locked for gradient calculation, and then loading and updates are anchored before and after it using Mem ramp-up and Bw peaks. The remaining unclassified windows are recorded as mixed segments but are still included in subsequent statistics, only with reduced weights, to ensure that there are no time gaps in the subsequent profiling matrix.

[0107] Once the phases are determined, the mean and fluctuation range of resource demand for each phase are calculated. The mean is taken as the expected value within the window of that phase, and the fluctuation range is represented by quantile bandwidth (e.g., P90−P10) to reduce the impact of outliers. For Mem, peak values ​​and step counts are additionally recorded to identify the risk of a continuous increase in occupancy; for Core, the mean and stability (which can be represented by the reciprocal of the variance) are recorded for subsequent time slice configuration; for Bw, the mean, peak values, and periodicity are recorded (the pulse period is obtained through autocorrelation or dominant frequency detection). If the span of a certain phase is too short to be statistically accurate, strafing aggregation is used to expand the window, or smooth extrapolation is performed from adjacent phases to logically avoid empty columns in the matrix due to data sparsity.

[0108] The above statistics are organized into a resource profile matrix R. The matrix is ​​arranged by rows corresponding to four stages (including additional rows for mixed stages if necessary), and by columns, the features are: Mem_mean, Mem_peak, Mem_range, Core_mean, Core_stability, Bw_mean, Bw_peak, and Bw_period. Matrix values ​​are standardized to maintain comparability while retaining a mirror copy of the original dimensions for boundary value conversion. R is the direct input for the subsequent S101 3D state space mapping and also provides an objective basis for the isolation parameters in S102. For example, gradient calculation segments with high Mem_peak but high Core_stability tend to be configured with higher memory limits and longer time slices; synchronization segments with high Bw_peak and short Bw_period are more suitable for peak limiting and staggering, and adjusting gradient accumulation.

[0109] In engineering scenarios, distributed training often involves heterogeneous nodes and uneven link distribution. To avoid a single slow node polluting the global statistics, this embodiment introduces a "bottleneck identification followed by aggregation" strategy: if there is a systematic deviation in the Bw or Core values ​​of multiple nodes for the same task, the bottleneck node is identified first. During stage statistics, the "global average feature" and "bottleneck feature" are calculated separately and represented as two columns in R. Subsequent scheduling can then determine whether to migrate the bottleneck or level out the parameters. Furthermore, when the batch size changes during training, the levels of Mem and Core values ​​will change instantaneously. In this embodiment, when a batch size event is detected within the stage window, the window is divided into sub-stages to prevent different batch sizes from being mixed into the same statistics, thus maintaining the physical interpretability of the profile.

[0110] To establish a clear trigger threshold for the division, this embodiment can introduce a stage discrimination score for disambiguation:

[0111] Score_phase = wm·g_mem + wc·s_core + wb·p_bw,

[0112] Where g_mem represents the memory growth rate characteristic (physical meaning: the first-order difference normalized value of Mem per unit time), s_core is the computing power stability index (physical meaning: the negative normalized value of Core variance), p_bw is the bandwidth pulse intensity (physical meaning: the normalized value of the bandwidth sequence main frequency amplitude), and wm, wc, and wb are weights (physical meaning: the relative importance of the three indicators in stage identification). The Score_phase for each corresponding stage is calculated on a sliding window, and the largest value is selected as the label for that window. A hysteresis threshold is applied to the label boundaries to reduce jitter. This definition conforms to the natural laws of training behavior: loading is observed by memory ramp-up, computation by computing power stability, and synchronization by bandwidth pulses.

[0113] Based on the processing chain of S201-S202 described above, the technical problem solved by this embodiment is that the original monitoring stream has problems such as inconsistent clocks, high noise, and inconsistencies across nodes. Directly using it for isolation and scheduling can lead to misjudgments. Through a sequence of steps—alignment, resampling, standardization, stage division, and statistical imaging—the noisy multi-source data is compressed into a matrix representation with clear stages, unified dimensions, and comparable characteristics. In a typical scenario, tenant A has a large Mem_range and high g_mem during the warm-up period. In this embodiment, this is reflected in R as a prominent loading stage feature, and a higher memory budget can be allocated in S102. Tenant B has high Bw_peak and p_bw in the model with frequent AllReduce, and R indicates that its synchronization segment has strong impulse characteristics. In S102, bandwidth shaping and queue priority can be set, and in S103, the candidate schemes tend to be staggered or gradient accumulation improved. Thus, the entire platform can maintain stable and traceable resource management logic even in a multi-tenant concurrent and stage non-stationary environment.

[0114] In one embodiment of the resource processing method for a multi-tenant large model training platform according to this application, it may further include the following:

[0115] Step S301: Calculate the resource consumption feature vectors of the training task at different stages based on the resource profile matrix, map the feature vectors to a three-dimensional state space containing the dimensions of video memory, computing power, and bandwidth, and divide the state space into grids to generate resource configuration units.

[0116] Step S302: Calculate the resource requirement trajectory of each tenant's training task, project the requirement trajectory into the grid cells of the state space, determine the resource boundary between tenants based on the occupancy of the grid cells, and generate a boundary parameter matrix including the upper limit of video memory, computing power quota, and bandwidth limit.

[0117] This embodiment focuses on steps S301 to S302, relying on the resource profile matrix R obtained in the preceding steps S201 to S202. The platform scenario involves multi-tenant distributed training and concurrent operation, with asynchronous stages and significant resource coupling. Therefore, it is necessary to translate the statistical profile matrix into a computable spatial trajectory and generate a boundary parameter matrix that can be directly used for isolation and scheduling.

[0118] In this embodiment, S301 first extracts the resource consumption feature vector v_s of each training task at each stage from R. The vector dimensions are selected to match the key bottlenecks: the memory dimension is taken as the composite index M_s of Mem_mean, Mem_peak, and Mem_range (the weights are given by the proportion of outliers in the stage, ensuring that the peak value receives higher weight); the computing power dimension is taken as Core_mean and corrected for stability to obtain C_s (high stability means closer to Core_mean, low stability means conservatively adjusted upward); the bandwidth dimension is taken as Bw_peak and supplemented with periodic intensity to form B_s (the peak value is used for strong pulse stages). After obtaining v_s=[M_s, C_s, B_s], it is mapped to the three-dimensional state space S. The coordinate system of S is normalized with hardware capacity, the M axis is GiB / device limit, the C axis is SM occupancy ratio, and the B axis is Gbps / link limit. All v_s are normalized to a [0,1] cube for easy cross-node comparison. To form the "minimum operable unit" in the subsequent process, S is divided into hierarchical meshes along each axis, with the step size determined by hardware-adjustable granularity. For example, memory is divided into 1 GiB or slice granularity, computing power into time slice tiers, and bandwidth into configurable steps of the shaper. Each mesh unit g(i,j,k) corresponds to a resource configuration unit, which can be directly attached to the parameter templates of container cgroups, GPU scheduling plugins, and virtual switches.

[0119] After grid partitioning, this embodiment requires constructing the resource requirement trajectory for each task. For each task, its stages (v_s) are concatenated along the training timeline, and long stages are further subdivided into segments to accommodate batch size variations or short-term anomalies. The trajectory is a sequence of time-labeled grid cells T=⟨g_t>, where each element represents the grid ID and dwell time of the stage feature vector projection. If a stage is identified as a "mixed segment," it is still projected, but marked as low confidence in the trajectory, and participates in subsequent boundary calculations with a decaying weight. For the multi-node characteristics of distributed tasks, topological reduction is performed before trajectory construction: if a bottleneck node exists within the same task, its v_s is entered into the principal component of the trajectory with "bottleneck priority," while non-bottleneck nodes are entered into the auxiliary component to indicate the transferable margin and avoid overly pessimistic or optimistic assessments.

[0120] Entering S302, the key lies in translating the overlap of multi-tenant trajectories in the state space into resource boundaries. This embodiment maintains occupancy U_g and overlap O_g on each grid cell. U_g is the proportion of the sum of the residence times of all trajectories within that cell to the scheduling cycle, reflecting "how common the demand is"; O_g is the number of tasks concurrently falling into that cell within the same time window, or the proportion of their resource overlap, reflecting "how acute the conflict is". When O_g is high and U_g has wide coverage, it indicates that the cell is a hotspot, and the boundaries need to be tightened; when O_g is low and U_g is high, it indicates normal but coexistent conditions, and the boundaries can be wider. Projecting each tenant's trajectory T onto these statistics yields its conflict risk and availability range in each cell.

[0121] The generation of the boundary parameter matrix Bnd follows a three-step process: hardware upper limit, image constraint, and conflict suppression. The first step uses the device's physical capacity to define an initial upper bound: M_max does not exceed the physical memory of a single card minus system reserves, C_quota does not exceed the sum of available time slices and time slices, and B_limit does not exceed a conservative proportion of the link's physical bandwidth. The second step tightens the constraints using image constraints: if M_s is close to its peak value at a certain stage, M_max is set to Mem_peak plus a safety margin; if C_s is consistently high, C_quota provides a higher but not excessive share; if B_s is a strong pulse with a short period, the initial value of B_limit is set to the integer upper bound of the pulse peak value, taking into account peak shifting. The third step adds conflict suppression: for hotspot units covered by the trajectory, the boundary of the corresponding dimension is reduced by the O_g weight, or a parameter substitution relationship is introduced, such as by increasing the gradient accumulation step to smooth the bandwidth peak, thus allowing B_limit to be adjusted without significantly reducing effective throughput. The output units of the boundary maintain physical dimensions, and each row is bound to a specific stage, facilitating direct pushdown of S102 to containers and switches.

[0122] To avoid boundary jitter caused by short-term noise, this embodiment uses hysteresis updates for each element of Bnd. If Bnd_old exists from the previous cycle and the current unit's O_g has not crossed the upper threshold, the new value is approached with a low learning rate; only when O_g crosses the upper threshold or U_g expands significantly is the rate tightening fast. Conversely, quotas are released only when O_g remains below the lower threshold for a period of time. This strategy ensures that boundary changes and observable conflicting changes are matched in time, preventing repeated rate limiting caused by jitter.

[0123] The above logic can be seen in two specific examples. Example 1: During the gradient calculation phase, tenant A's v_s exhibits high M_s, high C_s, and medium B_s. Its trajectory spends a long time in the high M / high C region of S, and its O_g is low in this region, similar to tenants B and C. The system allocates a higher C_quota and a near-Mem_peak M_max to A's Bnd during this phase. In the AllReduce segment, A and B's trajectories both fall into the high B unit, O_g increases, and Bnd tightens A's B_limit. Combined with queue shaping in S102, this avoids link congestion. Example 2: During the model saving phase, tenant D's trajectory falls into the medium B / medium M / low C unit, overlapping with another tenant E's checkpoint. O_g is high, but U_g is short. This embodiment chooses to apply only mild bandwidth limiting in the boundary matrix and labels the migration suggestion to the S103 candidate scheme, allowing the scheduler to decide whether to stagger the peak traffic.

[0124] To provide a computable criterion for boundary tightening, a cell-level tightening coefficient can be defined:

[0125] K_g = λ1·O_g + λ2·U_g·ρ − λ3·S_g,

[0126] Where λ1, λ2, and λ3 are weights (physical meaning: the relative impact of conflict intensity, coverage breadth, and stability), ρ is the saturation of the line or equipment where the unit is located (physical meaning: the degree to which actual resources approach the physical upper limit), and S_g is the historical stability (physical meaning: the reciprocal of the fluctuation in the unit's occupancy over past periods). When K_g exceeds the threshold, the boundary is lowered in the corresponding dimension for tenants mapped to g; if K_g is below the negative threshold, the boundary is allowed to be finely adjusted back up. Each term in this formula can be directly estimated by monitoring and profiling, and its directionality is intuitive: it tightens when there is a large overlap and the equipment is busy, and loosens when it is stable and idle.

[0127] In summary, this embodiment converts the statistical profile matrix into a 3D state trajectory and grid occupancy, and then derives the boundary parameter matrix for multi-tenants from occupancy and overlap. The output Bnd retains the stage differences and corresponds one-to-one with the isolated execution interface of S102, thus providing a clear feasible domain for candidate generation and selection in S103. Through this chain-like modeling, in the complex environment of multi-tenant asynchronous training, resource coupling, and heterogeneous hardware, resource boundaries can be quantified, interpreted, and stably pushed down for execution.

[0128] In one embodiment of the resource processing method for a multi-tenant large model training platform of this application, the method may further include the following:

[0129] Step S401: Deploy the tenant training process to run in an independent container, implement process isolation based on the container namespace, allocate computing core resources using a time-slice round-robin scheduling mechanism, divide independent communication queues in the virtual switch to achieve gradient synchronization traffic isolation, and generate isolation configuration parameters including process identifier, time slice ratio, and bandwidth quota.

[0130] Step S402: Use memory usage curve analysis and computational latency monitoring methods to identify resource competition relationships between training tasks, construct a task dependency graph based on the competition relationships, and use the dependency graph to guide the dynamic adjustment of isolation configuration parameters.

[0131] This embodiment focuses on steps S401 to S402, where multiple tenants are training concurrently within the same GPU cluster. The preceding steps have already provided the boundary parameters and conflict characteristics for each tenant at each stage. Here, these are implemented as isolated configurations across three layers: container, compute, and network. During runtime, closed-loop adjustments are made based on the competition relationship to prevent cross-tenant interference from spreading.

[0132] In this embodiment, in step S401, an independent container is first created for each training job, and the PID, IPC, Mount, UTS, and Network namespaces are enabled to separate the process tree, shared memory, file mounting, and network stack from each other. The container is based on a read-only base image with a writable layer superimposed on it, and independent drivers and library directories are mounted to prevent cross-influence between framework versions.

[0133] Subsequently, the initial budget is written to cgroups v2: CPU quota, memory limit, IO weight, and GPU device visibility list (passthrough or vGPU slice label). Core computing resources use time-slice round-robin scheduling: on the driver side, tenant CUDA streams are mapped to scheduling entities, defining the time slice length Ts and the concurrent kernel limit Kmax. The scheduler rounds kernel submissions according to a "token bucket + priority" strategy. Ts is set based on the computing power stability during the gradient calculation phase; higher stability results in a longer Ts to reduce switching overhead, while greater fluctuations result in a shorter Ts to constrain bursts. On the network side, after the container veth is connected to the virtual switch, it is tagged with TC class or VLAN by tenant ID, and an HTB hierarchical queue is configured. Synchronous traffic such as AllReduce enters the high-priority queue, with its peak bandwidth calculated by stage B_limit. Data loading and model saving use the low-priority channel.

[0134] The above landing parameters are summarized into an isolation configuration triplet.<ProcID, slice_ratio, bw_quota> Where ProcID is the training process identifier, slice_ratio is the time slice ratio (physical meaning: the proportion of the process occupied within the GPU rotation cycle), and bw_quota is the queue bandwidth quota (physical meaning: the maximum available bandwidth of the process or tenant on the switch).

[0135] Entering S402, this embodiment uses two indicators—memory usage curve and computation latency—to identify resource contention. The memory curve is periodically collected by probes within the container, taking the average and peak memory usage at each step; the computation latency consists of kernel queuing time and step time. If tenant A is detected to maintain a high memory usage on the same node and increase stepwise, while tenant B's kernel waiting latency ΔT is significantly prolonged, then contention is determined in the computation dimension for A→B; if B's ​​effective bandwidth decreases and the switch queue congestion flag is triggered within the same window, then contention is established in the communication dimension. To avoid mistaking internal model fluctuations for contention, this embodiment only performs horizontal comparisons within the same or adjacent training phases and uses detrending processing to remove periodic self-oscillations. The identified pairwise relationships are stored in a task dependency graph, with the vertices of the graph being a "tenant-node-device" ternary projection. The edge weights are a combination of three normalized indicators: memory stacking ratio Rmem, waiting latency increment ΔTnorm, and bandwidth yield ratio Rbw, which, after minimum-maximum scaling, fall within [0,1]. The larger the edge weight, the stronger the interference.

[0136] The dynamic adjustment of isolation parameters driven by the dependency graph follows the principle of "in-situ peak shaving and limited migration." When the sum of the incoming edge weights of a vertex exceeds a threshold, the system first attempts to adjust its slice_ratio and Kmax within the local node to limit kernel concurrency and shorten time slices, making queuing controllable. If the edge weights are concentrated in the communication dimension, its bw_quota is reduced or the gradient accumulation steps are increased to smooth bandwidth demand, and the priority of the affected tenant queue is increased on the switch to repair the fallback. If local adjustments are still insufficient, the scheduler refers to the community partitioning results of the dependency graph and selects low-coupling nodes to perform small-step migrations: moving data loading or auxiliary processes while retaining the main computing process to reduce interruptions. Each adjustment is recorded as a configuration version, timestamped, and written to the audit. In the next monitoring cycle, the edge weight changes are compared, and if they fall back, the parameters are gradually released and tightened to avoid long-term suppression.

[0137] Examples are given for two common scenarios.

[0138] Firstly, when tenant A performs large-batch training on a single machine with multiple GPUs, the memory curve plateaus but remains high. Meanwhile, tenant B begins to show an upward ΔT. In this embodiment, instead of directly reducing the batch size of A to avoid affecting convergence, we first reduce A's Kmax from multi-stream concurrency to single-stream and increase B's queue priority by one level to observe whether the edge weights fall back.

[0139] Secondly, when cross-machine communication is intensive, high Rbw edges (A↔C) appear on multiple nodes in the dependency graph. In this embodiment, the bw_quota of A is reduced first, and the synchronization interval of A is increased. If C is still under pressure, some workers of A are migrated to racks with less traffic. Before migration, container and queue configurations are warmed up to shorten the switching window. Through the above-explainable detection-graph construction-parameter tuning closed loop, the isolation configuration evolves with the contention situation in a fine-grained manner, providing a more stable foundation for subsequent global scheduling.

[0140] In one embodiment of the resource processing method for a multi-tenant large model training platform according to this application, it may further include the following:

[0141] Step S501: Using training nodes as graph vertices, establish connection edges between nodes based on resource competition relationships, calculate the weight coefficients of the edges, normalize the weight coefficients, and construct a conflict graph model that reflects the intensity of resource competition in the training task.

[0142] Step S502: Perform community partitioning and centrality calculation on the conflict graph, extract the node competition degree and edge competition intensity as conflict features, calculate the upper limit of batch size, computational resource quota ratio and gradient synchronization bandwidth threshold of the training task based on the conflict features, and generate an isolation parameter matrix.

[0143] This embodiment revolves around steps S501 to S502. Relying on the resource competition pairs identified in the preceding step S401 and the state space mapping results obtained in S301 to S302, in a shared cluster for multi-tenant distributed training, the original observations of the "competition relationship" are solidified into a computable conflict graph model, and the graph features are translated into an executable isolation parameter matrix for the container and network layer to be deployed in stages.

[0144] In this embodiment, the vertex set and edge construction rules of the conflict graph are first determined in S501. The vertices adopt the "training node perspective", and each vertex v represents the training entity of a tenant task on a physical node (which may include multi-card aggregation), ensuring consistency with the actual operable resource domain.

[0145] The establishment of edge e(u,v) is based on resource competition: if the indicators of two vertices show a correlation of "occupancy increase - the other party retreats" within the same time window, then a connection is established. To prevent misleading results from accidental synchronization, the correlation uses a detrended Pearson coefficient and a Granger causality test with a lag window as soft conditions, and is only calculated within the same or adjacent training phases. The edge weight prototype is derived from three normalized indicators: Rmem is the memory stacking ratio (physical meaning: the proportion of the total memory demand of the two vertices at the same node to the total physical memory), Rcomp is the normalized value of the waiting delay increment (physical meaning: the proportion of the kernel queuing time relative to the baseline due to the existence of the other party), and Rbw is the bandwidth retreat ratio (physical meaning: the relative extent to which the available bandwidth of the link is occupied by the other party). The initial edge weight can be a weighted sum w0 = a·Rmem + b·Rcomp + c·Rbw.

[0146] Since different nodes and links have different upper limits, this embodiment performs context normalization on w0: it performs scale shaping based on the node's resource saturation ρ (relative upper limit of memory / SM / bandwidth usage) and observation stability S (the reciprocal of the index fluctuation within the past window) to obtain w = (w0·ρ) / S, and then truncates w to [0,1]. The conflict graph G(V,E,w) constructed in this way can intuitively reflect the competition intensity and can be compared across nodes.

[0147] In step S502, this embodiment performs community partitioning and centrality analysis on the conflict graph to extract "who interferes most strongly with whom and who is the key influencer". Community partitioning can be performed using modularity maximization or spectral clustering, with the goal of grouping nodes with high edge weights into the same cluster. The resulting clusters represent potential resource conflict domains. Two types of centrality are calculated for each vertex: weighted centrality k_w (physical meaning: the total competition intensity between the vertex and its neighbors) and betweenness centrality b_w (physical meaning: the frequency with which the vertex is traversed on the conflict propagation path, indicating its degree as a "bottleneck / bridge"). The original edge weights are retained at the edge level as edge competition intensity. To reduce short-term fluctuations, this embodiment uses an exponential moving average for centrality, with the window length consistent with the scheduling period, avoiding drastic parameter fluctuations caused by instantaneous spikes.

[0148] Transforming conflict features into an isolation parameter matrix requires considering the coupling between training availability and physical constraints. This embodiment defines a parameter triplet for each vertex at the current stage: ⟨B_max, C_quota, L_bw>, corresponding to the maximum batch size, the proportion of computational resource quota, and the gradient synchronization bandwidth threshold, respectively. The transformation logic follows a monotonic relationship of "high contention—tightening, low contention—relaxing," but the response directions differ for each dimension: when both k_w and b_w are high and edge contention is concentrated in the computation dimension (Rcomp-dominated), C_quota is tightened first, specifically by shortening the channel time slice and lowering the concurrent kernel limit; if edge contention is concentrated in the communication dimension (Rbw-dominated), L_bw is lowered, and it is recommended to increase the gradient accumulation steps to smooth the peak value; if Rmem remains high and multiple points in the community have high k_w, B_max is limited or mixed precision / checkpointing is triggered to alleviate memory pressure.

[0149] When substitution relationships exist (such as increasing gradient accumulation in exchange for a reduction in peak bandwidth), parameter changes should be prioritized on the side with less impact on convergence to maintain semantic stability during training.

[0150] Considering the quantitative mapping between indicators and parameters, this embodiment can employ a simplified parameter convergence iteration:

[0151] Δθ k_w, b_w, Rmem, Rcomp, Rbw) − θ / θ_cap),

[0152] Where θ represents the target parameter to be adjusted (physical meaning: it can be the current value of C_quota or L_bw), and θ_cap is the upper bound of the hardware or policy for this dimension (physical meaning: such as the upper limit of the time slice allocation ratio, the maximum threshold that can be set on the link). Let Δθ be the expected percentage calculated from the conflict characteristics (physical meaning: a safe quota recommendation derived from the intensity of competition), and η be the learning rate (physical meaning: the parameter update step size). When Δθ is negative, it tightens; when it is positive, it pulls back. This form follows a natural rule: as competition intensifies, the expected percentage decreases, and the parameters converge to a more conservative range; as competition weakens, it gradually releases, avoiding excessive restriction over a long period.

[0153] The results are clear in both types of application scenarios.

[0154] Firstly, in a single-machine multi-tenant scenario, a high-weight community is formed on the graph. Node A has high k_w and low b_w, indicating that A and its neighbors have strong mutual interference but are not cross-domain bridges. In this embodiment, C_quota is reduced and L_bw is slightly limited for A, so that neighbors B and C obtain more stable queuing.

[0155] Secondly, in cross-rack distributed synchronization, if a node E has the highest b_w among multiple clusters, it is considered a conflict "bridge". The parameter matrix prioritizes tightening L_bw for E and suggests that S103 migrate some workers to a less coupled cluster. The overall edge weight of the graph falls back in the next cycle.

[0156] For each parameter generation, this embodiment records "vertex ID, cluster ID, k_w, b_w, edge weight decomposition, old / new parameter value, and timestamp" for auditing, which facilitates review and association with the billing side.

[0157] Through S501 to S502, the platform abstracts noisy competitive observations into a structured conflict graph, and then maps the graph theory features into an executable matrix of batch size, computing power quota, and bandwidth threshold, connecting the preceding profiling with subsequent scheduling, and constructing a consistent isolation control plane across the three dimensions of GPU, network, and video memory.

[0158] In one embodiment of the resource processing method for a multi-tenant large model training platform of this application, the method may further include the following:

[0159] Step S601: Generate multiple candidate resource configuration schemes for each training task based on the task type and training stage, calculate the matching degree between the candidate schemes and the isolation parameters, calculate the priority coefficient based on the matching degree score and the task service level, and filter and sort the candidate schemes according to the priority coefficient.

[0160] Step S602: Combine the priority score with the conflict feature to form the scheduling decision input, and use a multi-objective optimization algorithm to search for the optimal configuration combination in the candidate scheme space to generate a training resource scheduling scheme that includes node allocation, resource ratio and task order.

[0161] This embodiment focuses on steps S601-S602, relying on the resource profile matrix, boundary parameter matrix Bnd, and conflict graph features obtained in the preceding steps. The goal is to synthesize static constraints and dynamic conflict states into an "executable resource configuration combination," and to provide node allocation, resource ratio, and task order through an interpretable scoring and search process, satisfying the balance between fairness and efficiency in multi-tenant concurrent training.

[0162] In this embodiment, a candidate solution set is first constructed in S601. For each training task, based on its task type (pre-training / fine-tuning / evaluation) and current stage (loading / gradient calculation / updating / saving), a limited number of feasible adjustment dimensions are enumerated: the selection of the node set N (maintaining / expanding / cross-rack migration), the resource allocation level Q (number of GPU slices, time slice ratio, CPU and bandwidth limits), and the training level parameters A (batch size limit, gradient accumulation steps, synchronization interval). Boundary filtering is introduced during enumeration: no candidate may exceed [M_max, C_quota, B_limit] in Bnd, and topological constraints must be met (the communication domain should be as close as possible to the exchange domain). To avoid combinatorial explosion, the generator retains only 2-3 levels sensitive to the target stage for each dimension. For example, in the AllReduce intensive section, the two alternative paths of "improving gradient accumulation / keeping nodes unchanged" and "cross-rack migration / maintaining bandwidth limits" are prioritized.

[0163] After candidate generation, the matching degree is calculated. The matching degree measures the degree of fit between the candidate and the isolation parameters and the degree of boundary consumption. It is defined as the sum of two items: one measure is resource fit, such as C_quota utilization not exceeding the threshold, M_max margin being sufficient, and B_limit not being penetrated; the other measure is the potential contribution to conflict mitigation, which is estimated by the decrease in the weights of the task vertex and its neighbors on the conflict graph.

[0164] In this embodiment, the two items are normalized to the [0,1] interval and then added together to obtain the matching score Score_fit. Next, a service level factor is added, and a priority coefficient Pri = f(SLA weight, deadline urgency, historical fairness difference) is calculated. The fairness difference reflects the deviation between the tenant's cumulative resource acquisition percentage and their expected SLA; larger deviations are weighted accordingly. Finally, the ranking is based on Score = Score_fit × Pri. Within the same task, candidates are selected in descending order of Score, while across tasks, the top-ranked solutions for each task are retained to ensure global diversity.

[0165] Entering S602, priority scores and conflict features need to be combined into a solvable scheduling input. In this embodiment, each candidate is encoded as a vector x, containing node allocation, resource ratio, and training-level parameters. The objective function employs multi-objective optimization, using linear scaling as an engineering trade-off under the given conditions.

[0166] Max J(x) = α·Throughput(x) + β·Fairness(x) − γ·MigCost(x) − δ·Risk(x).

[0167] Where Throughput(x) is the expected effective training step increment (physical meaning: throughput estimate considering computing power and bandwidth constraints), Fairness(x) measures the reduction in the distance between resource allocation and SLA target (physical meaning: the degree to which long-term quota bias is corrected), MigCost(x) is the latency and cache miss loss required for migration and expansion, and Risk(x) evaluates the probability of residual conflicts after implementing the scheme based on the edge weights and centrality of the conflict graph. α, β, γ, and δ are trade-off coefficients, corresponding to the system's emphasis on throughput, fairness, cost, and risk, and their values ​​are derived from the platform's policy configuration. Priority scores are used as prior weights to apply to the upper limit of Throughput and Fairness, making high-priority tasks more likely to be selected, but without exceeding the risk and cost limits.

[0168] The search process employs a two-stage approach: "candidate combination + constraint screening." The first stage uses a bundle search to generate a global combination from the candidate tops of each task, with the bundle width *b* determined by the cluster size and scheduling cycle. The second stage performs hard constraint screening on the combinations, including node capacity, peak link bandwidth, GPU slice conflicts, and the total time slices not exceeding 1, eliminating infeasible items. For the remaining combinations, J(x) is calculated, and the globally largest or several solutions on the Pareto front are selected. To avoid frequent jitter, this embodiment introduces a cooling-off period and parameter hysteresis: if the J improvement of the new solution relative to the current solution is less than a threshold, and the conflict score does not worsen, the status quo is maintained and recorded as an observation state; a switch is triggered only once the conflict score crosses the upper threshold or J significantly improves, reducing back-and-forth migration.

[0169] To illustrate interpretability, two examples are given.

[0170] Firstly, tenant A is in the AllReduce segment, and the conflict graph has a high Rbw on its neighboring edges. Candidate 1 keeps the nodes unchanged, increases gradient accumulation, and decreases B_limit; Candidate 2 migrates two cards across racks and maintains the training parameters. Score_fit is higher for candidate 1 (parameter tuning within the boundary), and Pri considers A's SLA to be moderate. When calculating J, candidate 1 has a small MigCost and sufficient Risk decrease, and is ultimately adopted.

[0171] Secondly, tenant D is close to the SLA deadline and has a high Pri, but its node conflicts with tenant E in terms of computing power (high Rcomp). The candidate set suggests "increasing the time slice ratio" and "cross-node expansion". The former violates the hard boundary of C_quota and is filtered out, while the latter, although it has MigCost in J, has significant gains in Fairness and Throughput, and has a low risk of being rejected, thus becoming the output solution.

[0172] The final training resource scheduling scheme includes: the target node set for each task, the GPU slice and time slice ratio for each node, the queue bandwidth threshold, training parameter adjustments, and execution order. This scheme is delivered to the execution layer for implementation and auditing, for use in subsequent billing and policy backtracking, forming a closed loop from candidate selection to execution.

[0173] In one embodiment of the resource processing method for a multi-tenant large model training platform of this application, the method may further include the following:

[0174] Step S701: Based on the training resource scheduling scheme and the demand for expanding computing nodes, allocate computing resources and network identifiers to the nodes to be expanded, migrate training tasks to the target nodes in the order of dependencies, and generate audit data containing resource allocation records, task migration logs, and operation timestamps.

[0175] Step S702: Extract the GPU usage time, model parameter size, and total gradient communication amount of the training task from the audit data, calculate the training cost based on the resource usage, and generate a billing list that includes basic training costs, resource consumption costs, and communication transmission costs.

[0176] This embodiment focuses on steps S701-S702. The operational background is that the multi-tenant distributed training platform enters the execution closed loop after completing the scheduling decisions in S601-S602. Existing training resource scheduling schemes include node allocation, resource ratio, and task order. Currently, it is necessary to transform these "configuration intentions" into actual expansion and migration actions and form traceable audit data. Subsequently, billing and settlement are performed based on the audit data, and the cost items are calculated according to the natural correspondence between the actual resource usage and the actual usage, avoiding subjective estimations that are detached from physical quantities.

[0177] In this embodiment, the node expansion requirement is first calculated in step S701. The scheduling scheme provides a target computing power share and link threshold for each task. The execution layer generates an expansion plan ΔN based on the difference between the target and the current situation. Each node to be expanded needs to verify three availability items: GPU slice / time slice availability, network port and queue resources, and storage path capacity. If a node has insufficient GPU slices but sufficient time slices, time slice expansion is chosen instead of adding a new container to reduce image pulling and model reloading costs. Conversely, if the link queue is close to B_limit but GPUs are sufficient, other nodes within the same switching domain are selected first to reduce cross-domain communication costs. After the availability check, the system allocates resources to the node to be expanded: GPU / CPU / memory quotas are written to cgroups, slice identifiers or time slice ratios are set in the driver plugin, independent communication queues are allocated to the virtual switch and bandwidth limits are set, and the network side allocates tenant-isolated virtual network segments and routing labels to the new container to ensure topology consistency with existing containers.

[0178] The migration order is determined based on the dependency graph within the training task. Taking parameter server or full synchronization as an example, stateless or weakly stateful components (data loader, evaluation process) are migrated first, then some workers are migrated at the synchronization window boundary, and finally the state-heavy master worker is processed. The migration execution follows a "warm-up-switch-check" sequence: In the warm-up phase, the container is launched on the target node and the model weights and dependency libraries are preloaded; in the switch phase, there is a short pause at the global step boundary, the source-side gradient queue is frozen, the endpoint address of the parameter server or NCCL is updated, and traffic is switched to the target queue; in the check phase, the SM activity rate and bandwidth utilization before and after the migration are compared to see if they fall within the range specified by Bnd. If they deviate, a rollback or secondary reshaping is performed. Each operation generates an audit entry, with fields including resource allocation records (node ​​ID, GPU share, bandwidth quota, time slice ratio), task migration logs (source / destination node, component type, pause duration, number of retries), and operation timestamps and trigger reasons (e.g., "scheduling scheme v_k execution" or "conflict score exceeds threshold"). Audit entries are stored in a hashable chain to ensure consistency between subsequent cost calculations and policy backtracking.

[0179] In a typical scenario, tenant A is scheduled to increase gradient accumulation and expand its capacity by one GPU within the same rack due to an AllReduce conflict. The system detects that the current node has insufficient GPU slices but sufficient time slices, so it chooses to increase the time slice ratio of existing containers and start a new high-priority communication queue, without requiring a full migration; the audit log records the expansion type as "in-place," with a near-zero pause time. Another task, B, is migrated to an adjacent rack due to link congestion. Before execution, DSCP and HTB are pre-configured on the target switch, then switched at the synchronization point. After a successful rollback check, a "migration successful" log is written. If a rollback check fails, the audit retains a record of "rollback to v_{k-1} scheme" with a failure code for subsequent analysis.

[0180] Upon entering S702, the billing system extracts three basic quantities to support the costs from the audit data. GPU usage time is derived from the GPU quota and active time points per container, aggregated by tenant and task, reflecting the measurement caliber of basic training costs. Model parameter size is determined by the task metadata and the total number of parameters reported by the training framework, used to map the complexity factor of computing power proportion, avoiding unreasonable costs for small and large models with the same GPU usage time. Gradient communication volume is provided by virtual switches and RDMA counters, accumulated by tenant-isolated queues, reflecting communication transmission costs. To avoid duplicate metering, distributed tasks in multi-node scenarios are aggregated at the task granularity, eliminating intra-task loopbacks and local cache hits, retaining only valid communication bytes across devices and nodes.

[0181] The cost calculation follows a one-to-one mapping from physical quantities to cost items. Basic training costs are priced based on GPU usage time. Resource consumption costs combine the parameter scaling factor and the long-term average of SM activity rates to express the natural relationship that "more complex models consume more computing power for the same duration." Communication transmission costs are priced linearly or in segments based on the total amount of gradient communication. A congestion coefficient can be introduced during periods of link congestion, but this coefficient is derived from network saturation observations in audits and is not subjectively added. Based on this, a billing list is generated, including cost details, time intervals, resource usage summaries, and corresponding audit hashes for task and tenant dimensions, facilitating reconciliation and traceability.

[0182] To reduce fluctuations during the cycle, this embodiment introduces smoothing and anomaly verification at the billing settlement end. Smoothing is based on median filtering using a rolling window to avoid billing spikes caused by short-term migrations. Anomaly verification compares the audited resource allocation version with the collected actual usage curve. If a situation occurs where "usage is below the quota but costs are abnormally high," the operation log is traced back to verify whether there are duplicate billings or undeducted charges during failed rollbacks. After the list is generated, it is written back to the policy library. In the long term, it can serve as input for S601 priority, allowing high-consumption but low-SLA tasks to receive more conservative candidate scores in subsequent cycles, forming a closed-loop logic from execution—auditing—billing—policy, ensuring that the relationship between resources and costs is clear, interpretable, and stable in a multi-tenant environment.

[0183] To effectively address the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, and to provide technical support for large-scale model training, this application provides an embodiment of a resource processing device for a multi-tenant large-scale model training platform, which implements all or part of the resource processing method for the aforementioned multi-tenant large-scale model training platform. Referring to Figure 2, the resource processing device for the multi-tenant large-scale model training platform specifically includes the following components:

[0184] The resource partitioning module 10 is used to collect resource usage data of training tasks, divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage and model saving stage according to model training characteristics, normalize the resource demand data of each stage to generate a resource profile matrix, construct a state space containing video memory usage, computing power load and communication bandwidth based on the resource profile matrix, and dynamically partition the tenant training tasks according to the state space.

[0185] The resource isolation module 20 is used to perform multi-level resource isolation. It isolates the training process through container namespaces, isolates the computing core through time-slice round-robin, isolates gradient synchronization communication based on virtual switches, maps training instructions to independent queues according to tenant identifiers, identifies the competition relationship of training tasks using memory curves and computing latency detection methods, constructs a conflict graph with training nodes as vertices and competition relationships as edges, extracts conflict features from the conflict graph, and generates isolation parameters including batch size, computing quota, and synchronization bandwidth.

[0186] Resource scheduling module 30 is used to schedule training tasks based on candidate scheme generation and layer-by-layer screening methods, calculate task priority scores according to the isolation parameters, input the priority scores and conflict features into the scheduler to generate training resource allocation schemes, perform training node expansion and task migration, record graphics card usage data and training events to the auditing system, and perform billing and settlement according to training duration, model size and communication volume.

[0187] As described above, the resource processing device for a multi-tenant large-scale model training platform provided in this application can effectively express requirements through an innovatively designed resource profiling system, using stage division and state space. It constructs a multi-level isolation mechanism, combining conflict analysis and parameter optimization to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively solves the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training.

[0188] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, and to provide technical support for large-scale model training, this application provides an embodiment of an electronic device for implementing all or part of the resource processing method for the multi-tenant large-scale model training platform. The electronic device specifically includes the following components:

[0189] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the resource processing device of the multi-tenant large model training platform and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the resource processing method and the resource processing device of the multi-tenant large model training platform in the embodiments, the contents of which are incorporated herein, and repeated details will not be described again.

[0190] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0191] In practical applications, the resource processing method for a multi-tenant large model training platform can be partially executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0192] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0193] Figure 3 is a schematic block diagram of the system configuration of an electronic device 9600 according to an embodiment of this application. As shown in Figure 3, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0194] In one embodiment, the resource processing method for a multi-tenant large model training platform can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0195] Step S101: Collect resource usage data of training tasks, divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage and model saving stage according to model training characteristics, normalize the resource demand data of each stage to generate a resource profile matrix, construct a state space containing video memory usage, computing power load and communication bandwidth based on the resource profile matrix, and dynamically divide the tenant training tasks according to the state space.

[0196] Step S102: Perform multi-level resource isolation, isolate the training process through container namespaces, isolate the computing core using time-slice round-robin, isolate gradient synchronization communication based on virtual switches, map training instructions to independent queues according to tenant identifiers, identify the competition relationship of training tasks using memory curves and computation latency detection methods, construct a conflict graph with training nodes as vertices and competition relationships as edges, extract conflict features from the conflict graph, and generate isolation parameters including batch size, computation quota, and synchronization bandwidth;

[0197] Step S103: Schedule training tasks based on candidate scheme generation and layer-by-layer screening method, calculate task priority score according to the isolation parameter, input the priority score and conflict feature into the scheduler to generate training resource allocation scheme, perform training node expansion and task migration, record graphics card usage data and training events to the audit system, and perform billing and settlement according to training duration, model size and communication volume.

[0198] As described above, the electronic device provided in this application, through an innovatively designed resource profiling system, effectively expresses requirements via stage division and state space. It constructs a multi-level isolation mechanism, combining conflict analysis and parameter optimization to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively addresses the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training.

[0199] In another embodiment, the resource processing device for the multi-tenant large model training platform can be configured separately from the central processing unit 9100. For example, the resource processing device for the multi-tenant large model training platform can be configured as a chip connected to the central processing unit 9100, and the resource processing method function of the multi-tenant large model training platform can be implemented through the control of the central processing unit.

[0200] As shown in Figure 3, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily include all the components shown in Figure 3; furthermore, the electronic device 9600 may also include components not shown in Figure 3, as can be found in existing technologies.

[0201] As shown in Figure 3, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0202] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0203] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0204] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0205] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0206] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0207] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0208] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the resource processing method for a multi-tenant large model training platform with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the resource processing method for a multi-tenant large model training platform with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0209] Step S101: Collect resource usage data of training tasks, divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage and model saving stage according to model training characteristics, normalize the resource demand data of each stage to generate a resource profile matrix, construct a state space containing video memory usage, computing power load and communication bandwidth based on the resource profile matrix, and dynamically divide the tenant training tasks according to the state space.

[0210] Step S102: Perform multi-level resource isolation, isolate the training process through container namespaces, isolate the computing core using time-slice round-robin, isolate gradient synchronization communication based on virtual switches, map training instructions to independent queues according to tenant identifiers, identify the competition relationship of training tasks using memory curves and computation latency detection methods, construct a conflict graph with training nodes as vertices and competition relationships as edges, extract conflict features from the conflict graph, and generate isolation parameters including batch size, computation quota, and synchronization bandwidth;

[0211] Step S103: Schedule training tasks based on candidate scheme generation and layer-by-layer screening method, calculate task priority score according to the isolation parameter, input the priority score and conflict feature into the scheduler to generate training resource allocation scheme, perform training node expansion and task migration, record graphics card usage data and training events to the audit system, and perform billing and settlement according to training duration, model size and communication volume.

[0212] As described above, the computer-readable storage medium provided in this application, through an innovative design of a resource profiling system, achieves effective expression of requirements via stage division and state space. It constructs a multi-level isolation mechanism, combining conflict analysis and parameter optimization to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively addresses the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training.

[0213] Embodiments of this application also provide a computer program product capable of implementing all steps of the resource processing method for a multi-tenant large model training platform, where the execution subject is a server or client, as described in the above embodiments. When executed by a processor, this computer program / instruction implements the steps of the resource processing method for a multi-tenant large model training platform. For example, the computer program / instruction implements the following steps:

[0214] Step S101: Collect resource usage data of training tasks, divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage and model saving stage according to model training characteristics, normalize the resource demand data of each stage to generate a resource profile matrix, construct a state space containing video memory usage, computing power load and communication bandwidth based on the resource profile matrix, and dynamically divide the tenant training tasks according to the state space.

[0215] Step S102: Perform multi-level resource isolation, isolate the training process through container namespaces, isolate the computing core using time-slice round-robin, isolate gradient synchronization communication based on virtual switches, map training instructions to independent queues according to tenant identifiers, identify the competition relationship of training tasks using memory curves and computation latency detection methods, construct a conflict graph with training nodes as vertices and competition relationships as edges, extract conflict features from the conflict graph, and generate isolation parameters including batch size, computation quota, and synchronization bandwidth;

[0216] Step S103: Schedule training tasks based on candidate scheme generation and layer-by-layer screening method, calculate task priority score according to the isolation parameter, input the priority score and conflict feature into the scheduler to generate training resource allocation scheme, perform training node expansion and task migration, record graphics card usage data and training events to the audit system, and perform billing and settlement according to training duration, model size and communication volume.

[0217] As described above, the computer program product provided in this application, through an innovative design of a resource profiling system, achieves effective expression of requirements via stage division and state space. It constructs a multi-level isolation mechanism, combining conflict analysis and parameter optimization to establish reliable resource protection. Intelligent scheduling is introduced, ensuring the rationality of allocation through priority evaluation and scheme generation. This method effectively solves the shortcomings of traditional technologies in resource profiling, isolation mechanisms, and scheduling optimization, providing technical support for large-scale model training.

[0218] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0219] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0220] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0221] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0222] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A resource processing method for a large-scale model training platform for multi-tenancy, characterized in that, The method includes: collecting resource usage data for training tasks; dividing the usage data into parameter loading stage, gradient calculation stage, parameter update stage, and model saving stage based on model training characteristics; normalizing the resource requirement data for each stage to generate a resource profile matrix; constructing a state space containing memory usage, computing power load, and communication bandwidth based on the resource profile matrix; and dynamically dividing the tenant training task into resource boundaries based on the state space, including: calculating resource consumption feature vectors of the training task at different stages based on the resource profile matrix; mapping the feature vectors to a three-dimensional state space containing memory dimension, computing power dimension, and bandwidth dimension; and dividing the state space into grids to generate resource configuration units. The resource requirement trajectory of each tenant's training task is calculated, and the trajectory is projected onto the grid cells of the state space. Based on the occupancy of the grid cells, the resource boundaries between tenants are determined, generating a boundary parameter matrix that includes memory limits, computing power quotas, and bandwidth constraints. Multi-level resource isolation is implemented, isolating training processes through container namespaces, isolating computing cores using a time-slice round-robin scheduling mechanism, isolating gradient synchronization communication based on virtual switches, mapping training instructions to independent queues according to tenant identifiers, and identifying competition relationships among training tasks using memory curves and computational latency detection methods. This includes: deploying tenant training processes to run in independent containers, implementing process isolation based on container namespaces, and allocating computing core resources using a time-slice round-robin scheduling mechanism. In a virtual switch, independent communication queues are partitioned to achieve gradient synchronization traffic isolation, generating isolation configuration parameters including process identifiers, time slice ratios, and bandwidth quotas. Memory usage curve analysis and computational latency monitoring methods are used to identify resource contention relationships between training tasks. Based on these contention relationships, a task dependency graph is constructed, which guides the dynamic adjustment of the isolation configuration parameters. A conflict graph is constructed with training nodes as vertices and contention relationships as edges. Conflict features are extracted from this graph to generate isolation parameters including batch size, computational quotas, and synchronization bandwidth. Training tasks are scheduled based on candidate scheme generation and layer-by-layer screening methods. Task priority scores are calculated according to the isolation parameters, and the priority scores and conflict features are input into the system. The scheduler generates a training resource allocation scheme, including: generating multiple candidate resource configuration schemes for each training task based on task type and training stage; calculating the matching degree between the candidate schemes and isolation parameters; calculating priority coefficients based on matching degree scores and task service levels; and filtering and sorting the candidate schemes according to the priority coefficients. The priority scores and conflict features are combined to form a scheduling decision input. A multi-objective optimization algorithm is used to search for the optimal configuration combination in the candidate scheme space, generating a training resource scheduling scheme that includes node allocation, resource ratio, and task order. Training node expansion and task migration are performed, GPU usage data and training events are recorded to the auditing system, and billing is settled according to training duration, model size, and communication volume.

2. The resource processing method for a large-scale model training platform for multi-tenancy as described in claim 1, characterized in that, The resource usage data of the training task is collected and divided into parameter loading, gradient calculation, parameter update, and model saving stages based on model training characteristics. The resource requirement data of each stage is normalized to generate a resource profile matrix, including: collecting the GPU memory usage, computing core utilization, and network bandwidth consumption of the training task based on the resource monitoring module; aligning and labeling the monitoring data according to the time series to construct an original dataset containing timestamps, resource types, and numerical attributes; resampling and standardizing the original dataset; dividing the standardized data into parameter loading, gradient calculation, parameter update, and model saving stages according to the training progress; calculating the mean and fluctuation range of resource requirements for each stage; and organizing the calculation results into a resource profile matrix.

3. The resource processing method for a large-scale model training platform for multi-tenancy as described in claim 1, characterized in that, The construction of a conflict graph with training nodes as vertices and competition relationships as edges, and the extraction of conflict features from the conflict graph to generate isolation parameters including batch size, computational quota, and synchronization bandwidth, includes: using training nodes as graph vertices, establishing connection edges between nodes based on resource competition relationships, calculating the weight coefficients of the edges, normalizing the weight coefficients, and constructing a conflict graph model reflecting the intensity of resource competition in the training task; performing community partitioning and centrality calculation on the conflict graph, extracting the degree of node competition and edge competition intensity as conflict features, and calculating the upper limit of batch size, computational resource quota ratio, and gradient synchronization bandwidth threshold of the training task based on the conflict features, thereby generating an isolation parameter matrix.

4. The resource processing method for a large-scale model training platform for multi-tenancy as described in claim 1, characterized in that, The process of scaling up training nodes and migrating tasks involves recording GPU usage data and training events to the auditing system, and billing and settling accounts based on training duration, model size, and communication volume. This includes: allocating computing resources and network identifiers to nodes to be scaled up based on the training resource scheduling scheme and the computing node scaling requirements; migrating training tasks to target nodes in the order of dependencies; and generating audit data containing resource allocation records, task migration logs, and operation timestamps. From the audit data, the GPU usage duration, model parameter size, and total gradient communication volume of the training tasks are extracted. The training cost is calculated based on resource usage, and a billing list containing basic training costs, resource consumption costs, and communication transmission costs is generated.

5. A resource processing device for a large-scale model training platform for multi-tenants, characterized in that, The device includes: a resource partitioning module, used to collect resource usage data of training tasks, divide the usage data into parameter loading stage, gradient calculation stage, parameter update stage, and model saving stage according to model training characteristics, normalize the resource requirement data of each stage to generate a resource profile matrix, construct a state space containing memory usage, computing power load, and communication bandwidth based on the resource profile matrix, and dynamically partition the resource boundaries of tenant training tasks according to the state space, including: calculating the resource consumption feature vectors of training tasks at different stages based on the resource profile matrix, mapping the feature vectors to a three-dimensional state space containing memory dimension, computing power dimension, and bandwidth dimension, and performing grid partitioning of the state space to generate resource configuration units; The system calculates the resource requirement trajectory of each tenant's training task, projects this trajectory onto the grid cells of the state space, determines the resource boundaries between tenants based on the grid cell occupancy, and generates a boundary parameter matrix containing memory limits, computing power quotas, and bandwidth constraints. The resource isolation module performs multi-level resource isolation, isolating training processes through container namespaces, isolating computational cores using a time-slice round-robin scheduling method, isolating gradient synchronization communication based on virtual switches, mapping training instructions to independent queues according to tenant identifiers, and identifying competition relationships among training tasks using memory curves and computational latency detection methods. This includes deploying tenant training processes to independent containers, implementing process isolation based on container namespaces, and allocating resources using a time-slice round-robin scheduling mechanism. The computational core resources are allocated by dividing independent communication queues within a virtual switch to achieve gradient synchronization traffic isolation, generating isolation configuration parameters including process identifiers, time slice allocations, and bandwidth quotas. Memory usage curve analysis and computational latency monitoring methods are used to identify resource contention relationships between training tasks. Based on these contention relationships, a task dependency graph is constructed, which guides the dynamic adjustment of isolation configuration parameters. A conflict graph is constructed with training nodes as vertices and contention relationships as edges. Conflict features are extracted from the conflict graph to generate isolation parameters including batch size, computational quotas, and synchronization bandwidth. A resource scheduling module is used to schedule training tasks based on candidate scheme generation and layer-by-layer screening methods. Task priority scores are calculated based on the isolation parameters, and these priority scores are correlated with conflict... The feature input scheduler generates a training resource allocation scheme, including: generating multiple candidate resource configuration schemes for each training task based on task type and training stage; calculating the matching degree between the candidate schemes and isolation parameters; calculating the priority coefficient based on the matching degree score and task service level; and filtering and sorting the candidate schemes according to the priority coefficient; combining the priority score and conflict features to form the scheduling decision input; using a multi-objective optimization algorithm to search for the optimal configuration combination in the candidate scheme space; generating a training resource scheduling scheme that includes node allocation, resource ratio, and task order; performing training node expansion and task migration; recording graphics card usage data and training events to the auditing system; and billing and settling according to training duration, model size, and communication volume.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the resource processing method for a large model training platform for multi-tenancy as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the resource processing method for a large, multi-tenant model training platform as described in any one of claims 1 to 4.

Images