A cloud data server hybrid training method and system

Abstract

The application discloses a cloud data server hybrid training method and system, which is applied to a cloud data server cluster containing various computing resources. The method analyzes the model structure, dataset, time delay constraint and isolation constraint of the training task to generate a task image, collects the computing node resource configuration, running state and isolation capability to generate a resource image, and constructs a co-location interference degree index table of the task and resource combination based on historical monitoring data. During scheduling, the heterogeneous computing power utilization rate, estimated training time delay deviation and co-location interference degree are taken as indexes to perform weighted evaluation on the candidate resource allocation scheme to generate a hybrid training scheduling strategy, and resource isolation is implemented through container and accelerator multi-instance division. During running, the training time delay and actual co-location interference degree are continuously monitored, the scheduling weight and co-location interference degree index are dynamically adjusted according to the deviation, closed-loop optimization of multi-task hybrid training is realized, the heterogeneous resource utilization rate is improved, and time delay violation and co-location interference are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing and distributed machine learning, in particular to a cloud data server hybrid training method and system. BACKGROUND

[0002] With the increase of the size of deep learning models and the number of training tasks, the cloud data server cluster usually contains multiple types of processors, accelerators and network resources. When a large number of training tasks are running in the cloud data server cluster at the same time, the following technical problems exist:

[0003] The computing nodes in the cloud data server cluster differ in processor type, accelerator type, memory capacity, video memory capacity and network bandwidth. If a simple matching is made according to the resource quantity declared in the training task request during scheduling, some types of resources may be idle for a long time, while some types of resources may be congested for a long time, resulting in low utilization of heterogeneous computing resources.

[0004] Different training tasks have different constraints on training latency. Some training tasks have strict iteration time requirements, and some training tasks have loose training latency constraints. The existing scheduling method is usually difficult to estimate and control the training latency by combining the task characteristics and the cluster state, and some training tasks may have a significant deviation from the training latency constraint.

[0005] When multiple training tasks are running on the same computing node, they will compete for processor time slices, video memory space and network bandwidth. If there is a lack of description of resource isolation capability and quantitative indicators for co-location interference, it is difficult to evaluate the co-location interference during scheduling, which may lead to training task performance jitter and unstable training process.

[0006] In the prior art, even if a certain monitoring and rescheduling mechanism is used, the training latency deviation and co-location interference deviation are usually not associated with the update of the scheduling weight and the interference index as explicit feedback, which leads to the fact that the scheduling strategy cannot be adaptively adjusted according to the actual performance of the training process. SUMMARY

[0007] The present application aims to overcome the shortcomings of the prior art and provide a cloud data server hybrid training method, which comprises:

[0008] Step one, receiving a training task request, parsing model structure parameters, dataset parameters, training latency constraints, resource isolation constraints, determining a training phase identifier, and generating task portrait data corresponding to the training task;

[0009] Step two, collect resource configuration data and running state data of each computing node, determine resource isolation capability index, associate resource configuration data, running state data and resource isolation capability index to generate resource portrait data;

[0010] Step three, based on historical running monitoring data, statistics performance degradation ratio under combination of different task portrait data and different resource portrait data, normalize performance degradation ratio, generate co-location interference degree index table, co-location interference degree index value is stored in co-location interference degree index table;

[0011] Step four, based on task portrait data, resource portrait data and co-location interference degree index table, construct multiple candidate resource allocation schemes, calculate heterogeneous computing resource utilization rate index, estimate training time delay deviation index and co-location interference degree index for each candidate resource allocation scheme, weight heterogeneous computing resource utilization rate index, estimate training time delay deviation index and co-location interference degree index according to preset weight corresponding to training phase identifier, generate hybrid training scheduling strategy;

[0012] Step five, according to the hybrid training scheduling strategy, allocate target computing node and resource quota, create container form training execution unit on the target computing node, configure accelerator multi-instance division parameter and resource limit parameter, start training execution;

[0013] Step six, collect running monitoring data in the training execution process, calculate actual training time delay index and actual co-location interference degree index, obtain actual training time delay deviation index based on comparison of actual training time delay index and training time delay constraint, obtain co-location interference deviation index based on comparison of actual co-location interference degree index and co-location interference degree index value, when actual training time delay deviation index or co-location interference deviation index exceeds preset threshold, adjust preset weight and hybrid training scheduling strategy, and apply the adjustment result to subsequent training execution, and carry out closed loop optimization on the hybrid training process.

[0014] Further, the receiving training task request, analyzing model structure parameters, data set parameters, training time delay constraint, resource isolation constraint, determining training phase identifier, generating task portrait data corresponding to the training task, comprising:

[0015] Divide the training task into preheating phase, stable training phase and convergence phase, assign training phase identifier to each training task; analyze model layer number, parameter size and operator type as model structure parameters from training task request, analyze sample number, sample feature dimension and data preprocessing method as data set parameters, analyze target training time delay and training task priority as training time delay constraint, analyze resource isolation level requirement as resource isolation constraint;

[0016] Based on the model structure parameters, dataset parameters, training latency constraints, resource isolation constraints and training phase identifiers, parameters representing computing resource requirements, parameters representing storage resource requirements, parameters representing communication resource requirements, latency sensitivity weights and isolation sensitivity weights are calculated and associated with training task identifiers to form task profile data.

[0017] Further, the resource configuration data and the running state data of each computing node are collected, the resource isolation capability index is determined, and the resource configuration data, the running state data and the resource isolation capability index are associated to generate resource profile data, including:

[0018] The processor type, processor quantity, accelerator type, accelerator quantity, memory capacity, video memory capacity and network bandwidth capability of each computing node are obtained as resource configuration data;

[0019] The processor utilization, accelerator utilization, memory occupancy, video memory occupancy, current number of training tasks carried and network throughput of each computing node are periodically collected as running state data;

[0020] The resource isolation capability index of each computing node is determined, including accelerator multi-instance division support identifier, processor affinity control support identifier and network bandwidth limitation support identifier;

[0021] The resource configuration data, running state data and resource isolation capability index are associated according to the computing node identifier to form the resource profile data.

[0022] Further, based on the historical running monitoring data, the performance degradation proportion under the combination of different task profile data and different resource profile data is counted, the performance degradation proportion is normalized, and a co-location interference degree index table is generated, in which the co-location interference degree index value is stored, including:

[0023] The task profile data identifier, resource profile data identifier and training iteration latency and throughput under the corresponding combination are extracted from the historical running monitoring data;

[0024] Under the condition that there are co-located training tasks, the performance degradation proportion compared with the single running training task is calculated, and the performance degradation proportion is taken as the performance degradation degree;

[0025] The performance degradation degrees corresponding to the combination of different task profile data identifiers and different resource profile data identifiers are normalized to obtain the co-location interference degree index value;

[0026] The co-location interference degree index values are organized according to the task profile data identifier and the resource profile data identifier to generate the co-location interference degree index table.

[0027] Furthermore, multiple candidate resource allocation schemes are constructed based on task profile data, resource profile data, and a co-location interference index table. For each candidate resource allocation scheme, heterogeneous computing power resource utilization, estimated training latency deviation, and co-location interference are calculated. These indicators are then weighted according to preset weights corresponding to the training stage identifiers to generate a hybrid training scheduling strategy, including:

[0028] Based on resource profile data, a set of available target computing nodes is determined, and a set of training tasks is constructed based on task profile data. Candidate resource allocation schemes are generated between the set of available target computing nodes and the set of training tasks. The candidate resource allocation schemes include the mapping relationship between training tasks and target computing nodes and the resource quota corresponding to each mapping relationship.

[0029] For each candidate resource allocation scheme, calculate the heterogeneous computing power resource utilization rate index based on resource quota and resource profile data, estimate the training latency and calculate the estimated training latency deviation index based on task profile data and resource profile data, and find the co-location interference index value in the co-location interference index table based on the task profile data identifier and resource profile data identifier in the candidate resource allocation scheme to obtain the co-location interference index.

[0030] Based on the training phase identifier, select the corresponding preset weights, and weight the heterogeneous computing power resource utilization rate index, the estimated training latency deviation index, and the co-location interference index to obtain a comprehensive evaluation result. Select the candidate resource allocation scheme with the best comprehensive evaluation result from all candidate resource allocation schemes to generate a hybrid training scheduling strategy.

[0031] Furthermore, the process of allocating target computing nodes and resource quotas according to the hybrid training scheduling strategy, creating container-type training execution units on the target computing nodes, configuring accelerator multi-instance partitioning parameters and resource limit parameters, and starting training execution includes:

[0032] Based on the hybrid training scheduling strategy, a container-type training execution unit is created on the target computing node for each training task;

[0033] Set upper limits for the number of processor cores, processor time slice quota, memory capacity, video memory capacity, and network bandwidth for each training execution unit, and limit the resources of processor, memory, video memory, and network bandwidth through a control group mechanism;

[0034] On accelerators that support multiple instance partitioning, the number of accelerator instances and the memory partitioning of each accelerator instance are configured according to the hybrid training scheduling strategy, and different training execution units are bound to different accelerator instances; within each training execution unit, training parallel parameters are configured according to the hybrid training scheduling strategy and training execution is started.

[0035] Furthermore, the process of collecting runtime monitoring data during training execution, calculating the actual training latency index and the actual co-location interference index, obtaining the actual training latency deviation index based on the comparison between the actual training latency index and the training latency constraint, and obtaining the co-location interference deviation index based on the comparison between the actual co-location interference index and the co-location interference index value, includes:

[0036] Within a preset monitoring period, the training iteration latency, throughput, processor utilization, accelerator utilization, memory usage, GPU memory usage, and task co-location relationships of each training execution unit are collected to form runtime monitoring data.

[0037] The average training iteration latency and training iteration latency variance of each training task are calculated based on the operation monitoring data. The average training iteration latency is compared with the training latency constraint to obtain the actual training latency deviation index.

[0038] Based on the performance degradation ratio recorded in the operation monitoring data during the co-location operation, the performance degradation ratio is normalized to obtain the actual co-location interference index. Based on the comparison between the actual co-location interference index and the co-location interference index value, the co-location interference deviation index is obtained.

[0039] Furthermore, when the actual training latency deviation index or co-location interference deviation index exceeds a preset threshold, the preset weights and hybrid training scheduling strategy are adjusted, and the adjustment results are applied to subsequent training execution to perform closed-loop optimization of the hybrid training process, including:

[0040] When the actual training latency deviation index exceeds the corresponding preset threshold, the weight of the estimated training latency deviation index in the preset weight is increased, and the weight of the heterogeneous computing power resource utilization index in the preset weight is decreased.

[0041] When the co-location interference deviation index exceeds the corresponding preset threshold, the weight value of the co-location interference index in the preset weight is increased, and the weight value of the heterogeneous computing power resource utilization index in the preset weight is decreased.

[0042] Based on the adjusted preset weights, an updated hybrid training scheduling strategy is generated and used in subsequent training executions.

[0043] Furthermore, the step of generating an updated hybrid training scheduling strategy based on the adjusted preset weights and using the updated hybrid training scheduling strategy in subsequent training execution includes:

[0044] Select the task profile data identifier, resource profile data identifier, and training iteration latency and throughput under the corresponding combination from the latest monitoring period from the operation monitoring data, and update the performance degradation level.

[0045] The updated performance degradation level is normalized, the co-location interference index value is updated, and the co-location interference index table is refreshed.

[0046] When generating subsequent hybrid training scheduling strategies, the refreshed co-location interference index table is used to adaptively optimize the subsequent hybrid training process.

[0047] A cloud data server hybrid training system, applying the cloud data server hybrid training method, includes: a training task access and profiling module, a resource representation and monitoring module, a co-location interference index management module, a hybrid training scheduling strategy generation module, a hybrid training execution and isolation control module, a running monitoring and adaptive adjustment module, and a data processing module.

[0048] The training task access and profiling module, resource representation and monitoring module, co-location interference index management module, hybrid training scheduling strategy generation module, hybrid training execution and isolation control module, and operation monitoring and adaptive adjustment module are respectively connected to the data processing module.

[0049] The training task access and profiling module mentioned above is used to receive training task requests, parse model structure parameters, dataset parameters, training latency constraints, resource isolation constraints, determine training stage identifiers, and generate task profiling data.

[0050] The resource characterization and monitoring module is used to collect resource configuration data and operating status data of each computing node, determine resource isolation capability indicators, and generate resource profile data.

[0051] The co-location interference index management module is used to generate a co-location interference index table and store the co-location interference index values ​​in the co-location interference index table based on the performance degradation ratio statistically based on historical operation monitoring data.

[0052] The hybrid training scheduling strategy generation module is used to construct candidate resource allocation schemes based on task profile data, resource profile data and co-location interference index table, calculate heterogeneous computing power resource utilization index, estimate training latency deviation index and co-location interference index, and generate hybrid training scheduling strategy according to preset weights.

[0053] The hybrid training execution and isolation control module is used to allocate target computing nodes and resource quotas according to the hybrid training scheduling strategy, create container-type training execution units, configure accelerator multi-instance partitioning parameters and resource limit parameters, and start training execution.

[0054] The aforementioned operation monitoring and adaptive adjustment module is used to collect operation monitoring data, calculate the actual training latency index and the actual co-location interference index, obtain the actual training latency deviation index based on the comparison between the actual training latency index and the training latency constraint, obtain the co-location interference deviation index based on the comparison between the actual co-location interference index and the co-location interference index value, and adjust the preset weights and hybrid training scheduling strategy when the actual training latency deviation index or the co-location interference deviation index exceeds the preset threshold, and update the co-location interference index table.

[0055] The beneficial effects of this invention are:

[0056] By unifying the modeling of task profile data and resource profile data, training tasks and heterogeneous computing resources can be abstracted from the level of describable fields, which is beneficial for subsequent fine-grained scheduling.

[0057] By using co-location interference index values ​​and co-location interference index tables, the performance degradation between co-location training tasks is quantified, transforming the degree of resource isolation from an abstract requirement into an assessable metric.

[0058] By differentiating between the estimated training latency deviation index and the actual training latency deviation index, and by setting preset weights for different training stages, a differentiated trade-off between training latency, resource utilization, and co-location interference is achieved at different training stages.

[0059] When the actual training latency deviation index and co-location interference deviation index exceed the preset threshold, the preset weights and hybrid training scheduling strategy are adjusted, and the co-location interference index table is updated. This achieves adaptive closed-loop optimization based on operation monitoring data, which improves the stability and resource utilization efficiency of the hybrid training process. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a hybrid training method using a cloud-based data server.

[0061] Figure 2 Flowchart for generating resource profile data. Detailed Implementation

[0062] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0063] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0064] Example 1

[0065] like Figure 1 As shown, a cloud data server hybrid training method includes:

[0066] Step 1: Receive training task requests, parse model structure parameters, dataset parameters, training latency constraints, and resource isolation constraints, determine the training stage identifier, and generate task profile data corresponding to the training task.

[0067] Step 2: Collect resource configuration data and operating status data of each computing node, determine resource isolation capability indicators, and associate resource configuration data, operating status data and resource isolation capability indicators to generate resource profile data;

[0068] Step 3: Based on historical operation monitoring data, calculate the performance degradation ratio under different combinations of task profile data and resource profile data, normalize the performance degradation ratio, generate a co-location interference index table, and store the co-location interference index value in the co-location interference index table.

[0069] Step 4: Based on task profile data, resource profile data, and co-location interference index table, construct multiple candidate resource allocation schemes. For each candidate resource allocation scheme, calculate the heterogeneous computing power resource utilization rate, estimated training latency deviation, and co-location interference index. Based on the preset weights corresponding to the training stage identifier, weight the heterogeneous computing power resource utilization rate, estimated training latency deviation, and co-location interference index to generate a hybrid training scheduling strategy.

[0070] Step 5: Allocate target computing nodes and resource quotas according to the hybrid training scheduling strategy, create container-type training execution units on the target computing nodes, configure accelerator multi-instance partitioning parameters and resource limit parameters, and start training execution;

[0071] Step six involves collecting runtime monitoring data during training execution, calculating the actual training latency index and the actual co-location interference index, obtaining the actual training latency deviation index based on a comparison between the actual training latency index and the training latency constraint, and obtaining the co-location interference deviation index based on a comparison between the actual co-location interference index and the co-location interference index value. When either the actual training latency deviation index or the co-location interference deviation index exceeds a preset threshold, the preset weights and the hybrid training scheduling strategy are adjusted, and the adjustment results are applied to subsequent training executions to perform closed-loop optimization of the hybrid training process.

[0072] Specifically, the training task access and profiling module receives training task requests and parses them to obtain:

[0073] Model structure parameters include the number of model layers, the size of model parameters, and the type of dominant operator.

[0074] Dataset parameters include the number of samples, sample feature dimensions, and data preprocessing methods.

[0075] Training latency constraints include the target training iteration time or the target total training duration and the priority of training tasks;

[0076] Resource isolation constraints: including resource isolation level requirements.

[0077] Based on the type and process of the training task, the training task is divided into a warm-up phase, a stable training phase, and a convergence phase. For example, the range of iteration rounds or loss variation conditions corresponding to different phases can be pre-configured for a certain type of model. When a training task is connected, its training phase identifier is initialized to a preset phase, and the training phase identifier can be updated according to the training progress rules during the training process.

[0078] Based on model structure parameters, dataset parameters, training latency constraints, resource isolation constraints, and training phase identifiers, task profile data is constructed.

[0079] The corresponding baseline level of computing resource requirements is looked up in a pre-established baseline table based on the model parameter size and the dominant operator type. The baseline table pre-assigns different model feature combinations to several discrete levels; for example, the first level represents low computing resource requirements, the second level represents medium requirements, and the third level represents high requirements. Depending on the batch size or data augmentation overhead, the baseline level can be adjusted upwards or downwards to obtain parameters characterizing the computing resource requirements, still expressed as discrete levels.

[0080] The estimated storage space requirement for the dataset is obtained by estimating the number of samples, the dimensionality of sample features, the storage requirement of each feature, and the amount of intermediate data generated during data preprocessing. This estimated storage space requirement is then compared with a pre-set storage capacity threshold to map the requirement to a storage resource requirement level, which serves as a parameter characterizing the storage resource demand.

[0081] Based on the model structure and expected parallel processing method, the amount of data transmitted during each gradient synchronization or activation is estimated, and combined with the synchronization frequency, the communication resource requirement is obtained. This communication requirement is compared with a preset network bandwidth threshold and mapped to a communication resource requirement level, serving as a parameter characterizing the communication resource demand.

[0082] The target training iteration time is compared with the cluster's historical average iteration time. When the target training iteration time is significantly less than the historical average iteration time, or when the training task has a high priority, the latency sensitivity weight is set to a higher level; when the target training iteration time is close to or greater than the historical average iteration time, the latency sensitivity weight is set to a medium or low level. The specific weight values ​​for each level can be set through a configuration table during deployment.

[0083] Based on resource isolation level requirements, strong isolation requirements are mapped to the highest isolation sensitivity weight, medium isolation requirements to a medium isolation sensitivity weight, and weak isolation requirements to a lower isolation sensitivity weight. The specific weight values ​​for each level can also be determined through configuration. These parameters and weights are then associated with the training task identifier to form task profile data.

[0084] like Figure 2 As shown, the resource characterization and monitoring module performs the following operations on each computing node in the cloud data server cluster:

[0085] Collect the processor type and number, accelerator type and number, memory capacity, video memory capacity, and network bandwidth capacity of each computing node, and combine the above information into resource configuration data.

[0086] Periodically collect processor utilization, accelerator utilization, memory usage, GPU memory usage, number of training tasks currently being carried, and network throughput of each computing node to form operational status data.

[0087] Determine resource isolation capability metrics for each computing node. These metrics include:

[0088] Does the system support multi-instance partitioning on a single physical accelerator? Does it support limiting the set of processor cores that a training execution unit can use through processor affinity control? Does it support setting a network bandwidth limit for the training execution unit? The multi-instance partitioning of the accelerator is used to divide a single physical accelerator into multiple accelerator instances, with different accelerator instances isolated from each other in terms of memory quotas and access boundaries. An accelerator instance refers to a logical accelerator unit with independent memory quotas and scheduling boundaries, obtained by partitioning from a single physical accelerator using the multi-instance partitioning method.

[0089] Resource configuration data, operational status data, and resource isolation capability indicators are associated according to the computing node identifier to form resource profile data, which is used to comprehensively characterize the static resource capabilities and dynamic load status of each computing node.

[0090] The co-location interference index management module constructs a co-location interference index table based on historical operation monitoring data. It extracts the task profile data identifier, resource profile data identifier, training iteration latency, and throughput corresponding to each record from the historical operation monitoring data.

[0091] For each combination of task profile data identifier and resource profile data identifier, the following statistics are compiled:

[0092] The reference iteration delay when running alone is the average iteration delay of the combination without co-location; the co-location iteration delay when running with co-location is the average iteration delay of the combination with co-location.

[0093] Calculate the difference between the co-set iteration delay and the reference iteration delay, and then perform a ratio operation between this difference and the reference iteration delay to obtain the performance degradation ratio. The calculation of the performance degradation ratio includes the following steps:

[0094] First, calculate the difference between the co-location iteration delay and the reference iteration delay. Use this difference as the dividend and the reference iteration delay as the divisor to perform a ratio operation, obtaining a dimensionless value as the performance degradation ratio. A performance degradation ratio of zero indicates that the iteration delays of co-location operation and individual operation are basically the same. A performance degradation ratio greater than zero and a larger value indicates that the performance degradation caused by co-location operation is more significant.

[0095] Based on the statistical results of historical performance degradation rates, the highest quantile in the statistical distribution of performance degradation rates is selected as the maximum expected value of the performance degradation rate. For each performance degradation rate, the ratio is calculated by using the performance degradation rate as the dividend and the maximum expected value as the divisor. The resulting ratio, limited to the range of zero to one, serves as the co-location interference index value. When the performance degradation rate is less than or equal to the maximum expected value, the ratio is between zero and one; when the performance degradation rate is greater than the maximum expected value, the co-location interference index value can be directly set to one, indicating that the degree of co-location interference has reached the preset upper limit.

[0096] Organize the co-location interference index values ​​according to the task profile data identifier and the resource profile data identifier, and generate a co-location interference index table so that the co-location interference index value corresponding to any combination of task profile and resource profile can be quickly queried during scheduling.

[0097] The hybrid training scheduling strategy generation module generates hybrid training scheduling strategies based on task profile data, resource profile data, and co-location interference index table.

[0098] Based on resource profile data, a set of available target computing nodes is determined, and a set of training tasks to be scheduled is constructed based on task profile data. Based on the task resource requirement level and node resource capabilities, multiple candidate resource allocation schemes are generated between the set of available target computing nodes and the set of training tasks. Each scheme includes a mapping relationship between training tasks and target computing nodes, as well as the resource quota corresponding to each mapping relationship.

[0099] For each candidate resource allocation scheme, the ratios of the allocated processor cores to the total number of processor cores, the allocated accelerators to the total number of accelerators, and the allocated memory and GPU memory capacity to the total capacity are calculated on each computing node. Based on preset weights, these ratios are weighted to obtain the resource utilization value for each node. Then, a weighted average of the resource utilization values ​​of all nodes is calculated based on the resource capacity of each node to obtain the heterogeneous computing power resource utilization index for that candidate resource allocation scheme.

[0100] For each training task in the candidate resource allocation scheme, the training iteration latency is estimated based on the task profile data identifier and the resource profile data identifier of the target computing node:

[0101] If there is a combination of the same task profile data identifier and resource profile data identifier in the historical records, the average training iteration latency of that combination will be used as the estimated training iteration latency.

[0102] If no perfect matching combination exists, then based on the model parameter size, dataset size, and resource configuration data, calculate one or more historical combinations that have smaller differences from the current combination in the above dimensions in the historical records, and use the average training iteration latency of these combinations as the estimated training iteration latency.

[0103] The estimated training iteration latency is compared with the training latency constraint. First, the difference between the estimated training iteration latency and the training latency constraint is calculated. Then, this difference is used as the dividend, and the training latency constraint is used as the divisor to perform a ratio operation, yielding the estimated training latency deviation ratio. The estimated training latency deviation ratio reflects the degree of deviation of the expected training iteration latency from the training latency constraint under the current resource allocation scheme. When the estimated training iteration latency is not greater than the training latency constraint, the estimated training latency deviation ratio can be considered zero or truncated to zero. Subsequently, according to a preset mapping relationship, the estimated training latency deviation ratio is mapped to a value between zero and one, serving as the estimated training latency deviation index.

[0104] For each candidate resource allocation scheme, the set of co-located training tasks on each target computing node is statistically analyzed. For each training task in the set, the co-location interference index value is looked up in the co-location interference index table based on the task profile data identifier and the resource profile data identifier of the node. The co-location interference value of the node can be obtained by averaging the index values ​​of all training tasks on the same node. The co-location interference values ​​of all nodes are then weighted and averaged according to the resource capacity of each node to obtain the co-location interference index of the candidate scheme.

[0105] Based on the training phase identifier, select the corresponding weight reorganization from the preset weight configuration. For each candidate solution:

[0106] The heterogeneous computing power resource utilization rate is regarded as a positive indicator; the estimated training delay deviation and co-location interference are regarded as negative indicators. They can be converted into "the smaller the better" evaluation values ​​before being included in the weighting.

[0107] The above evaluation values ​​are weighted and summed according to preset weights to obtain the comprehensive evaluation value of the candidate schemes. The scheme with the best comprehensive evaluation value among all candidate resource allocation schemes is selected as the current hybrid training scheduling strategy.

[0108] The hybrid training execution and isolation control module performs the following operations based on the hybrid training scheduling strategy:

[0109] 1) Create a container-based training execution unit on the target computing node for each training task;

[0110] 2) Configure the upper limit of the number of processor cores, processor time slice quota, memory capacity, video memory capacity and network bandwidth for each training execution unit, and implement resource limits through the control group mechanism;

[0111] 3) On physical accelerators that support multi-instance partitioning, the number of accelerator instances is determined according to the hybrid training scheduling strategy, and corresponding GPU memory capacity is allocated to each accelerator instance. Different training execution units are bound to different accelerator instances to achieve isolation at the accelerator resource level.

[0112] 4) Configure the number of data parallel processes, model parallelism mode, and gradient synchronization strategy within each training execution unit, and start training execution.

[0113] In this approach, a single physical accelerator is divided into several accelerator instances. Each instance only exposes its corresponding memory quota and computing resource boundaries to the bound training execution unit, thereby reducing resource contention between training tasks.

[0114] The operation monitoring and adaptive adjustment module monitors the running status during training execution and performs adaptive adjustments based on the monitoring results.

[0115] Within a preset monitoring period, the training iteration latency, throughput, processor utilization, accelerator utilization, memory usage, GPU memory usage, and co-location relationship with other training execution units of each training execution unit are collected to form operational monitoring data.

[0116] Based on the operational monitoring data, the average training iteration latency within the monitoring period is calculated for each training task. This average training iteration latency is then compared to the training latency constraint. First, the difference between the average training iteration latency and the training latency constraint is calculated. Then, this difference is used as the dividend, and the training latency constraint is used as the divisor to perform a ratio calculation, yielding the actual training latency deviation ratio. The actual training latency deviation ratio represents the degree of deviation of the actual training iteration latency from the training latency constraint. When the average training iteration latency is not greater than the training latency constraint, the actual training latency deviation ratio can be considered zero or truncated to zero. According to a preset mapping relationship, the actual training latency deviation ratio is mapped to a value between zero and one, serving as the actual training latency deviation index.

[0117] Based on operational monitoring data, the iteration latency of individual and co-location operations within the latest monitoring period is statistically analyzed, and the corresponding performance degradation ratio is calculated. Using the same normalization method as when constructing the co-location interference index, the performance degradation ratio is mapped to a value between zero and one, serving as the actual co-location interference index.

[0118] The actual co-location interference level is compared with the corresponding co-location interference index value in the co-location interference index table. First, the difference between the actual co-location interference level and the co-location interference index value is calculated. Then, this difference is used as the dividend, and the co-location interference index value is used as the divisor to perform a ratio calculation, yielding the co-location interference deviation ratio. The co-location interference deviation ratio is used to represent the degree of deviation of the actual co-location interference level from historical statistical results. When the actual co-location interference level is not greater than the co-location interference index value, the co-location interference deviation ratio can be considered zero or truncated to zero. According to a preset mapping relationship, the co-location interference deviation ratio is mapped to a value between zero and one, serving as the co-location interference deviation index.

[0119] When the actual training latency deviation index of a training task exceeds the corresponding preset threshold, the operation monitoring and adaptive adjustment module increases the weight value of the estimated training latency deviation index in the preset weight and correspondingly reduces the weight value of the heterogeneous computing power resource utilization index in the preset weight.

[0120] When the co-location interference deviation index of a training task exceeds the corresponding preset threshold, the weight value of the co-location interference index in the preset weight is increased, and the weight value of the heterogeneous computing power resource utilization index in the preset weight is reduced accordingly.

[0121] The adjusted preset weights are used in the subsequent hybrid training scheduling strategy generation process, making subsequent scheduling pay more attention to the indicators with larger current deviations.

[0122] To ensure that the co-location interference index reflects the latest operational status, combinations of task profile data identifiers and resource profile data identifiers from the latest monitoring period are selected from the operational monitoring data. The performance degradation ratio of the corresponding combinations is calculated and weighted with the historical performance degradation level to obtain the updated performance degradation level. Using the same normalization method as when constructing the co-location interference index, the updated performance degradation level is mapped to the new co-location interference index value, replacing the corresponding entry in the co-location interference index table.

[0123] The updated co-location interference index table is used in the subsequent hybrid training scheduling strategy generation process, so that the scheduling decision continuously utilizes the latest co-location interference statistics.

[0124] Example 2

[0125] A cloud data server hybrid training system, applying the cloud data server hybrid training method, includes: a training task access and profiling module, a resource representation and monitoring module, a co-location interference index management module, a hybrid training scheduling strategy generation module, a hybrid training execution and isolation control module, a running monitoring and adaptive adjustment module, and a data processing module.

[0126] The training task access and profiling module, resource representation and monitoring module, co-location interference index management module, hybrid training scheduling strategy generation module, hybrid training execution and isolation control module, and operation monitoring and adaptive adjustment module are respectively connected to the data processing module.

[0127] The training task access and profiling module mentioned above is used to receive training task requests, parse model structure parameters, dataset parameters, training latency constraints, resource isolation constraints, determine training stage identifiers, and generate task profiling data.

[0128] The resource characterization and monitoring module is used to collect resource configuration data and operating status data of each computing node, determine resource isolation capability indicators, and generate resource profile data.

[0129] The co-location interference index management module is used to generate a co-location interference index table and store the co-location interference index values ​​in the co-location interference index table based on the performance degradation ratio statistically based on historical operation monitoring data.

[0130] The hybrid training scheduling strategy generation module is used to construct candidate resource allocation schemes based on task profile data, resource profile data and co-location interference index table, calculate heterogeneous computing power resource utilization index, estimate training latency deviation index and co-location interference index, and generate hybrid training scheduling strategy according to preset weights.

[0131] The hybrid training execution and isolation control module is used to allocate target computing nodes and resource quotas according to the hybrid training scheduling strategy, create container-type training execution units, configure accelerator multi-instance partitioning parameters and resource limit parameters, and start training execution.

[0132] The aforementioned operation monitoring and adaptive adjustment module is used to collect operation monitoring data, calculate the actual training latency index and the actual co-location interference index, obtain the actual training latency deviation index based on the comparison between the actual training latency index and the training latency constraint, obtain the co-location interference deviation index based on the comparison between the actual co-location interference index and the co-location interference index value, and adjust the preset weights and hybrid training scheduling strategy when the actual training latency deviation index or the co-location interference deviation index exceeds the preset threshold, and update the co-location interference index table.

[0133] Example 3

[0134] Based on the cloud data server hybrid training method and system described in Embodiments 1 and 2 above, this embodiment provides a specific application scenario to illustrate the specific usage of each indicator and parameter in a real environment.

[0135] The cloud data server cluster consists of three computing nodes, denoted as node N1, node N2, and node N3. The resource configuration data for each node is as follows:

[0136] Node N1:

[0137] Processor: 32-core general-purpose processor;

[0138] Accelerator: 4 high-end GPUs, supporting multi-instance partitioning;

[0139] Memory capacity: 512GB;

[0140] Video memory capacity: 80GB per GPU;

[0141] Network bandwidth capability: High-bandwidth network interface.

[0142] Node N2:

[0143] Processor: 24-core general-purpose processor;

[0144] Accelerator: 4 mid-range GPUs, does not support multi-instance partitioning;

[0145] Memory capacity: 256GB;

[0146] Video memory capacity: 24GB per GPU;

[0147] Network bandwidth capability: Medium bandwidth network interface.

[0148] Node N3:

[0149] Processor: 16-core general-purpose processor;

[0150] Accelerator: 2 mid-range GPUs, does not support multi-instance partitioning;

[0151] Memory capacity: 128GB;

[0152] Video memory capacity: 24GB per GPU;

[0153] Network bandwidth capability: Medium bandwidth network interface.

[0154] The resource characterization and monitoring module collects operational status data from the three nodes. For example, at a certain moment:

[0155] Node N1: Processor utilization is 40%, accelerator utilization is 30%, memory usage is 35%, GPU memory usage is 20%, currently carrying 0 training tasks, and network throughput is low.

[0156] Node N2: Processor utilization is 50%, accelerator utilization is 60%, memory utilization is 70%, GPU memory utilization is 65%, and it is currently supporting 2 training tasks.

[0157] Node N3: Processor utilization is 20%, accelerator utilization is 10%, memory usage is 25%, GPU memory usage is 15%, and it is currently hosting 1 training task.

[0158] The resource isolation capability index is defined as follows:

[0159] Node N1: Accelerator multi-instance partitioning is marked as supported, processor affinity control is marked as supported, and network bandwidth limiting is marked as supported.

[0160] Node N2: Accelerator multi-instance partitioning is marked as not supported, processor affinity control is marked as supported, and network bandwidth limiting is marked as supported.

[0161] Node N3: Accelerator multi-instance partitioning support is marked as not supported, processor affinity control support is marked as supported, and network bandwidth limiting support is marked as supported.

[0162] The system associates resource configuration data, running status data, and resource isolation capability indicators according to the computing node identifier to form resource profile data for nodes N1, N2, and N3. The resource profile data contains their respective resource capacity, real-time load, and isolation capability information.

[0163] In the above cluster environment, the training task access and profiling module received three training task requests:

[0164] Training Task T1: Model Structure Parameters: Large-scale Transformer model with many layers and a large parameter scale, mainly using matrix multiplication and attention operators; Dataset Parameters: Large number of text samples with high feature dimensionality, using complex data augmentation; Training Latency Constraints: The target training iteration time is short, and the training task has the highest priority; Resource Isolation Constraints: Request a strong isolation level.

[0165] Training Task T2: Model structure parameters: medium-sized convolutional neural network model, mainly using convolution operators; Dataset parameters: medium number of image samples, medium feature dimension, and relatively simple data preprocessing method; Training latency constraints: the target training iteration time is slightly lower than the cluster average, and the training task priority is medium; Resource isolation constraints: request a medium isolation level.

[0166] Training Task T3: Model Structure Parameters: Small to medium-scale recommendation model with many embedding layers, mainly using embedding lookup and simple multiplication and addition operations; Dataset Parameters: Medium number of user behavior samples, medium feature dimension, and few preprocessing operations; Training Latency Constraints: Target training iteration time is close to or slightly higher than the cluster average, and the training task priority is low; Resource Isolation Constraints: Request weak isolation level.

[0167] The training task access and profiling module marks all three training tasks as being in the preheating stage based on the task type, and assigns the training stage identifier as "preheating stage".

[0168] Subsequently, the module generates task profile data according to the logic given in the implementation method:

[0169] For T1: Due to the large model parameter size and the use of matrix multiplication and attention operators, the parameters representing computational resource requirements are set to a high level based on the resource requirement benchmark table; due to the large number of text samples and high feature dimensionality, the parameters representing storage resource requirements are set to a high level; due to the expected use of data parallelism and frequent gradient synchronization, the parameters representing communication resource requirements are set to a high level; due to the target training iteration time being significantly lower than the cluster's historical average iteration time and the highest priority of the training task, the latency sensitivity weight is set to a high level; and due to the strong isolation constraint, the isolation sensitivity weight is set to the highest level.

[0170] For T2: The model size and parameter size are moderate, and the parameters used to characterize the computational resource requirements are set to a moderate level; the number of image samples is moderate, and the parameters used to characterize the storage resource requirements are set to a moderate level; data parallelism is expected to be used, and the synchronization frequency is moderate, so the parameters used to characterize the communication resource requirements are set to a moderate level; the target training iteration time is slightly lower than the historical average iteration time, and the latency sensitivity weight is set to a moderately high level; the resource isolation constraint is a moderate isolation level, and the isolation sensitivity weight is set to a moderate level.

[0171] For T3: The model size is small to medium, so the parameters representing computational resource requirements are set to a medium-low level; the dataset sample size is medium, so the parameters representing storage resource requirements are set to a medium level; the communication requirements are low, so the parameters representing communication resource requirements are set to a low level; the target training iteration time is close to or slightly above average, so the latency sensitivity weight is set to a medium-low level; the resource isolation constraint is a weak isolation level, so the isolation sensitivity weight is set to a low level.

[0172] The training task access and profiling module associates the above parameters and weights with the training task identifiers corresponding to T1, T2, and T3 to form task profiling data for T1, T2, and T3.

[0173] The co-location interference index management module constructs a co-location interference index table based on historical operation monitoring data. For this embodiment, it is assumed that the historical operation records include the following typical cases (this is just an example; the actual index table can contain many more combinations):

[0174] The reference iteration delay for task T1 when running alone on node N1 is a certain time value;

[0175] When task T1 is co-located with task T2 on node N1, the iteration latency increases compared to the reference iteration latency, and the performance degradation rate falls in the medium to high range.

[0176] When T1 type tasks are co-located on node N2, the performance degradation rate is close to the preset upper limit due to the limitations of mid-range GPU performance and the high load.

[0177] When T2 type tasks are co-located with other medium-sized tasks on node N2, the performance degradation rate is in the medium range.

[0178] When T3 type tasks are co-located on node N3, the overall performance degradation rate is relatively low.

[0179] The co-location interference index management module normalizes the aforementioned performance degradation ratios. For example, it normalizes lower performance degradation ratios to smaller co-location interference index values ​​and performance degradation ratios close to the upper limit to larger co-location interference index values, and stores the results in the co-location interference index table.

[0180] For example, the following abstract co-location interference index value can be obtained (for illustrating relative size only):

[0181] The co-location interference index value of "T1 type task + node N1" is in the medium range;

[0182] The co-location interference index value of "T1 type task + node N2" is close to the high range;

[0183] The co-location interference index value of "T2 type task + node N2" is in the medium range;

[0184] The co-location interference index value of “T3 type task + node N3” is in the low range.

[0185] At the current moment, the resource profile data provided by the resource characterization and monitoring module shows that node N1 has significant resource redundancy and strong isolation capabilities, while nodes N2 and N3 are already carrying some training tasks. The hybrid training scheduling strategy generation module constructs several candidate resource allocation schemes, including:

[0186] Option S1: T1 runs entirely on node N1, which allocates multiple GPU instances and a significant amount of processor resources to T1; T2 is assigned to node N2, using some GPU and processor resources; T3 is assigned to node N3, using some of node N3's resources.

[0187] Scheme S2: T1 and T2 are co-located and run on node N1, with the GPU and processor resources of node N1 allocated to T1 and T2 respectively; T3 is still allocated to node N3.

[0188] For each candidate solution, the hybrid training scheduling strategy generation module calculates three metrics according to the logic described above:

[0189] Heterogeneous computing power resource utilization index

[0190] For scheme S1, the GPU and processor resource utilization of node N1 is at a medium level, and the resource utilization of nodes N2 and N3 is also at a medium level.

[0191] In scheme S2, the resource utilization of node N1 is improved, and node N2 no longer carries T2, resulting in a higher overall resource utilization compared to scheme S1.

[0192] Estimate training latency bias:

[0193] For T1, in scheme S1, T1 uses the high-end GPU of node N1 and is not co-located with other high-load tasks. Historical monitoring data shows that under this configuration, the estimated training iteration latency is close to the target training iteration time, and the estimated training latency deviation index is low.

[0194] In scheme S2, T1 and T2 are co-located and run on node N1 and share some resources. Historical monitoring data shows that under similar co-location conditions, the iteration latency of T1 will increase significantly, and the estimated training latency deviation index is higher than that of scheme S1.

[0195] For T2 and T3, estimations were also made based on historical data and resource profiles to obtain corresponding estimated training latency deviation indices.

[0196] Co-location interference index:

[0197] For scheme S1, T1 is not co-located with other training tasks on node N1, and the co-location interference index is taken as a low value according to the co-location interference index table; T2 may be co-located with existing tasks on node N2, and the co-location interference index is medium; T3 is co-located on node N3, and the interference index is low; after combination, the co-location interference index of scheme S1 is in the medium to low range.

[0198] For scheme S2, T1 and T2 are co-located on node N1. According to the co-location interference index table, the co-location interference index value of "T1 type task + node N1" in the co-location scenario is moderately high, which makes the overall co-location interference index of scheme S2 higher than that of scheme S1.

[0199] Since all three training tasks are currently in the warm-up phase, and T1 has high latency sensitivity and isolation sensitivity weights, the preset weights assign higher weights to the estimated training latency deviation index and the co-location interference index during the warm-up phase, and relatively lower weights to the heterogeneous computing power resource utilization index.

[0200] Under the above weight configuration, the hybrid training scheduling strategy generation module performs a weighted comprehensive evaluation of scheme S1 and scheme S2 respectively. Although the heterogeneous computing power resource utilization rate of scheme S2 is higher, the estimated training delay deviation and co-location interference of scheme S2 are significantly higher than those of scheme S1. Therefore, the comprehensive evaluation results show that scheme S1 is better.

[0201] Therefore, the hybrid training scheduling strategy generation module selects scheme S1 as the current hybrid training scheduling strategy.

[0202] The hybrid training execution and isolation control module performs the following operations based on the hybrid training scheduling strategy:

[0203] On node N1, a container-type training execution unit is created for training task T1, configuring the maximum number of processor cores, processor time slice quota, maximum memory capacity, and maximum GPU memory capacity. Since node N1 supports multi-instance partitioning of accelerators, each high-end GPU is divided into multiple accelerator instances, and a suitable number of accelerator instances are allocated to T1, configured in conjunction with the computing resource requirement parameters in the task profile data.

[0204] On node N2, a training execution unit is created for training task T2, and the corresponding control group parameters are configured to limit the number of available processor cores and memory capacity, and to set the network bandwidth limit, so that T2 is subject to resource isolation control on node N2.

[0205] On node N3, create a training execution unit for training task T3 and set resource limit parameters for processor, video memory and network bandwidth. Since T3 is less sensitive to resource isolation, the resource limit parameters can be appropriately relaxed.

[0206] Within the aforementioned training execution unit, the hybrid training execution and isolation control module configures the number of data parallel processes, gradient synchronization frequency, and other training parallel parameters according to the hybrid training scheduling strategy, and then initiates training execution.

[0207] After training has been running for a period of time, the operation monitoring and adaptive adjustment module collects operation monitoring data within a monitoring period, and obtains the following results (for example):

[0208] The average training iteration latency of T1 is slightly higher than the training latency constraint, and the corresponding actual training latency deviation index is in the medium range.

[0209] T2 is affected by the existing tasks on node N2, and the average training iteration latency is significantly higher than the training latency constraint. The corresponding actual training latency deviation index is close to the upper limit, and the average training iteration latency fluctuates greatly.

[0210] T3 runs stably on node N3, with low actual training latency deviation.

[0211] By comparing the performance degradation rate during co-location operation with the co-location interference index value in the co-location interference index table, it was found that the actual co-location interference index of T2 was higher than the index value, causing the co-location interference deviation index of T2 to exceed the preset threshold.

[0212] Based on this, the operation monitoring and adaptive adjustment module executes the following adaptive adjustment logic:

[0213] Preset weight adjustment:

[0214] Since the actual training latency deviation index and co-location interference deviation index of T2 both exceed the preset threshold, the operation monitoring and adaptive adjustment module increases the weight value of the estimated training latency deviation index in the preset weight, while increasing the weight value of the co-location interference index and correspondingly reducing the weight value of the heterogeneous computing power resource utilization index.

[0215] Co-location interference index table update:

[0216] The operation monitoring and adaptive adjustment module selects a combination of task profile data identifiers and resource profile data identifiers related to T2 from the latest monitoring cycle, recalculates the performance degradation degree, and normalizes the new performance degradation degree and the historical performance degradation degree by weighting the average according to a certain ratio to obtain the updated co-location interference index value.

[0217] The co-location interference index management module refreshes the corresponding entry in the co-location interference index table with the new co-location interference index value.

[0218] A new round of scheduling strategy generation:

[0219] In the next scheduling, the hybrid training scheduling strategy generation module uses the adjusted preset weights and the refreshed co-location interference index table to re-evaluate the candidate resource allocation schemes.

[0220] In the new comprehensive evaluation process, the resource allocation scheme that still co-locates T2 with high-load tasks and causes significant interference will result in a lower comprehensive evaluation result due to the increased weight of the estimated training latency deviation index and the co-location interference index.

[0221] The system may choose to migrate T2 from node N2 to node N1 or node N3, or adjust the resource quota in node N2 to give T2 more video memory and processor resources, so as to reduce the actual training latency deviation and co-location interference deviation.

[0222] Through one or more of the above monitoring and adaptive adjustment processes, the cloud data server cluster gradually adjusts the operating status of T1, T2 and T3 to a more balanced state between training latency constraints, co-location interference control and heterogeneous computing resource utilization, while ensuring that resource isolation capabilities are fully utilized.

[0223] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A cloud-based data server hybrid training method, characterized in that, include: Step 1: Receive training task requests, parse model structure parameters, dataset parameters, training latency constraints, and resource isolation constraints, determine the training stage identifier, and generate task profile data corresponding to the training task. Step 2: Collect resource configuration data and operating status data of each computing node, determine resource isolation capability indicators, and associate resource configuration data, operating status data and resource isolation capability indicators to generate resource profile data; Step 3: Based on historical operation monitoring data, calculate the performance degradation ratio under different combinations of task profile data and resource profile data, normalize the performance degradation ratio, generate a co-location interference index table, and store the co-location interference index value in the co-location interference index table. Step 4: Based on task profile data, resource profile data, and co-location interference index table, construct multiple candidate resource allocation schemes. For each candidate resource allocation scheme, calculate the heterogeneous computing power resource utilization rate, estimated training latency deviation, and co-location interference index. Based on the preset weights corresponding to the training stage identifier, weight the heterogeneous computing power resource utilization rate, estimated training latency deviation, and co-location interference index to generate a hybrid training scheduling strategy. Step 5: Allocate target computing nodes and resource quotas according to the hybrid training scheduling strategy, create container-type training execution units on the target computing nodes, configure accelerator multi-instance partitioning parameters and resource limit parameters, and start training execution; Step 6: During the training execution process, collect operation monitoring data, calculate the actual training latency index and the actual co-location interference index, obtain the actual training latency deviation index based on the comparison between the actual training latency index and the training latency constraint, obtain the co-location interference deviation index based on the comparison between the actual co-location interference index and the co-location interference index value, and when the actual training latency deviation index or the co-location interference deviation index exceeds the preset threshold, adjust the preset weights and the hybrid training scheduling strategy, and apply the adjustment results to subsequent training execution to perform closed-loop optimization of the hybrid training process.

2. The cloud data server hybrid training method according to claim 1, characterized in that, The process of receiving training task requests, parsing model structure parameters, dataset parameters, training latency constraints, and resource isolation constraints, determining training phase identifiers, and generating task profile data corresponding to the training task includes: The training task is divided into a warm-up phase, a stable training phase, and a convergence phase, and a training phase identifier is assigned to each training task. The model layer number, parameter size, and operator type are parsed from the training task request as model structure parameters, the number of samples, sample feature dimensions, and data preprocessing method are parsed as dataset parameters, the target training latency and training task priority are parsed as training latency constraints, and the resource isolation level requirements are parsed as resource isolation constraints. Based on model structure parameters, dataset parameters, training latency constraints, resource isolation constraints, and training phase identifiers, parameters for representing computing resource requirements, parameters for representing storage resource requirements, parameters for representing communication resource requirements, latency sensitivity weights, and isolation sensitivity weights are calculated and associated with training task identifiers to form task profile data.

3. The cloud data server hybrid training method according to claim 1, characterized in that, The process of collecting resource configuration data and operational status data from each computing node, determining resource isolation capability indicators, and correlating resource configuration data, operational status data, and resource isolation capability indicators to generate resource profile data includes: Obtain the processor type, number of processors, accelerator type, number of accelerators, memory capacity, video memory capacity, and network bandwidth capacity of each computing node as resource configuration data; The processor utilization, accelerator utilization, memory usage, GPU memory usage, number of training tasks currently being carried, and network throughput of each computing node are periodically collected as runtime status data. For each compute node, resource isolation capability indicators are determined. These indicators include accelerator multi-instance partitioning support identifier, processor affinity control support identifier, and network bandwidth limitation support identifier. Resource configuration data, operational status data, and resource isolation capability indicators are linked according to the computing node identifier to form resource profile data.

4. The cloud data server hybrid training method according to claim 1, characterized in that, The aforementioned method involves calculating the performance degradation ratio under different combinations of task profile data and resource profile data based on historical operation monitoring data, normalizing the performance degradation ratio, and generating a co-location interference index table. This co-location interference index table stores co-location interference index values, including: Extract task profile data identifiers, resource profile data identifiers, and training iteration latency and throughput under corresponding combinations from historical operation monitoring data; Under the condition of co-located training tasks, calculate the performance degradation ratio compared with running the training task alone, and use the performance degradation ratio as the degree of performance degradation; The performance degradation degree corresponding to different combinations of task profile data identifiers and different resource profile data identifiers is normalized to obtain the co-location interference index value. Organize the co-location interference index values ​​according to the task profile data identifier and the resource profile data identifier, and generate the co-location interference index table.

5. The cloud data server hybrid training method according to claim 1, characterized in that, The method involves constructing multiple candidate resource allocation schemes based on task profile data, resource profile data, and a co-location interference index table. For each candidate resource allocation scheme, heterogeneous computing power resource utilization, estimated training latency deviation, and co-location interference are calculated. These indicators are then weighted according to preset weights corresponding to the training stage identifiers to generate a hybrid training scheduling strategy, including: The available target computing node set is determined based on resource profile data, and the training task set is constructed based on task profile data. Candidate resource allocation schemes are generated between the available target computing node set and the training task set. The candidate resource allocation schemes include the mapping relationship between training tasks and target computing nodes and the resource quota corresponding to each mapping relationship. For each candidate resource allocation scheme, calculate the heterogeneous computing power resource utilization rate index based on resource quota and resource profile data, estimate the training latency and calculate the estimated training latency deviation index based on task profile data and resource profile data, and find the co-location interference index value in the co-location interference index table based on the task profile data identifier and resource profile data identifier in the candidate resource allocation scheme to obtain the co-location interference index. Based on the training phase identifier, select the corresponding preset weights, and weight the heterogeneous computing power resource utilization rate index, the estimated training latency deviation index, and the co-location interference index to obtain a comprehensive evaluation result. Select the candidate resource allocation scheme with the best comprehensive evaluation result from all candidate resource allocation schemes to generate a hybrid training scheduling strategy.

6. The cloud data server hybrid training method according to claim 1, characterized in that, The process of allocating target computing nodes and resource quotas according to a hybrid training scheduling strategy, creating container-type training execution units on the target computing nodes, configuring accelerator multi-instance partitioning parameters and resource limit parameters, and starting training execution includes: Based on the hybrid training scheduling strategy, a container-type training execution unit is created on the target computing node for each training task; Set upper limits on the number of processor cores, processor time slice quota, memory capacity, video memory capacity, and network bandwidth for each training execution unit, and limit the resources of processor, memory, video memory, and network bandwidth through a control group mechanism; On accelerators that support multi-instance partitioning, the number of accelerator instances and the memory partitioning of each accelerator instance are configured according to the hybrid training scheduling strategy, and different training execution units are bound to different accelerator instances; within each training execution unit, training parallel parameters are configured according to the hybrid training scheduling strategy and training execution is started.

7. The cloud data server hybrid training method according to claim 1, characterized in that, The process of collecting runtime monitoring data during training execution, calculating the actual training latency index and the actual co-location interference index, obtaining the actual training latency deviation index based on the comparison between the actual training latency index and the training latency constraint, and obtaining the co-location interference deviation index based on the comparison between the actual co-location interference index and the co-location interference index value, includes: Within a preset monitoring period, the training iteration latency, throughput, processor utilization, accelerator utilization, memory usage, GPU memory usage, and task co-location relationships of each training execution unit are collected to form runtime monitoring data. The average training iteration latency and training iteration latency variance of each training task are calculated based on the operation monitoring data. The average training iteration latency is compared with the training latency constraint to obtain the actual training latency deviation index. Based on the performance degradation ratio recorded in the operation monitoring data during the co-location operation, the performance degradation ratio is normalized to obtain the actual co-location interference index. Based on the comparison between the actual co-location interference index and the co-location interference index value, the co-location interference deviation index is obtained.

8. The cloud data server hybrid training method according to claim 1, characterized in that, When the actual training latency deviation index or co-location interference deviation index exceeds a preset threshold, the preset weights and hybrid training scheduling strategy are adjusted, and the adjustment results are applied to subsequent training execution to perform closed-loop optimization of the hybrid training process, including: When the actual training latency deviation index exceeds the corresponding preset threshold, the weight of the estimated training latency deviation index in the preset weight is increased, and the weight of the heterogeneous computing power resource utilization index in the preset weight is decreased. When the co-location interference deviation index exceeds the corresponding preset threshold, the weight value of the co-location interference index in the preset weight is increased, and the weight value of the heterogeneous computing power resource utilization index in the preset weight is decreased. Based on the adjusted preset weights, an updated hybrid training scheduling strategy is generated and used in subsequent training executions.

9. The cloud data server hybrid training method according to claim 8, characterized in that, The step of generating an updated hybrid training scheduling strategy based on the adjusted preset weights and using the updated hybrid training scheduling strategy in subsequent training execution includes: Select the task profile data identifier, resource profile data identifier, and training iteration latency and throughput under the corresponding combination from the latest monitoring period from the operation monitoring data, and update the performance degradation level. The updated performance degradation level is normalized, the co-location interference index value is updated, and the co-location interference index table is refreshed. When generating subsequent hybrid training scheduling strategies, the refreshed co-location interference index table is used to adaptively optimize the subsequent hybrid training process.

10. A cloud-based data server hybrid training system, characterized in that, The cloud data server hybrid training method according to any one of claims 1-9 includes: a training task access and profiling module, a resource characterization and monitoring module, a co-location interference index management module, a hybrid training scheduling strategy generation module, a hybrid training execution and isolation control module, a running monitoring and adaptive adjustment module, and a data processing module. The training task access and profiling module, resource representation and monitoring module, co-location interference index management module, hybrid training scheduling strategy generation module, hybrid training execution and isolation control module, and operation monitoring and adaptive adjustment module are respectively connected to the data processing module. The training task access and profiling module mentioned above is used to receive training task requests, parse model structure parameters, dataset parameters, training latency constraints, resource isolation constraints, determine training stage identifiers, and generate task profiling data. The resource characterization and monitoring module is used to collect resource configuration data and operating status data of each computing node, determine resource isolation capability indicators, and generate resource profile data. The co-location interference index management module is used to generate a co-location interference index table and store the co-location interference index values ​​in the co-location interference index table based on the performance degradation ratio statistically based on historical operation monitoring data. The hybrid training scheduling strategy generation module is used to construct candidate resource allocation schemes based on task profile data, resource profile data and co-location interference index table, calculate heterogeneous computing power resource utilization index, estimate training latency deviation index and co-location interference index, and generate hybrid training scheduling strategy according to preset weights. The hybrid training execution and isolation control module is used to allocate target computing nodes and resource quotas according to the hybrid training scheduling strategy, create container-type training execution units, configure accelerator multi-instance partitioning parameters and resource limit parameters, and start training execution. The aforementioned operation monitoring and adaptive adjustment module is used to collect operation monitoring data, calculate the actual training latency index and the actual co-location interference index, obtain the actual training latency deviation index based on the comparison between the actual training latency index and the training latency constraint, obtain the co-location interference deviation index based on the comparison between the actual co-location interference index and the co-location interference index value, and adjust the preset weights and hybrid training scheduling strategy when the actual training latency deviation index or the co-location interference deviation index exceeds the preset threshold, and update the co-location interference index table.

Images