Method for model training based on k8s

Abstract

The application provides a model training method based on k8s, relates to the technical field of model training, and comprises the following steps: creating a business queue in a k8s cluster, distributing a training task to a corresponding queue, identifying a training stage feature and collecting performance data by a customized controller, determining an initial batch allocation value according to the feature and the data, continuously adjusting the batch allocation value based on an actual calculation time delay, finally updating resource requirement information in a yaml file, and selecting a training node meeting the resource requirement to execute a model training task. The application can optimize resource allocation, improve training efficiency, and reduce resource waste.

Description

Technical Field

[0001] This invention relates to model training techniques, and more particularly to a method for model training based on k8s. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning model training has become a key business for many enterprises and research institutions. Kubernetes (k8s), as an open-source container orchestration platform, has gradually become the infrastructure platform for model training tasks due to its excellent container management, scheduling, and scaling capabilities. In practical applications, model training often requires a large amount of computing resources, including CPU, GPU, and memory, and the training process is complex, involving multiple stages such as data preparation, model configuration, and parameter tuning.

[0003] Current Kubernetes-based model training systems typically employ static resource allocation, requiring resource requirements to be pre-specified before training begins and remaining unchanged throughout the training process. While simple and direct, this approach ignores the varying resource requirements at different stages of deep learning training, leading to inefficient resource utilization. The main shortcomings and deficiencies of existing technologies include:

[0004] Existing Kubernetes-based model training systems lack the ability to identify and optimize features at different training stages. They cannot dynamically adjust resource allocation according to the characteristics of different stages in the training process, which may lead to resource surplus in some stages and resource shortage in others.

[0005] Existing systems typically employ fixed batch allocation strategies that do not take into account the performance differences and real-time load status of different training nodes. This can lead to poor-performing nodes becoming the bottleneck for overall training speed in heterogeneous computing environments, thus reducing training efficiency.

[0006] The static preset method of resource requirements lacks flexibility and cannot dynamically adjust resource allocation based on actual performance data during training, resulting in low resource utilization, increased training costs, and difficulty in achieving efficient scheduling and resource sharing of training tasks among different business priority queues. Summary of the Invention

[0007] This invention provides a method for model training based on k8s, which can solve the problems in the prior art.

[0008] A first aspect of the present invention provides a method for model training based on Kubernetes, comprising:

[0009] Create a business queue in the Kubernetes cluster, assign the model training task to the corresponding business queue according to the training task type, generate the YAML file of the model training task, and submit the YAML file to the custom controller in the Kubernetes cluster.

[0010] The custom controller identifies the training phase characteristics of the model training task and collects performance data of the training nodes based on the training phase characteristics.

[0011] The initial batch allocation value of the training node is determined based on the training phase characteristics and the performance data. The batch allocation value is continuously adjusted based on the actual computation latency of the training node when executing the initial batch allocation value, until the maximum difference in the single batch computation time of the training node is less than the dynamic latency threshold. The adjusted batch allocation value is written into the YAML file, and the corresponding resource requirements are calculated based on the adjusted batch allocation value. The resource requirement information in the YAML file is then updated.

[0012] Based on the updated resource requirement information, a training node that meets the resource requirements is selected from the business queue to execute the model training task.

[0013] In one alternative implementation,

[0014] The steps of creating a business queue in the Kubernetes cluster, allocating model training tasks to the corresponding business queues according to the training task type, generating a YAML file for the model training task, and submitting the YAML file to the custom controller in the Kubernetes cluster include:

[0015] In a Kubernetes cluster, a multi-level business queue is created, and the resource quota of the multi-level business queue is configured through a resource weight matrix. Based on the resource weight matrix, a resource borrowing matrix between queues is constructed, and the value of each element in the resource borrowing matrix is ​​determined by the resource utilization rate of the source queue, the resource utilization rate of the target queue, and the historical borrowing success rate.

[0016] Obtain feature data for the model training task, generate a multi-dimensional feature vector based on the feature data, and generate a model training task profile vector by weighted combination of the multi-dimensional feature vectors.

[0017] Calculate the cosine similarity between the model training task profile vector and the feature vectors of each business queue in the multi-level business queue. Multiply the cosine similarity by the priority of the corresponding business queue to obtain the queue matching degree. Assign the model training task to the business queue with the highest matching degree according to the queue matching degree. When the business queue with the highest matching degree has insufficient resources, determine the available queues and borrowing quotas according to the resource borrowing matrix to realize cross-queue resource allocation.

[0018] Write the queue allocation information for the model training task into a YAML file and submit it to the custom controller.

[0019] In one alternative implementation,

[0020] The steps for collecting performance data of training nodes based on the features of the training phase include:

[0021] The training phase features include data loading phase features, model calculation phase features, and parameter update phase features; a training phase feature vector is constructed, which contains the performance indicators corresponding to each phase feature.

[0022] Based on the performance loss gradient of each feature in the feature vector of the training stage, the weight coefficients of each stage feature in the feature vector of the training stage are dynamically updated, and a performance data collection scheme is formulated according to the updated weight coefficients.

[0023] Calculate the predicted feature trend value based on the historical feature sequence of the feature vector during the training phase; use the deviation between the current feature vector and the predicted feature trend value as the feature change amplitude; dynamically optimize the sampling window size and sampling frequency of the performance data acquisition scheme according to the fluctuation law of the feature change amplitude; and perform performance data acquisition according to the optimized acquisition scheme.

[0024] The collected performance data is subjected to multi-dimensional quality assessment to obtain performance data quality assessment results; the performance data quality assessment results are fed back to the custom controller to optimize the feature recognition process during the training phase.

[0025] In one alternative implementation,

[0026] Based on the performance loss gradient of each feature in the training phase feature vector, the weight coefficients of each phase feature in the training phase feature vector are dynamically updated, and the performance data acquisition scheme is formulated according to the updated weight coefficients. The steps include:

[0027] Based on the historical change characteristics of the performance loss gradient, the gradient influence factor is determined. The weight coefficients of each stage feature in the training stage feature vector are dynamically updated according to the performance loss gradient and the gradient influence factor to obtain the updated weight coefficients.

[0028] A data acquisition sensitivity matrix is ​​constructed based on the updated weight coefficients, which represents the degree of correlation between features and performance indicators in each training stage; a time-series sensitivity feature is constructed based on the historical sequence of the data acquisition sensitivity matrix, which represents the change pattern of the data acquisition sensitivity matrix over time; and a stage coupling feature is constructed based on the correlation between different training stages in the data acquisition sensitivity matrix.

[0029] The time-series sensitivity features and the stage coupling features are fused to construct a multi-level optimization objective function. The multi-level optimization objective function includes a sensitivity weighting term and a fluctuation penalty term in the main objective function, and an upper limit for resource utilization and a lower limit for sampling frequency in the constraint function. The multi-level optimization objective function is solved using the adaptive step-size Lagrange multiplier method, wherein the multiplier update rate is dynamically adjusted according to the degree of constraint violation. The search space is optimized based on historical performance data and task similarity, and a performance data acquisition scheme is output when the stability of the solution meets the preset conditions.

[0030] In one alternative implementation,

[0031] The steps of determining the initial batch allocation value for training nodes based on the training phase characteristics and the performance data, and continuously adjusting the batch allocation value based on the actual computation latency of the training nodes when executing the initial batch allocation value, until the maximum difference in the single batch computation time of the training nodes is less than the dynamic latency threshold, include:

[0032] The computational load characteristics of the training nodes are constructed based on the training phase characteristics and the performance data, wherein the computational load characteristics characterize the resource consumption level of each training phase; and the initial batch allocation value of the training nodes is determined according to the computational load characteristics.

[0033] The single-batch computation time of the training node when executing the initial batch allocation value is obtained, and the maximum difference between the single-batch computation times of different training nodes is calculated. Based on the maximum difference and the spectral analysis results of the performance fluctuation of the training node, an adaptive batch adjustment amount is constructed. The batch adjustment amount is negatively correlated with the stability of the performance fluctuation frequency and has a piecewise nonlinear relationship with the degree of deviation of the single-batch computation time from the average value.

[0034] A dynamic latency threshold is constructed, which decays exponentially with training time and has a minimum threshold limit; when the maximum difference is greater than the dynamic latency threshold, the current batch allocation value of the training node is adjusted based on the batch adjustment amount until the maximum difference of the single batch calculation time is less than or equal to the dynamic latency threshold.

[0035] The adjusted batch allocation execution results are fed back to the performance data acquisition scheme to optimize the performance data acquisition process in the subsequent training phase.

[0036] In one alternative implementation,

[0037] The steps to construct a dynamic latency threshold include:

[0038] Obtain the computational power parameters of the training nodes, and calculate the initial threshold of the training nodes based on the computational power parameters;

[0039] Obtain the current gradient variance, current loss value, and historical adjustment frequency of the training node, and calculate a custom decay coefficient based on the current gradient variance, the current loss value, and the historical adjustment frequency;

[0040] The dynamic latency threshold is obtained by multiplying the initial threshold by an exponential decay function of the training time, wherein the decay coefficient in the exponential decay function is determined by the custom decay coefficient; when the dynamic latency threshold is less than the minimum threshold limit, the minimum threshold limit is used as the dynamic latency threshold.

[0041] In one alternative implementation,

[0042] Based on the updated resource requirement information, the step of selecting a training node that meets the resource requirements in the business queue to execute the model training task includes:

[0043] A performance index vector for training nodes is constructed, which includes computational power and historical stability. An adaptive weight matrix is ​​constructed based on the performance index vector, and the weight coefficients of the adaptive weight matrix are dynamically adjusted according to the computational and communication density of the training task. The adaptive weight matrix is ​​multiplied by the performance index vector to obtain a node fitness matrix, which represents the degree of fitness of different training nodes to the model training task.

[0044] Based on the updated resource requirement information, a set of candidate nodes that meet the basic resource conditions is selected, and the set of candidate nodes is sorted based on the node adaptability matrix.

[0045] Based on the sorted candidate node set, the network topology distance between candidate nodes is calculated, and an initial topology affinity matrix is ​​constructed based on the network topology distance. The initial topology affinity matrix is ​​weighted and fused with the historical communication efficiency matrix to obtain an optimized topology affinity matrix. Based on the optimized topology affinity matrix and the updated resource requirement information, the node combination score is calculated. The training node combination with the highest node combination score and meeting the resource requirements is selected to perform the model training task.

[0046] A second aspect of the present invention provides an electronic device, comprising:

[0047] processor;

[0048] Memory used to store processor-executable instructions;

[0049] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0050] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0051] The beneficial effects of this application are as follows:

[0052] By creating business queues in the Kubernetes cluster and allocating model training tasks according to the type of training task, orderly management and resource allocation of different training tasks are achieved, improving resource utilization efficiency and the standardization of task processing.

[0053] Based on the identification of training phase features and the collection of performance data by the custom controller, the changes in computational requirements during model training can be accurately grasped, providing data support for subsequent batch allocation and making resource allocation more intelligent and precise.

[0054] By dynamically adjusting the batch allocation value until the latency threshold requirement is met, and updating the resource requirement information accordingly, load balancing among training nodes is achieved. This effectively solves the problems of wasted computing resources and low training efficiency caused by traditional fixed batch allocation, and significantly improves the overall performance and resource utilization of model training. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the method for model training based on k8s according to an embodiment of the present invention;

[0056] Figure 2 Flowchart for dynamic adjustment and optimization of training node batches. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0059] Figure 1 This is a flowchart illustrating the method for model training based on Kubernetes according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Create a business queue in the Kubernetes cluster, assign the model training task to the corresponding business queue according to the training task type, generate the YAML file of the model training task, and submit the YAML file to the custom controller in the Kubernetes cluster.

[0061] The custom controller identifies the training phase characteristics of the model training task and collects performance data of the training nodes based on the training phase characteristics.

[0062] The initial batch allocation value of the training node is determined based on the training phase characteristics and the performance data. The batch allocation value is continuously adjusted based on the actual computation latency of the training node when executing the initial batch allocation value, until the maximum difference in the single batch computation time of the training node is less than the dynamic latency threshold. The adjusted batch allocation value is written into the YAML file, and the corresponding resource requirements are calculated based on the adjusted batch allocation value. The resource requirement information in the YAML file is then updated.

[0063] Based on the updated resource requirement information, a training node that meets the resource requirements is selected from the business queue to execute the model training task.

[0064] This implementation details the complete technical process from creating a business queue to executing training tasks. Adaptive resource scheduling based on a Kubernetes cluster allocates training tasks to appropriate computing resources and dynamically adjusts batch allocation according to task characteristics, achieving efficient utilization of computing resources.

[0065] When creating business queues in a Kubernetes cluster, queues are categorized based on the computational characteristics of training tasks, such as image recognition queues, natural language processing queues, and recommendation queues. During queue creation, the business queue resource type is defined using Kubernetes' Custom Resource Definition (CRD) mechanism, including attributes such as queue name, resource cap, priority, and reserved resource ratio. For example, for an image recognition queue, the definition includes: queue name "image-recognition-queue", resource cap of 40% of the total cluster resources, priority of 8 (range 0-10), and reserved resource ratio of 15%. These custom resource instances are created via the Kubernetes API, and the corresponding queue controller is started to monitor the queue status. The queue controller is responsible for maintaining resource usage statistics, task queuing order, and resource allocation strategies within the queue.

[0066] When assigning model training tasks to corresponding business queues based on training task type, the model architecture, dataset features, and training parameters of the training task are analyzed. For example, for an image classification task based on a convolutional neural network, the features include: model type "CNN", 85 million parameters, training dataset size of 120GB, training batch size of 64, and learning rate of 0.001. According to the preset task classification rules, it is determined that this task should be assigned to the "image-recognition-queue". The classification rules adopt a decision tree structure, first dividing into broad categories based on model type, and then further subdividing based on indicators such as parameter count and data scale. A standard YAML file is generated for each task, containing task metadata, resource requirements, training configuration, and other information. The YAML file structure includes: apiVersion, kind, metadata (name, namespace, labels, etc.), and spec (model configuration, training parameters, resource requests, etc.). For example, a YAML file for a CNN training task contains: task name "cnn-training-task-0042", namespace "deep-learning", tags {type:"image",priority:"high"}, initial resource requests {cpu:"16",memory:"64Gi",gpu:"4"}, and training parameters {epochs:50,batch_size:64,learning_rate:0.001}, etc.

[0067] After submitting the YAML file to the custom controller in the Kubernetes cluster, the controller first verifies the format and content validity of the YAML file, ensuring that all required fields are correctly filled in. The custom controller is implemented based on the Kubernetes Operator pattern, extending native Kubernetes functionality by registering CRDs and corresponding controller logic. The controller listens for CRD resource change events related to the training task, triggering the processing flow upon receiving a newly submitted YAML file. The core components of the controller include: a resource listener, a state manager, a scheduler, and an executor. The resource listener is responsible for monitoring CRD resource changes; the state manager maintains the task's lifecycle state; the scheduler is responsible for resource allocation decisions; and the executor is responsible for starting the task container on the selected node.

[0068] The custom controller identifies the training phase characteristics of the training task, a crucial step for achieving refined resource management. The controller analyzes information such as model structure, parameter count, and optimizer type in the YAML file, and combines this with historical task data to identify the characteristics of the task at different training phases. Training phases typically include: data loading, forward propagation, backpropagation, and parameter update. For each phase, its resource sensitivity characteristics are identified, such as CPU-intensive, memory-intensive, computationally intensive, or I / O-intensive. For example, for the aforementioned CNN training task, the controller identifies: the data loading phase as I / O-intensive (I / O sensitivity 0.85), the forward propagation phase as computationally intensive (computational sensitivity 0.92), the backpropagation phase as a hybrid of computation and memory (computational sensitivity 0.88, memory sensitivity 0.75), and the parameter update phase as communication-intensive (communication sensitivity 0.79). These feature values ​​are calculated using a pre-trained feature extraction model trained on historical task data, capable of inferring the resource sensitivity characteristics of each phase from the model structure and training parameters.

[0069] When collecting performance data from training nodes during the training phase, the controller deploys a lightweight monitoring component to candidate training nodes. This component collects metrics such as CPU utilization, memory usage, GPU utilization, I / O rate, and network traffic. The sampling frequency is dynamically adjusted based on the sensitivity of different stages, increasing the sampling frequency for highly sensitive metrics. For example, during the data loading phase, I / O-related metrics are sampled twice per second, while other metrics are sampled once every 5 seconds; during the computationally intensive forward propagation phase, the sampling frequency for GPU utilization is increased to three times per second. The collected raw data undergoes preprocessing, including denoising, standardization, and feature extraction, to form a structured performance dataset. For example, the performance data of a node during the forward propagation phase might include: average GPU utilization of 92%, GPU memory usage of 85%, PCIe bandwidth utilization of 78%, and an average processing time of 210 milliseconds for a single batch of data.

[0070] The performance prediction model takes node performance data and training phase features as input and outputs the expected time for each node to process different batch sizes. The prediction model is trained on historical task data and employs a gradient boosting tree algorithm, capable of handling heterogeneous features and capturing non-linear relationships. For distributed training tasks, an appropriate number of batches needs to be allocated to each participating node. The initial batch allocation is calculated based on the relative proportion of node performance. For example, for a task with a total batch size of 128, if there are two training nodes A and B with a performance ratio of 3:2, the initial allocation would be 76 samples for node A and 52 samples for node B.

[0071] Monitor the actual processing time of each node for allocated batches, calculate the computation time per batch (total time divided by the number of batches), and identify the maximum difference between different nodes. For example, node A takes 380 milliseconds to process 76 samples, with a single batch time of 5 milliseconds; node B takes 312 milliseconds to process 52 samples, with a single batch time of 6 milliseconds. The maximum difference is 1 millisecond. Construct a dynamic latency threshold, which adjusts as training progresses. The initial threshold is typically set to 10% of the expected single batch time and decreases with the number of training iterations, but does not fall below the minimum threshold (usually 20% of the initial threshold). When the maximum difference exceeds the dynamic threshold, adjust the batch allocation, transferring some batches from slower nodes to faster nodes. The adjustment amount is related to the magnitude of the difference and node stability. The adjustment process continues until the maximum difference in single batch computation time is less than the dynamic threshold, or the maximum number of adjustments is reached.

[0072] The adjusted batch allocation values ​​are written to a YAML file, and the corresponding resource requirements are calculated based on these values, updating the resource requirement information in the YAML file. Using a resource prediction model, the required number of CPU cores, memory capacity, and GPUs are estimated based on the batch size and model characteristics. For example, for an adjusted configuration where node A processes 80 samples and node B processes 48 samples, the corresponding resource requirements are updated as follows: node A requires {cpu:"18",memory:"72Gi",gpu:"4"}, and node B requires {cpu:"14",memory:"56Gi",gpu:"3"}. The resource specification section of the YAML file is updated using the Kubernetes API to provide a basis for subsequent resource allocation.

[0073] Based on the updated resource requirements, training nodes that meet the resource requirements are selected from the business queue to execute the model training task. The available training node resource pool in the queue is queried, and a set of nodes that meet the updated resource requirements is selected. Selection criteria include hardware specification matching, current load level, and historical performance stability. A matching score between nodes and tasks is calculated, comprehensively considering resource satisfaction, network topology, and historical execution efficiency. The node combination with the highest score is selected, and the task is assigned to these nodes for execution via the Kubernetes scheduler. For example, nodes node-gpu-12 and node-gpu-15 are selected from the "image-recognition-queue" queue to execute the aforementioned CNN training task. The resource configurations of these two nodes meet the requirements of nodes A and B respectively, and they have high matching scores.

[0074] In one optional implementation, the steps of creating a business queue in the Kubernetes cluster, allocating model training tasks to the corresponding business queues according to the training task type, generating a YAML file for the model training task, and submitting the YAML file to a custom controller in the Kubernetes cluster include:

[0075] In a Kubernetes cluster, a multi-level business queue is created, and the resource quota of the multi-level business queue is configured through a resource weight matrix. Based on the resource weight matrix, a resource borrowing matrix between queues is constructed, and the value of each element in the resource borrowing matrix is ​​determined by the resource utilization rate of the source queue, the resource utilization rate of the target queue, and the historical borrowing success rate.

[0076] Obtain feature data for the model training task, generate a multi-dimensional feature vector based on the feature data, and generate a model training task profile vector by weighted combination of the multi-dimensional feature vectors.

[0077] Calculate the cosine similarity between the model training task profile vector and the feature vectors of each business queue in the multi-level business queue. Multiply the cosine similarity by the priority of the corresponding business queue to obtain the queue matching degree. Assign the model training task to the business queue with the highest matching degree according to the queue matching degree. When the business queue with the highest matching degree has insufficient resources, determine the available queues and borrowing quotas according to the resource borrowing matrix to realize cross-queue resource allocation.

[0078] Write the queue allocation information for the model training task into a YAML file and submit it to the custom controller.

[0079] For example, creating multi-level service queues in a Kubernetes cluster can be achieved through custom resource definitions (CRDs). Service queues are divided into three levels: high priority, medium priority, and low priority. Each level can contain multiple specific service queues. The resource quotas for these queues are configured using a resource weight matrix. The resource weight matrix is ​​a two-dimensional array where rows represent different service queues and columns represent resource types such as CPU, memory, and GPU. For example, for a machine learning inference task queue, the CPU weight can be set to 0.6, the memory weight to 0.3, and the GPU weight to 0.1; while for a large-scale training task queue, the CPU weight can be set to 0.2, the memory weight to 0.2, and the GPU weight to 0.6. The actual resource quota for each queue is calculated based on these weight values ​​and the total cluster resources.

[0080] Based on the resource weight matrix, a resource borrowing matrix is ​​constructed between queues. The resource borrowing matrix is ​​an N×N matrix, where N is the number of queues. Each element B[i][j] in the matrix represents a feasibility score for borrowing resources from queue i to queue j. This score is determined by the resource utilization rate of the source queue, the resource utilization rate of the target queue, and the historical borrowing success rate. The resource utilization rate of the source queue is calculated as a percentage of the allocated resources to the total quota (e.g., CPU cores used / total cores). The resource utilization rate of the target queue is calculated as a percentage of the resources currently in use to the total quota. The historical borrowing success rate is calculated by dividing the number of successful borrowings in the past 24 hours by the total number of borrowing requests. Specifically, when the resource utilization rate of source queue i is below a threshold (e.g., 30%), the resource utilization rate of target queue j is above a threshold (e.g., 80%), and the historical borrowing success rate is greater than a certain value (e.g., 60%), the value of B[i][j] will be higher, indicating a higher feasibility of borrowing resources from queue i to queue j. Maintain a sliding window to record borrowing information over the past 24 hours, which is used to calculate the historical borrowing success rate.

[0081] When a new model training task is submitted, the first step is to acquire the task's feature data. This feature data includes model type (e.g., CNN, RNN, Transformer), estimated training time, dataset size, batch size, and computational complexity. Based on this feature data, a multi-dimensional feature vector is generated. For example, for an image classification task, the feature vector might include multiple dimensions such as model type (CNN=1, others=0), dataset size (medium=0.5), and computational complexity (high=0.8). This multi-dimensional feature vector is then weighted and combined to generate a model training task profile vector. The weighting process considers the importance of different features; for example, model type has a weight of 0.4, dataset size has a weight of 0.3, and computational complexity has a weight of 0.3. After weighted combination, a unified task profile vector is formed, which comprehensively represents the characteristics of the training task.

[0082] The cosine similarity is calculated between the task profile vector of the model training and the feature vectors of each business queue in the multi-level business queue. Each business queue has a predefined feature vector representing the type of task that the queue is suitable for processing. For example, the feature vector of a GPU-intensive task queue has a higher value in the GPU resource requirement dimension. The calculated cosine similarity reflects the degree of matching between the task and the queue, with a value between -1 and 1; the closer to 1, the better the match. This cosine similarity is multiplied by the priority weight of the corresponding business queue to obtain the final queue matching degree. For example, if the cosine similarity with a high-priority queue is 0.8 and the priority weight of that queue is 1.2, the final matching degree is 0.96.

[0083] Model training tasks are assigned to the business queue with the highest matching degree based on queue matching. If the business queue has sufficient resources, the task is created directly in that queue; if resources are insufficient, a resource borrowing mechanism is triggered. The resource borrowing matrix is ​​queried to find available queues and borrowing quotas. For example, if the current queue needs 4 GPUs but only 2 are available, the resource borrowing matrix will find the most suitable queue to lend resources to. If that queue has 2 idle GPUs and the borrowing conditions are met, these 2 GPUs will be temporarily allocated to the current queue. This cross-queue resource allocation mechanism significantly improves resource utilization.

[0084] After determining the queue allocation and resource borrowing scheme, a YAML file for the model training task is generated. This YAML file contains information such as queue identifier, resource requirements, container image, mounted volume, and environment variables. This YAML file is then submitted to a custom controller in the Kubernetes cluster. The custom controller parses the YAML file, creates the corresponding Pod based on the queue information and resource availability, and monitors the task execution status. When resource borrowing occurs, the controller records the borrowing and returns the resources upon task completion.

[0085] To ensure the accuracy of the resource borrowing matrix, the values ​​in the matrix are updated periodically. The update frequency can be set to once per hour, adjusting the matrix element values ​​based on recent resource usage and borrowing success rates. Furthermore, the queue feature vector is also fine-tuned based on historical task execution data, making the matching between queues and tasks more precise.

[0086] This invention establishes a multi-level business queue and resource borrowing mechanism to achieve intelligent allocation of model training tasks and dynamic resource allocation, thereby improving cluster resource utilization and task processing efficiency, reducing queuing time for training tasks, enhancing the execution stability of model training tasks, and providing a more efficient and flexible scheduling strategy for AI model training.

[0087] In one optional implementation, the step of collecting performance data of training nodes for the training phase features includes:

[0088] The training phase features include data loading phase features, model calculation phase features, and parameter update phase features; a training phase feature vector is constructed, which contains the performance indicators corresponding to each phase feature.

[0089] Based on the performance loss gradient of each feature in the feature vector of the training stage, the weight coefficients of each stage feature in the feature vector of the training stage are dynamically updated, and a performance data collection scheme is formulated according to the updated weight coefficients.

[0090] Calculate the predicted feature trend value based on the historical feature sequence of the feature vector during the training phase; use the deviation between the current feature vector and the predicted feature trend value as the feature change amplitude; dynamically optimize the sampling window size and sampling frequency of the performance data acquisition scheme according to the fluctuation law of the feature change amplitude; and perform performance data acquisition according to the optimized acquisition scheme.

[0091] The collected performance data is subjected to multi-dimensional quality assessment to obtain performance data quality assessment results; the performance data quality assessment results are fed back to the custom controller to optimize the feature recognition process during the training phase.

[0092] For example, the training phase features mainly include three core parts: data loading phase features, model computation phase features, and parameter update phase features. Data loading phase features include metrics such as data reading rate, data preprocessing time, data augmentation time, and data queue length, reflecting the performance of the data preparation stage during training. For example, in image classification tasks, a data reading rate of 1000 images per second and an average preprocessing time of 0.2 seconds per batch can be monitored. Model computation phase features include metrics such as forward propagation time, computation density, memory access patterns, and peak computational utilization, reflecting the model's execution efficiency on computing hardware. For example, for convolutional neural networks, a forward propagation time of 0.5 seconds per batch and a GPU computational utilization of 78% can be monitored. Parameter update phase features include metrics such as backpropagation time, gradient computation time, parameter update time, and communication overhead, reflecting the performance of the model weight update process. For example, a backpropagation time of 0.7 seconds per batch and a parameter update time of 0.1 seconds per batch can be monitored.

[0093] Based on the feature metrics of the three stages mentioned above, a training stage feature vector is constructed. This feature vector is a multi-dimensional array containing the performance metric values ​​corresponding to the features of each stage. For example, a training stage feature vector can be represented as: [Data read rate = 1000 images / second, preprocessing time = 0.2 seconds / batch, forward propagation time = 0.5 seconds / batch, GPU utilization = 78%, backpropagation time = 0.7 seconds / batch, parameter update time = 0.1 seconds / batch]. These metrics are collected by a monitoring agent deployed on the training nodes, with the initial collection frequency set to once every 5 seconds.

[0094] The performance loss gradient of each feature in the feature vector during the training phase represents the degree to which a change in a performance metric affects the overall training speed. It is calculated by recording the ratio of the change in performance metric to the change in training speed over a period of time. For example, if a 10% decrease in data loading rate leads to an 8% decrease in overall training speed, the performance loss gradient for data loading rate is 0.8. The performance loss gradient is calculated for each performance metric individually; a higher value indicates a greater impact of that metric on training performance. Based on the calculated performance loss gradient, the weight coefficients of features at each stage are adjusted. For example, if the performance loss gradient is 0.8 in the data loading stage, 0.5 in the model calculation stage, and 0.3 in the parameter update stage, the weight coefficient of the features in the data loading stage will be increased to 0.5, while the weight coefficients of other stages will be appropriately decreased. The updated weight coefficients are used to formulate a performance data collection plan, with a focus on strengthening the monitoring of high-weight stages.

[0095] Feature trend predictions are calculated based on historical feature sequences of feature vectors from the training phase. Specifically, a sliding window is used to collect feature vector sequences from the past 30 minutes, and time-series analysis is used to predict feature values ​​for the next 10 minutes. For example, if GPU utilization shows a fluctuating upward trend over the past 30 minutes, gradually increasing from 65% to 78%, it is predicted that GPU utilization will reach 85% within the next 10 minutes. The deviation between the current feature vector and the predicted feature trend is calculated as the feature change amplitude. For example, if the current GPU utilization is 78% and the predicted value is 85%, the change amplitude is 7%. Based on the fluctuation pattern of the feature change amplitude, the sampling window size and sampling frequency of the performance data acquisition scheme are dynamically optimized. When the feature change amplitude is large (e.g., exceeding 10%) and fluctuates frequently, the sampling window is reduced to 2 seconds and the sampling frequency is increased to once per second; when the feature change amplitude is small (e.g., below 5%) and relatively stable, the sampling window is expanded to 10 seconds and the sampling frequency is reduced to once every 10 seconds to reduce monitoring overhead. Performance data acquisition is then performed according to the optimized acquisition scheme, employing different acquisition strategies for features at different stages.

[0096] The collected performance data undergoes multi-dimensional quality assessment, including completeness assessment, consistency assessment, timeliness assessment, and noise level assessment. Completeness assessment checks for missing values ​​and calculates the data completeness rate. For example, if 985 out of 1000 collected data points are valid, the completeness rate is 98.5%. Consistency assessment checks if the data matches historical trends and calculates outliers. For example, if the current GPU utilization is 20%, while the historical average for the same period is 75%, the data is marked as an outlier. Timeliness assessment checks for data acquisition latency to ensure the data reflects the training status promptly. For example, if the data presentation time lags by more than 2 seconds compared to the actual occurrence time, the data's reliability is reduced. Noise level assessment checks for data volatility and filters out short-term noise. For example, if GPU utilization frequently fluctuates between 75% and 76% within a short period, these minor fluctuations are smoothed out. Based on these assessment dimensions, a comprehensive performance data quality score is given, ranging from 0 to 100. When the quality score is below 60, the current acquisition scheme is flagged as problematic, and acquisition parameters need adjustment.

[0097] The performance data quality assessment results are fed back to a custom controller to optimize the feature recognition process during the training phase. The custom controller adjusts the parameters of the feature extraction algorithm based on the quality assessment results. For example, when the data completeness rate is below 90%, the controller reduces the feature dimensions, focusing on core metrics that are easier to collect; when the data noise level is high, the controller increases the window size for smoothing processing. Through this feedback mechanism, the recognition accuracy of features during the training phase is continuously optimized, improving the targeting and effectiveness of subsequent performance data collection.

[0098] This invention achieves refined monitoring and optimization of the model training process by dynamically sensing the characteristics of the training phase and collecting corresponding performance data. This effectively improves the efficiency of deep learning training, reduces the consumption of computing resources, shortens the model training time, and can promptly identify performance bottlenecks in the training process, providing a precise basis for resource allocation and scheduling of training tasks.

[0099] In one optional implementation, the steps of dynamically updating the weight coefficients of each feature in the training phase feature vector based on the performance loss gradient of each feature, and formulating a performance data acquisition scheme based on the updated weight coefficients, include:

[0100] Based on the historical change characteristics of the performance loss gradient, the gradient influence factor is determined. The weight coefficients of each stage feature in the training stage feature vector are dynamically updated according to the performance loss gradient and the gradient influence factor to obtain the updated weight coefficients.

[0101] A data acquisition sensitivity matrix is ​​constructed based on the updated weight coefficients, which represents the degree of correlation between features and performance indicators in each training stage; a time-series sensitivity feature is constructed based on the historical sequence of the data acquisition sensitivity matrix, which represents the change pattern of the data acquisition sensitivity matrix over time; and a stage coupling feature is constructed based on the correlation between different training stages in the data acquisition sensitivity matrix.

[0102] The time-series sensitivity features and the stage coupling features are fused to construct a multi-level optimization objective function. The multi-level optimization objective function includes a sensitivity weighting term and a fluctuation penalty term in the main objective function, and an upper limit for resource utilization and a lower limit for sampling frequency in the constraint function. The multi-level optimization objective function is solved using the adaptive step-size Lagrange multiplier method, wherein the multiplier update rate is dynamically adjusted according to the degree of constraint violation. The search space is optimized based on historical performance data and task similarity, and a performance data acquisition scheme is output when the stability of the solution meets the preset conditions.

[0103] For example, hourly performance loss gradient data over the past 24 hours is collected to form a time series. The performance loss gradient represents the degree of impact of a feature change on the overall training performance. For instance, if a 10% decrease in data loading speed leads to a 7% decrease in overall training speed, the performance loss gradient is 0.7. The trend of these gradient changes is analyzed, and their fluctuation amplitude and direction are calculated. If the gradient value continues to rise and fluctuates little, it indicates that the feature's impact on performance is becoming increasingly stable and important, and its gradient impact factor is set to a higher value, such as 0.9. If the gradient value fluctuates greatly, it indicates that the feature's impact on performance is unstable, and its gradient impact factor is set to a lower value, such as 0.5. In a certain training task, the performance loss gradient during the data loading phase is 0.7 and relatively stable, and its gradient impact factor is calculated to be 0.85; while the performance loss gradient during the parameter update phase is 0.4 and fluctuates greatly, and its gradient impact factor is calculated to be 0.6.

[0104] The weight coefficients of features at each stage in the feature vector during the training phase are dynamically updated based on the performance loss gradient and gradient influence factor. The update method uses a weighted average, where the new weight equals the old weight multiplied by a decay factor (e.g., 0.7), plus the performance loss gradient multiplied by the gradient influence factor, and then multiplied by a gain factor (e.g., 0.3). For the data loading phase, if the original weight is 0.4, the updated weight is 0.4×0.7 + 0.7×0.85×0.3 ≈ 0.46; for the model calculation phase, if the original weight is 0.35, the performance loss gradient is 0.6, and the gradient influence factor is 0.8, the updated weight is 0.35×0.7 + 0.6×0.8×0.3 ≈ 0.39; for the parameter update phase, if the original weight is 0.25, the updated weight is 0.25×0.7 + 0.4×0.6×0.3 ≈ 0.25. After the weights are updated, they need to be normalized to ensure that the sum of all weights is 1. The normalized weights are as follows: 0.42 for the data loading stage, 0.35 for the model calculation stage, and 0.23 for the parameter update stage.

[0105] A sensitivity matrix is ​​constructed based on the updated weight coefficients. This matrix is ​​a two-dimensional array, where rows represent different training stages (data loading, model computation, parameter updates) and columns represent different performance metrics (CPU utilization, memory utilization, GPU utilization, network throughput, etc.). The value of each element in the matrix represents the sensitivity of a specific training stage to a specific performance metric, determined by the correlation between the stage's weight and the performance metric. For example, the data loading stage has a sensitivity of 0.85 to disk I / O and 0.2 to GPU utilization; the model computation stage has a sensitivity of 0.9 to GPU utilization and 0.15 to network throughput; and the parameter update stage has a sensitivity of 0.75 to network throughput and 0.6 to memory utilization.

[0106] Temporal sensitivity features are constructed based on the historical sequence of the acquisition sensitivity matrix. The acquisition sensitivity matrix for each hour over the past 12 hours is saved, and the changing trends of matrix elements over time are analyzed. Temporal sensitivity features include three aspects: rate of change, periodicity, and trend direction. The rate of change is obtained by calculating the sensitivity difference between consecutive time points; for example, the sensitivity to disk I / O during the data loading phase increased from 0.78 to 0.85 over the past 4 hours, with a rate of change of 0.0175 / hour. Periodicity is identified by analyzing recurring patterns in sensitivity values; for example, the sensitivity to GPU utilization during the model calculation phase peaks every 4 hours. The trend direction is determined by the sign of the slope in linear regression: positive values ​​indicate an upward trend, and negative values ​​indicate a downward trend.

[0107] Stage coupling features are constructed based on the correlation between different training stages in the sensitivity matrix. Specifically, the correlation coefficients of sensitivity values ​​between different training stages are calculated to form a stage coupling matrix. For example, the coupling degree between the data loading stage and the model calculation stage is 0.4, indicating a moderate correlation; the coupling degree between the model calculation stage and the parameter update stage is 0.7, indicating a high correlation. Furthermore, the existence of leader-lag relationships in the temporal changes of different stages is analyzed. For instance, sensitivity changes in the data loading stage often lead those in the model calculation stage by 2 hours; this relationship is recorded as the time dependency feature between stages.

[0108] The temporal sensitivity features and stage coupling features are fused to form the basis for constructing the optimization objective function. Feature fusion uses a weighted combination method, with a weight of 0.6 for temporal features and 0.4 for coupling features. Taking the sensitivity to disk I / O during the data loading stage as an example, if its temporal sensitivity is 0.85 and the stage coupling feature value is 0.3, the fused feature value is 0.85 × 0.6 + 0.3 × 0.4 = 0.63. Fusion feature values ​​are calculated for all training stages and performance metrics to form a fusion feature matrix. Based on the fusion feature matrix, a multi-level optimization objective function is constructed. This function consists of a main objective function and constraint functions. The main objective function includes a sensitivity weighting term and a fluctuation penalty term. The sensitivity weighting term uses the fused feature values ​​as weight coefficients, multiplied by the sampling frequency of each performance metric, and then summed. For example, if the fusion sensitivity of GPU utilization is 0.8 and the sampling frequency is 0.5 times / second, its contribution to the sensitivity weighting term is 0.8 × 0.5 = 0.4. This design allows for more sampling resources for highly sensitive metrics. The fluctuation penalty term is the sum of squares of the amplitude of changes in sampling frequency, aiming to reduce drastic fluctuations in the sampling scheme. The main objective function combines these two terms through a subtraction relationship: Main Objective Function = Sensitivity Weighted Term - Fluctuation Penalty Term × Penalty Coefficient (typically 0.3). The constraint function includes an upper limit for resource utilization and a lower limit for sampling frequency. The upper limit for resource utilization ensures that the data acquisition process does not excessively consume resources, expressed as the weighted sum of the sampling frequencies of all metrics not exceeding a threshold (e.g., 5%). The lower limit for sampling frequency ensures that key metrics have sufficient sampling points, such as GPU utilization sampling frequency not less than once per second. This structure allows the optimization process to balance data acquisition comprehensiveness, resource consumption, and stability, providing a clear mathematical framework for subsequent solutions.

[0109] An adaptive-step Lagrange multiplier method is used to solve the multi-level optimization objective function. The sampling frequency of each performance metric is initialized to a moderate value (e.g., once every 5 seconds), and the initial value of the Lagrange multiplier is set to 0.5. In each iteration, the sampling frequency is updated according to the gradient of the objective function, and the update step size is dynamically adjusted based on the historical iteration results. When the improvement effect after several consecutive iterations is small, the step size is reduced to improve accuracy; when the objective function value is still improving rapidly, a larger step size is maintained to accelerate convergence. The multiplier update rate is dynamically adjusted according to the degree of constraint violation; the more severe the constraint violation, the larger the update rate. For example, when the sampling scheme causes the CPU utilization to reach 5.5%, exceeding the upper limit by 0.5%, the corresponding multiplier update rate is set to 0.2; while when the sampling frequency is only 0.1 times / second higher than the lower limit, the corresponding multiplier update rate is set to 0.05.

[0110] The system retrieves historical tasks similar to the current task from the historical task database (with a similarity threshold of 0.8) and analyzes their optimal sampling schemes as the initial point and reference range for the current search. For example, if the optimal GPU sampling frequency for similar image classification tasks in the past is concentrated every 2-3 seconds, the search range will be narrowed down to the vicinity of this interval. The stability of the solution is measured by the variance of the continuous iteration results. When the variance of the sampling scheme for 10 consecutive iterations is less than a preset threshold (e.g., 0.01), the solution process is considered to have converged, and the final performance data acquisition scheme is output. The final output acquisition scheme includes the sampling frequency, sampling duration, and triggering conditions for each performance indicator, such as GPU utilization being sampled every 2 seconds for 30 minutes; and network throughput being sampled every 1 second during the parameter update phase for 5 minutes.

[0111] This invention achieves precise monitoring of the deep learning training process by dynamically updating feature weights during the training phase based on performance loss gradients and optimizing the performance data acquisition scheme. This improves the targeting of performance data acquisition and the efficiency of resource utilization, provides high-quality data support for training process optimization, reduces monitoring overhead, and enables more accurate identification of performance bottlenecks and corresponding optimization measures.

[0112] In one optional implementation, the step of determining an initial batch allocation value for a training node based on the training phase characteristics and the performance data, and continuously adjusting the batch allocation value based on the actual computation latency of the training node when executing the initial batch allocation value, until the maximum difference in the single-batch computation time of the training node is less than a dynamic latency threshold, includes:

[0113] The computational load characteristics of the training nodes are constructed based on the training phase characteristics and the performance data, wherein the computational load characteristics characterize the resource consumption level of each training phase; and the initial batch allocation value of the training nodes is determined according to the computational load characteristics.

[0114] The single-batch computation time of the training node when executing the initial batch allocation value is obtained, and the maximum difference between the single-batch computation times of different training nodes is calculated. Based on the maximum difference and the spectral analysis results of the performance fluctuation of the training node, an adaptive batch adjustment amount is constructed. The batch adjustment amount is negatively correlated with the stability of the performance fluctuation frequency and has a piecewise nonlinear relationship with the degree of deviation of the single-batch computation time from the average value.

[0115] A dynamic latency threshold is constructed, which decays exponentially with training time and has a minimum threshold limit; when the maximum difference is greater than the dynamic latency threshold, the current batch allocation value of the training node is adjusted based on the batch adjustment amount until the maximum difference of the single batch calculation time is less than or equal to the dynamic latency threshold.

[0116] The adjusted batch allocation execution results are fed back to the performance data acquisition scheme to optimize the performance data acquisition process in the subsequent training phase.

[0117] Combination Figure 2 The flowchart illustrating the dynamic adjustment and optimization process for training nodes in batches is provided. For example, when constructing the computational load characteristics of training nodes based on training phase features and performance data, resource consumption data for each training node during data loading, forward propagation, backpropagation, and parameter update phases is collected. The computational load characteristic is a multi-dimensional vector containing metrics such as CPU utilization, memory usage, GPU utilization, and memory bandwidth utilization for each training phase. For example, for node A, its computational load characteristics are: [CPU utilization during data loading phase = 45%, GPU utilization during forward propagation phase = 85%, GPU utilization during backpropagation phase = 92%, network bandwidth utilization during parameter update phase = 60%]. These raw data are normalized, mapping the values ​​of each metric to the range of 0 to 1 for easier subsequent calculations. In the case of multiple training nodes, an independent computational load characteristic vector is constructed for each node.

[0118] The overall computing power score of each node is calculated, considering multiple factors: GPU computing performance (weight 0.5), memory bandwidth (weight 0.3), and CPU processing power (weight 0.2). For example, if node A has a GPU performance score of 0.8, a memory bandwidth score of 0.7, and a CPU score of 0.6, its overall computing power score is 0.8 × 0.5 + 0.7 × 0.3 + 0.6 × 0.2 = 0.73. Initial batches are allocated proportionally based on each node's overall score and the total batch size. Assuming a total training batch size of 128, and two nodes A and B with overall scores of 0.73 and 0.62 respectively, the initial batch allocation for node A is 128 × 0.73 / (0.73 + 0.62) = 69, and the initial batch allocation for node B is 128 × 0.62 / (0.73 + 0.62) = 59. The discreteness constraint of batch size is also considered to ensure that the allocated batches are integers and meet the requirements of model training.

[0119] After allocating initial batches, monitor the time required for each node to complete a batch of training. For example, node A takes 210 milliseconds to process a batch of 69 samples, while node B takes 190 milliseconds to process a batch of 59 samples. Calculate the single batch time: node A takes approximately 210 / 69 ≈ 3.04 milliseconds per sample, and node B takes approximately 190 / 59 ≈ 3.22 milliseconds per sample. The maximum difference in single batch computation time between different nodes is |3.04 - 3.22| = 0.18 milliseconds per sample.

[0120] An adaptive batch adjustment is constructed based on the spectral analysis results of the maximum difference and the performance fluctuations of the training nodes. Spectral analysis is performed on the performance data of each node to extract the frequency characteristics of performance fluctuations. The spectral analysis uses a sliding window method with a window size of 20 training iterations, identifying the main fluctuation frequencies and intensities by calculating the power spectral density. For example, if node A's performance fluctuation spectrum has a peak at 0.05Hz with an intensity of 0.8, it indicates that its performance fluctuates significantly approximately every 20 seconds; while node B's fluctuation spectrum is relatively flat, with a maximum peak intensity of only 0.3, indicating relatively stable performance. The stability index of performance fluctuations is defined as 1 minus the intensity of the dominant frequency; the stability of node A is 1-0.8=0.2, and the stability of node B is 1-0.3=0.7. The batch adjustment is negatively correlated with the stability of the performance fluctuation frequency; the lower the stability, the smaller the adjustment, to avoid performance oscillations caused by over-adjustment. Meanwhile, the batch adjustment amount and the degree of deviation of the single batch calculation time from the average value exhibit a piecewise non-linear relationship: when the deviation is within the range of 0-10%, the adjustment coefficient is 0.5 times the deviation; when the deviation is within the range of 10%-30%, the adjustment coefficient is 0.8 times the deviation; and when the deviation exceeds 30%, the adjustment coefficient is 1.2 times the deviation. This piecewise design makes the adjustment more conservative for small deviations and more aggressive for large deviations.

[0121] A dynamic latency threshold is constructed, which decays exponentially with training time and has a minimum threshold limit. The initial value of the dynamic latency threshold is set to 15% of the average computation time per batch, and it decays with the number of training iterations. The decay formula is exponential: threshold = initial threshold × decay coefficient raised to the power of the number of iterations, where the decay coefficient is typically set to 0.995. For example, if the initial threshold is 0.3 ms / sample, after 100 iterations, the threshold decreases to 0.3 × 0.995^100 ≈ 0.18 ms / sample. To prevent the threshold from being too small and causing frequent adjustments, a minimum threshold limit is set to 20% of the initial threshold, i.e., 0.06 ms / sample. When the calculated threshold is less than this limit, the minimum threshold is used.

[0122] When the maximum difference exceeds the dynamic latency threshold, the current batch allocation value of the training node is adjusted based on the batch adjustment amount. Continuing the previous example, if the current maximum difference of 0.18 ms / sample is greater than the dynamic threshold of 0.06 ms / sample, batch adjustment is triggered. Assuming that node A's single batch time is 8% lower than the average and its stability is 0.2, its adjustment coefficient is 0.08 × 0.5 × 0.2 = 0.008; node B's single batch time is 12% higher than the average and its stability is 0.7, its adjustment coefficient is 0.12 × 0.8 × 0.7 = 0.0672. A batch of 0.0672 × 59 ≈ 4 samples is reduced from node B and allocated to node A. After adjustment, node A has 69 + 4 = 73 batches, and node B has 59 - 4 = 55 batches. The single batch computation time after adjustment is continuously monitored. If the maximum difference is still greater than the dynamic threshold, adjustments continue until the condition is met (less than or equal to the threshold) or the maximum number of adjustments is reached (e.g., 10 times).

[0123] The effects of each batch adjustment are recorded, including changes in single-batch computation time and training throughput before and after the adjustment. This data is used to optimize performance data collection strategies, such as increasing the sampling frequency for nodes with large fluctuations or adjusting the monitoring density for specific training phases. For example, if node A is found to fluctuate significantly during the parameter update phase, the sampling frequency for that phase will be increased from once every 5 seconds to once every 2 seconds. The optimized collection scheme can more accurately capture performance bottlenecks, providing higher-quality decision-making basis for the next round of batch adjustments.

[0124] This invention achieves efficient and balanced utilization of computing resources in distributed training tasks by constructing computational load characteristics and dynamically adjusting batch allocation. It effectively solves the problem of uneven load in heterogeneous computing environments, reduces training time, improves resource utilization, and can adaptively cope with performance fluctuations during training, providing stable and efficient technical support for the training of large-scale deep learning models.

[0125] In one alternative implementation, the step of constructing the dynamic delay threshold includes:

[0126] Obtain the computational power parameters of the training nodes, and calculate the initial threshold of the training nodes based on the computational power parameters;

[0127] Obtain the current gradient variance, current loss value, and historical adjustment frequency of the training node, and calculate a custom decay coefficient based on the current gradient variance, the current loss value, and the historical adjustment frequency;

[0128] The dynamic latency threshold is obtained by multiplying the initial threshold by an exponential decay function of the training time, wherein the decay coefficient in the exponential decay function is determined by the custom decay coefficient; when the dynamic latency threshold is less than the minimum threshold limit, the minimum threshold limit is used as the dynamic latency threshold.

[0129] For example, the computational power parameters of a training node include hardware metrics such as the number of processor cores, processor frequency, memory capacity, memory bandwidth, accelerator type, and its computational capability. These parameters are obtained through tools provided by the underlying hardware interface or operation. For instance, a training node might have the following computational power parameters: an 8-core processor, a 3.2GHz frequency, 32GB of memory, and an accelerator single-precision computing capability of 14 TFLOPS. To facilitate subsequent calculations, these heterogeneous parameters are uniformly converted into a standardized computational power score. The conversion method involves comparing each parameter with a preset benchmark value, calculating the relative performance ratio, and then performing a weighted sum according to predefined weights. For example, the processor weight is 0.2, the memory weight is 0.3, and the accelerator weight is 0.5. Assuming that the node's processor performance is 1.5 times the benchmark, its memory performance is 1.2 times the benchmark, and its accelerator performance is 2.0 times the benchmark, then its standardized computational power score is 1.5 × 0.2 + 1.2 × 0.3 + 2.0 × 0.5 = 1.66.

[0130] The initial threshold represents the upper limit of the allowable difference in computational latency between nodes during the initial training phase. It is closely related to the heterogeneity of the nodes and the distribution of computational power. The calculation method is to multiply the difference between the highest and lowest computational power scores by a scaling factor, and then divide by the number of nodes in the cluster. The scaling factor is typically set between 0.1 and 0.2, reflecting the tolerance for initial computational imbalance. For example, in a four-node cluster, if the highest computational power score is 2.0 and the lowest is 1.2, with a difference of 0.8, and the scaling factor is set to 0.15, then the initial threshold is 0.8 × 0.15 / 4 = 0.03, indicating that the allowable difference in computation time per batch is 3% of the average time. Furthermore, the impact of model complexity on the initial threshold is considered. Complex models typically have higher initial thresholds to accommodate their computational characteristics. For example, for a large Transformer model, multiplying the calculated initial threshold by 1.5 yields an adjusted initial threshold of 0.045.

[0131] The current gradient variance reflects the stability of model training and is obtained by calculating the statistical variance of the gradients of each parameter in the most recent 10 batches. For example, if the gradient variance of the most recent 10 batches is 0.005, it indicates relatively stable training; if it is 0.05, it indicates large training fluctuations. The current loss value represents the convergence degree of model training and is obtained by directly reading the loss function value during training. For example, the loss value is 2.5 in the initial training stage, and decreases to 0.3 in the later stages of training. The historical adjustment frequency reflects the frequency of batch allocation adjustments and is calculated by counting the number of batch adjustments in the past 50 training iterations. For example, if 8 batch adjustments were made in the past 50 iterations, the historical adjustment frequency is 8 / 50 = 0.16, indicating that an adjustment was made on average every 6.25 iterations.

[0132] A custom decay factor is calculated based on the current gradient variance, current loss value, and historical adjustment frequency. This custom decay factor controls the rate at which the dynamic latency threshold decays over training time; it should be dynamically adjusted according to the training state. The calculation method involves multiplying the base decay factor by three adjustment factors. The base decay factor is typically set to 0.995, representing the standard decay rate under normal conditions. The gradient variance adjustment factor is directly proportional to the gradient variance; the larger the gradient variance, the larger the adjustment factor (and the slower the decay). For example, when the gradient variance is 0.005, the corresponding adjustment factor is 0.98; when the gradient variance is 0.05, the adjustment factor is 1.02. The loss value adjustment factor is directly proportional to the ratio of the current loss value to the initial loss value; the more the loss value decreases, the smaller the adjustment factor (and the faster the decay). For example, if the current loss value is 50% of the initial value, the corresponding adjustment factor is 0.99; if it is 10% of the initial value, the adjustment factor is 0.97. The historical adjustment frequency adjustment factor is inversely proportional to the adjustment frequency; the more frequent the adjustments, the larger the adjustment factor (and the slower the decay). For example, when the adjustment frequency is 0.05, the corresponding adjustment factor is 0.98; when the adjustment frequency is 0.2, the adjustment factor is 1.03. Multiplying the base attenuation coefficient by the three adjustment factors, for example, 0.995×0.98×0.99×1.03≈0.993, yields the final custom attenuation coefficient.

[0133] The dynamic latency threshold is obtained by multiplying the initial threshold by an exponential decay function of the training time. The exponential decay function controls the rate at which the threshold gradually decreases during training. Specifically, this function is a power of the number of training iterations over a custom decay coefficient. To implement this calculation, a global iteration counter is maintained, incrementing from 0 at the start of training. After the t-th iteration, the dynamic latency threshold is calculated as the initial threshold multiplied by the power of t over the custom decay coefficient. In practice, logarithmic calculations are used to avoid numerical precision issues caused by large exponential operations. First, the logarithm of the exponent is calculated, i.e., t multiplied by the natural logarithm of the custom decay coefficient, and then the exponent is reduced to obtain the final result. For example, with an initial threshold of 0.045 and a custom decay coefficient of 0.993, after 100 training iterations, the calculation steps are: (1) calculate ln(0.993)≈-0.007; (2) calculate 100×(-0.007)=-0.7; (3) calculate e^(-0.7)≈0.497; (4) calculate 0.045×0.497≈0.022. Therefore, the dynamic delay threshold is 0.022. In practice, to improve computational efficiency, the dynamic delay threshold can be updated every fixed number of iterations (e.g., 10 times) instead of being recalculated for each iteration. For long-running training tasks, the baseline for exponential calculation is periodically reset to avoid accumulated errors. For example, every 1000 iterations, the current threshold is set as a new reference value, and the iteration count is reset, with the exponential decay calculation continuing from this new starting point. This segmented calculation method effectively avoids the numerical instability problem that occurs during long-term training. After each calculation, the system checks whether the obtained dynamic latency threshold is less than the preset minimum threshold limit. The minimum threshold limit is typically set to 15% to 20% of the initial threshold to prevent frequent adjustments due to an excessively small threshold. For example, if the initial threshold is 0.045, the minimum threshold limit can be set to 0.045 × 0.15 = 0.00675. If the calculated dynamic latency threshold is less than 0.00675, 0.00675 will be used as the actual dynamic latency threshold to ensure stability. Furthermore, different decay strategies are set according to the training progress. In the early stages of training (e.g., the first 10% of iterations), a slower decay rate is used by adjusting the custom decay coefficient to its square root; in the middle stages of training, a standard decay rate is used; and in the later stages of training (e.g., the last 20% of iterations), when the model is close to convergence, threshold decay is completely stopped, maintaining the minimum threshold limit to avoid unnecessary batch adjustments due to minor computational fluctuations.

[0134] The effectiveness of the dynamic latency threshold is evaluated periodically and fine-tuned based on actual training conditions. For example, if batch adjustments are found to be too frequent, the minimum threshold limit can be temporarily increased; if a long-term load imbalance is found between nodes, the attenuation coefficient can be temporarily decreased. Furthermore, a threshold parameter library is maintained for different types of model training tasks, recording the optimal threshold parameter settings for various models in historical training, providing more accurate initial parameters for new tasks.

[0135] This invention achieves precise balancing of computational load in distributed training by constructing a dynamic latency threshold that adaptively adjusts based on training state. This effectively reduces waiting time during training, improves the efficiency of computing resource utilization, and can automatically adjust the tolerance for imbalance according to the training progress, providing a more efficient and stable training environment for large-scale deep learning models.

[0136] In one optional implementation, the step of selecting a training node that meets the resource requirements in the service queue to perform the model training task based on the updated resource requirement information includes:

[0137] A performance index vector for training nodes is constructed, which includes computational power and historical stability. An adaptive weight matrix is ​​constructed based on the performance index vector, and the weight coefficients of the adaptive weight matrix are dynamically adjusted according to the computational and communication density of the training task. The adaptive weight matrix is ​​multiplied by the performance index vector to obtain a node fitness matrix, which represents the degree of fitness of different training nodes to the model training task.

[0138] Based on the updated resource requirement information, a set of candidate nodes that meet the basic resource conditions is selected, and the set of candidate nodes is sorted based on the node adaptability matrix.

[0139] Based on the sorted candidate node set, the network topology distance between candidate nodes is calculated, and an initial topology affinity matrix is ​​constructed based on the network topology distance. The initial topology affinity matrix is ​​weighted and fused with the historical communication efficiency matrix to obtain an optimized topology affinity matrix. Based on the optimized topology affinity matrix and the updated resource requirement information, the node combination score is calculated. The training node combination with the highest node combination score and meeting the resource requirements is selected to perform the model training task.

[0140] For example, the performance metric vector includes two dimensions: computing power and historical stability. The computing power dimension integrates hardware metrics such as the number of processor cores, processor frequency, number of accelerators, accelerator computing power, and memory bandwidth, which are standardized and converted into scores between 0 and 1. For instance, a training node with a 32-core, 2.6GHz processor, four accelerators (each with 12 TFLOPS of computing power and 900GB / s memory bandwidth) receives a computing power score of 0.85 after standardization. The historical stability dimension is calculated by analyzing the node's task completion, resource fluctuations, and failure rate over the past 30 days. For example, a node that completed 95% of its assigned tasks, had a resource fluctuation rate of 5%, and a failure rate of 2% over the past 30 days receives a weighted historical stability score of 0.92. Newly added nodes without historical data are assigned a lower initial stability score (e.g., 0.7) and updated quickly after each task execution. Ultimately, the performance metric vector for this training node is [0.85, 0.92].

[0141] An adaptive weight matrix is ​​constructed based on performance metric vectors. This adaptive weight matrix represents the importance of different performance metrics to a specific training task, and its weight coefficients are dynamically adjusted according to the computational and communication density of the training task. First, the characteristics of the training task are analyzed, including the number of model parameters, batch size, forward propagation computational complexity, backpropagation gradient computational complexity, and communication data volume, to derive the task's computational and communication density metrics. For example, for a large Transformer model training task with 1 billion parameters and a batch size of 128, the analysis yields a computational density metric of 0.8 and a communication density metric of 0.6. For computationally intensive tasks (computational density > 0.7), the weight of computational capability in the weight matrix is ​​increased; for communication-intensive tasks (communication density > 0.7), the weight of historical stability in the weight matrix is ​​increased. Specifically, the adjustment method involves multiplying the base weights by the square of the density metric and then normalizing. For example, the initial weight matrix is ​​[[0.7,0.3],[0.4,0.6]], which becomes [[0.7×0.8×0.8,0.3×0.6×0.6],[0.4×0.8×0.8,0.6×0.6×0.6]] after adjustment, i.e., [[0.448,0.108],[0.256,0.216]]. After normalization, the final adaptive weight matrix is ​​obtained as [[0.806,0.194],[0.542,0.458]].

[0142] Multiplying the performance metric vector of each node by the adaptive weight matrix yields a numerical value representing the node's fitness. For example, for a node with a performance metric vector of [0.85, 0.92], multiplying it by the adaptive weight matrix [[0.806, 0.194], [0.542, 0.458]] results in [0.85×0.806+0.92×0.194, 0.85×0.542+0.92×0.458]=[0.685+0.178, 0.461+0.422]=[0.863, 0.883]. This indicates that the node's computational fitness for the current training task is 0.863, and its communication fitness is 0.883. The same calculation is performed for each node in the cluster, forming a complete node fitness matrix.

[0143] Based on the updated resource requirements, including minimum processor core count, minimum memory capacity, and minimum number of accelerators, all training nodes in the cluster are traversed, and nodes that meet these conditions are selected as a candidate node set. For example, if the task requires at least 16 processor cores, 64GB of memory, and 2 accelerators, nodes that do not meet these conditions will be excluded. The selected candidate node set is then sorted based on the node fitness matrix. The sorting method calculates the overall fitness score for each node, which is the weighted sum of fitness and communication fitness, with the weights being computational density and communication density, respectively. For example, for a node fitness [0.863, 0.883], with a computational density of 0.8 and a communication density of 0.6, the overall fitness score is 0.863 × 0.8 + 0.883 × 0.6 = 0.690 + 0.530 = 1.220. The candidate nodes are then sorted from highest to lowest overall fitness score.

[0144] Based on the sorted set of candidate nodes, the network topology distance between candidate nodes is calculated. The network topology distance represents the quality of network connectivity between nodes and is calculated by measuring network latency, bandwidth, and hop count. A pre-defined probing tool is used to measure the bidirectional network latency and bandwidth between candidate nodes, recording the average and variance. For example, if the average network latency between node A and node B is 0.5 milliseconds, the bandwidth is 10 Gbps, and the hop count is 2, the network topology distance after standardization is 0.2 (a smaller value indicates better connection quality). The network topology distance is calculated for each pair of candidate nodes, constructing an initial topology affinity matrix. The topology affinity value is defined as 1 minus the standardized network topology distance; for example, the topology affinity between node A and node B is 1 - 0.2 = 0.8.

[0145] The historical communication efficiency matrix records the actual communication efficiency of node pairs in past training tasks, calculated by monitoring inter-node communication throughput, latency, and error rate. For example, the average communication efficiency between node A and node B when performing similar training tasks in the past is 0.85 (a higher value indicates higher efficiency). The initial topology affinity and historical communication efficiency are weighted and fused, with the fusion weight dynamically adjusted based on the reliability of the historical data. For example, if the historical data is relatively reliable (based on data volume and timeliness assessment), a weight of 0.4 is assigned to the initial topology affinity, and a weight of 0.6 is assigned to the historical communication efficiency, resulting in an optimized topology affinity of 0.8 × 0.4 + 0.85 × 0.6 = 0.32 + 0.51 = 0.83.

[0146] Node combination scores are calculated based on the optimized topology affinity matrix and updated resource requirement information. Node combinations are generated from the top-ranked candidate nodes according to the number of nodes required for the task. For each node combination, its overall topology affinity (a weighted average of affinity scores for all nodes) and resource matching degree (the degree to which node resources match task requirements) are calculated. For example, a node combination containing nodes A, B, and C has an overall topology affinity of 0.81 and a resource matching degree of 0.95. The node combination score is calculated by a weighted sum of the overall topology affinity and resource matching degree. The weights are adjusted according to task characteristics; typically, communication-intensive tasks are given higher weight for topology affinity, while computation-intensive tasks are given higher weight for resource matching degree. For example, for a task with a computational intensity of 0.8 and a communication intensity of 0.6, the resource matching weight can be set to 0.65 and the topology affinity weight can be set to 0.35, resulting in a node combination score of 0.95×0.65+0.81×0.35=0.618+0.284=0.902.

[0147] The scores of all candidate node combinations are compared, and the combination with the highest score is selected. To ensure the stability of the selection, historical performance fluctuations between combinations with similar scores (difference less than 5%) are also considered, with combinations exhibiting smaller fluctuations given priority. After determining the final node combinations, tasks are assigned to these nodes for execution, and node performance and communication efficiency are continuously monitored during task execution, with relevant historical data updated to provide a more accurate reference for subsequent task assignments.

[0148] This invention constructs a performance index vector and an adaptive weight matrix, combined with network topology analysis and historical communication efficiency, to achieve precise selection and combination of distributed training nodes. This effectively improves model training efficiency and resource utilization, reduces communication overhead, and enhances the stability of the training process, providing an efficient and reliable node selection mechanism for large-scale distributed deep learning.

[0149] A second aspect of the present invention provides an electronic device, comprising:

[0150] processor;

[0151] Memory used to store processor-executable instructions;

[0152] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0153] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0154] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for model training based on k8s, characterized in that, The method comprises the following steps: creating a business queue in a k8s cluster, assigning a model training task to a corresponding business queue according to the type of the training task, generating a yaml file of the model training task, and submitting the yaml file to a custom controller in the k8s cluster; the custom controller identifies the training phase characteristics of the model training task, and collects performance data of the training node according to the training phase characteristics; determining the initial batch allocation value of the training node according to the training phase characteristics and the performance data, continuously adjusting the batch allocation value based on the actual calculation time delay of the training node when executing the initial batch allocation value, until the maximum difference of the single batch calculation time length of the training node is less than the dynamic time delay threshold; write the adjusted batch allocation value to the yaml file, and calculate the corresponding resource demand according to the adjusted batch allocation value, and update the resource demand information in the yaml file; based on the updated resource demand information, select a training node that meets the resource demand in the business queue to execute the model training task.

2. The method of claim 1, wherein, The step of creating a business queue in a k8s cluster, assigning a model training task to a corresponding business queue according to the type of the training task, generating a yaml file of the model training task, and submitting the yaml file to a custom controller in the k8s cluster comprises: creating a multi-level business queue in a k8s cluster, the resource quota of the multi-level business queue is configured through a resource weight matrix; based on the resource weight matrix, a resource borrowing matrix between queues is constructed, the value of each element in the resource borrowing matrix is determined by the source queue resource utilization, the target queue resource utilization and the historical borrowing success rate; obtain the feature data of the model training task, generate a multi-dimensional feature vector based on the feature data; combine the multi-dimensional feature vector to generate a model training task portrait vector; calculate the cosine similarity of the model training task portrait vector and the feature vector of each business queue in the multi-level business queue, multiply the cosine similarity by the priority of the corresponding business queue to obtain the queue matching degree; according to the queue matching degree, the model training task is assigned to the business queue with the highest matching degree; when the resource of the business queue with the highest matching degree is insufficient, the borrowable queue and the borrowing amount are determined according to the resource borrowing matrix, and the resource allocation across queues is realized; write the queue allocation information of the model training task into the yaml file and submit it to the custom controller.

3. The method of claim 1, wherein, The step of collecting performance data of the training node according to the training phase characteristics comprises: the training phase characteristics include data loading phase characteristics, model calculation phase characteristics and parameter updating phase characteristics; a training phase feature vector is constructed, which contains the performance indicators corresponding to each phase feature; based on the performance loss gradient of each feature in the training phase feature vector, dynamically update the weight coefficient of each phase feature in the training phase feature vector, and formulate a performance data collection scheme according to the updated weight coefficient; calculating a feature trend prediction value based on a historical feature sequence of the training phase feature vector; taking a deviation of a current feature vector from the feature trend prediction value as a feature change amplitude, and dynamically optimizing a sampling window size and a sampling frequency of the performance data collection scheme according to a fluctuation rule of the feature change amplitude; and performing performance data collection according to the optimized collection scheme; performing multi-dimensional quality evaluation on the collected performance data to obtain a performance data quality evaluation result; and feeding back the performance data quality evaluation result to the self-defined controller for optimizing a recognition process of a training phase feature.

4. The method of claim 3, wherein, dynamically updating a weight coefficient of each phase feature in the training phase feature vector based on a performance loss gradient of each feature in the training phase feature vector, and formulating steps of a performance data collection scheme according to the updated weight coefficient, including: determining a gradient influence factor based on a historical change feature of the performance loss gradient, and dynamically updating the weight coefficient of each phase feature in the training phase feature vector based on the performance loss gradient and the gradient influence factor to obtain the updated weight coefficient; constructing a collection sensitivity matrix based on the updated weight coefficient, the collection sensitivity matrix representing a correlation degree between each training phase feature and a performance index; constructing a time sequence sensitivity feature based on a historical sequence of the collection sensitivity matrix, the time sequence sensitivity feature representing a change rule of the collection sensitivity matrix over time; and constructing a phase coupling feature based on a correlation relationship between different training phases; performing feature fusion on the time sequence sensitivity feature and the phase coupling feature to construct a multi-level optimization objective function; the multi-level optimization objective function including a sensitivity weighting item and a fluctuation penalty item in a main objective function, and an upper limit of resource utilization and a lower limit of sampling frequency in a constraint function; and solving the multi-level optimization objective function by using a Lagrange multiplier method with an adaptive step length, wherein a multiplier update rate is dynamically adjusted according to a constraint violation degree; and optimizing a search space based on historical performance data and task similarity, and outputting a performance data collection scheme when stability of a solution meets a preset condition.

5. The method of claim 1, wherein, the steps of determining an initial batch allocation value of a training node based on the training phase feature and the performance data, and continuously adjusting the batch allocation value based on an actual calculation time delay of the training node when the initial batch allocation value is executed until a maximum difference of single batch calculation time lengths of the training node is less than a dynamic time delay threshold, including: constructing a calculation load feature of a training node based on the training phase feature and the performance data, the calculation load feature representing a resource consumption level of each training phase; and determining an initial batch allocation value of a training node according to the calculation load feature; obtaining a single batch calculation duration of the training node when performing the initial batch allocation value, calculating a maximum difference value of the single batch calculation duration among different training nodes; constructing an adaptive batch adjustment amount based on the maximum difference value and a frequency spectrum analysis result of training node performance fluctuation, the batch adjustment amount being negatively correlated with stability of performance fluctuation frequency and being in a segmented nonlinear relationship with a degree of deviation from an average value of the single batch calculation duration; constructing a dynamic latency threshold value, the dynamic latency threshold value exponentially decaying with training time and being limited by a minimum threshold value; when the maximum difference value is greater than the dynamic latency threshold value, adjusting a current batch allocation value of the training node based on the batch adjustment amount until the maximum difference value of the single batch calculation duration is less than or equal to the dynamic latency threshold value; feeding back an execution result of the adjusted batch allocation to a performance data acquisition scheme for optimizing a performance data acquisition process in a subsequent training phase.

6. The method of claim 5, wherein, The step of constructing the dynamic latency threshold value comprises: obtaining a computing power parameter of the training node, and calculating an initial threshold value of the training node based on the computing power parameter; obtaining a current gradient variance, a current loss value and a historical adjustment frequency of the training node, and calculating a self-defined decay coefficient based on the current gradient variance, the current loss value and the historical adjustment frequency; multiplying the initial threshold value by an exponential decay function of training time to obtain the dynamic latency threshold value, wherein a decay coefficient in the exponential decay function is determined by the self-defined decay coefficient; when the dynamic latency threshold value is less than a minimum threshold limit, taking the minimum threshold limit as the dynamic latency threshold value.

7. The method of claim 1, wherein, The step of selecting, based on the updated resource requirement information, a training node satisfying the resource requirement from the service queue to execute the model training task comprises: constructing a performance index vector of the training node, the performance index vector including computing capability and historical stability; constructing an adaptive weight matrix based on the performance index vector, wherein a weight coefficient of the adaptive weight matrix is dynamically adjusted according to a calculation-intensive degree and a communication-intensive degree of the training task; multiplying the adaptive weight matrix and the performance index vector to obtain a node fitness matrix, the node fitness matrix representing an adaptation degree of different training nodes to the model training task; screening a candidate node set satisfying a basic resource condition according to the updated resource requirement information, and sorting the candidate node set based on the node fitness matrix; calculating a network topology distance among the candidate nodes based on the sorted candidate node set, constructing an initial topology affinity matrix based on the network topology distance, and weighting and fusing the initial topology affinity matrix and a historical communication efficiency matrix to obtain an optimized topology affinity matrix; calculating a node combination score based on the optimized topology affinity matrix and the updated resource requirement information; and selecting a training node combination with the highest node combination score and satisfying the resource requirement to execute the model training task.

8. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method of any one of claims 1 to 7.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 7.

Images