Method for carrying out model training based on k8s

Abstract

The invention provides a method for carrying out model training based on k8s, which relates to the technical field of model training, and comprises the following steps: creating a service queue in a k8s cluster, allocating training tasks to a corresponding queue, identifying training stage characteristics and collecting performance data by a user-defined controller, determining an initial batch allocation value according to the characteristics and the data, and carrying out model training according to the initial batch allocation value. And continuously adjusting the batch allocation value based on the actual calculation time delay, finally updating the resource demand information in the yaml file, and selecting the training node meeting the resource demand to execute the model training task. According to the invention, resource allocation can be optimized, training efficiency is improved, and resource waste is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to model training technology, in particular to a method for model training based on k8s. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, deep learning model training has become a key business for many enterprises and research institutions. Kubernetes (k8s) as an open-source container orchestration platform, due to its excellent container management, scheduling and expansion capabilities, has gradually become the infrastructure platform for model training tasks. In practical applications, model training often requires a large amount of computing resources, including CPU, GPU, memory, etc., and the training process is complex, involving data preparation, model configuration, parameter optimization and other aspects.

[0003] The current model training system based on k8s usually adopts a static resource allocation method, which requires pre-specifying resource requirements before starting the training task, and keeping it unchanged throughout the training process. This method is simple and direct, but it ignores the differences in resource requirements at different stages of the deep learning training process, resulting in low resource utilization efficiency. The main defects and deficiencies of the prior art include: The existing model training system based on k8s lacks the ability to identify and optimize the characteristics of the training phase, and cannot dynamically adjust resource allocation according to the characteristics of different stages of the training process, resulting in resource surplus in some stages and resource shortage in other stages.

[0004] The existing system usually adopts a fixed batch allocation strategy, which does not take into account the performance differences and real-time load status of different training nodes, resulting in poor performance of the nodes in a heterogeneous computing environment, which may become the bottleneck of the overall training speed, reducing the training efficiency.

[0005] The static presetting of resource requirements lacks flexibility and cannot dynamically adjust resource allocation according to actual performance data during the training process, resulting in low resource utilization, increased training cost, and difficulty in achieving efficient scheduling and resource sharing of training tasks among different business priority queues. SUMMARY

[0006] The embodiments of the present application provide a method for model training based on k8s, which can solve the problems in the prior art.

[0007] In a first aspect of the embodiments of the present application, a method for model training based on k8s is provided, comprising: creating a business queue in a k8s cluster, assigning model training tasks to corresponding business queues according to the type of the training task, generating a yaml file of the model training task, and submitting the yaml file to a custom controller in the k8s cluster. The custom controller identifies a training phase feature of the model training task, collects performance data of a training node for the training phase feature; According to the training phase feature and the performance data, an initial batch allocation value of the training node is determined, and the batch allocation value is continuously adjusted based on the actual calculation time delay of the training node when the initial batch allocation value is executed until the maximum difference of single batch calculation time length of the training node is less than a dynamic time delay threshold; the adjusted batch allocation value is written into the yaml file, and the corresponding resource requirement is calculated according to the adjusted batch allocation value, and the resource requirement information in the yaml file is updated; Based on the updated resource requirement information, a training node meeting the resource requirement is selected in the service queue to execute the model training task.

[0008] In an optional implementation, In the k8s cluster, a service queue is created, model training tasks are allocated to corresponding service queues according to the type of training task, a yaml file of the model training task is generated, and the step of submitting the yaml file to the custom controller in the k8s cluster includes: A multi-level service queue is created in the k8s cluster, and the resource quota of the multi-level service queue is configured through a resource weight matrix; a resource borrowing matrix between queues is constructed based on the resource weight matrix, and the value of each element in the resource borrowing matrix is determined by the source queue resource utilization, the target queue resource utilization and the historical borrowing success rate; Obtain the feature data of the model training task, generate a multi-dimensional feature vector based on the feature data, and combine the multi-dimensional feature vector to generate a model training task portrait vector; Calculate the cosine similarity of the model training task portrait vector and the feature vector of each service queue in the multi-level service queue, multiply the cosine similarity by the priority of the corresponding service queue to obtain the queue matching degree, allocate the model training task to the service queue with the highest matching degree according to the queue matching degree, and when the service queue with the highest matching degree is insufficient in resources, determine the borrowable queue and the borrowing amount according to the resource borrowing matrix to realize cross-queue resource allocation; Write the queue allocation information of the model training task into the yaml file and submit it to the custom controller.

[0009] In an optional implementation, The step of collecting performance data of a training node for the training phase feature includes: The training phase features include data loading phase features, model calculation phase features, and parameter updating phase features; a training phase feature vector is constructed, and the training phase feature vector contains performance indicators corresponding to each phase feature; Based on the performance loss gradient of each feature in the training phase feature vector, the weight coefficients of each phase feature in the training phase feature vector are dynamically updated, and a performance data collection scheme is formulated according to the updated weight coefficients; Based on the historical feature sequence of the training phase feature vector, a feature trend prediction value is calculated; a deviation between the current feature vector and the feature trend prediction value is taken as a feature change amplitude, the sampling window size and the sampling frequency of the performance data collection scheme are dynamically optimized according to the fluctuation rule of the feature change amplitude; and performance data collection is performed according to the optimized collection scheme. The collected performance data is subjected to multi-dimensional quality evaluation to obtain performance data quality evaluation results; and the performance data quality evaluation results are fed back to the self-defined controller for optimizing the identification process of the training phase features.

[0010] In an optional implementation, The step of dynamically updating the weight coefficients of each phase feature in the training phase feature vector based on the performance loss gradient of each feature in the training phase feature vector and formulating a performance data collection scheme according to the updated weight coefficients comprises: A gradient influence factor is determined based on the historical change characteristics of the performance loss gradient, the weight coefficients of each phase feature in the training phase feature vector are dynamically updated based on the performance loss gradient and the gradient influence factor, and the updated weight coefficients are obtained; A collection sensitivity matrix is constructed based on the updated weight coefficients, the collection sensitivity matrix represents the correlation degree between each training phase feature and a performance indicator; a time sequence sensitivity feature is constructed based on the historical sequence of the collection sensitivity matrix, the time sequence sensitivity feature represents the change rule of the collection sensitivity matrix over time; and a phase coupling feature is constructed based on the correlation between different training phases in the collection sensitivity matrix; The time sequence sensitivity feature and the phase coupling feature are fused to construct a multi-level optimization objective function; the multi-level optimization objective function includes a sensitivity weighting item and a fluctuation penalty item in a main objective function, and an upper limit of resource utilization and a lower limit of sampling frequency in a constraint function; a Lagrange multiplier method with an adaptive step size is used to solve the multi-level optimization objective function, wherein the multiplier update rate is dynamically adjusted according to the constraint violation degree; a search space is optimized based on historical performance data and task similarity, and a performance data collection scheme is output when the stability of the solution meets a preset condition.

[0011] In an optional implementation, The step of determining an initial batch assignment value of the training node according to the training phase features and the performance data, and continuously adjusting the batch assignment value based on the actual computation latency of the training node when executing the initial batch assignment value until the maximum difference of single batch computation time length of the training node is less than a dynamic latency threshold value comprises: constructing a computation load feature of the training node based on the training phase features and the performance data, the computation load feature representing the resource consumption level of each training phase; determining an initial batch assignment value of the training node according to the computation load feature; obtaining the single batch computation time length of the training node when executing the initial batch assignment value, calculating the maximum difference of single batch computation time length among different training nodes; constructing an adaptive batch adjustment amount based on the maximum difference and the frequency spectrum analysis result of the performance fluctuation of the training node, the batch adjustment amount being negatively correlated with the stability of the performance fluctuation frequency and being in a segmented nonlinear relationship with the degree of deviation from the average value of the single batch computation time length; constructing a dynamic latency threshold value, the dynamic latency threshold value exponentially decaying with the training time and being limited by a minimum threshold value; when the maximum difference is greater than the dynamic latency threshold value, adjusting the current batch assignment value of the training node based on the batch adjustment amount until the maximum difference of single batch computation time length is less than or equal to the dynamic latency threshold value; feeding the adjusted batch assignment execution result back to the performance data collection scheme for optimizing the performance data collection process of the subsequent training phase.

[0012] In an optional implementation, The step of constructing a dynamic latency threshold value comprises: obtaining a computation power parameter of the training node, calculating an initial threshold value of the training node based on the computation power parameter; obtaining the current gradient variance, the current loss value and the historical adjustment frequency of the training node, calculating a self-defined decay coefficient based on the current gradient variance, the current loss value and the historical adjustment frequency; multiplying the initial threshold value by an exponential decay function of the training time to obtain a dynamic latency threshold value, the decay coefficient in the exponential decay function being determined by the self-defined decay coefficient; when the dynamic latency threshold value is less than a minimum threshold value limit, taking the minimum threshold value limit as the dynamic latency threshold value.

[0013] In an optional implementation, The step of selecting a training node satisfying the resource requirement in the service queue to execute the model training task based on the updated resource requirement information comprises: construct a performance index vector of the training nodes, the performance index vector including a computing capability and a historical stability; construct an adaptive weight matrix based on the performance index vector, a weight coefficient of the adaptive weight matrix being dynamically adjusted according to a computation intensity and a communication intensity of a training task; multiply the adaptive weight matrix by the performance index vector to obtain a node adaptation degree matrix, the node adaptation degree matrix representing an adaptation degree of different training nodes to the model training task; filter a candidate node set meeting a basic resource condition according to the updated resource requirement information, and sort the candidate node set based on the node adaptation degree matrix; based on the sorted candidate node set, calculate a network topology distance between candidate nodes, construct an initial topology affinity matrix based on the network topology distance, and obtain an optimized topology affinity matrix by weighted fusion of the initial topology affinity matrix and a historical communication efficiency matrix; calculate a node combination score based on the optimized topology affinity matrix and the updated resource requirement information; and select a training node combination with the highest node combination score and meeting the resource requirement to execute the model training task.

[0014] In a second aspect of the embodiment of the present application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0015] In a third aspect of the embodiment of the present application, a computer-readable storage medium is provided, which stores computer program instructions, the computer program instructions being executed by a processor to implement the method described above.

[0016] The present application has the following beneficial effects: By creating a business queue in the k8s cluster and assigning model training tasks according to the training task type, the present application realizes the ordered management and resource allocation of different training tasks, and improves the resource utilization efficiency and the standardization of task processing.

[0017] Based on the recognition of training phase features and the collection of performance data by the custom controller, the present application can accurately grasp the changes in computing requirements in the model training process, providing data support for subsequent batch allocation, and making the resource allocation more intelligent and accurate.

[0018] By dynamically adjusting the batch allocation value until the delay threshold requirement is met, and updating the resource requirement information accordingly, the present application realizes the load balancing between training nodes, effectively solves the problems of waste of computing resources and low training efficiency caused by traditional fixed batch allocation, and significantly improves the overall performance of model training and resource utilization. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of a method for model training based on k8s according to an embodiment of the present application; Figure 2 A flowchart of a process for dynamic adjustment and optimization of training nodes. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0021] The technical solutions of the present application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and some embodiments can not be described again for the same or similar concepts or processes.

[0022] Figure 1 A flowchart of a method for model training based on k8s according to an embodiment of the present application is shown in FIG. 1, which includes the following steps. Figure 1 A business queue is created in a k8s cluster, model training tasks are assigned to corresponding business queues according to training task types, a yaml file of the model training task is generated, and the yaml file is submitted to a custom controller in the k8s cluster. The custom controller identifies the training phase characteristics of the model training task, and collects performance data of training nodes for the training phase characteristics. An initial batch allocation value of the training nodes is determined according to the training phase characteristics and the performance data, the batch allocation value is continuously adjusted based on the actual calculation time delay of the training nodes when executing the initial batch allocation value, until the maximum difference of single batch calculation time of the training nodes is less than a dynamic time delay threshold, the adjusted batch allocation value is written into the yaml file, the corresponding resource requirement is calculated according to the adjusted batch allocation value, and the resource requirement information in the yaml file is updated. Based on the updated resource requirement information, a training node meeting the resource requirement is selected from the business queue to execute the model training task.

[0023] ​This implementation details the complete technical process from creating business queues to executing training tasks. Adaptive resource scheduling based on k8s clusters allocates training tasks to appropriate computing resources and dynamically adjusts batch allocation according to task characteristics, achieving efficient utilization of computing resources.

[0024] When creating business queues in a k8s cluster, the queues are divided into multiple categories based on the computational characteristics of training tasks, such as image recognition queues, natural language processing queues, and recommendation queues. During the creation of the queue, the k8s custom resource definition (CRD) mechanism is used to define the resource type of the business queue, including queue name, resource upper limit, priority, and reserved resource proportion. For example, for an image recognition queue, the definition includes: queue name "image-recognition-queue", resource upper limit is 40% of the total cluster resources, priority is 8 (value range 0-10), and reserved resource proportion is 15%. These custom resource instances are created through the k8s API, and the corresponding queue controller is started to monitor the queue state. The queue controller is responsible for maintaining resource usage statistics, task queuing order, and resource allocation strategies within the queue.

[0025] When assigning model training tasks to corresponding business queues based on training task types, analyze the model architecture, dataset characteristics, and training parameters of the training task. For example, for an image classification task based on a convolutional neural network, the characteristics include: model type "CNN", parameter quantity 85 million, training dataset size 120GB, training batch size 64, and learning rate 0.001. According to the preset task classification rules, it is determined that the task should be assigned to the "image-recognition-queue" queue. The classification rules use a decision tree structure, first dividing by model type, then subdividing by parameter quantity, data size, and other indicators. A standard yaml file is generated for each task, containing task metadata, resource requirements, training configuration, and other information. The yaml file structure includes: apiVersion, kind, metadata (name, namespace, labels, etc.), and spec (model configuration, training parameters, resource requests, etc.). For example, the yaml file of a certain CNN training task contains: task name "cnn-training-task-0042", namespace "deep-learning", labels include {type:"image",priority:"high"}, initial resource request {cpu:"16",memory:"64Gi",gpu:"4"}, training parameters include {epochs:50,batch_size:64,learning_rate:0.001}, and other information.

[0026] After submitting the yaml file to the custom controller in the k8s cluster, the controller first verifies the format and content of the yaml file to ensure that all required fields are correctly filled out. The custom controller is implemented based on the Operator pattern of k8s, which extends the native functions of k8s by registering CRD and corresponding controller logic. The controller listens to the CRD resource change events related to training tasks and triggers the processing flow when it receives a newly submitted yaml file. The core components of the controller include: resource listener, state manager, scheduler, and executor. The resource listener is responsible for listening to changes in CRD resources; the state manager maintains the lifecycle state of the task; the scheduler is responsible for resource allocation decisions; and the executor is responsible for starting the task container on the selected node.

[0027] The custom controller identifies the characteristics of the training phase of the training task, which is the key to implementing fine-grained resource management. The controller analyzes the model structure, parameter quantity, optimizer type, and other information in the yaml file, and combines historical task data to identify the characteristics of the task in different training phases. The training phase usually includes: data loading phase, forward propagation phase, backward propagation phase, and parameter update phase. For each phase, identify its resource-sensitive characteristics, such as CPU-intensive, memory-intensive, computation-intensive, or IO-intensive. For example, for the aforementioned CNN training task, the controller identifies that: the data loading phase is IO-intensive (IO sensitivity 0.85), the forward propagation phase is computation-intensive (computation sensitivity 0.92), the backward propagation phase is a mix of computation and memory (computation sensitivity 0.88, memory sensitivity 0.75), and the parameter update phase is communication-intensive (communication sensitivity 0.79). These characteristic values are calculated by a pre-trained feature extraction model, which is trained based on historical task data and can infer the resource-sensitive characteristics of each phase from the model structure and training parameters.

[0028] When collecting performance data of training nodes according to the characteristics of the training phase, the controller deploys a lightweight monitoring component to the candidate training nodes. The monitoring component collects node CPU usage, memory occupancy, GPU utilization, IO rate, network traffic, and other indicators. The sampling frequency is dynamically adjusted according to the sensitivity characteristics of different phases, and the sampling frequency of sensitive indicators is increased. For example, in the data loading phase, the sampling frequency of IO-related indicators is 2 times per second, and the sampling frequency of other indicators is 1 time per 5 seconds; in the computation-intensive forward propagation phase, the sampling frequency of GPU utilization is increased to 3 times per second. The raw data collected is preprocessed, including denoising, standardization, and feature extraction, to form a structured performance data set. For example, the performance data of a node in the forward propagation phase includes: average GPU utilization 92%, GPU memory occupancy 85%, PCIe bandwidth usage 78%, and average time to process a single batch of data 210 milliseconds.

[0029] The performance prediction model takes node performance data and training phase features as input, and outputs the expected time for the node to process different batch sizes. The prediction model is trained based on historical task data using the gradient boosting tree algorithm, which can handle heterogeneous features and capture nonlinear relationships. For distributed training tasks, appropriate batch sizes need to be allocated to each participating node. The initial batch allocation is based on the relative proportion of node performance. For example, for a task with a total batch size of 128, if there are two training nodes A and B with a performance ratio of 3:2, the initial allocation is 76 samples for node A and 52 samples for node B.

[0030] The actual time for each node to process the allocated batch is monitored, and the single batch calculation time (total time divided by batch number) is calculated, and the maximum difference between different nodes is found. For example, node A takes 380 milliseconds to process 76 samples, and the single batch time is 5 milliseconds; node B takes 312 milliseconds to process 52 samples, and the single batch time is 6 milliseconds. The maximum difference is 1 millisecond. A dynamic latency threshold is constructed, which adjusts as the training progresses. The initial threshold is usually set to 10% of the expected single batch time, and decreases as the number of training iterations increases, but does not fall below the minimum threshold (usually 20% of the initial threshold). When the maximum difference exceeds the dynamic threshold, adjust the batch allocation, transfer part of the batch from the slower node to the faster node. The adjustment amount is related to the difference size and node stability. The adjustment process continues until the maximum difference in single batch calculation time is less than the dynamic threshold, or the maximum number of adjustments is reached.

[0031] Write the adjusted batch allocation value to the yaml file, and calculate the corresponding resource requirements based on the adjusted batch allocation value, and update the resource requirement information in the yaml file. Through the resource prediction model, estimate the required CPU cores, memory capacity and GPU number based on the batch size and model characteristics. For example, for the configuration of node A processing 80 samples and node B processing 48 samples after adjustment, the corresponding resource requirements are updated as follows: node A needs {cpu:"18", memory:"72Gi", gpu:"4"}, node B needs {cpu:"14", memory:"56Gi", gpu:"3"}. Use k8sAPI to update the resource specification part in the yaml file to provide a basis for subsequent resource allocation.

[0032] Based on the updated resource requirement information, a training node that meets the resource requirement is selected from the service queue to perform the model training task. The available training node resource pool in the queue is queried, and a node set that meets the updated resource requirement is screened out. The screening criteria include hardware specification matching degree, current load level and historical performance stability. The matching score of the computing node and the task is calculated, considering the resource satisfaction, network topology relationship and historical execution efficiency. The node combination with the highest score is selected, and the task is allocated to these nodes for execution through the k8s scheduler. For example, nodes node-gpu-12 and node-gpu-15 are selected from the "image-recognition-queue" queue to perform the aforementioned CNN training task. The resource configurations of the two nodes meet the requirements of nodes A and B, respectively, and have a high matching score.

[0033] In an optional embodiment, a service queue is created in the k8s cluster, the model training task is allocated to the corresponding service queue according to the type of the training task, a yaml file of the model training task is generated, and the step of submitting the yaml file to the custom controller in the k8s cluster includes: A multi-level service queue is created in the k8s cluster, and the resource quota of the multi-level service queue is configured through a resource weight matrix. A resource borrowing matrix between queues is constructed based on the resource weight matrix, and the value of each element in the resource borrowing matrix is determined by the source queue resource utilization, the target queue resource utilization and the historical borrowing success rate. Feature data of the model training task is obtained, and a multi-dimensional feature vector is generated based on the feature data. The multi-dimensional feature vector is combined to generate a model training task portrait vector. The cosine similarity of the model training task portrait vector and the feature vector of each service queue in the multi-level service queue is calculated, the cosine similarity is multiplied by the priority of the corresponding service queue to obtain a queue matching degree, the model training task is allocated to the service queue with the highest matching degree according to the queue matching degree, and when the service queue with the highest matching degree is insufficient in resources, the borrowable queue and the borrowing amount are determined according to the resource borrowing matrix to realize cross-queue resource allocation. The queue allocation information of the model training task is written into a yaml file and submitted to the custom controller.

[0034] For example, when creating a multi-level business queue in a Kubernetes cluster, it can be implemented through custom resource definition (CRD). The business queue is divided into three levels: high priority, medium priority, and low priority, and each level can contain multiple specific business queues. The resource quota of these queues is configured through a resource weight matrix. The resource weight matrix is a two-dimensional array, with rows representing different business queues and columns representing resource types such as CPU, memory, and GPU. For example, for a machine learning inference task queue, the CPU weight can be set to 0.6, the memory weight to 0.3, and the GPU weight to 0.1; while for a large-scale training task queue, the CPU weight can be set to 0.2, the memory weight to 0.2, and the GPU weight to 0.6. According to these weight values and the total resource amount of the cluster, the actual resource quota of each queue is calculated.

[0035] Based on the resource weight matrix, a resource borrowing matrix between queues is constructed. The resource borrowing matrix is an N x N matrix, where N is the number of queues, and each element B[i][j] in the matrix represents the feasibility score of borrowing resources from queue i to queue j. This score is determined by the source queue resource utilization, the target queue resource utilization, and the historical borrowing success rate. The source queue resource utilization is calculated by the percentage of allocated resources to the total quota (e.g., CPU core usage / total core number), the target queue resource utilization is calculated by the percentage of currently used resources to its total quota, and the historical borrowing success rate is calculated by the number of successful borrowing times in the past 24 hours divided by the total borrowing request times. Specifically, when the resource utilization of source queue i is lower than a threshold (e.g., 30%), the resource utilization of target queue j is higher than a threshold (e.g., 80%), and the historical borrowing success rate is greater than a certain value (e.g., 60%), the value of B[i][j] will be higher, indicating that the feasibility of borrowing resources from queue i to queue j is higher. A sliding window is maintained to record the borrowing situation in the past 24 hours, which is used to calculate the historical borrowing success rate.

[0036] When a new model training task is submitted to the system, the first step is to obtain the feature data of the task. The feature data includes the model type (such as CNN, RNN, Transformer), estimated training duration, dataset size, batch size, computational complexity, and so on. Based on these feature data, a multi-dimensional feature vector is generated. For example, for an image classification task, the feature vector contains model type features (CNN = 1, others = 0), dataset size features (medium = 0.5), computational complexity features (high = 0.8), and other dimensions. The multi-dimensional feature vector is combined by weighting, generating a model training task portrait vector. The weighting process considers the importance of different features, such as the weight of model type is 0.4, the weight of dataset size is 0.3, and the weight of computational complexity is 0.3. After weighting and combining, a unified task portrait vector is formed, which fully represents the characteristics of the training task.

[0037] The cosine similarity between the model training task portrait vector and the feature vector of each business queue in the multi-level business queue is calculated. Each business queue has a pre-defined feature vector representing the type of tasks that the queue is suitable for handling. For example, the feature vector of a GPU-intensive task queue has a high value in the GPU resource demand dimension. The calculated cosine similarity reflects the matching degree between the task and the queue, with a value range of -1 to 1, and the closer to 1 indicates the higher matching. Multiply this cosine similarity with the priority weight of the corresponding business queue to get the final queue matching degree. For example, if the cosine similarity with the high-priority queue is 0.8 and the priority weight of this queue is 1.2, the final matching degree is 0.96.

[0038] According to the queue matching degree, the model training task is assigned to the business queue with the highest matching degree. If the business queue has enough resources, the task is created directly in the queue; if the resources are insufficient, the resource borrowing mechanism is triggered. Query the resource borrowing matrix to find out the available queues and borrowing limits. For example, if the current queue needs 4 GPUs but only has 2 available, it will find the most suitable queue to borrow resources according to the resource borrowing matrix, and if the queue has 2 idle GPUs and the borrowing conditions are met, it will temporarily allocate the 2 GPUs to the current queue. This cross-queue resource allocation mechanism significantly improves resource utilization.

[0039] After determining the queue allocation and resource borrowing scheme, a YAML file for the model training task is generated. The YAML file contains information such as queue identification, resource requirements, container image, mounted volume, environment variables, etc. This YAML file is submitted to the custom controller in the Kubernetes cluster. The custom controller parses the YAML file and creates the corresponding Pod according to the queue information and resource conditions, and monitors the task execution status. When resource borrowing occurs, the controller will record the borrowing situation and return the resources after the task is completed.

[0040] To ensure the accuracy of the resource borrowing matrix, the values in the matrix will be updated regularly. The update frequency can be set to once an hour, and the matrix element values will be adjusted according to the resource usage and borrowing success in the recent period of time. In addition, the queue feature vector will also be fine-tuned according to the historical task execution situation, so that the matching between the queue and the task is more accurate.

[0041] The present application realizes intelligent allocation of model training tasks and dynamic allocation of resources by establishing a multi-level service queue and a resource borrowing mechanism, improves the utilization rate of cluster resources and the efficiency of task processing, reduces the queuing waiting time of training tasks, enhances the execution stability of model training tasks, and provides a more efficient and flexible scheduling strategy for AI model training.

[0042] In an optional embodiment, the step of collecting performance data of the training phase feature collection training node includes: The training phase features include data loading phase features, model calculation phase features, and parameter updating phase features. A training phase feature vector is constructed, which contains performance indicators corresponding to each phase feature. Based on the performance loss gradient of each feature in the training phase feature vector, the weight coefficients of each phase feature in the training phase feature vector are dynamically updated, and a performance data collection scheme is formulated according to the updated weight coefficients; Based on the historical feature sequence of the training phase feature vector, a feature trend prediction value is calculated. The deviation of the current feature vector from the feature trend prediction value is taken as the feature change amplitude, and the sampling window size and sampling frequency of the performance data collection scheme are dynamically optimized according to the fluctuation rule of the feature change amplitude. The performance data collection is performed according to the optimized collection scheme. The collected performance data is subjected to multi-dimensional quality evaluation to obtain performance data quality evaluation results. The performance data quality evaluation results are fed back to the self-defined controller for optimizing the identification process of the training phase features.

[0043] Exemplarily, the training stage features mainly include three core parts: data loading stage features, model computing stage features, and parameter updating stage features. The data loading stage features include data reading rate, data preprocessing time, data augmentation time consumption, data queue length, and other indicators, reflecting the performance of the data preparation link in the training process. For example, in the image classification task, the data reading rate can be monitored as 1000 images per second, and the average preprocessing time is 0.2 seconds per batch. The model computing stage features include forward propagation time, computation density, memory access pattern, computation peak utilization rate, and other indicators, reflecting the execution efficiency of the model on the computing hardware. For example, for a convolutional neural network, the forward propagation time can be monitored as 0.5 seconds per batch, and the GPU computation utilization rate is 78%. The parameter updating stage features include backpropagation time, gradient computation time, parameter update time, communication overhead, and other indicators, reflecting the performance of the model weight update process. For example, the backpropagation time can be monitored as 0.7 seconds per batch, and the parameter update time is 0.1 seconds per batch.

[0044] Based on the feature indicators of the above three stages, a training stage feature vector is constructed. The feature vector is a multi-dimensional array containing the performance indicator values corresponding to the features of each stage. For example, a training stage feature vector can be represented as: [data reading rate = 1000 images / second, preprocessing time = 0.2 seconds / batch, forward propagation time = 0.5 seconds / batch, GPU utilization rate = 78%, backpropagation time = 0.7 seconds / batch, parameter update time = 0.1 seconds / batch]. These indicators are obtained by the monitoring agent deployed on the training node, and the collection frequency is initially set to once every 5 seconds.

[0045] The performance loss gradient of each feature in the training stage feature vector represents the degree of influence of the change of a certain performance indicator on the overall training speed. The calculation method is to record the ratio of the change amount of the performance indicator to the change amount of the training speed within a certain period of time. For example, if the data reading rate decreases by 10% and the overall training speed decreases by 8%, the performance loss gradient of the data reading rate is 0.8. The performance loss gradient of each performance indicator is calculated, and the higher the value, the greater the influence of the indicator on the training performance. According to the calculated performance loss gradient, the weight coefficients of each stage feature are adjusted. For example, when the performance loss gradient of the data loading stage is 0.8, the performance loss gradient of the model computing stage is 0.5, and the performance loss gradient of the parameter updating stage is 0.3, the weight coefficient of the data loading stage feature is increased to 0.5, and the weight coefficients of the other stages are appropriately reduced. The updated weight coefficients are used to develop a performance data collection scheme, focusing on monitoring the high-weight stages.

[0046] The feature trend prediction value is calculated based on the historical feature sequence of the feature vector in the training phase. The specific method is to use a sliding window to collect the feature vector sequence in the last 30 minutes, and predict the feature value in the next 10 minutes through time series analysis method. For example, if it is observed that the GPU utilization rate presents a fluctuating upward trend in the past 30 minutes, gradually rising from 65% to 78%, it is predicted that the GPU utilization rate will reach 85% in the next 10 minutes. The deviation of the current feature vector from the feature trend prediction value is calculated as the feature change amplitude. For example, if the current GPU utilization rate is 78% and the predicted value is 85%, the change amplitude is 7%. According to the fluctuation rule of the feature change amplitude, the sampling window size and sampling frequency of the performance data collection scheme are dynamically optimized. When the feature change amplitude is large (such as more than 10%) and fluctuates frequently, the sampling window is reduced to 2 seconds and the sampling frequency is increased to once per second; when the feature change amplitude is small (such as less than 5%) and relatively stable, the sampling window is expanded to 10 seconds and the sampling frequency is reduced to once every 10 seconds, to reduce the monitoring overhead. According to the optimized collection scheme, performance data collection is performed, and different collection strategies are adopted for different phase features.

[0047] The collected performance data is subjected to multi-dimensional quality evaluation, including integrity evaluation, consistency evaluation, timeliness evaluation and noise level evaluation. The integrity evaluation checks whether there are missing values in the data and calculates the data integrity rate. For example, if 985 out of 1000 data points collected are valid, the integrity rate is 98.5%. The consistency evaluation checks whether the data is consistent with the historical trend and calculates the degree of data outliers. For example, if the current GPU utilization rate is 20% and the historical average at the same stage is 75%, the data will be marked as an abnormal point. The timeliness evaluation checks the data collection delay to ensure that the data can timely reflect the training state. For example, if the data collection presents a time lag of more than 2 seconds compared with the actual occurrence time, the credibility of the data will be reduced. The noise level evaluation checks the fluctuation degree of the data and filters short-term noise. For example, if the GPU utilization rate frequently fluctuates between 75%-76% in a short time, the small fluctuations will be smoothed. Based on the above evaluation dimensions, a comprehensive performance data quality score is given, ranging from 0 to 100 points. When the quality score is less than 60 points, it indicates that there is a problem with the current collection scheme, and the collection parameters need to be adjusted.

[0048] The performance data quality evaluation results are fed back to the custom controller for optimizing the recognition process of the training phase features. The custom controller adjusts the parameters of the feature extraction algorithm according to the quality evaluation results. For example, when the data integrity rate is less than 90%, the controller will reduce the feature dimension and focus on the core indicators that are easier to collect; when the data noise level is high, the controller will increase the window size of the smoothing processing. Through this feedback mechanism, the recognition accuracy of the training phase features is continuously optimized, and the pertinence and effectiveness of the subsequent performance data collection are improved.

[0049] The application realizes fine monitoring and optimization of the model training process by dynamically sensing the training phase features and collecting corresponding performance data, effectively improves the deep learning training efficiency, reduces the calculation resource consumption, shortens the model training time, and can timely find the performance bottleneck in the training process, providing accurate basis for resource allocation and scheduling of the training task.

[0050] In an optional embodiment, based on the performance loss gradient of each feature in the training phase feature vector, the weight coefficients of each phase feature in the training phase feature vector are dynamically updated, and the steps of formulating a performance data collection scheme according to the updated weight coefficients include: determining a gradient influence factor based on the historical change characteristics of the performance loss gradient, dynamically updating the weight coefficients of each phase feature in the training phase feature vector based on the performance loss gradient and the gradient influence factor, and obtaining updated weight coefficients; based on the updated weight coefficients, constructing a collection sensitivity matrix, the collection sensitivity matrix representing the correlation degree between each training phase feature and the performance index; based on the historical sequence of the collection sensitivity matrix, constructing a time sequence sensitivity feature, the time sequence sensitivity feature representing the change law of the collection sensitivity matrix with time; based on the correlation relationship between different training phases in the collection sensitivity matrix, constructing a phase coupling feature; performing feature fusion on the time sequence sensitivity feature and the phase coupling feature to construct a multi-level optimization objective function; the multi-level optimization objective function includes a sensitivity weighting item and a fluctuation penalty item in the main objective function, and a resource utilization rate upper limit and a sampling frequency lower limit in the constraint function; the multi-level optimization objective function is solved by using a Lagrange multiplier method with an adaptive step length, wherein the multiplier update rate is dynamically adjusted according to the constraint violation degree; based on the historical performance data and the task similarity, the search space is optimized, and when the stability of the solution meets the preset condition, the performance data collection scheme is output.

[0051] For example, the performance loss gradient data of each hour in the past 24 hours can be collected to form a time series. The performance loss gradient represents the degree of influence of a certain feature change on the overall training performance. For example, if a 10% decrease in data loading speed leads to a 7% decrease in overall training speed, the performance loss gradient is 0.7. By analyzing the trend of these gradients, the fluctuation amplitude and directionality can be calculated. If the gradient value continues to rise and the fluctuation is small, it indicates that the feature has a more stable and important influence on the performance, and the gradient influence factor can be set to a higher value, such as 0.9. If the gradient value fluctuates greatly, it indicates that the feature has an unstable influence on the performance, and the gradient influence factor can be set to a lower value, such as 0.5. In a certain training task, the performance loss gradient of the data loading stage is 0.7 and relatively stable, and the gradient influence factor is calculated as 0.85. The performance loss gradient of the parameter update stage is 0.4 and fluctuates greatly, and the gradient influence factor is calculated as 0.6.

[0052] The weight coefficients of each stage feature in the training stage feature vector are dynamically updated according to the performance loss gradient and the gradient influence factor. The updating method uses a weighted average method, that is, the new weight is equal to the old weight multiplied by a decay factor (such as 0.7) plus the performance loss gradient multiplied by the gradient influence factor and then multiplied by a gain factor (such as 0.3). For the data loading stage, if the original weight is 0.4, the updated weight is 0.4 x 0.7 + 0.7 x 0.85 x 0.3 ≈ 0.46. For the model calculation stage, if the original weight is 0.35, the performance loss gradient is 0.6, and the gradient influence factor is 0.8, the updated weight is 0.35 x 0.7 + 0.6 x 0.8 x 0.3 ≈ 0.39. For the parameter update stage, if the original weight is 0.25, the updated weight is 0.25 x 0.7 + 0.4 x 0.6 x 0.3 ≈ 0.25. After the weight is updated, normalization processing is required to ensure that the sum of all weights is 1. The normalized weights are: data loading stage 0.42, model calculation stage 0.35, and parameter update stage 0.23.

[0053] Based on the updated weight coefficients, a collection sensitivity matrix is constructed. This matrix is a two-dimensional array, with rows representing different training stages (data loading, model calculation, parameter update) and columns representing different performance indicators (CPU usage, memory usage, GPU utilization, network throughput, etc.). The value of each element in the matrix represents the sensitivity of a specific training stage to a specific performance indicator, which is determined by the weight of that stage and the correlation of that performance indicator. For example, the sensitivity of the data loading stage to disk I / O is 0.85, and the sensitivity to GPU utilization is 0.2. The sensitivity of the model calculation stage to GPU utilization is 0.9, and the sensitivity to network throughput is 0.15. The sensitivity of the parameter update stage to network throughput is 0.75, and the sensitivity to memory usage is 0.6.

[0054] The time-series sensitivity features are constructed based on the historical sequence of the collection sensitivity matrix. The collection sensitivity matrix of each hour in the past 12 hours is saved, and the trend of the matrix elements over time is analyzed. The time-series sensitivity features include three aspects: change rate, periodicity, and trend direction. The change rate is obtained by calculating the sensitivity difference between consecutive time points, such as the sensitivity of disk I / O in the data loading stage increased from 0.78 to 0.85 in the past 4 hours, with a change rate of 0.0175 / hour. The periodicity is identified by analyzing the repeating patterns of sensitivity values, such as the sensitivity of GPU utilization in the model computation stage appears a peak every 4 hours. The trend direction is determined by the slope sign of linear regression, positive value indicates upward trend, negative value indicates downward trend.

[0055] The stage coupling features are constructed based on the correlation between different training stages in the collection sensitivity matrix. Specifically, the correlation coefficient of sensitivity values between different training stages is calculated to form a stage coupling matrix. For example, the coupling degree between the data loading stage and the model computation stage is 0.4, indicating a moderate correlation; the coupling degree between the model computation stage and the parameter update stage is 0.7, indicating a higher correlation. In addition, it is analyzed whether there is a lead-lag relationship between the time-series changes of different stages, such as the sensitivity change of the data loading stage often leads the model computation stage by 2 hours, this relationship is recorded as the time-dependent feature between stages.

[0056] The time-sensitive features and stage coupling features are fused as the basis for constructing the optimization objective function. The feature fusion adopts a weighted combination method, with the time-sensitive feature weight being 0.6 and the coupling feature weight being 0.4. Taking the sensitivity of the data loading stage to the disk I / O as an example, if its time-sensitive sensitivity is 0.85 and the stage coupling feature value is 0.3, the fused feature value is 0.85*0.6+0.3*0.4=0.63. The fused feature values are calculated for all training stages and performance indicators to form a fused feature matrix. Based on the fused feature matrix, a multi-level optimization objective function is constructed. The function is composed of a main objective function and a constraint function. The main objective function includes a sensitivity weighted item and a fluctuation penalty item. The sensitivity weighted item uses the fused feature value as the weight coefficient, and the sampling frequency of each performance indicator is multiplied and summed. For example, if the fused sensitivity of GPU utilization is 0.8 and the sampling frequency is 0.5 times per second, its contribution to the sensitivity weighted item is 0.8*0.5=0.4. This design allows high-sensitivity indicators to obtain more sampling resources. The fluctuation penalty item is the square sum of the sampling frequency variation amplitude, aiming to reduce the drastic fluctuations of the sampling scheme. The main objective function combines these two items through a subtraction relationship: main objective function=sensitivity weighted item-fluctuation penalty item*penalty coefficient (usually 0.3). The constraint function includes the resource utilization upper limit and the sampling frequency lower limit. The resource utilization upper limit ensures that the acquisition process does not excessively consume resources, and is represented as the weighted sum of all indicator sampling frequencies not exceeding the threshold (such as 5%). The sampling frequency lower limit ensures that key indicators have sufficient sampling points, such as the GPU utilization sampling frequency not being less than once per second. This structure balances the comprehensiveness of acquisition, resource consumption, and stability, providing a clear mathematical framework for subsequent solution.

[0057] The multi-level optimization objective function is solved by using the Lagrange multiplier method with adaptive step size. The sampling frequency of each performance indicator is initialized to a medium value (such as once every 5 seconds), and the initial value of the Lagrange multiplier is set to 0.5. In each iteration, the sampling frequency is updated according to the gradient of the objective function, and the step size is dynamically adjusted according to the historical iteration effect. When the improvement effect of consecutive iterations is small, the step size is reduced to improve accuracy; when the objective function value is still rapidly improving, the larger step size is maintained to speed up convergence. The multiplier update rate is dynamically adjusted according to the constraint violation degree, and the more serious the constraint violation, the larger the update rate. For example, when the sampling scheme causes the CPU usage to reach 5.5%, exceeding the upper limit by 0.5%, the corresponding multiplier update rate will be set to 0.2; when the sampling frequency is only 0.1 times per second higher than the lower limit, the corresponding multiplier update rate will be set to 0.05.

[0058] Retrieving a historical task similar to the current task from the historical task library (similarity threshold set to 0.8), analyzing its optimal sampling scheme as the initial point and reference range of the current search. For example, if the optimal GPU sampling frequency of a similar image classification task in history is concentrated in every 2-3 seconds, the search range will be narrowed to the vicinity of this interval. The stability of the solution is measured by the variance of the results of consecutive iterations. When the variance of the sampling scheme of 10 consecutive iterations is less than a preset threshold (such as 0.01), it is considered that the solution process converges, and the final performance data collection scheme is output. The final output of the collection scheme includes the sampling frequency, sampling duration and trigger condition of each performance indicator, such as GPU utilization sampling every 2 seconds, lasting 30 minutes; network throughput sampling every 1 second during parameter update phase, lasting 5 minutes.

[0059] The present application dynamically updates the training phase feature weight based on the performance loss gradient and optimizes the performance data collection scheme, realizes the accurate monitoring of the deep learning training process, improves the pertinence and resource utilization efficiency of performance data collection, provides high-quality data support for training process optimization, reduces the monitoring overhead, enables more accurate identification of performance bottlenecks and takes corresponding optimization measures.

[0060] In an optional implementation, the step of determining an initial batch allocation value of the training node according to the training phase feature and the performance data, and continuously adjusting the batch allocation value based on the actual calculation time delay of the training node when executing the initial batch allocation value until the maximum difference value of the single batch calculation time length of the training node is less than a dynamic time delay threshold value comprises: constructing a calculation load feature of the training node based on the training phase feature and the performance data, the calculation load feature representing the resource consumption level of each training phase; determining an initial batch allocation value of the training node according to the calculation load feature; obtaining the single batch calculation time length of the training node when executing the initial batch allocation value, calculating the maximum difference value of the single batch calculation time length among different training nodes; constructing an adaptive batch adjustment amount based on the maximum difference value and the frequency spectrum analysis result of the performance fluctuation of the training node, the batch adjustment amount being negatively correlated with the stability of the performance fluctuation frequency and being in a segmented nonlinear relationship with the degree of deviation from the average value of the single batch calculation time length; constructing a dynamic time delay threshold value, the dynamic time delay threshold value exponentially decaying with the training time and being limited by a minimum threshold value; when the maximum difference value is greater than the dynamic time delay threshold value, adjusting the current batch allocation value of the training node based on the batch adjustment amount until the maximum difference value of the single batch calculation time length is less than or equal to the dynamic time delay threshold value; feeding the adjusted batch allocation execution result back to the performance data collection scheme for optimizing the performance data collection process of the subsequent training phase.

[0061] Combining Figure 2 The training node batch dynamic adjustment and optimization flow chart is described. For example, based on the training phase features and performance data, the computing load features of the training nodes are constructed, and the resource consumption data of each training node in the data loading, forward propagation, backward propagation and parameter updating stages is collected. The computing load feature is a multi-dimensional vector, which includes CPU usage, memory occupation, GPU utilization, memory bandwidth usage and other indicators in each training stage. For example, for node A, the computing load feature is: [data loading stage CPU usage = 45%, forward propagation stage GPU utilization = 85%, backward propagation stage GPU utilization = 92%, parameter updating stage network bandwidth usage = 60%]. The original data is normalized to map the values of each indicator to the range of 0 to 1, which is convenient for subsequent calculation. For the case of multiple training nodes, an independent computing load feature vector is constructed for each node.

[0062] The comprehensive computing capability score of each node is calculated, considering multiple factors: GPU computing performance weight 0.5, memory bandwidth weight 0.3, CPU processing capability weight 0.2. For example, if the GPU performance score of node A is 0.8, the memory bandwidth score is 0.7, and the CPU score is 0.6, then the comprehensive computing capability score is 0.8x0.5+0.7x0.3+0.6x0.2=0.73. According to the comprehensive score of each node and the total batch size, the initial batch is allocated in proportion. Assuming that the total training batch is 128, and the cluster has two nodes A and B, their comprehensive scores are 0.73 and 0.62 respectively, then the initial batch allocation of node A is 128x0.73 / (0.73+0.62)=69, and the initial batch allocation of node B is 128x0.62 / (0.73+0.62)=59. The discreteness constraint of batch size is also considered to ensure that the allocated batch is an integer and meets the requirements of model training.

[0063] After allocating the initial batch, the time required for each node to complete a batch training is monitored. For example, node A processes a batch of 69 samples in 210 milliseconds, and node B processes a batch of 59 samples in 190 milliseconds. The single batch time is calculated, which is 210 / 69≈3.04 milliseconds / sample for node A and 190 / 59≈3.22 milliseconds / sample for node B. The maximum difference in single batch calculation time between different nodes is |3.04-3.22|=0.18 milliseconds / sample.

[0064] An adaptive batch adjustment amount is constructed based on the maximum difference and the spectral analysis result of the performance fluctuation of the training nodes. The performance data of each node is subjected to spectral analysis to extract the frequency characteristics of the performance fluctuation. The spectral analysis adopts a sliding window method, and the window size is 20 training iterations. The main fluctuation frequency and intensity are identified by calculating the power spectral density. For example, if the performance fluctuation spectrum of node A has a peak value at 0.05 Hz, and the intensity is 0.8, it indicates that the performance of node A fluctuates significantly every 20 seconds or so. The performance fluctuation spectrum of node B is relatively flat, and the maximum peak intensity is only 0.3, indicating that the performance of node B is relatively stable. The stability index of performance fluctuation is defined as 1 minus the main frequency intensity. The stability of node A is 1-0.8=0.2, and the stability of node B is 1-0.3=0.7. The batch adjustment amount is negatively correlated with the stability of the performance fluctuation frequency. The lower the stability, the smaller the adjustment amount, so as to avoid performance shock caused by excessive adjustment. At the same time, the batch adjustment amount and the deviation of the single batch calculation time from the average value have a segmented nonlinear relationship. When the deviation is within the range of 0-10%, the adjustment coefficient is 0.5 times the deviation. When the deviation is within the range of 10%-30%, the adjustment coefficient is 0.8 times the deviation. When the deviation exceeds 30%, the adjustment coefficient is 1.2 times the deviation. This segmented design makes the adjustment more conservative when the deviation is small, and the adjustment more aggressive when the deviation is large.

[0065] A dynamic latency threshold is constructed, which decays exponentially with the training time and has a minimum threshold limit. The initial value of the dynamic latency threshold is set to 15% of the average calculation time of a single batch, and it decays as the number of training iterations increases. The decay formula adopts an exponential form, threshold = initial threshold x decay coefficient iteration number power, and the decay coefficient is usually set to 0.995. For example, if the initial threshold is 0.3 ms / sample, after 100 iterations, the threshold decreases to 0.3 x 0.995^100 ≈ 0.18 ms / sample. To prevent the threshold from being too small and causing frequent adjustments, the minimum threshold limit is set to 20% of the initial threshold, i.e. 0.06 ms / sample. When the calculated threshold is less than this limit, the minimum threshold is used.

[0066] When the maximum difference is greater than the dynamic latency threshold, the current batch assignment value of the training node is adjusted based on the batch adjustment amount. Continuing the above example, if the current maximum difference 0.18 ms / sample is greater than the dynamic threshold 0.06 ms / sample, batch adjustment is triggered. Assuming that the single batch time of node A is lower than the average value 8% and the stability is 0.2, the adjustment coefficient is 0.08x0.5x0.2=0.008; the single batch time of node B is higher than the average value 12% and the stability is 0.7, the adjustment coefficient is 0.12x0.8x0.7=0.0672. From node B, 0.0672x59≈4 samples of batch are reduced and assigned to node A. After adjustment, the batch of node A is 69+4=73 and the batch of node B is 59-4=55. The single batch calculation time after adjustment is continuously monitored, and if the maximum difference is still greater than the dynamic threshold, the adjustment is continued until the condition (less than or equal to the threshold) is met or the maximum adjustment number (such as 10 times) is reached.

[0067] The effect of each batch adjustment is recorded, including the single batch calculation time before and after adjustment, the training throughput change, etc. These data are used to optimize the performance data collection strategy, such as increasing the sampling frequency of nodes with larger fluctuations, or adjusting the monitoring density of specific training stages. For example, if it is found that node A fluctuates significantly during the parameter update stage, the sampling frequency of this stage will be increased from every 5 seconds to every 2 seconds. The optimized collection scheme can more accurately capture performance bottlenecks and provide higher quality decision-making basis for the next round of batch adjustment.

[0068] The present application realizes efficient and balanced utilization of computing resources in distributed training tasks by constructing computing load characteristics and dynamically adjusting batch allocation, effectively solves the problem of uneven load in heterogeneous computing environment, reduces training time, improves resource utilization, and can adapt to performance fluctuations in the training process, providing stable and efficient technical support for training of large-scale deep learning models.

[0069] In an optional embodiment, the step of constructing a dynamic latency threshold comprises: obtaining the computing power parameters of the training nodes, and calculating the initial threshold of the training nodes based on the computing power parameters; obtaining the current gradient variance, the current loss value and the historical adjustment frequency of the training nodes, and calculating a self-defined decay coefficient based on the current gradient variance, the current loss value and the historical adjustment frequency; multiplying the initial threshold by an exponential decay function of training time to obtain a dynamic latency threshold, wherein the decay coefficient in the exponential decay function is determined by the self-defined decay coefficient; when the dynamic latency threshold is less than a minimum threshold limit, the minimum threshold limit is taken as the dynamic latency threshold.

[0070] For example, the computing power parameters of a training node include the number of processor cores, processor frequency, memory capacity, memory bandwidth, accelerator type and its computing power, and other hardware indicators. These parameters are obtained through the tools provided by the underlying hardware interface or operation. For example, for a training node, the computing power parameters are: 8-core processor, 3.2 GHz frequency, 32 GB memory, and accelerator single-precision computing power of 14 TFLOPS. In order to facilitate subsequent calculations, these heterogeneous parameters are uniformly converted into standardized computing power scores. The conversion method is to compare each parameter with the pre-set reference value, calculate the relative performance ratio, and then perform weighted summation according to the pre-defined weight. For example, the processor weight is 0.2, the memory weight is 0.3, and the accelerator weight is 0.5. Assuming that the processor performance of the node is 1.5 times the reference value, the memory performance is 1.2 times the reference value, and the accelerator performance is 2.0 times the reference value, then the standardized computing power score is 1.5 x 0.2 + 1.2 x 0.3 + 2.0 x 0.5 = 1.66.

[0071] The initial threshold value represents the upper limit of the allowed inter-node computing time difference at the initial stage of training, which is closely related to the heterogeneity of the nodes and the computing power distribution. The calculation method is to multiply the difference between the highest computing power score and the lowest computing power score by a proportion factor, and then divide by the number of nodes in the cluster. The proportion factor is usually set between 0.1 and 0.2, reflecting the tolerance for initial computing imbalance. For example, in a four-node cluster, if the highest computing power score is 2.0 and the lowest is 1.2, the difference is 0.8, and the proportion factor is set to 0.15, then the initial threshold value is 0.8 x 0.15 / 4 = 0.03, indicating that the allowed difference of single batch computing time is 3% of the average time. In addition, the influence of model complexity on the initial threshold value is also considered, and a higher initial threshold value is usually set for complex models to adapt to their computing characteristics. For example, for a large Transformer model, multiply the initial threshold value obtained by 1.5 to get the adjusted initial threshold value of 0.045.

[0072] The current gradient variance reflects the stability of model training, which is obtained by calculating the statistical variance of each parameter gradient in the last 10 batches. For example, if the gradient variance of the last 10 batches is 0.005, it indicates that the training is relatively stable; if it is 0.05, it indicates that the training fluctuates greatly. The current loss value represents the convergence degree of model training, which is obtained by directly reading the loss function value in the training process. For example, the loss value in the initial training stage is 2.5, and it decreases to 0.3 in the later training stage. The historical adjustment frequency reflects the frequency of batch allocation adjustment, which is calculated by counting the number of batch adjustments in the past 50 training iterations. For example, if 8 batch adjustments were made in the past 50 iterations, the historical adjustment frequency is 8 / 50 = 0.16, indicating that an adjustment is made on average every 6.25 iterations.

[0073] The custom decay coefficient is calculated based on the current gradient variance, the current loss value and the historical adjustment frequency. The custom decay coefficient controls the rate of decay of the dynamic latency threshold with respect to training time, which should be dynamically adjusted according to the training status. The calculation method is to multiply the base decay coefficient with three adjustment factors. The base decay coefficient is usually set to 0.995, representing the standard decay rate under normal circumstances. The gradient variance adjustment factor is proportional to the gradient variance, the larger the gradient variance, the larger the adjustment factor (the slower the decay). For example, when the gradient variance is 0.005, the corresponding adjustment factor is 0.98; when the gradient variance is 0.05, the adjustment factor is 1.02. The loss value adjustment factor is proportional to the ratio of the current loss value and the initial loss value, the more the loss value decreases, the smaller the adjustment factor (the faster the decay). For example, if the current loss value is 50% of the initial value, the corresponding adjustment factor is 0.99; if it is 10% of the initial value, the adjustment factor is 0.97. The historical adjustment frequency adjustment factor is inversely proportional to the adjustment frequency, the more frequent the adjustment, the larger the adjustment factor (the slower the decay). For example, when the adjustment frequency is 0.05, the corresponding adjustment factor is 0.98; when the adjustment frequency is 0.2, the adjustment factor is 1.03. Multiply the base decay coefficient with the three adjustment factors, for example, 0.995x0.98x0.99x1.03≈0.993, to get the final custom decay coefficient.

[0074] The initial threshold is multiplied by an exponential decay function of the training time to obtain the dynamic latency threshold. The exponential decay function is used to control the rate at which the threshold decreases as training progresses. The function is specifically the number of training iterations to the power of a custom decay coefficient. To implement this calculation, a global iteration counter is maintained, starting at 0 at the beginning of training and incremented at the end of each iteration. At the end of the tth iteration, the dynamic latency threshold is calculated as the initial threshold multiplied by t to the power of the custom decay coefficient. In implementation, logarithmic calculations are used to avoid numerical precision issues caused by large exponentials. First, the logarithm of the exponential part is calculated, which is t times the natural logarithm of the custom decay coefficient. Then, the result is exponentiated to obtain the final result. For example, for an initial threshold of 0.045 and a custom decay coefficient of 0.993, after 100 training iterations, the calculation steps are: (1) calculate ln(0.993) ≈ -0.007; (2) calculate 100 * (-0.007) = -0.7; (3) calculate e^(-0.7) ≈ 0.497; (4) calculate 0.045 * 0.497 ≈ 0.022. Thus the dynamic latency threshold is 0.022. In practice, to improve computational efficiency, the dynamic latency threshold can be updated every fixed number of iterations (e.g., 10) rather than being recalculated every iteration. For long-running training tasks, the base point for exponential calculation is periodically reset to avoid cumulative errors. For example, every 1000 iterations, the current threshold is set as the new reference value, and the iteration count is reset to continue the exponential decay calculation from this new starting point. This segmented calculation method effectively avoids numerical instability problems that occur during long training. After each calculation, the dynamic latency threshold obtained is checked to see if it is less than a pre-set minimum threshold limit. The minimum threshold limit is usually set to 15% to 20% of the initial threshold to prevent the threshold from being too small and causing frequent adjustments. For example, if the initial threshold is 0.045, the minimum threshold limit can be set to 0.045 * 0.15 = 0.00675. If the calculated dynamic latency threshold is less than 0.00675, 0.00675 will be used as the actual dynamic latency threshold to ensure stability. In addition, different decay strategies are set according to the training progress. In the early stages of training (e.g., the first 10% of iterations), a slower decay rate is used by adjusting the custom decay coefficient to its square root; in the middle of training, the standard decay rate is used; in the later stages of training (e.g., the last 20% of iterations), when the model is close to convergence, the threshold decay is completely stopped and maintained at the minimum threshold limit to avoid unnecessary batch adjustments caused by small calculation fluctuations.

[0075] Periodically evaluate the effectiveness of the dynamic latency threshold, and fine-tune according to the actual situation of training. For example, when it is found that the batch adjustment is too frequent, the minimum threshold limit can be temporarily increased; when it is found that the load between nodes is long-term unbalanced, the decay coefficient can be temporarily reduced. In addition, for different types of model training tasks, a threshold parameter library is maintained to record the optimal threshold parameter settings of various models in historical training, providing more accurate initial parameters for new tasks.

[0076] The application realizes the accurate balance of computing load in distributed training by constructing a dynamic latency threshold based on adaptive adjustment of training state, effectively reduces the waiting time in the training process, improves the utilization efficiency of computing resources, and can automatically adjust the tolerance to imbalance according to the training process, providing a more efficient and stable training environment for large-scale deep learning models.

[0077] In an optional implementation, based on the updated resource demand information, the step of selecting training nodes in the service queue that meet the resource demand to execute the model training task includes: Construct a performance index vector of the training nodes, the performance index vector including computing power and historical stability; construct an adaptive weight matrix based on the performance index vector, the weight coefficients of the adaptive weight matrix being dynamically adjusted according to the computation intensity and communication intensity of the training task; multiply the adaptive weight matrix by the performance index vector to obtain a node adaptation degree matrix, the node adaptation degree matrix representing the adaptation degree of different training nodes to the model training task; According to the updated resource demand information, filter a candidate node set that meets the basic resource condition, and sort the candidate node set based on the node adaptation degree matrix; Based on the sorted candidate node set, calculate the network topology distance between the candidate nodes, construct an initial topology affinity matrix based on the network topology distance, and obtain an optimized topology affinity matrix by weighted fusion of the initial topology affinity matrix and a historical communication efficiency matrix; calculate a node combination score based on the optimized topology affinity matrix and the updated resource demand information; select a training node combination with the highest node combination score and meeting the resource demand to execute the model training task.

[0078] For example, the performance indicator vector includes two dimensions of computing power and historical stability. The computing power dimension integrates hardware indicators such as the number of processor cores, processor frequency, number of accelerators, accelerator computing power, and memory bandwidth, and is converted into a score between 0 and 1 through standardization processing. For example, a processor of a certain training node has 32 cores, a frequency of 2.6 GHz, is equipped with 4 accelerators, and a single accelerator has a computing power of 12 TFLOPS and a memory bandwidth of 900 GB / s. After standardization processing, the computing power score is 0.85. The historical stability dimension is calculated by analyzing the task completion, resource fluctuation, and failure rate of the node in the past 30 days. For example, a certain node has completed 95% of the assigned tasks in the past 30 days, the resource fluctuation rate is 5%, and the failure rate is 2%. After weighted calculation, the historical stability score is 0.92. For nodes that newly join the cluster and have no historical data, a lower initial stability score (such as 0.7) is assigned, and the score is quickly updated after the node performs tasks. Finally, the performance indicator vector of the training node is [0.85, 0.92].

[0079] An adaptive weight matrix is constructed based on the performance indicator vector. The adaptive weight matrix is used to represent the importance of different performance indicators to a specific training task, and its weight coefficients are dynamically adjusted according to the computation intensity and communication intensity of the training task. First, the characteristics of the training task are analyzed, including the model parameter quantity, batch size, forward propagation computation complexity, backward propagation gradient computation quantity, and communication data quantity, to obtain the computation intensity and communication intensity indicators of the task. For example, for a large Transformer model training task with a parameter quantity of 1 billion and a batch size of 128, the computation intensity indicator is 0.8 and the communication intensity indicator is 0.6. For computation-intensive tasks (computation intensity > 0.7), the weight of computing power in the weight matrix is increased; for communication-intensive tasks (communication intensity > 0.7), the weight of historical stability in the weight matrix is increased. The specific adjustment method is to multiply the basic weight by the square of the intensity indicator, and then normalize it. For example, the initial weight matrix is [[0.7, 0.3], [0.4, 0.6]], which is adjusted to [[0.7×0.8×0.8, 0.3×0.6×0.6], [0.4×0.8×0.8, 0.6×0.6×0.6]], i.e. [[0.448, 0.108], [0.256, 0.216]]. After normalization, the final adaptive weight matrix is [[0.806, 0.194], [0.542, 0.458]].

[0080] The performance indicator vector of each node is multiplied by the adaptive weight matrix to obtain a value representing the node's fitness. For example, for a node with a performance indicator vector of [0.85, 0.92], multiplying it with the adaptive weight matrix [[0.806, 0.194], [0.542, 0.458]] results in [0.85 x 0.806 + 0.92 x 0.194, 0.85 x 0.542 + 0.92 x 0.458] = [0.685 + 0.178, 0.461 + 0.422] = [0.863, 0.883]. This indicates that the node's computational fitness for the current training task is 0.863 and its communication fitness is 0.883. The same calculation is performed for each node in the cluster to form a complete node fitness matrix.

[0081] According to the updated resource requirement information, including minimum processor core number, minimum memory capacity, minimum accelerator number, and other hard conditions. All training nodes in the cluster are traversed, and nodes that meet these conditions are selected as a candidate node set. For example, if the task requires at least a 16-core processor, 64GB of memory, and 2 accelerators, nodes that do not meet these conditions are excluded. For the selected candidate node set, the node fitness matrix is sorted. The sorting method is to calculate the comprehensive fitness score of each node, which is the weighted sum of the computational fitness and the communication fitness, with weights of computational intensity and communication intensity. For example, for the node fitness [0.863, 0.883], the computational intensity is 0.8 and the communication intensity is 0.6, the comprehensive fitness score is 0.863 x 0.8 + 0.883 x 0.6 = 0.690 + 0.530 = 1.220. The candidate nodes are sorted in descending order of comprehensive fitness score.

[0082] Based on the sorted candidate node set, the network topology distance between candidate nodes is calculated. The network topology distance represents the quality of network connection between nodes, which is calculated by measuring the network delay, bandwidth, and hop count between nodes. The preset probe tool is used to measure the bidirectional network delay and bandwidth between candidate nodes, and the average and variance are recorded. For example, the average network delay between node A and node B is 0.5 milliseconds, the bandwidth is 10 Gbps, and the network hop count is 2, and after standardization, the network topology distance value is 0.2 (the smaller the value, the better the connection quality). Calculate the network topology distance for each pair of candidate nodes to construct an initial topology affinity matrix. The topology affinity value is defined as 1 minus the standardized network topology distance, for example, the topology affinity of node A and node B is 1 - 0.2 = 0.8.

[0083] The historical communication efficiency matrix records the actual communication efficiency of node pairs in past training tasks, which is calculated by monitoring the communication throughput, delay and error rate between nodes. For example, the average communication efficiency of node A and node B in the past when performing similar training tasks is 0.85 (the larger the value, the higher the efficiency). The initial topology affinity is weighted and fused with the historical communication efficiency, and the fusion weight is dynamically adjusted according to the reliability of the historical data. For example, if the historical data is relatively reliable (based on data volume and timeliness evaluation), the initial topology affinity is given a weight of 0.4, and the historical communication efficiency is given a weight of 0.6, and the optimized topology affinity is 0.8*0.4+0.85*0.6=0.32+0.51=0.83.

[0084] The node combination score is calculated based on the optimized topology affinity matrix and the updated resource demand information. According to the number of nodes required by the task, the node combination is generated from the candidate nodes with high ranking. For each node combination, the overall topology affinity (weighted average of the affinity of all node pairs) and the resource matching degree (the degree of matching between node resources and task requirements) are calculated. For example, a node combination contains nodes A, B and C, and the overall topology affinity is 0.81 and the resource matching degree is 0.95. The node combination score is calculated by weighted sum of the overall topology affinity and the resource matching degree, and the weight is adjusted according to the characteristics of the task. Generally, communication-intensive tasks give higher weight to topology affinity, and computation-intensive tasks give higher weight to resource matching degree. For example, for a task with computation intensity 0.8 and communication intensity 0.6, the resource matching degree weight can be set to 0.65 and the topology affinity weight can be set to 0.35, and the node combination score is 0.95*0.65+0.81*0.35=0.618+0.284=0.902.

[0085] The scores of all candidate node combinations are compared, and the combination with the highest score is selected. To ensure the stability of the selection, the historical performance fluctuations between combinations with similar scores (difference less than 5%) are also considered, and the combination with smaller fluctuations is preferred. After determining the final node combination, the task is assigned to these nodes for execution, and the node performance and communication efficiency are continuously monitored during the task execution process, and the relevant historical data is updated to provide more accurate reference for subsequent task allocation.

[0086] The present application realizes accurate selection and combination of distributed training nodes by constructing performance index vector and adaptive weight matrix, combining network topology analysis and historical communication efficiency, effectively improves model training efficiency and resource utilization, reduces communication overhead, enhances the stability of the training process, and provides an efficient and reliable node selection mechanism for large-scale distributed deep learning.

[0087] In a second aspect, the embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored by the memory to perform the method as described above.

[0088] In a third aspect, the present application provides a computer readable storage medium, having stored thereon computer program instructions, which when executed by a processor implement the method as described above.

[0089] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which, when executed by a processor, perform various aspects of the present application.

[0090] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for model training based on k8s, characterized in that, The method comprises the following steps: creating a business queue in a k8s cluster, assigning a model training task to a corresponding business queue according to the type of the training task, generating a yaml file of the model training task, and submitting the yaml file to a custom controller in the k8s cluster; the custom controller identifies the training phase characteristics of the model training task, and collects performance data of the training node according to the training phase characteristics; determining the initial batch allocation value of the training node according to the training phase characteristics and the performance data, continuously adjusting the batch allocation value based on the actual calculation time delay of the training node when executing the initial batch allocation value, until the maximum difference of the single batch calculation time length of the training node is less than the dynamic time delay threshold; write the adjusted batch allocation value to the yaml file, and calculate the corresponding resource demand according to the adjusted batch allocation value, and update the resource demand information in the yaml file; based on the updated resource demand information, select a training node that meets the resource demand in the business queue to execute the model training task.

2. The method of claim 1, wherein, The step of creating a business queue in a k8s cluster, assigning a model training task to a corresponding business queue according to the type of the training task, generating a yaml file of the model training task, and submitting the yaml file to a custom controller in the k8s cluster comprises: creating a multi-level business queue in a k8s cluster, the resource quota of the multi-level business queue is configured through a resource weight matrix; based on the resource weight matrix, a resource borrowing matrix between queues is constructed, the value of each element in the resource borrowing matrix is determined by the source queue resource utilization, the target queue resource utilization and the historical borrowing success rate; obtain the feature data of the model training task, generate a multi-dimensional feature vector based on the feature data; combine the multi-dimensional feature vector to generate a model training task portrait vector; calculate the cosine similarity of the model training task portrait vector and the feature vector of each business queue in the multi-level business queue, multiply the cosine similarity by the priority of the corresponding business queue to obtain the queue matching degree; according to the queue matching degree, the model training task is assigned to the business queue with the highest matching degree; when the resource of the business queue with the highest matching degree is insufficient, the borrowable queue and the borrowing amount are determined according to the resource borrowing matrix, and the resource allocation across queues is realized; write the queue allocation information of the model training task into the yaml file and submit it to the custom controller.

3. The method of claim 1, wherein, The step of collecting performance data of the training node according to the training phase characteristics comprises: the training phase characteristics include data loading phase characteristics, model calculation phase characteristics and parameter updating phase characteristics; a training phase feature vector is constructed, which contains the performance indicators corresponding to each phase feature; based on the performance loss gradient of each feature in the training phase feature vector, dynamically update the weight coefficient of each phase feature in the training phase feature vector, and formulate a performance data collection scheme according to the updated weight coefficient; calculating a feature trend prediction value based on a historical feature sequence of the training phase feature vector; taking a deviation of a current feature vector from the feature trend prediction value as a feature change amplitude, and dynamically optimizing a sampling window size and a sampling frequency of the performance data collection scheme according to a fluctuation rule of the feature change amplitude; and performing performance data collection according to the optimized collection scheme; performing multi-dimensional quality evaluation on the collected performance data to obtain a performance data quality evaluation result; and feeding back the performance data quality evaluation result to the self-defined controller for optimizing a recognition process of a training phase feature.

4. The method of claim 3, wherein, dynamically updating a weight coefficient of each phase feature in the training phase feature vector based on a performance loss gradient of each feature in the training phase feature vector, and formulating steps of a performance data collection scheme according to the updated weight coefficient, including: determining a gradient influence factor based on a historical change feature of the performance loss gradient, and dynamically updating the weight coefficient of each phase feature in the training phase feature vector based on the performance loss gradient and the gradient influence factor to obtain the updated weight coefficient; constructing a collection sensitivity matrix based on the updated weight coefficient, the collection sensitivity matrix representing a correlation degree between each training phase feature and a performance index; constructing a time sequence sensitivity feature based on a historical sequence of the collection sensitivity matrix, the time sequence sensitivity feature representing a change rule of the collection sensitivity matrix over time; and constructing a phase coupling feature based on a correlation relationship between different training phases; performing feature fusion on the time sequence sensitivity feature and the phase coupling feature to construct a multi-level optimization objective function; the multi-level optimization objective function including a sensitivity weighting item and a fluctuation penalty item in a main objective function, and an upper limit of resource utilization and a lower limit of sampling frequency in a constraint function; and solving the multi-level optimization objective function by using a Lagrange multiplier method with an adaptive step length, wherein a multiplier update rate is dynamically adjusted according to a constraint violation degree; and optimizing a search space based on historical performance data and task similarity, and outputting a performance data collection scheme when stability of a solution meets a preset condition.

5. The method of claim 1, wherein, the steps of determining an initial batch allocation value of a training node based on the training phase feature and the performance data, and continuously adjusting the batch allocation value based on an actual calculation time delay of the training node when the initial batch allocation value is executed until a maximum difference of single batch calculation time lengths of the training node is less than a dynamic time delay threshold, including: constructing a calculation load feature of a training node based on the training phase feature and the performance data, the calculation load feature representing a resource consumption level of each training phase; and determining an initial batch allocation value of a training node according to the calculation load feature; obtaining a single batch calculation duration of the training node when performing the initial batch allocation value, calculating a maximum difference value of the single batch calculation duration among different training nodes; constructing an adaptive batch adjustment amount based on the maximum difference value and a frequency spectrum analysis result of training node performance fluctuation, the batch adjustment amount being negatively correlated with stability of performance fluctuation frequency and being in a segmented nonlinear relationship with a degree of deviation from an average value of the single batch calculation duration; constructing a dynamic latency threshold value, the dynamic latency threshold value exponentially decaying with training time and being limited by a minimum threshold value; when the maximum difference value is greater than the dynamic latency threshold value, adjusting a current batch allocation value of the training node based on the batch adjustment amount until the maximum difference value of the single batch calculation duration is less than or equal to the dynamic latency threshold value; feeding back an execution result of the adjusted batch allocation to a performance data acquisition scheme for optimizing a performance data acquisition process in a subsequent training phase.

6. The method of claim 5, wherein, The step of constructing the dynamic latency threshold value comprises: obtaining a computing power parameter of the training node, and calculating an initial threshold value of the training node based on the computing power parameter; obtaining a current gradient variance, a current loss value and a historical adjustment frequency of the training node, and calculating a self-defined decay coefficient based on the current gradient variance, the current loss value and the historical adjustment frequency; multiplying the initial threshold value by an exponential decay function of training time to obtain the dynamic latency threshold value, wherein a decay coefficient in the exponential decay function is determined by the self-defined decay coefficient; when the dynamic latency threshold value is less than a minimum threshold limit, taking the minimum threshold limit as the dynamic latency threshold value.

7. The method of claim 1, wherein, The step of selecting, based on the updated resource requirement information, a training node satisfying the resource requirement from the service queue to execute the model training task comprises: constructing a performance index vector of the training node, the performance index vector including computing capability and historical stability; constructing an adaptive weight matrix based on the performance index vector, wherein a weight coefficient of the adaptive weight matrix is dynamically adjusted according to a calculation-intensive degree and a communication-intensive degree of the training task; multiplying the adaptive weight matrix and the performance index vector to obtain a node fitness matrix, the node fitness matrix representing an adaptation degree of different training nodes to the model training task; screening a candidate node set satisfying a basic resource condition according to the updated resource requirement information, and sorting the candidate node set based on the node fitness matrix; calculating a network topology distance among the candidate nodes based on the sorted candidate node set, constructing an initial topology affinity matrix based on the network topology distance, and weighting and fusing the initial topology affinity matrix and a historical communication efficiency matrix to obtain an optimized topology affinity matrix; calculating a node combination score based on the optimized topology affinity matrix and the updated resource requirement information; and selecting a training node combination with the highest node combination score and satisfying the resource requirement to execute the model training task.

8. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method of any one of claims 1 to 7.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1 to 7.

Images