AI model training acceleration method and system based on computing power service scheduling

Abstract

The present invention provides an AI model training acceleration method and system based on computing power service scheduling, which relates to the field of computing power service technology. First, the historical training process data of the AI ​​model and the structural characteristics of the current training task model are collected to construct a computing power demand prediction model. The structural characteristics of the current task model are input to generate a computing power resource demand distribution sequence, covering the demand change curves for the number of computing cores, memory bandwidth and data transmission rate in each stage of training. Then, according to the sequence, a matching computing power resource combination scheme is screened from the computing power service cluster, and the computing nodes are scheduled to execute the training tasks. The actual computing power resource consumption data is collected in real time. Finally, the actual data is compared with the predicted sequence, a deviation value is generated and the number of enabled computing nodes is dynamically adjusted, thereby realizing accurate prediction and dynamic optimization scheduling of computing power demand, improving computing power resource utilization, and accelerating AI model training.

Description

Technical Field

[0001] The present invention relates to the field of computing power service technology, and more specifically, to an AI model training acceleration method and system based on computing power service scheduling. Background Art

[0002] With the booming development of artificial intelligence (AI) technology, AI model training has become a core driver of innovation and application across various fields. With the increasing complexity of models, such as the increase in the number of layers in deep neural networks and the rapid expansion of parameter sizes, the demand for computing resources has also increased exponentially. Traditional AI model training methods often use fixed-configuration computing resources for training, lacking dynamic awareness and precise matching of computing resource requirements during training.

[0003] Most existing computing power scheduling methods are based on simple empirical rules or static resource allocation strategies, which fail to fully consider the unique computing power requirements of different model structures during the training process. For example, for convolutional neural networks and recurrent neural networks, there are significant differences in their requirements for the number of computing cores, memory bandwidth, and data transmission rates, but traditional methods have difficulty in performing targeted resource scheduling based on these differences. In addition, during the training process, due to factors such as model convergence speed and data loading efficiency, the actual computing power resource consumption will deviate from the initial estimate. Existing methods lack an effective dynamic adjustment mechanism, resulting in low computing power resource utilization, extended training time, and even possible training interruption due to insufficient resources, seriously affecting the efficiency and quality of AI model training. Summary of the Invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides an AI model training acceleration method based on computing power service scheduling, the method comprising:

[0005] Collect historical training process data of the AI ​​model and the model structure characteristics of the current training task. The historical training process data includes the computing resource consumption sequence and training efficiency indicators of different model structures during the training process. The model structure characteristics include network layer type combinations, inter-layer connection relationships, and parameter scale distribution characteristics;

[0006] Building a computing power demand prediction model based on the historical training process data, wherein the input of the computing power demand prediction model is the model structure characteristics, and the output is the computing power resource demand distribution sequence at different stages of the training process;

[0007] Inputting the model structure characteristics of the current training task into the computing power demand prediction model to generate a computing power resource demand distribution sequence for the current training task, wherein the computing power resource demand distribution sequence includes a demand change curve for the number of computing cores, memory bandwidth, and data transmission rate at each stage of training;

[0008] Filtering a matching computing resource combination solution from the computing service cluster according to the computing resource demand distribution sequence, wherein the computing resource combination solution includes a plurality of computing nodes with different configurations and their enabled quantities in each training stage;

[0009] Schedule the computing nodes in the computing service cluster to execute the current training task according to the computing resource combination plan, and collect the actual computing resource consumption data during the training process in real time;

[0010] The actual computing power resource consumption data is compared and analyzed with the computing power resource demand distribution sequence to generate a resource demand prediction deviation value, and the number of enabled computing nodes in the computing power resource combination plan is dynamically adjusted according to the resource demand prediction deviation value.

[0011] On the other hand, an embodiment of the present invention also provides an AI model training acceleration system based on computing power service scheduling, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0012] Based on the above aspects, the embodiments of the present invention can accurately grasp the computing power resource consumption patterns and training efficiency indicators of different model structures during the training process by collecting historical training process data of the AI ​​model and the model structure characteristics of the current training task. The computing power demand prediction model constructed based on these historical training process data uses the model structure characteristics as input to generate a computing power resource demand distribution sequence at different stages of the training process, thereby achieving accurate prediction of computing power demand. The matching computing power resource combination scheme is screened according to the computing power resource demand distribution sequence to ensure that the configuration of the computing node is highly consistent with the requirements of the training task, thereby improving the utilization efficiency of computing power resources. During the training process, the actual computing power resource consumption data is collected in real time and compared with the prediction sequence for analysis. The number of enabled computing nodes is dynamically adjusted according to the generated resource demand prediction deviation value, thereby achieving dynamic optimization and scheduling of computing power resources, effectively responding to changes in computing power demand caused by various factors during the training process, further improving training efficiency, shortening training time, and reducing training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the execution flow of the AI ​​model training acceleration method based on computing power service scheduling provided by an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of exemplary hardware and software components of an AI model training acceleration system based on computing power service scheduling provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of an AI model training acceleration method based on computing power service scheduling provided by an embodiment of the present invention. The AI ​​model training acceleration method based on computing power service scheduling is introduced in detail below.

[0016] Step S110: Collect historical training process data of the AI ​​model and model structure characteristics of the current training task. The historical training process data includes the computing resource consumption sequence and training efficiency indicators of different model structures during the training process. The model structure characteristics include network layer type combinations, inter-layer connection relationships, and parameter scale distribution characteristics.

[0017] When accelerating AI model training, the primary task is to collect two key types of data. First, historical AI model training data records the various aspects of training for different model architectures. Computing resource consumption sequences reflect the computing resource usage of different model architectures during training. These sequences include information such as compute core utilization, memory usage, and data transfer throughput, recorded chronologically. For example, in the training of an image recognition AI model, compute core utilization may be relatively low at the beginning, as the model is initializing parameters and the computational workload is relatively small. As training progresses, compute core utilization gradually increases, as the model begins performing a significant number of computational operations. Memory usage also changes as training progresses, increasing when loading training data and storing intermediate results. Data transfer throughput reflects the speed at which data is transferred between different components, such as the speed of data transfer from storage devices to the compute cores.

[0018] Training efficiency metrics reflect the model training process from another perspective, including the iteration time per round, gradient update speed, and the number of model convergence iterations. The iteration time per round can help assess the training speed. If it takes too long, it may indicate insufficient computing resources or an overly complex model structure. The gradient update speed affects the extent of model parameter adjustments. An appropriate gradient update speed can enable faster model convergence. The number of model convergence iterations indicates the number of iterations required for the model to converge. Excessive iterations may indicate that the model is struggling to converge, and adjustments to the training strategy are necessary.

[0019] On the other hand, it is necessary to collect the model structure characteristics of the current training task. Model structure characteristics include the combination of network layer types, the connection relationship between layers, and the parameter scale distribution characteristics. The network layer type combination specifies the types of network layers used in the model and their arrangement order. In a typical convolutional neural network, there may be multiple convolutional layers to extract image features; followed by pooling layers to reduce the size of feature maps and reduce computational complexity; then fully connected layers to integrate and classify the extracted features; and finally, the activation function layer outputs the classification results. Different combinations of network layer types will have a significant impact on the performance and computational requirements of the model.

[0020] Inter-layer connectivity describes how each network layer is interconnected. Some layers may be connected sequentially, where the output of the previous layer directly serves as the input of the next layer. Other layers may have skip connections, allowing information to be passed directly between layers, which can alleviate problems such as vanishing gradients. Parameter size distribution characteristics focus on the proportion of parameters in each layer to the total parameters and how feature map size changes with network depth. Some layers may have a large number of parameters, which plays a key role in the model's expressiveness. Feature map size may gradually decrease at different layers to reduce computational effort.

[0021] Step S111: Extract historical training process data corresponding to different model structures from the model training log database. The computing power resource consumption sequence in the historical training process data includes computing core utilization, memory occupancy and data transmission throughput recorded by timestamp. The training efficiency indicators include the time consumption of each round of iteration, gradient update speed and number of model convergence iterations.

[0022] The model training log database stores detailed information about the training process of a large number of different model structures. Through query and filtering operations, you can obtain the historical training process data corresponding to the required different model structures.

[0023] Compute core utilization in the computing resource consumption series is recorded by timestamp, reflecting the degree of compute core utilization at different times. Compute core utilization fluctuates at different stages of training. During model initialization, compute core utilization may be low, as this is primarily for preparatory work; during peak training, compute core utilization reaches higher levels. Memory usage is also recorded by timestamp, reflecting the model's memory requirements during training. Memory usage increases significantly when loading large amounts of training data or storing intermediate results. Data transfer throughput records the speed at which data is transferred between different components, such as from the hard drive to memory and from memory to the compute core.

[0024] The training efficiency metric, "Per-round iteration time," can help assess training efficiency. If each iteration takes too long, you may need to check whether computing resources are sufficient or the model structure is overly complex. The gradient update rate affects the adjustment range of model parameters. An appropriate gradient update rate can help the model converge faster. The model convergence iteration count, which indicates the number of iterations required for the model to converge, is an important indicator of model training effectiveness. If the model converges too many times, you may need to adjust the training strategy or optimize the model structure.

[0025] Step S112: Standardize the computing power resource consumption sequence in the historical training process data, unify the resource consumption measurement units under different hardware environments, and generate a standardized resource consumption sequence.

[0026] Because different hardware environments may result in different units of measurement for computing resource consumption, standardization of computing resource consumption data is necessary. Different servers may have different computing core performance, memory capacity, and data transmission bandwidth, and these differences can affect resource consumption measurement. Through standardization, resource consumption data from different hardware environments can be unified under the same metric, facilitating subsequent analysis and comparison.

[0027] Normalization involves converting data such as compute core utilization, memory usage, and data transfer throughput. For compute core utilization, adjustments can be made based on the performance of the compute cores of different hardware, ensuring comparability across different hardware environments. For memory usage, the memory capacity of different hardware can be considered and converted into a relative utilization ratio. For data transfer throughput, normalization can be performed based on the data transfer bandwidth of different hardware to reflect actual transmission efficiency. The resulting standardized resource consumption series can more accurately reflect the resource consumption of the model across different hardware environments.

[0028] Step S113: Parse the model definition file of the current training task and identify the network layer type combination declared in the model definition file, including the types and arrangement order of convolutional layers, pooling layers, fully connected layers, and activation function layers.

[0029] The model definition file for the current training task describes the model's structure in detail. By parsing this file, you can accurately identify the network layer type combination. Model definition files typically use a set format and syntax to describe the model's structure, and the parsing process follows these rules.

[0030] Convolutional layers are commonly used in deep learning networks, extracting features from input data. When parsing the model definition file, you can determine the type of convolutional layer, such as standard convolutional layer or depthwise separable convolutional layer, as well as its parameters, such as kernel size and stride. Pooling layers are used to reduce the size of feature maps and reduce computational complexity. Common pooling layer types include max pooling and average pooling. The parsing process will identify the pooling layer type used and its associated parameters.

[0031] The fully connected layer integrates and classifies the extracted features. The parsing process determines the number of neurons in the fully connected layer and their connection structure. The activation function layer introduces nonlinearity. Common activation functions include ReLU and Sigmoid. The parsing process identifies the type of activation function used. The order of these network layers is also determined, which helps us understand the model's operating principles and computational flow.

[0032] Step S114: extracting the connection relationship between each layer in the network layer type combination, and constructing an inter-layer connection relationship graph, wherein the inter-layer connection relationship graph includes the data flow direction from the input layer to the output layer and the feature mapping dimension change relationship.

[0033] After identifying the network layer type combinations, we need to further extract the connection relationships between the layers and construct an inter-layer connection relationship diagram. The inter-layer connection relationship diagram can intuitively show the flow direction of data between different network layers and the changes in feature dimensions.

[0034] By analyzing the relevant information in the model definition file, you can determine how each layer is connected. Some network layers may be connected sequentially, where the output of the previous layer directly serves as the input of the next layer; while other network layers may have skip connections, allowing information to pass directly between layers. When constructing a diagram of the inter-layer connections, you can graphically represent each network layer, using arrows to indicate the direction of data flow.

[0035] The inter-layer connectivity diagram also records changes in feature map dimensions. The size and dimensions of feature maps may vary across different network layers. For example, in a convolutional layer, the convolution operation changes the size and number of channels of the feature map; in a pooling layer, the size of the feature map can be further reduced. By recording these changes, we can better understand how the model processes features at different layers.

[0036] Step S115: Count the number of parameters and feature map size of each network layer, and calculate the parameter scale distribution characteristics, which include the proportion of parameters of each layer to the total parameters and the change trend of feature map size along the network depth.

[0037] The purpose of counting the number of parameters and feature map sizes for each network layer is to calculate the parameter size distribution characteristics. The number of parameters for each network layer can be determined by the parameter settings in the model definition file. Different types of network layers have different parameter calculation methods. For example, the number of parameters for convolutional layers is related to the size of the convolution kernel and the number of channels, while the number of parameters for fully connected layers is related to the number of neurons.

[0038] Feature map size reflects the representation of features at different network layers. In convolutional and pooling layers, feature map size changes as network depth increases. By counting the feature map sizes of each layer, we can observe how feature map size changes along the network depth.

[0039] When calculating parameter size distribution features, we can calculate the proportion of parameters in each layer to the total parameters. This helps us understand the importance of different layers in the model. Some layers may have a large number of parameters and play a key role in the model's expressiveness; while other layers may have fewer parameters but play an important role in feature extraction or information integration. Furthermore, by analyzing the changing trend of feature map size along the depth of the network, we can understand how the model processes features at different layers, such as whether there is excessive feature compression or information loss.

[0040] Step S116: Pack the network layer type combination, inter-layer connection relationship diagram and parameter scale distribution characteristics to generate the model structure characteristics of the current training task.

[0041] After collecting and calculating the network layer type combinations, inter-layer connection diagrams, and parameter scale distribution characteristics, they need to be packaged together to generate the model structure characteristics of the current training task. This model structure characteristic will serve as the input for the subsequent computing power demand prediction model.

[0042] The packaging process integrates the network layer type combination, the inter-layer connection diagram, and the parameter size distribution characteristics. The network layer type combination provides information about the types and order of network layers used in the model; the inter-layer connection diagram shows the connection between each layer and the direction of data flow; the parameter size distribution characteristics reflect the importance of each layer's parameters and the changing trends of feature map sizes. By packaging this information together, a comprehensive description of the model structure of the current training task can be achieved.

[0043] Step S120: Construct a computing power demand prediction model based on the historical training process data, wherein the input of the computing power demand prediction model is the model structure feature, and the output is the computing power resource demand distribution sequence at different stages of the training process.

[0044] After collecting historical training data and the model structure characteristics of the current training task, the next step is to build a computing power demand prediction model based on the historical training process data. The purpose of this computing power demand prediction model is to predict the distribution sequence of computing power resource requirements at different stages of the training process based on the input model structure characteristics.

[0045] First, historical training data must be processed for use in model construction. This data includes the computing resource consumption sequences and training efficiency indicators of different model structures during training. This data contains rich information that can help the model learn the relationship between different model structures and computing resource requirements.

[0046] Step S121: Perform time series segmentation processing on the standardized resource consumption sequence in the historical training process data, and divide the complete training process of each AI model into an initial convergence stage, a rapid iteration stage, and a fine tuning stage, each stage corresponding to a sub-resource consumption sequence.

[0047] To better analyze and model, it's necessary to segment the standardized resource consumption sequences in historical training data into time series. During AI model training, resource consumption varies at different stages. The complete training process can be divided into the initial convergence phase, the rapid iteration phase, and the fine-tuning phase.

[0048] The initial convergence phase is when model training begins. Model parameters are randomly initialized, requiring extensive computation to adjust the parameters and achieve convergence. During this initial convergence phase, compute core utilization may gradually increase, and memory usage will also increase as training data is loaded and intermediate results are stored. The rapid iteration phase is when model parameters are rapidly adjusted. At this point, the model has a solid foundation, and continuous iterative parameter updates can rapidly improve model performance. During this rapid iteration phase, compute core utilization and data transfer throughput may reach high levels. The fine-tuning phase is when the model nears convergence. Fine-tuning the model parameters is necessary to improve accuracy. During this fine-tuning phase, compute core utilization and memory usage may remain relatively stable, but a certain amount of computing resources are still required.

[0049] Each phase corresponds to a sub-resource consumption sequence, which includes information such as computing core utilization, memory usage, and data transfer throughput within that phase. By analyzing the sub-resource consumption sequences for each phase, we can better understand the resource requirements of the model at different stages.

[0050] Step S122: extracting characteristic statistics of each sub-resource consumption sequence to generate a stage resource characteristic vector, wherein the characteristic statistics include sequence mean, peak value, fluctuation variance and trend slope.

[0051] After segmenting the standardized resource consumption sequence, it is necessary to extract the characteristic statistics of each sub-resource consumption sequence to generate the stage resource feature vector. The characteristic statistics can reflect the characteristics and regularities of the sub-resource consumption sequence.

[0052] The series mean represents the average level of a sub-resource consumption series, helping to understand the overall resource consumption situation during a particular period. The peak value reflects the maximum value in the sub-resource consumption series, reflecting the maximum resource demand that may occur during that period. The fluctuation variance measures the degree of fluctuation in the sub-resource consumption series; a larger variance indicates more drastic fluctuations in resource consumption. The trend slope indicates the changing trend of the sub-resource consumption series. A positive trend slope indicates an upward trend in resource consumption, while a negative trend slope indicates a downward trend in resource consumption.

[0053] By calculating these characteristic statistics, each sub-resource consumption sequence can be converted into a stage resource characteristic vector, which contains the main characteristic information of resource consumption in that stage.

[0054] Step S123: The model structure features in the historical training process data are associated with the corresponding stage resource feature vectors and stored to construct a target associated data set.

[0055] After generating the stage resource feature vectors, the model structure features from the historical training process data need to be associated and stored with the corresponding stage resource feature vectors to construct the target associated dataset. The model structure features describe the model's structural information, including the combination of network layer types, inter-layer connectivity, and parameter size distribution characteristics; the stage resource feature vectors reflect the resource consumption characteristics of the model at different training stages.

[0056] By associating and storing these two, we can establish a connection between the model structure and resource requirements. In the subsequent modeling process, we can use this target-associated dataset to learn the mapping relationship between the model structure and resource requirements.

[0057] Step S124: Divide the target association data set into a training set and a validation set, wherein the training set is used for model parameter learning, and the validation set is used for model generalization ability evaluation.

[0058] To train and evaluate the computing power demand prediction model, the target-related dataset needs to be divided into a training set and a validation set. The training set is used to learn model parameters. By using the data in the training set, the model can adjust its parameters and learn the mapping relationship between the model structure and resource requirements.

[0059] The validation set is used to evaluate the model's generalization ability. Generalization ability refers to the model's performance on unseen data. By using the validation set, we can test whether the model can accurately predict the computing resource requirements of different model structures. If the model performs well on the validation set, it indicates that the model has good generalization ability. If the model performs poorly on the validation set, it may be necessary to adjust the model structure or training strategy.

[0060] Step S125: Construct a deep learning network including an input layer, a feature fusion layer, a time series prediction layer and an output layer as the basic architecture of the computing power demand prediction model. The number of neurons in the input layer matches the dimension of the model structure characteristics, and the number of neurons in the output layer matches the dimension of the stage resource feature vector.

[0061] The basic architecture for building a computing power demand prediction model is a deep learning network consisting of an input layer, a feature fusion layer, a time series prediction layer, and an output layer. The number of neurons in the input layer matches the dimensionality of the model's structural features, ensuring that the input layer can receive and process information from the model's structural features.

[0062] The feature fusion layer fuses and transforms the high-dimensional features of the input layer. Within the feature fusion layer, multiple fully connected sublayers can be configured, each containing a set number of neurons. Using the ReLU activation function for nonlinear transformation, the high-dimensional features of the input layer are mapped into a low-dimensional fused feature space, thereby extracting more valuable feature information.

[0063] The time series prediction layer uses a two-layer bidirectional long short-term memory network structure to capture the long-term dependencies of resource requirements and bidirectional time series characteristics during training. Long short-term memory networks are effective in processing time series data. Memory cells can remember past information and make predictions based on current input. A dropout layer is placed between the two layers of the bidirectional long short-term memory network to prevent overfitting of the deep learning network by randomly dropping some neuronal connections.

[0064] The output layer uses a fully connected structure, with the number of neurons matching the dimensions of the stage resource feature vector. A linear activation function is used to output the predicted stage resource feature vector, which contains the model's resource demand prediction information at different training stages.

[0065] Step S1251: Determine the number of neurons in the input layer, which is equal to the sum of the one-hot encoding dimension of the network layer type combination in the model structure characteristics, the adjacency matrix dimension of the inter-layer connection relationship graph, and the statistical dimension of the parameter scale distribution characteristics.

[0066] When building a deep learning network, it is necessary to determine the number of neurons in the input layer. The number of neurons in the input layer should match the dimensionality of the model's structural features to ensure that they can accurately receive and process information from the model's structural features.

[0067] Model structure features include the network layer type combination, the inter-layer connection graph, and the parameter size distribution. The one-hot encoding dimension of the network layer type combination reflects the type and arrangement of different network layer types. The adjacency matrix dimension of the inter-layer connection graph reflects the connection method and information transfer relationship between layers. The statistical dimension of the parameter size distribution feature includes information such as the importance of each layer parameter and the changing trend of the feature graph size.

[0068] Adding up the numbers of these three dimensions gives us the number of neurons in the input layer, which allows the input layer to fully receive information about the structural characteristics of the model.

[0069] Step S1252: multiple fully connected sublayers are set in the feature fusion layer, each fully connected sublayer contains a preset number of neurons, and nonlinear transformation is performed through the ReLU activation function to map the high-dimensional features of the input layer to the low-dimensional fusion feature space.

[0070] In the feature fusion layer, multiple fully connected sublayers are set up to process the high-dimensional features of the input layer. Each fully connected sublayer contains a preset number of neurons, which are connected to the neurons in the input layer through connection weights.

[0071] The ReLU activation function performs a nonlinear transformation, mapping the high-dimensional features of the input layer into a low-dimensional fused feature space. The ReLU activation function is simple and efficient, introducing nonlinear factors and enabling the model to learn more complex feature representations. In the feature fusion layer, through the processing of multiple fully connected sublayers, more valuable feature information can be gradually extracted, reducing feature redundancy and noise.

[0072] Step S1253: A batch normalization layer is set between the feature fusion layer and the time series prediction layer to standardize the fused feature vector and accelerate the training convergence speed of the deep learning network.

[0073] To accelerate the training convergence of the deep learning network, a batch normalization layer is placed between the feature fusion layer and the time series prediction layer. The batch normalization layer normalizes the fused feature vector so that its mean is zero and its variance is one.

[0074] Batch normalization reduces the distribution differences in feature vectors, making model training more stable. During training, data from different batches may have different distributions, which can lead to unstable model training. The batch normalization layer can eliminate these distribution differences, allowing the model to converge faster.

[0075] Step S1254: The time series prediction layer adopts a two-layer bidirectional long short-term memory network structure. Each layer of the long short-term memory network contains a preset number of memory units to capture the long-term dependency of resource requirements and bidirectional time series characteristics during the training process.

[0076] The time series prediction layer uses a two-layer bidirectional long short-term memory network structure. This structure effectively captures the long-term dependencies of resource requirements and bidirectional time series characteristics during training. Long short-term memory networks use memory cells to store and process time series data, remembering past information and making predictions based on current input.

[0077] Each layer of the LSTM network contains a preset number of memory units, which can learn patterns and regularities in time series data. Bidirectional LSTM networks can simultaneously consider both forward and reverse information in the time series, thereby better capturing changing trends in resource demand.

[0078] Step S1255: A dropout layer is set between the two layers of the bidirectional long short-term memory network to prevent overfitting of the deep learning network by randomly discarding some neuron connections.

[0079] To prevent overfitting of the deep learning network, a dropout layer is placed between the two layers of the bidirectional long short-term memory network. Overfitting refers to the phenomenon that the model performs well on the training set but performs poorly on unseen data.

[0080] The dropout layer randomly discards some neuron connections, preventing the model from over-relying on certain neurons during training, thereby improving the model's generalization ability. During training, the dropout layer randomly selects a portion of neuron connections to discard, allowing the model to learn more robust feature representations.

[0081] Step S1256: The output layer adopts a fully connected structure, the number of neurons matches the dimension of the stage resource feature vector, and the predicted stage resource feature vector is output through a linear activation function.

[0082] The output layer adopts a fully connected structure, and the number of neurons matches the dimension of the stage resource feature vector. This ensures that the output layer can accurately output the predicted stage resource feature vector.

[0083] The predicted stage resource feature vector is output through the linear activation function. The linear activation function can maintain the linear relationship of the output, so that the output of the model can directly reflect the predicted value of resource demand.

[0084] Step S1257: Use the Xavier initialization method to initialize the weight parameters of the deep learning network, set the network optimizer to the Adam optimizer, the learning rate to the preset initial learning rate, and the training rounds to the preset maximum training rounds.

[0085] In order to train a deep learning network better, the network's weight parameters need to be initialized. Using the Xavier initialization method can make the initial values ​​of the weight parameters have a suitable distribution, thereby accelerating the training convergence of the network.

[0086] Set the network optimizer to the Adam optimizer. The Adam optimizer is an adaptive learning rate optimizer that automatically adjusts the learning rate based on parameter updates, making model training more stable and efficient. Set the learning rate to the preset initial learning rate. An appropriate learning rate allows the model to better adjust parameters during training. Set the number of training rounds to the preset maximum number of training rounds. Through multiple training rounds, the model can fully learn the information in the data and improve its performance.

[0087] Step S126: Use the training set to train the deep learning network, and adjust the network weight parameters through the back propagation algorithm to minimize the mean square error between the predicted stage resource feature vector output by the deep learning network and the real stage resource feature vector.

[0088] After building the infrastructure of a deep learning network, the network is trained using a training set. Through the backpropagation algorithm, the network can adjust its weight parameters based on the data in the training set.

[0089] The basic idea of ​​the backpropagation algorithm is to calculate the gradient of the error between the predicted and actual resource feature vectors output by the network with respect to the network's weight parameters, and then update the weight parameters based on the gradient. By continuously iteratively updating the weight parameters, the mean squared error between the predicted and actual resource feature vectors output by the deep learning network is minimized.

[0090] Step S127: Use the validation set to evaluate the prediction performance of the trained deep learning network. When the prediction mean square error is lower than a preset threshold, stop training to obtain the final computing power demand prediction model.

[0091] The prediction performance of the trained deep learning network is evaluated using the validation set. The prediction accuracy of the model can be evaluated by calculating the mean squared error between the predicted and true resource feature vectors.

[0092] When the predicted mean squared error is lower than the preset threshold, the model's prediction performance has met the requirements and training can be stopped to obtain the final computing power demand prediction model. If the predicted mean squared error is higher than the preset threshold, the model structure or training strategy may need to be adjusted before continuing training.

[0093] Step S130: Input the model structure characteristics of the current training task into the computing power demand prediction model to generate a computing power resource demand distribution sequence for the current training task. The computing power resource demand distribution sequence includes the demand change curve for the number of computing cores, memory bandwidth and data transmission rate at each stage of training.

[0094] After obtaining the final computing power demand prediction model, the model structure characteristics of the current training task are input into the model to generate the computing power resource demand distribution sequence of the current training task. This computing power resource demand distribution sequence includes the demand change curves for the number of computing cores, memory bandwidth, and data transmission rate at each stage of training.

[0095] First, the model structure features of the current training task need to be preprocessed to ensure that they can be correctly processed by the computing power demand prediction model.

[0096] Step S131: Perform feature normalization on the model structure features of the current training task to obtain standardized model structure features so that the numerical range of each feature dimension is consistent with the numerical range of the model structure features used when constructing the computing power demand prediction model.

[0097] To make the model structure features of the current training task comparable with those used in building the computing power demand prediction model, feature normalization is required. The purpose of feature normalization is to adjust the numerical range of each feature dimension to be consistent with the numerical range of the features used in building the model.

[0098] By standardizing the model's structural features, including network layer type combinations, inter-layer connectivity, and parameter size distribution, we can eliminate dimensional differences between features, enabling the model to more accurately process and analyze these features. This standardized model structure is more conducive to input and processing into the computing power demand forecasting model.

[0099] Step S132: input the standardized model structure features into the input layer of the computing power demand prediction model, perform nonlinear transformation on the model structure features through the feature fusion layer, and generate a fused feature vector.

[0100] The standardized model structure features are input into the input layer of the computing power demand prediction model. The number of neurons in the input layer matches the dimensions of the model structure features, enabling it to receive and process these feature information.

[0101] In the feature fusion layer, multiple fully connected sublayers and the ReLU activation function perform nonlinear transformations on the model's structural features. Neurons in the fully connected sublayers are connected to neurons in the input layer via connection weights, mapping the input's high-dimensional features into a low-dimensional fused feature space. The ReLU activation function introduces nonlinearity, enabling the model to learn more complex feature representations. Processing in the feature fusion layer generates a fused feature vector that contains more valuable feature information.

[0102] Step S133: Input the fused feature vector into the time series prediction layer of the computing power demand prediction model, model the resource demand change trend of each stage of the training process through the long short-term memory network, and output the predicted stage resource feature vector of each stage.

[0103] The fused feature vector is input into the time series prediction layer of the computing power demand forecasting model. The time series prediction layer uses a two-layer bidirectional long short-term memory network structure to capture the long-term dependencies and bidirectional time series characteristics of resource requirements during training.

[0104] Long-Short-Term Memory (LSTM) networks use memory cells to store and process time series data, enabling them to remember past information and make predictions based on current input. During this process, the LSTM network learns the changing patterns and regularities of resource demand during training and outputs resource feature vectors for each predicted stage. These vectors contain information about the model's resource demand forecasts at different training stages.

[0105] Step S134: parsing the prediction stage resource feature vectors of each stage, and extracting feature statistics, wherein the feature statistics include calculating the mean core utilization, the peak value of the memory usage, and the data transmission throughput trend slope.

[0106] The resource feature vectors of each prediction phase are parsed to extract characteristic statistics, including the average computing core utilization, peak memory usage, and data transmission throughput trend slope.

[0107] The mean compute core utilization reflects the average utilization of the compute cores during that phase, the peak memory usage represents the maximum memory usage during that phase, and the data throughput trend slope reflects the changing trend of data throughput during that phase. By extracting these characteristic statistics, we can more intuitively understand the resource requirements of the model at different training phases.

[0108] Step S135: Calculate the number of computing cores required for each training stage based on the computing core utilization average and the preset single-core computing capability parameters; calculate the memory bandwidth and data transmission rate required for each training stage based on the memory occupancy peak and the data transmission throughput trend slope.

[0109] The number of cores required for each training phase can be calculated based on the average core utilization and the preset per-core computing power parameter. The per-core computing power parameter represents the computing power of a single core per unit time. By calculating the relationship between the average core utilization and the per-core computing power parameter, we can determine the number of cores required to meet the computing needs during each phase.

[0110] The memory bandwidth and data transfer rate required for each training phase can be calculated based on the peak memory usage and the data transfer throughput trend slope. The peak memory usage reflects the maximum memory usage during that phase, which can be used to determine the required memory bandwidth to ensure timely reading and writing of memory data. The data transfer throughput trend slope reflects the changing trend of data transfer throughput, which can be used to determine the required data transfer rate to meet data transfer requirements between different components.

[0111] Step S136: Plot the relationship between the number of computing cores, memory bandwidth, and data transmission rate at each training stage and time into a curve, and combine them to generate a computing resource demand distribution sequence for the current training task.

[0112] Plot the relationship between the number of computing cores, memory bandwidth, and data transfer rate at each training stage over time. These curves can intuitively show how the model's demand for computing cores, memory bandwidth, and data transfer rate changes at different training stages.

[0113] By combining these curves, we generate a distribution sequence of computing resource requirements for the current training task. This distribution sequence includes curves showing changes in the number of computing cores, memory bandwidth, and data transfer rates required at each training stage.

[0114] Step S140: Filter a matching computing resource combination solution from the computing resource service cluster according to the computing resource demand distribution sequence, where the computing resource combination solution includes a plurality of computing nodes with different configurations and their enabled numbers in each training stage.

[0115] After obtaining the computing resource demand distribution sequence for the current training task, it is necessary to select a matching computing resource combination solution from the computing service cluster based on this computing resource demand distribution sequence. The computing service cluster is a resource pool containing multiple computing nodes with different configurations, each with different parameters such as computing power, memory capacity, and data transmission rate.

[0116] First, you need to obtain the resource configuration list of the computing service cluster and understand the hardware parameters of each computing node.

[0117] Step S141: Obtain a resource configuration list of the computing power service cluster, wherein the resource configuration list includes hardware parameters of each computing node, including computing core type, number of cores, memory capacity, memory bandwidth, and network interface rate.

[0118] Obtaining the resource configuration list of the computing service cluster is the basis for screening and matching computing resource combinations. The resource configuration list records the hardware parameters of each computing node in detail, including computing core type, number of cores, memory capacity, memory bandwidth, and network interface speed.

[0119] The compute core type determines the computing power and computational methods of a compute node. Different types of compute cores have different performance characteristics. The core count indicates the number of compute cores in a compute node; a larger number of cores indicates greater computing power. Memory capacity determines the amount of data a compute node can store, while memory bandwidth affects the speed at which data can be read and written to memory. Network interface speed indicates the speed at which data can be transmitted between the compute node and the external network.

[0120] Step S142: Decompose the computing power resource demand distribution sequence into an initial convergence phase demand subsequence, a rapid iteration phase demand subsequence, and a fine tuning phase demand subsequence according to the training phase.

[0121] In order to more accurately screen matching computing resources, it is necessary to decompose the computing resource demand distribution sequence into the initial convergence stage demand subsequence, the rapid iteration stage demand subsequence, and the fine-tuning stage demand subsequence according to the training stage.

[0122] Each phase's requirement subsequence contains information about the number of computing cores, memory bandwidth, and data transfer rate required for that phase. By breaking down the requirement subsequences, we can select appropriate computing nodes based on the requirements of each phase.

[0123] Step S143: For each demand subsequence, select a set of candidate computing nodes from the computing power service cluster whose hardware parameters meet the requirements of the number of computing cores, memory bandwidth, and data transmission rate at this stage.

[0124] For each demand subsequence, we need to filter out candidate compute nodes from the computing service cluster whose hardware parameters meet the requirements of that stage. First, we parse the demand subsequence to extract the minimum required values ​​for the number of computing cores, memory bandwidth, and data transfer rate for that stage.

[0125] Next, the resource configuration list of the computing service cluster is traversed to read the hardware parameters of each compute node. For each compute node, the system checks whether its number of compute cores is greater than or equal to the minimum required number of compute cores, whether its memory bandwidth is greater than or equal to the minimum required memory bandwidth, and whether its network interface speed is greater than or equal to the minimum required data transfer speed. Compute nodes that meet all three conditions are marked as candidate nodes. The identification and complete hardware parameters of all candidate nodes are collected to form the candidate compute node set for this training phase.

[0126] Step S1431: parse the requirement subsequence to extract the minimum required value for the number of computing cores, the minimum required value for the memory bandwidth, and the minimum required value for the data transmission rate in the training phase.

[0127] Parsing the requirement subsequence is a key step in screening candidate compute nodes. By analyzing the requirement subsequence, we can extract the minimum requirements for the number of computing cores, memory bandwidth, and data transfer rate during the training phase.

[0128] These minimum requirements serve as the basis for screening candidate compute nodes. Only compute nodes whose hardware parameters meet these minimum requirements are selected. By accurately extracting these minimum requirements, we can ensure that the selected candidate compute nodes can meet the computing power resource requirements of the current phase.

[0129] Step S1432: traverse the resource configuration list of the computing power service cluster and read the hardware parameters of each computing node, including the number of computing cores, memory bandwidth and network interface rate.

[0130] Traverse the resource configuration list of the computing power service cluster and read the hardware parameters of each computing node. The resource configuration list records the hardware information of each computing node in detail. By traversing the list, you can obtain parameters such as the number of computing cores, memory bandwidth, and network interface speed of each computing node.

[0131] These parameters are important for determining whether a compute node meets requirements. By reading these parameters, you can gain a comprehensive understanding of the performance of each compute node.

[0132] Step S1433: For each computing node, determine whether the number of computing cores is greater than or equal to the minimum required value of the number of computing cores, whether the memory bandwidth is greater than or equal to the minimum required value of the memory bandwidth, and whether the network interface rate is greater than or equal to the minimum required value of the data transmission rate.

[0133] For each computing node, a series of judgments need to be made. The first step is to determine whether the number of computing cores is greater than or equal to the minimum required number of computing cores. If this condition is not met, it means that the computing capacity of the computing node may not meet the requirements of this stage.

[0134] Determine whether its memory bandwidth is greater than or equal to the minimum required memory bandwidth. If this condition is not met, memory read and write speeds may be too slow, affecting training efficiency. Determine whether its network interface rate is greater than or equal to the minimum required data transmission rate. If this condition is not met, data transmission delays may occur, affecting model training results.

[0135] Step S1434: Mark the computing nodes that meet the above three conditions as candidate nodes, and collect the identifiers of all computing nodes marked as candidate nodes and their complete hardware parameters to form a set of candidate computing nodes for this training phase.

[0136] Compute nodes that meet the minimum requirements for the number of computing cores, memory bandwidth, and network interface speed are marked as candidate nodes. These candidate nodes have the potential to meet the computing resource requirements of this stage.

[0137] Collect all the compute node identifiers and their complete hardware parameters marked as candidate nodes to form the candidate compute node set for this training phase. This candidate compute node set includes all the compute nodes that may meet the requirements of this phase.

[0138] Step S1435: sort the candidate computing node set in descending order by the number of computing cores, and sort them in descending order by memory bandwidth when the number of computing cores is the same, to generate an ordered list of candidate computing nodes.

[0139] To more easily select the appropriate computing node, the candidate computing nodes need to be sorted. First, sort them in descending order by the number of computing cores. The computing nodes with more computing cores are ranked first, because the more computing cores there are, the stronger the computing power is.

[0140] When the number of compute cores is the same, nodes with higher memory bandwidth are sorted in descending order. This is because higher memory bandwidth can increase data read and write speeds, which is beneficial for model training. This sorting generates an ordered list of candidate compute nodes, which helps to more intuitively compare and select appropriate compute nodes.

[0141] Step S144: performing a performance-cost ratio evaluation on each computing node in the candidate computing node set, where the performance-cost ratio is a ratio of a computing performance index of a node to a resource leasing cost.

[0142] Evaluate the performance-to-cost ratio of each compute node in the candidate compute node set. This ratio is the ratio of a node's computing performance to its resource rental cost. The computing performance metric reflects the computing power of a compute node by comprehensively considering factors such as the number of compute cores, core type, and memory bandwidth.

[0143] The resource rental cost represents the cost of using the compute node. By calculating the performance-to-cost ratio, you can evaluate the cost-effectiveness of each compute node. A higher performance-to-cost ratio indicates better computing performance for the compute node at the same cost.

[0144] Step S145: Select a preset number of computing nodes from the candidate computing node set according to the performance-cost ratio to form a computing power resource sub-plan for this stage, and the computing power resource sub-plan includes the identifiers of the selected computing nodes and the number of enabled nodes.

[0145] A preset number of computing nodes are selected from the candidate computing node set based on their performance-cost ratio to form the computing power resource sub-plan for this phase. When selecting, computing nodes with high performance-cost ratios are prioritized to ensure that resource rental costs are reduced while meeting computing power requirements.

[0146] The computing resource sub-plan includes the IDs of the selected compute nodes and their enabled quantity. The ID uniquely identifies each compute node, and the enabled quantity indicates the number of compute nodes required for that phase. By selecting the appropriate compute nodes and enabled quantity, you can optimize the computing resource allocation for that phase.

[0147] Step S146: Combine the computing power resource sub-plans of the initial convergence stage, the rapid iteration stage, and the fine-tuning stage in chronological order to generate a computing power resource combination plan covering the entire training process.

[0148] The computing resource sub-plans for the initial convergence phase, rapid iteration phase, and fine-tuning phase are combined in chronological order to generate a computing resource combination plan that covers the entire training process. This computing resource combination plan includes the computing nodes required and the number of nodes to be activated at different training stages, which can meet the computing resource requirements of the model throughout the training process.

[0149] By rationally combining computing resource sub-plans at different stages, efficient scheduling of computing resources can be achieved, thereby improving model training efficiency.

[0150] Step S150: Schedule the computing nodes in the computing service cluster to execute the current training task according to the computing resource combination plan, and collect the actual computing resource consumption data during the training process in real time.

[0151] After obtaining a computing resource combination plan, it is necessary to schedule the computing nodes in the computing service cluster according to the plan to execute the current training task. At the same time, it is necessary to collect the actual computing resource consumption data during the training process in real time for subsequent analysis and adjustment.

[0152] First, it is necessary to parse the computing resource combination plan and extract the computing node identification, enabled number and stage start timestamp corresponding to each training stage.

[0153] Step S151: Analyze the computing resource combination plan and extract the computing node identifier, enabled quantity and stage start timestamp corresponding to each training stage.

[0154] Analyzing the computing resource combination plan is a prerequisite for scheduling computing nodes. By analyzing the plan, we can extract the computing node identification, activation quantity, and phase start timestamp corresponding to each training phase.

[0155] The compute node identifier uniquely identifies each compute node, the enabled quantity indicates the number of compute nodes to be used in that phase, and the phase start timestamp indicates when the phase began. By extracting this information, we can accurately determine which compute nodes to use and when to enable them in different training phases.

[0156] Step S152: before the start timestamp of the initial convergence phase arrives, a resource reservation request is sent to the resource scheduling interface of the computing power service cluster, where the resource reservation request includes the identifiers and enabled number of computing nodes required for the initial convergence phase.

[0157] Before the initial convergence phase start timestamp arrives, a resource reservation request needs to be sent to the resource scheduling interface of the computing power service cluster. The resource reservation request contains the identifiers and number of enabled computing nodes required for the initial convergence phase.

[0158] By sending a resource reservation request, you can reserve the required computing resources from the computing service cluster in advance, ensuring that the required computing nodes can be obtained in time at the beginning of the initial convergence phase. The resource scheduling interface will allocate and manage resources based on the request.

[0159] Step S153: When the training task is started, the reserved computing nodes are activated through the resource scheduling interface, and the model parameters and initial training data of the current training task are distributed to each computing node.

[0160] When a training task starts, the reserved compute nodes are activated through the resource scheduling interface. The activated compute nodes can begin executing the training task. Simultaneously, the model parameters and initial training data for the current training task are distributed to each compute node.

[0161] Model parameters are the initial state of the model, and initial training data is the data used to train the model. By distributing this information to each computing node, they can begin model training.

[0162] Step S154: Each computing node performs model training operations according to the preset distributed training protocol. During the training process, the actual computing power resource consumption data is collected in real time through the node's built-in resource monitoring module. The actual computing power resource consumption data includes computing core utilization, memory usage and data transmission rate.

[0163] Each computing node performs model training operations according to a preset distributed training protocol. The distributed training protocol specifies how computing nodes collaborate to perform model training, such as data parallelism and model parallelism.

[0164] During training, the node's built-in resource monitoring module collects real-time data on actual computing resource consumption, including core utilization, memory usage, and data transfer rate. This data reflects the actual resource usage of the compute node during training.

[0165] Step S155: The actual computing power resource consumption data is uploaded to the central monitoring node after being marked with a timestamp. The central monitoring node aggregates and stores the received actual computing power resource consumption data to form an actual computing power resource consumption sequence.

[0166] The actual computing resource consumption data is timestamped and uploaded to the central monitoring node. The timestamp records the time of data collection, facilitating subsequent analysis and comparison.

[0167] The central monitoring node aggregates and stores the received actual computing resource consumption data to form an actual computing resource consumption sequence. This actual computing resource consumption sequence contains the actual resource consumption of each computing node during the entire training process.

[0168] Step S156: When the training process enters the rapid iteration stage and the fine-tuning stage, the resource reservation, node activation and data collection operations are repeated so that the computing nodes in each stage are enabled on time and complete actual computing resource consumption data is collected.

[0169] When the training process enters the rapid iteration phase and the fine-tuning phase, resource reservation, node activation, and data collection operations need to be repeated. Before the start of each phase, a resource reservation request is sent to the resource scheduling interface of the computing power service cluster to reserve the computing nodes required for that phase.

[0170] When this phase begins, the reserved compute nodes are activated, and the model parameters and training data are distributed to each compute node. During training, actual computing resource consumption data is collected in real time and uploaded to the central monitoring node. By repeating these operations, we can ensure that the compute nodes in each phase are activated on time and that complete actual computing resource consumption data is collected.

[0171] Step S160: Compare and analyze the actual computing power resource consumption data with the computing power resource demand distribution sequence to generate a resource demand prediction deviation value, and dynamically adjust the number of enabled computing nodes in the computing power resource combination plan according to the resource demand prediction deviation value.

[0172] After collecting actual computing resource consumption data, it is necessary to compare and analyze it with the computing resource demand distribution sequence. This comparison and analysis can generate a resource demand forecast deviation value, which can be used to dynamically adjust the number of computing nodes enabled in the computing resource combination plan.

[0173] First, it is necessary to extract the actual resource consumption feature vector of each training stage from the actual computing power resource consumption data.

[0174] Step S161: extracting the actual resource consumption feature vector of each training stage from the actual computing power resource consumption data, wherein the actual resource consumption feature vector includes the actual mean value of the computing core utilization, the actual peak value of the memory occupancy, and the actual trend slope of the data transmission rate.

[0175] The actual resource consumption feature vectors for each training phase are extracted from the actual computing resource consumption data. These feature vectors contain information such as the actual mean of computing core utilization, the actual peak of memory usage, and the actual trend slope of data transfer rate.

[0176] The actual mean compute core utilization reflects the actual average utilization of the compute cores during that phase. The actual peak memory usage represents the actual maximum memory usage during that phase. The actual data transfer rate trend slope reflects the actual change in the data transfer rate during that phase. By extracting these feature vectors, we can more accurately understand the actual resource consumption of the model at different training phases.

[0177] Step S162: extracting a predicted resource requirement feature vector of each training stage from the computing power resource requirement distribution sequence, wherein the predicted resource requirement feature vector includes a predicted value of the number of computing cores, a predicted value of memory bandwidth, and a predicted value of data transmission rate.

[0178] The predicted resource demand feature vector for each training phase is extracted from the computing power resource demand distribution sequence. The predicted resource demand feature vector contains information such as the predicted value of the number of computing cores, the predicted value of memory bandwidth, and the predicted value of data transmission rate.

[0179] These predictions are derived from a computing power demand prediction model and reflect the model's predicted resource demand at different training stages. By extracting these feature vectors, we can compare and analyze them with the actual resource consumption feature vectors.

[0180] Step S163: Calculate the difference between the actual resource consumption feature vector and the corresponding feature component in the predicted resource demand feature vector to generate a resource demand prediction deviation value, wherein the resource demand prediction deviation value includes a computing core deviation value, a memory bandwidth deviation value, and a data transmission rate deviation value.

[0181] The difference between the actual resource consumption feature vector and the corresponding feature component in the predicted resource demand feature vector is calculated to generate the resource demand forecast deviation value. These deviation values ​​include the computing core deviation value, memory bandwidth deviation value, and data transmission rate deviation value.

[0182] The compute core deviation value reflects the difference between actual compute core usage and predicted compute core demand. The memory bandwidth deviation value reflects the difference between actual memory bandwidth usage and predicted memory bandwidth demand. The data transfer rate deviation value represents the difference between actual data transfer rate and predicted data transfer rate. By calculating these deviation values, you can understand the degree of deviation between predicted and actual values.

[0183] Step S164: When the computing core deviation value is positive and the absolute value exceeds the preset deviation threshold, it is determined that the current number of enabled computing nodes is insufficient and the number of enabled computing nodes needs to be increased; when the computing core deviation value is negative and the absolute value exceeds the preset deviation threshold, it is determined that the current number of enabled computing nodes is excessive and the number of enabled computing nodes needs to be reduced.

[0184] The appropriate number of active compute nodes is determined based on the positive and absolute value of the compute core deviation. If the compute core deviation is positive and its absolute value exceeds the preset deviation threshold, it indicates that the actual compute core utilization is higher than the predicted value. This may indicate that the current number of active compute nodes is insufficient and requires an increase.

[0185] If the compute core deviation is negative and its absolute value exceeds the preset deviation threshold, it indicates that the actual compute core utilization is lower than the predicted value. This may indicate that there are too many active compute nodes and that the number of active compute nodes needs to be reduced. The preset deviation threshold is a pre-set value used to determine whether the deviation has reached a level that requires adjustment.

[0186] Step S165: Calculate the adjustment value of the number of enabled computing nodes that needs to be increased or decreased according to the absolute value of the computing core deviation value and the preset adjustment step parameter.

[0187] The amount of compute node activation to increase or decrease is calculated based on the absolute value of the compute core deviation and the preset adjustment step size parameter. The adjustment step size parameter indicates the increment by which the number of compute nodes activated is adjusted each time.

[0188] If the absolute value of the compute core deviation is large, it indicates a significant deviation and requires a significant adjustment to the number of enabled compute nodes. If the absolute value of the compute core deviation is small, it indicates a minor deviation and requires a smaller adjustment to the number of enabled compute nodes. By properly calculating the adjustment value, the number of enabled compute nodes can be brought closer to actual demand.

[0189] Step S166: Apply the adjustment value of the number of enabled computing nodes to the computing power resource sub-plan of the current training phase, update the number of enabled computing nodes, and synchronize the adjusted computing power resource sub-plan to the resource scheduling interface of the computing power service cluster.

[0190] Apply the compute node activation quantity adjustment value to the computing resource sub-plan for the current training phase to update the number of enabled compute nodes. By updating the enabled quantity, the computing resource sub-plan can be made more consistent with actual resource requirements.

[0191] The adjusted computing resource sub-plan is synchronized to the resource scheduling interface of the computing service cluster. The resource scheduling interface will reallocate and manage computing nodes according to the new plan to ensure that the model can obtain appropriate computing resources during subsequent training.

[0192] For example, step S1651: setting a first threshold and a second threshold for calculating the core deviation value, wherein the first threshold is smaller than the second threshold.

[0193] To fine-tune the number of active compute nodes, we set a first and second threshold for the compute core deviation value, with the first threshold being smaller than the second. These two thresholds divide the compute core deviation value into different ranges, and we adopt different adjustment strategies for each range.

[0194] Step S1652: When the absolute value of the calculated core deviation value is less than or equal to the first threshold, it is determined that the deviation is within an acceptable range, and the value of the enabled number of computing nodes is adjusted to zero.

[0195] When the absolute value of the computing core deviation is less than or equal to the first threshold, it indicates that the deviation is within an acceptable range and there is no need to adjust the number of enabled computing nodes. In this case, the adjustment value of the number of enabled computing nodes is set to zero, and the current number of enabled computing nodes remains unchanged.

[0196] Step S1653: When the absolute value of the calculated core deviation value is greater than the first threshold and less than or equal to the second threshold, the preset basic adjustment step parameter is used as the adjustment value of the number of enabled computing nodes.

[0197] If the absolute value of the core computation deviation is greater than the first threshold and less than or equal to the second threshold, the deviation exceeds the acceptable range, but the degree of deviation is not significant. In this case, the preset basic adjustment step size parameter is used as the value for adjusting the number of enabled compute nodes, and a smaller adjustment is made.

[0198] Step S1654: When the absolute value of the calculated core deviation value is greater than the second threshold, the ratio of the absolute value of the deviation value to the second threshold is calculated, and the ratio is rounded up and multiplied by the basic adjustment step parameter to obtain the adjustment value of the number of enabled computing nodes.

[0199] If the absolute value of the compute core deviation exceeds the second threshold, the deviation is significant and requires a larger adjustment. The ratio of the absolute value of the deviation to the second threshold is calculated, rounded up, and multiplied by the basic adjustment step size parameter to obtain the adjustment value for the number of active compute nodes. This approach allows for a reasonable adjustment based on the magnitude of the deviation.

[0200] Step S1655: If the computing core deviation value is positive, the computing node enabled quantity adjustment value is positive, indicating that the number of computing nodes enabled needs to be increased; if the computing core deviation value is negative, the computing node enabled quantity adjustment value is negative, indicating that the number of computing nodes enabled needs to be reduced.

[0201] The sign of the active compute node quantity adjustment value is determined by the positive or negative value of the compute core deviation. If the compute core deviation value is positive, it indicates that the actual compute core utilization is higher than the predicted value, and the number of active compute nodes needs to be increased. In this case, the active compute node quantity adjustment value is positive.

[0202] If the computing core deviation value is negative, it means that the actual computing core utilization is lower than the predicted value. The number of enabled computing nodes needs to be reduced. In this case, the adjustment value of the number of enabled computing nodes is negative.

[0203] Step S1656: Add the calculated adjustment value of the number of enabled computing nodes to the current number of enabled computing nodes to obtain the adjusted number of enabled computing nodes, so that the adjusted number of enabled computing nodes is not less than the preset minimum number of enabled computing nodes and not greater than the total available number of computing nodes of this type in the computing power service cluster.

[0204] Add the calculated adjustment value for the number of enabled compute nodes to the current number of enabled compute nodes to obtain the adjusted number of enabled compute nodes. During the adjustment process, ensure that the adjusted number of enabled compute nodes is no less than the preset minimum number to ensure proper model training. Furthermore, the adjusted number of enabled compute nodes is no greater than the total available number of compute nodes of that type in the computing service cluster to avoid exceeding resource limits. This approach allows you to rationally adjust the number of enabled compute nodes and improve model training efficiency.

[0205] Figure 2 A schematic diagram of exemplary hardware and software components of an AI model training acceleration system 100 based on computing service scheduling, which can implement the concepts of the present application, is shown in some embodiments of the present application. For example, the processor 120 can be used in the AI ​​model training acceleration system 100 based on computing service scheduling and is used to perform the functions of the present application.

[0206] The AI ​​model training acceleration system 100 based on computing power service scheduling can be a general-purpose server or a special-purpose server, both of which can be used to implement the AI ​​model training acceleration method based on computing power service scheduling of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0207] For example, the AI ​​model training acceleration system 100 based on computing power service scheduling may include a network port 110 connected to the network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as disks, ROMs, or RAMs, or any combination thereof. Exemplarily, the AI ​​model training acceleration system 100 based on computing power service scheduling may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The AI ​​model training acceleration system 100 based on computing power service scheduling also includes an I / O interface 150 between the computer and other input and output devices.

[0208] For ease of explanation, only one processor is described in the AI ​​model training acceleration system 100 based on computing power service scheduling. However, it should be noted that the AI ​​model training acceleration system 100 based on computing power service scheduling in this application may also include multiple processors, so the steps performed by one processor described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the AI ​​model training acceleration system 100 based on computing power service scheduling executes steps A and B, it should be understood that steps A and B may also be executed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0209] In addition, an embodiment of the present invention also provides a readable storage medium, which has computer-executable instructions preset in the readable storage medium. When the processor executes the computer-executable instructions, the above-mentioned AI model training acceleration method based on computing power service scheduling is implemented.

[0210] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for accelerating AI model training based on computing power service scheduling, characterized in that: The method comprises: Collect historical training process data of the AI ​​model and the model structure characteristics of the current training task. The historical training process data includes the computing resource consumption sequence and training efficiency indicators of different model structures during the training process. The model structure characteristics include network layer type combinations, inter-layer connection relationships, and parameter scale distribution characteristics; Building a computing power demand prediction model based on the historical training process data, wherein the input of the computing power demand prediction model is the model structure characteristics, and the output is the computing power resource demand distribution sequence at different stages of the training process; Inputting the model structure characteristics of the current training task into the computing power demand prediction model to generate a computing power resource demand distribution sequence for the current training task, wherein the computing power resource demand distribution sequence includes a demand change curve for the number of computing cores, memory bandwidth, and data transmission rate at each stage of training; Filtering a matching computing resource combination solution from the computing service cluster according to the computing resource demand distribution sequence, wherein the computing resource combination solution includes a plurality of computing nodes with different configurations and their enabled quantities in each training stage; Schedule the computing nodes in the computing service cluster to execute the current training task according to the computing resource combination plan, and collect the actual computing resource consumption data during the training process in real time; The actual computing power resource consumption data is compared and analyzed with the computing power resource demand distribution sequence to generate a resource demand prediction deviation value, and the number of enabled computing nodes in the computing power resource combination plan is dynamically adjusted according to the resource demand prediction deviation value.

2. The AI ​​model training acceleration method based on computing power service scheduling according to claim 1 is characterized in that: The collected AI model's historical training process data and the model structure features of the current training task include: Extracting historical training process data corresponding to different model structures from the model training log database. The computing resource consumption sequence in the historical training process data includes computing core utilization, memory usage, and data transmission throughput recorded by timestamp. The training efficiency indicators include the time consumption per iteration, gradient update speed, and number of model convergence iterations. Standardizing the computing power resource consumption sequence in the historical training process data, unifying the resource consumption measurement units under different hardware environments, and generating a standardized resource consumption sequence; Parse the model definition file of the current training task and identify the network layer type combination declared in the model definition file, including the type and arrangement order of convolutional layers, pooling layers, fully connected layers, and activation function layers; Extracting the connection relationship between each layer in the network layer type combination and constructing an inter-layer connection relationship graph, wherein the inter-layer connection relationship graph includes the data flow direction from the input layer to the output layer and the feature mapping dimension change relationship; Count the number of parameters and feature map sizes of each network layer and calculate the parameter scale distribution characteristics, which include the proportion of parameters of each layer to the total parameters and the changing trend of feature map size along the depth of the network; The network layer type combination, inter-layer connection relationship diagram and parameter scale distribution characteristics are packaged to generate the model structure characteristics of the current training task.

3. The AI ​​model training acceleration method based on computing power service scheduling according to claim 1 is characterized in that: The constructing of a computing power demand prediction model based on the historical training process data includes: Performing time series segmentation processing on the standardized resource consumption sequence in the historical training process data, dividing the complete training process of each AI model into an initial convergence phase, a rapid iteration phase, and a fine-tuning phase, with each phase corresponding to a sub-resource consumption sequence; Extract characteristic statistics of each sub-resource consumption sequence to generate a stage resource characteristic vector. The characteristic statistics include sequence mean, peak value, fluctuation variance and trend slope; Associating and storing the model structure features in the historical training process data with the corresponding stage resource feature vectors to construct a target associated data set; Dividing the target association data set into a training set and a validation set, wherein the training set is used for model parameter learning and the validation set is used for model generalization ability evaluation; Construct a deep learning network consisting of an input layer, a feature fusion layer, a time series prediction layer, and an output layer as the basic architecture of the computing power demand prediction model. The number of neurons in the input layer matches the dimension of the model structure characteristics, and the number of neurons in the output layer matches the dimension of the stage resource feature vector. Using the training set to train the deep learning network, adjusting the network weight parameters through a back-propagation algorithm, so as to minimize the mean square error between the predicted stage resource feature vector output by the deep learning network and the actual stage resource feature vector; The prediction performance of the trained deep learning network is evaluated using the validation set. When the prediction mean square error is lower than a preset threshold, the training is stopped to obtain the final computing power demand prediction model.

4. The AI ​​model training acceleration method based on computing power service scheduling according to claim 3 is characterized in that: The deep learning network comprising an input layer, a feature fusion layer, a time series prediction layer, and an output layer is constructed as the basic architecture of the computing power demand prediction model, including: Determine the number of neurons in the input layer, where the number of neurons is equal to the sum of the one-hot encoding dimension of the network layer type combination in the model structure characteristics, the adjacency matrix dimension of the inter-layer connection relationship graph, and the statistic dimension of the parameter scale distribution characteristic; In the feature fusion layer, multiple fully connected sublayers are set. Each fully connected sublayer contains a preset number of neurons. The ReLU activation function is used to perform nonlinear transformation to map the high-dimensional features of the input layer to the low-dimensional fusion feature space. A batch normalization layer is set between the feature fusion layer and the time series prediction layer to standardize the fused feature vector and accelerate the training convergence speed of the deep learning network; The time series prediction layer uses a two-layer bidirectional long short-term memory network structure. Each layer of the long short-term memory network contains a preset number of memory units to capture the long-term dependency of resource requirements and bidirectional time series characteristics during the training process. A dropout layer is set between the two layers of bidirectional long short-term memory networks to prevent overfitting of the deep learning network by randomly discarding some neuron connections; The output layer adopts a fully connected structure, the number of neurons matches the dimension of the stage resource feature vector, and the predicted stage resource feature vector is output through a linear activation function; The weight parameters of the deep learning network are initialized using the Xavier initialization method. The network optimizer is set to the Adam optimizer, the learning rate is the preset initial learning rate, and the training rounds are set to the preset maximum training rounds.

5. The AI ​​model training acceleration method based on computing power service scheduling according to claim 1 is characterized in that: The step of inputting the model structure features of the current training task into the computing power demand prediction model to generate a computing power resource demand distribution sequence of the current training task includes: Perform feature normalization on the model structure features of the current training task to obtain standardized model structure features so that the numerical range of each feature dimension is consistent with the numerical range of the model structure features used when building the computing power demand prediction model; Input the standardized model structure features into the input layer of the computing power demand prediction model, perform nonlinear transformation on the model structure features through the feature fusion layer, and generate a fused feature vector; Input the fused feature vector into the time series prediction layer of the computing power demand prediction model, model the resource demand change trend of each stage of the training process through the long short-term memory network, and output the predicted stage resource feature vector of each stage; Analyzing the prediction stage resource feature vectors of each stage and extracting feature statistics, wherein the feature statistics include the mean of computing core utilization, the peak of memory usage, and the trend slope of data transmission throughput; Calculating the number of computing cores required for each training phase based on the average computing core utilization and a preset single-core computing capacity parameter; Calculate the memory bandwidth and data transfer rate required for each training stage based on the peak memory usage and the data transfer throughput trend slope; The relationship between the number of computing cores, memory bandwidth, and data transmission rate at each stage of training and time is plotted into a curve, which is combined to generate a distribution sequence of computing power resource requirements for the current training task.

6. The AI ​​model training acceleration method based on computing power service scheduling according to claim 1 is characterized in that: The step of screening a matching computing resource combination solution from the computing service cluster according to the computing resource demand distribution sequence includes: Obtain a resource configuration list for the computing service cluster, the resource configuration list including the hardware parameters of each computing node, including computing core type, number of cores, memory capacity, memory bandwidth, and network interface rate; Decomposing the computing power resource demand distribution sequence into an initial convergence phase demand subsequence, a rapid iteration phase demand subsequence, and a fine tuning phase demand subsequence according to the training phase; For each demand subsequence, select a set of candidate computing nodes from the computing service cluster whose hardware parameters meet the requirements for the number of computing cores, memory bandwidth, and data transmission rate at that stage; Performing a performance-cost ratio evaluation on each computing node in the candidate computing node set, where the performance-cost ratio is a ratio of a computing performance index of the node to a resource leasing cost; Selecting a preset number of computing nodes from the candidate computing node set according to the performance-cost ratio to form a computing power resource sub-plan for this stage, wherein the computing power resource sub-plan includes identifiers of the selected computing nodes and the number of enabled nodes; The computing power resource sub-plans of the initial convergence stage, rapid iteration stage, and fine-tuning stage are combined in chronological order to generate a computing power resource combination plan covering the entire training process.

7. The AI ​​model training acceleration method based on computing power service scheduling according to claim 6 is characterized in that: For each demand subsequence, a set of candidate computing nodes whose hardware parameters meet the requirements of the number of computing cores, memory bandwidth, and data transmission rate at that stage are screened from the computing power service cluster, including: Parse the requirement subsequence to extract the minimum required values ​​for the number of computing cores, the minimum required value for memory bandwidth, and the minimum required value for data transmission rate in this training phase; Traverse the resource configuration list of the computing power service cluster and read the hardware parameters of each computing node, including the number of computing cores, memory bandwidth, and network interface speed; For each computing node, determine whether the number of computing cores is greater than or equal to the minimum required number of computing cores, whether the memory bandwidth is greater than or equal to the minimum required memory bandwidth, and whether the network interface rate is greater than or equal to the minimum required data transmission rate; The computing nodes that meet the above three conditions are marked as candidate nodes, and the identifiers and complete hardware parameters of all computing nodes marked as candidate nodes are collected to form the candidate computing node set for this training phase; The candidate computing node set is sorted in descending order by the number of computing cores, and when the number of computing cores is the same, it is sorted in descending order by memory bandwidth to generate an ordered list of candidate computing nodes.

8. The AI ​​model training acceleration method based on computing power service scheduling according to claim 1 is characterized in that: The computing nodes in the computing service cluster are scheduled to execute the current training task according to the computing resource combination scheme, and the actual computing resource consumption data during the training process is collected in real time, including: Analyze the computing resource combination plan and extract the computing node identification, enabled number and stage start timestamp corresponding to each training stage; Before the initial convergence phase start timestamp arrives, a resource reservation request is sent to the resource scheduling interface of the computing power service cluster. The resource reservation request includes the identifiers and the number of enabled computing nodes required for the initial convergence phase. When a training task is started, the reserved computing nodes are activated through the resource scheduling interface, and the model parameters and initial training data of the current training task are distributed to each computing node; Each computing node performs model training operations according to the preset distributed training protocol. During the training process, the node's built-in resource monitoring module collects actual computing power resource consumption data in real time. The actual computing power resource consumption data includes computing core utilization, memory usage, and data transmission rate. The actual computing power resource consumption data is uploaded to the central monitoring node after being marked with a timestamp, and the central monitoring node aggregates and stores the received actual computing power resource consumption data to form an actual computing power resource consumption sequence; When the training process enters the rapid iteration stage and fine-tuning stage, resource reservation, node activation, and data collection operations are repeated to ensure that the computing nodes in each stage are enabled on time and complete actual computing resource consumption data is collected.

9. The AI ​​model training acceleration method based on computing power service scheduling according to claim 1 is characterized in that: The comparing and analyzing the actual computing resource consumption data with the computing resource demand distribution sequence to generate a resource demand prediction deviation value, and dynamically adjusting the number of enabled computing nodes in the computing resource combination solution according to the resource demand prediction deviation value, includes: Extracting an actual resource consumption feature vector of each training stage from the actual computing power resource consumption data, wherein the actual resource consumption feature vector includes an actual mean value of computing core utilization, an actual peak value of memory usage, and an actual trend slope of data transmission rate; Extracting a predicted resource requirement feature vector for each training stage from the computing power resource requirement distribution sequence, the predicted resource requirement feature vector including a predicted value of the number of computing cores, a predicted value of memory bandwidth, and a predicted value of data transmission rate; Calculating the difference between the actual resource consumption feature vector and the corresponding feature components in the predicted resource demand feature vector to generate a resource demand prediction deviation value, wherein the resource demand prediction deviation value includes a computing core deviation value, a memory bandwidth deviation value, and a data transmission rate deviation value; When the computing core deviation value is positive and the absolute value exceeds the preset deviation threshold, it is determined that the number of currently enabled computing nodes is insufficient and the number of enabled computing nodes needs to be increased; When the computing core deviation value is negative and the absolute value exceeds the preset deviation threshold, it is determined that the number of currently enabled computing nodes is excessive and the number of enabled computing nodes needs to be reduced; Calculate the adjustment value of the number of enabled computing nodes that needs to be increased or decreased according to the absolute value of the computing core deviation value and a preset adjustment step parameter; The adjustment value of the number of enabled computing nodes is applied to the computing power resource sub-plan of the current training phase, the number of enabled computing nodes is updated, and the adjusted computing power resource sub-plan is synchronized to the resource scheduling interface of the computing power service cluster.

10. An AI model training acceleration system based on computing power service scheduling, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the AI ​​model training acceleration method based on computing power service scheduling as described in any one of claims 1 to 9.

Images