A method and system for dynamic resource allocation of computing power services applied to AI model training

Abstract

The application provides a kind of computing power service dynamic resource allocation method and system applied to AI model training, first, the real-time computing power resource use data (including computing node load, memory occupation, data transmission delay) and model training state data (including training task stage identifier, model parameter update frequency, training data batch processing progress) in AI model training are collected, then the correlation characteristics of two types of data are extracted, the computing power resource demand correlation characteristic set is generated, the computing power resource dynamic allocation decision model including resource allocation priority judgment, adjustment amplitude calculation, scheduling time selection unit is constructed based on the set, the computing power resource allocation scheme (including computing node number, memory capacity, data transmission bandwidth adjustment instruction) is output through the model, the resource is scheduled according to the scheme, and the characteristic set is updated by collecting new data, so as to realize the dynamic accurate allocation of computing power resource, improve the resource utilization and training efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a computing power service dynamic resource allocation method and system applied to AI model training. BACKGROUND

[0002] In the process of rapid development of artificial intelligence technology, the demand for computing power resources for AI model training is increasingly large and complex. Currently, AI model training is usually carried out in a computing power service cluster environment, but the existing computing power resource allocation method has many limitations.

[0003] The traditional computing power resource allocation method mostly adopts a static allocation strategy, that is, a fixed number of computing nodes, memory capacity and data transmission bandwidth are allocated to the training task according to experience or preliminary estimation before the training starts. The above allocation method cannot be dynamically adjusted according to the actual situation in the training process. In the training process, the demand for computing power resources is significantly different at different training stages. For example, in the model parameter initialization stage, the demand for computing resources may be low, while in the key stage of model convergence, the demand for computing resources and memory resources will increase sharply. At the same time, the loading and processing speed of training data will also affect the demand for network resources. The existing static allocation method cannot timely perceive these changes, resulting in insufficient computing power in the stage of high resource demand, affecting the training efficiency and model quality; while in the stage of low resource demand, the computing power resources are idle and wasted, increasing the training cost. SUMMARY

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present application, the embodiments of the present application provide a computing power service dynamic resource allocation method applied to AI model training, which comprises:

[0005] Collecting real-time computing power resource usage data and model training state data in the AI model training process, the real-time computing power resource usage data including computing node load condition, memory occupation level and data transmission delay condition, and the model training state data including training task stage identifier, model parameter update frequency and training data batch processing progress;

[0006] Performing associated feature extraction on the real-time computing power resource usage data and the model training state data to generate a computing power resource demand associated feature set, the computing power resource demand associated feature set including computing resource demand associated features, storage resource demand associated features and network resource demand associated features;

[0007] Constructing a computing power resource dynamic allocation decision model based on the computing power resource demand associated feature set, the computing power resource dynamic allocation decision model including a resource allocation priority judgment unit, a resource adjustment amplitude calculation unit and a resource scheduling timing selection unit;

[0008] outputting, by the computing power resource dynamic allocation decision model, a computing power resource allocation scheme, the computing power resource allocation scheme comprising a computing node quantity adjustment instruction, a memory capacity allocation instruction, and a data transmission bandwidth adjustment instruction;

[0009] performing a computing power resource scheduling operation according to the computing power resource allocation scheme, and collecting resource-scheduled computing power resource usage data and model training progress data as new real-time computing power resource usage data and new model training state data for updating the computing power resource demand correlation feature set.

[0010] In still another aspect, the embodiments of the present application also provide a computing power service dynamic resource allocation system applied to AI model training, comprising a processor, a machine-readable storage medium, the machine-readable storage medium being connected with the processor, the machine-readable storage medium being used for storing programs, instructions or codes, and the processor being used for executing the programs, instructions or codes in the machine-readable storage medium to realize the above-mentioned method.

[0011] Based on the above aspects, the embodiments of the present application comprehensively and accurately grasp the actual consumption of computing power resources and the training progress of the model in the training process by collecting real-time computing power resource usage data and model training state data in the AI model training process, and then extract correlation features from the two types of data to generate a computing power resource demand correlation feature set that can accurately reflect the demand change law of computing power resources from multiple dimensions such as computing, storage and network, so that resource allocation is more targeted and scientific. The computing power resource dynamic allocation decision model constructed based on the feature set integrates resource allocation priority judgment, resource adjustment amplitude calculation and resource scheduling timing selection functions, and can quickly and accurately make resource allocation decisions according to real-time data, ensure that the number of computing nodes, memory capacity and data transmission bandwidth are adjusted at the right time and with the right amplitude, perform scheduling operations according to the computing power resource allocation scheme output by the decision model, and update the feature set with new data after scheduling, forming a closed-loop dynamic adjustment mechanism that can continuously adapt to changes in computing power resource demand in the training process, effectively improving the utilization rate of computing power resources, shortening the training time of AI models, reducing training costs, and improving the training quality and stability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is an execution flow schematic diagram of the computing power service dynamic resource allocation method applied to AI model training provided by the embodiments of the present application.

[0013] Figure 2 is a schematic diagram of exemplary hardware and software components of the computing power service dynamic resource allocation system applied to AI model training provided by the embodiments of the present application. DETAILED DESCRIPTION

[0014] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is a flowchart illustrating a dynamic resource allocation method for computing power services applied to AI model training, provided by an embodiment of the present invention. The following is a detailed description of this dynamic resource allocation method for computing power services applied to AI model training.

[0015] Step S110: Collect real-time computing resource usage data and model training status data during the AI ​​model training process. The real-time computing resource usage data includes the computing node load, memory usage level, and data transmission latency. The model training status data includes the training task stage identifier, model parameter update frequency, and training data batch processing progress.

[0016] In real-world AI model training scenarios, in order to achieve dynamic resource allocation of computing power services, it is necessary to collect relevant real-time computing power resource usage data and model training status data.

[0017] For real-time computing resource usage data, the load of computing nodes is a crucial indicator of the workload of each node. Different computing nodes exhibit varying load states due to differences in the complexity of their computational tasks and hardware performance. For example, some computing nodes may be responsible for handling complex convolution operations related to image recognition, which require a large amount of floating-point calculations, resulting in a higher load for these nodes; while other computing nodes may only perform simple data preprocessing tasks, leading to a relatively lower load.

[0018] Memory usage reflects the extent to which system memory resources are utilized. During AI model training, model parameters, intermediate calculation results, and training data all need to be stored in memory. If memory usage is too high, it may lead to insufficient memory, affecting the normal progress of training and even causing program crashes. For example, when training a large-scale deep learning model, the number of parameters is enormous, requiring a large amount of memory for storage, and the memory usage level will increase significantly.

[0019] Data transmission latency reflects the time spent transmitting data between different components. In a distributed training environment, data needs to be transmitted between multiple computing nodes. If the data transmission latency is too high, it will lead to synchronization problems during the training process, reducing the efficiency of collaboration between computing nodes and thus affecting the overall training efficiency. For example, in a cluster composed of multiple computing nodes, when a computing node completes a part of the computation task, it needs to transmit the result to other nodes for further processing. If the data transmission latency is too high, other nodes will have to wait a long time before they can continue working.

[0020] For model training state data, the training task phase identifier identifies the current stage of training. Different training stages have different demands for computing resources. In the initialization stage of the model, the main task is to randomly initialize the parameters of the model, and the demand for computing resources is relatively low; in the training stage of the model, a large number of forward propagation and back propagation calculations are required, and the demand for computing resources will increase significantly; in the evaluation stage of the model, the performance of the model needs to be evaluated, and there is a certain requirement for computing resources and data transmission resources.

[0021] The model parameter update frequency reflects the frequency of parameter changes in the training process. High update frequency means that the model is constantly learning and adjusting, and requires more computing resources to support. For example, in the early stage of training, the parameters of the model may need to be updated frequently to quickly converge to a better state, so the model parameter update frequency will be relatively high; as the training progresses, the model gradually converges, and the parameter update frequency will gradually decrease.

[0022] The training data batch processing progress shows the processing progress of the current training data batch. In AI model training, the training data is usually divided into multiple batches for processing, and after the data processing of each batch is completed, the parameters of the model are updated. By understanding the training data batch processing progress, it can be determined whether the training is progressing normally and whether resource allocation needs to be adjusted. For example, if a batch of data processing takes too long, it may be due to the large amount of data in that batch or insufficient computing resources, so resource allocation needs to be adjusted accordingly.

[0023] In order to collect the above data, for the computing node load situation, the CPU usage rate, GPU usage rate and other indicators can be monitored in real time by installing monitoring software on each computing node; for memory occupancy level, the memory usage can be obtained through the system's memory management tool; for data transmission delay condition, the data transmission time between different nodes can be measured through network monitoring tools. For model training state data, information such as training task phase identifier, model parameter update frequency and training data batch processing progress can be obtained from the training framework. The collected data will be stored in a data warehouse for subsequent processing and analysis.

[0024] Step S120: Perform correlation feature extraction on the real-time computing resource usage data and the model training state data to generate a computing resource demand correlation feature set, which includes computing resource demand correlation features, storage resource demand correlation features, and network resource demand correlation features.

[0025] After collecting real-time computing resource usage data and model training state data, it is necessary to extract associated features from the real-time computing resource usage data and model training state data. The purpose is to find the internal relationship between the real-time computing resource usage data and the model training state data, so as to more accurately reflect the demand of AI model training for different types of computing resources.

[0026] Step S121: Time series division is performed on the real-time computing resource usage data. The real-time computing resource usage data is divided into a plurality of continuous computing data windows according to a fixed time interval. Each computing data window contains a computing node load change sequence, a memory occupancy level change sequence, and a data transmission delay condition change sequence in the corresponding time interval.

[0027] Time series division of real-time computing resource usage data is an important step of associated feature extraction. By dividing the data into a plurality of continuous computing data windows according to a fixed time interval, the continuous time series data can be discretized, which is convenient for subsequent analysis and processing.

[0028] When dividing the computing data window, a suitable time interval needs to be selected. The selection of the time interval should be determined according to the specific application scenario and the characteristics of the data. If the time interval is too small, the number of computing data windows will be too large, increasing the complexity of subsequent processing. If the time interval is too large, some important information may be lost, and the change of the data cannot be accurately reflected.

[0029] Each computing data window contains a computing node load change sequence, a memory occupancy level change sequence, and a data transmission delay condition change sequence in the corresponding time interval. The computing node load change sequence records the change of the load of each computing node in the time interval. By analyzing the node load change sequence, the fluctuation of the computing node load can be understood, and whether there is a sudden high load condition can be judged. For example, in a certain time window, the load of a certain computing node suddenly increases, which may be because the node receives a complex computing task, or the system has an abnormal condition.

[0030] The memory occupancy level change sequence reflects the change trend of memory usage in the time interval. By analyzing the memory occupancy level change sequence, the use of memory can be predicted, and the allocation of memory resources can be prepared in advance. For example, if it is found that the memory occupancy level continuously increases in a certain time window, it may be necessary to increase the memory resources or optimize the memory usage strategy in time.

[0031] The data transmission delay condition change sequence records the change of data transmission delay in the time interval. By analyzing the data transmission delay condition change sequence, the regularity of data transmission delay can be found out, and the data transmission strategy can be optimized. For example, if it is found that the data transmission delay appears periodic fluctuation in a certain time window, it may be caused by the limitation of network bandwidth or the problem of data transmission protocol, at this time, network configuration can be considered to be adjusted or data transmission protocol can be optimized.

[0032] Step S122: Phase feature labeling is performed on the model training state data, and the model training state data is divided into multiple training phase data units according to the training task phase identifier, and each training phase data unit contains the model parameter update frequency and training data batch processing progress of the corresponding training phase.

[0033] The phase feature labeling of the model training state data is to divide the training process into different phases in order to better analyze the demand for computing resources in each phase. According to the training task phase identifier, the model training state data can be divided into multiple training phase data units.

[0034] Each training phase data unit contains the model parameter update frequency and training data batch processing progress of the corresponding training phase. The model parameter update frequency may be different in different training phases. In the initial training phase of the model, in order to enable the model to quickly converge to a better state, a higher model parameter update frequency is usually required; as the training proceeds, the model gradually converges, and the parameter update frequency gradually decreases. By analyzing the model parameter update frequency in different training phases, the learning speed and convergence of the model can be understood.

[0035] The training data batch processing progress reflects the processing progress of the training data in this training phase. Different training phases may have different data processing needs. In the pre-training phase of the model, a large amount of initial data may need to be processed to enable the model to learn some basic features; while in the fine-tuning phase of the model, only a small amount of specific data may need to be processed to further optimize the model. By analyzing the training data batch processing progress, the order and resource allocation of data processing can be reasonably arranged.

[0036] Step S123: Time alignment processing is performed on the computing power data window and the training phase data unit, so that each training phase data unit corresponds to at least one computing power data window, and the time coverage of the two remains consistent.

[0037] In order to accurately analyze the correlation between real-time computing resource usage data and model training state data, it is necessary to perform time alignment processing on the computing power data window and the training phase data unit. The purpose of time alignment processing is to ensure that each training phase data unit can correspond to a corresponding computing power data window, so that the resource requirements of the training phase can be matched with the actual resource usage.

[0038] When performing time alignment processing, it is necessary to ensure that each training phase data unit corresponds to at least one computing power data window, and the time coverage of the two is consistent, so as to ensure that when analyzing the resource requirements of each training phase, the actual resource usage in the phase can be referred to. For example, if a training phase lasts for a long time, it may correspond to multiple computing power data windows. By analyzing these computing power data windows, the resource requirements of the training phase can be more accurately understood.

[0039] Step S124: calculating a first correlation coefficient between the time-aligned training phase data unit and the computing node load condition in the computing power data window, a second correlation coefficient between the time-aligned training phase data unit and the memory occupation level in the computing power data window, and a third correlation coefficient between the time-aligned training phase data unit and the data transmission delay condition in the computing power data window.

[0040] Step S1241: calculating a first correlation coefficient between the time-aligned training phase data unit and the computing node load condition in the computing power data window.

[0041] To calculate the first correlation coefficient, the relevant data features need to be extracted first. Specifically, the model parameter update frequency is extracted from the training phase data unit, and its average value in the corresponding training phase is calculated to obtain the parameter update frequency average, because the model parameter update frequency reflects the frequency of parameter changes in the training process, and taking the average value can more stably reflect the overall update of the phase. For example, in a training phase, the model parameter update frequency may fluctuate, but by calculating the average value, a relatively stable index can be obtained.

[0042] At the same time, the computing node load condition is extracted from the computing power data window, and its maximum value in the corresponding computing power data window is calculated to obtain the node load rate peak. The computing node load condition may change at different times, and taking the maximum value can reflect the highest level of computing node load in the computing power data window. For example, in a certain computing power data window, the load of the computing node may be low at some time, but it reaches a high value at a certain time, and this maximum value can reflect the maximum pressure that the computing node may face in that time period.

[0043] Since the average parameter update frequency and the peak node load rate may have different data dimensions, data standardization is necessary to ensure calculation accuracy, bringing them into the same data dimension range. Standardization eliminates the dimensional differences between different data characteristics, avoiding calculation errors caused by these differences. For example, if the average parameter update frequency ranges from 0 to 100, while the peak node load rate ranges from 0 to 1, direct calculation without standardization may lead to inaccurate results.

[0044] After standardization, the product of the average standardized parameter update frequency and the peak standardized node load rate is calculated to obtain a preliminary correlation value. This preliminary correlation value reflects a preliminary degree of correlation between the model parameter update frequency and the computing node load.

[0045] To further refine the initial correlation value, a correlation coefficient correction factor needs to be set. This correction factor is positively correlated with the duration of the data units during the training phase. This is because the longer the training phase lasts, the more complex the correlation between model parameter updates and computation node load may become, requiring appropriate correction. Multiplying the initial correlation value by the correlation coefficient correction factor yields the first correlation coefficient.

[0046] Step S1242: Calculate the second correlation coefficient between the time-aligned training phase data unit and the memory usage level in the computing power data window.

[0047] Calculating the second correlation coefficient requires analyzing the relationship between the progress of training data batch processing within the training phase data unit and the memory usage level in the computing power data window. Different training data batch processing progresses may have different impacts on memory usage. For example, processing a large batch may require more memory to store the data and intermediate calculation results, thus increasing memory usage.

[0048] We can statistically analyze the changes in memory usage at different stages of the training data batch processing progress. For example, we can divide the training data batch processing progress into different stages, such as the start stage, the middle stage, and the end stage, and statistically analyze the average and maximum memory usage at each stage. Then, we can calculate the correlation between these characteristics and the training data batch processing progress to obtain the correlation coefficient, i.e., the second correlation coefficient.

[0049] Step S1243: Calculate the third correlation coefficient between the time-aligned training phase data unit and the data transmission latency status in the computing power data window.

[0050] When calculating the third correlation coefficient, the relationship between the training task phase identifier in the training phase data unit and the data transmission delay condition in the computing power data window is analyzed. Different training task phases may have different requirements for data transmission, resulting in differences in data transmission delay conditions. For example, during the training phase of the model, gradient information may need to be frequently transmitted between computing nodes, at which time the real-time requirement for data transmission is higher, and if the data transmission delay is too high, the efficiency of training will be affected; while in the evaluation phase of the model, the real-time requirement for data transmission is relatively low.

[0051] By analyzing the training task phase identifier and the data transmission delay condition in detail, the internal relationship between them is found out. According to the different training task phases, the average value, fluctuation range and other characteristics of the data transmission delay in the corresponding phase can be counted, and then the correlation coefficient between the two, i.e. the third correlation coefficient, is calculated.

[0052] Step S125: determining the first correlation coefficient as the computing resource demand correlation feature, the second correlation coefficient as the storage resource demand correlation feature, and the third correlation coefficient as the network resource demand correlation feature, and combining to form a computing power resource demand correlation feature set.

[0053] After calculating the first correlation coefficient, the second correlation coefficient and the third correlation coefficient, they are respectively determined as the computing resource demand correlation feature, the storage resource demand correlation feature and the network resource demand correlation feature, because the first correlation coefficient reflects the correlation between model parameter update and computing node load in the model training process, which is closely related to the demand for computing resources; the second correlation coefficient reflects the correlation between training data batch processing progress and memory occupation level, which is related to the demand for storage resources; and the third correlation coefficient reflects the correlation between the training task phase and the data transmission delay condition, which is related to the demand for network resources.

[0054] The computing resource demand correlation feature, the storage resource demand correlation feature and the network resource demand correlation feature are combined together to form a computing power resource demand correlation feature set. The computing power resource demand correlation feature set contains the demand information of AI model training for computing resources, storage resources and network resources.

[0055] Step S130: constructing a computing power resource dynamic allocation decision model based on the computing power resource demand correlation feature set, wherein the computing power resource dynamic allocation decision model comprises a resource allocation priority judgment unit, a resource adjustment amplitude calculation unit and a resource scheduling timing selection unit.

[0056] Step S131: Determine the input layer parameters of the computing resource dynamic allocation decision model as the computing resource demand associated features, the storage resource demand associated features, and the network resource demand associated features in the computing resource demand associated feature set.

[0057] The input layer parameters of the computing resource dynamic allocation decision model are directly related to whether the model can accurately make resource allocation decisions. The computing resource demand associated features, the storage resource demand associated features, and the network resource demand associated features in the computing resource demand associated feature set are used as input layer parameters because these features comprehensively reflect the demand of AI model training for different types of computing resources. By inputting these features into the model, the model can determine the demand priority, adjustment range, and scheduling timing of different types of resources based on this information.

[0058] Step S132: Set the judgment rule of the resource allocation priority judgment unit, which includes that when the computing resource demand associated feature is higher than the first preset threshold, the computing resource allocation priority is the highest, when the storage resource demand associated feature is higher than the second preset threshold, the storage resource allocation priority is the highest, and when the network resource demand associated feature is higher than the third preset threshold, the network resource allocation priority is the highest.

[0059] The function of the resource allocation priority judgment unit is to determine which type of computing resource has the highest allocation priority under the current situation. To achieve this function, the corresponding judgment rule needs to be set. By setting the first preset threshold, the second preset threshold, and the third preset threshold, the computing resource demand associated features, the storage resource demand associated features, and the network resource demand associated features can be compared.

[0060] When the computing resource demand associated feature is higher than the first preset threshold, it indicates that the current model training has a very urgent demand for computing resources, and the computing resource allocation priority is the highest. For example, if at a certain moment, the value of the computing resource demand associated feature greatly exceeds the first preset threshold, it may mean that the model is in a training phase that requires a large amount of computation, such as large-scale matrix operations, etc., and at this time, computing resources need to be allocated preferentially.

[0061] When the storage resource demand associated feature is higher than the second preset threshold, it indicates that the current demand for storage resources is high, and the storage resource allocation priority is the highest. For example, when the training data volume increases significantly, more memory is needed to store data and model parameters, and the storage resource demand associated feature may exceed the second preset threshold at this time, and storage resources need to be allocated preferentially.

[0062] When the network resource demand correlation feature is higher than the third preset threshold, it indicates that the current demand for network resources is large, and the network resource allocation priority is the highest. For example, in a distributed training environment, if data needs to be frequently transmitted between multiple computing nodes, the network resource demand correlation feature may exceed the third preset threshold, at which time network resources need to be allocated in priority.

[0063] Step S133: Configure the calculation logic of the resource adjustment amplitude calculation unit, which is to determine the resource adjustment amplitude according to the difference ratio of the computing resource demand correlation feature and the corresponding preset threshold. The greater the difference ratio, the greater the resource adjustment amplitude.

[0064] Step S1331: Determine the calculation method of the computing resource adjustment amplitude, which is to divide the difference between the computing resource demand correlation feature and the first preset threshold by the first preset threshold, and then multiply the result by a reference adjustment coefficient to obtain the computing resource adjustment amplitude.

[0065] For the computing resource adjustment amplitude, divide the difference between the computing resource demand correlation feature and the first preset threshold by the first preset threshold to obtain a difference ratio. The difference ratio reflects the deviation between the computing resource demand correlation feature and the preset threshold. For example, if the value of the computing resource demand correlation feature is much larger than the first preset threshold, the difference ratio will be larger, indicating that the current demand for computing resources far exceeds the preset level.

[0066] Multiply the difference ratio by a reference adjustment coefficient to obtain the computing resource adjustment amplitude. The reference adjustment coefficient is a pre-set coefficient that can be adjusted according to the specific application scenario and the resource situation of the system. By multiplying the reference adjustment coefficient, the computing resource adjustment amplitude can be appropriately scaled to ensure that the adjustment amplitude is within a reasonable range.

[0067] Step S1332: Determine the calculation method of the storage resource adjustment amplitude, which is to divide the difference between the storage resource demand correlation feature and the second preset threshold by the second preset threshold, and then multiply the result by a reference adjustment coefficient to obtain the storage resource adjustment amplitude.

[0068] Similar to the calculation method of the computing resource adjustment amplitude, for the storage resource adjustment amplitude, divide the difference between the storage resource demand correlation feature and the second preset threshold by the second preset threshold to obtain the corresponding difference ratio. The difference ratio reflects the deviation between the storage resource demand correlation feature and the preset threshold.

[0069] Multiply the difference ratio by a reference adjustment coefficient to obtain the storage resource adjustment amplitude. Similarly, the reference adjustment coefficient can appropriately scale the storage resource adjustment amplitude to ensure that the adjustment amplitude conforms to the actual situation of the system.

[0070] Step S1333: The calculation method of the network resource adjustment range is that the difference between the network resource demand correlation feature and the third preset threshold is divided by the third preset threshold, and then multiplied by a reference adjustment coefficient to obtain the network resource adjustment range.

[0071] For the network resource adjustment range, the difference between the network resource demand correlation feature and the third preset threshold is divided by the third preset threshold to obtain a corresponding difference ratio. The difference ratio reflects the difference degree between the network resource demand correlation feature and the preset threshold.

[0072] The difference ratio is multiplied by the reference adjustment coefficient to obtain the network resource adjustment range. In this way, the allocation of network resources can be reasonably adjusted according to the actual demand of network resources.

[0073] Step S1334: The value range of the reference adjustment coefficient is set, and the value range is determined according to the total resource quota of the AI model training task.

[0074] The value range of the reference adjustment coefficient needs to be determined according to the total resource quota of the AI model training task. The total resource quota refers to the total amount of all resources allocated by the system for the training task, including computing resources, storage resources, and network resources. If the total resource quota is sufficient, the reference adjustment coefficient can be appropriately large, so that a large adjustment can be made when the resource demand is large. If the total resource quota is limited, the reference adjustment coefficient needs to be small to avoid excessive adjustment leading to unreasonable resource allocation.

[0075] Step S1335: When the computing resource demand correlation feature is less than or equal to the corresponding preset threshold, the corresponding resource adjustment range is set to zero.

[0076] If the computing resource demand correlation feature is less than or equal to the corresponding preset threshold, it means that the current demand for this type of resource is within the normal range, and resource adjustment is not needed. For example, if the computing resource demand correlation feature is less than or equal to the first preset threshold, the computing resource does not need to be adjusted, and the computing resource adjustment range is set to zero at this time.

[0077] Step S1336: When the computing resource adjustment range, the storage resource adjustment range, or the network resource adjustment range is greater than the maximum adjustment threshold, it is limited to the maximum adjustment threshold.

[0078] To avoid the adverse effects of large resource adjustment range on the system, a maximum adjustment threshold needs to be set. If the calculation resource adjustment range, the storage resource adjustment range, or the network resource adjustment range is greater than the maximum adjustment threshold, it is limited to the maximum adjustment threshold. For example, if the calculation resource adjustment range exceeds the maximum adjustment threshold, it may cause the calculation resource allocation of the system to be too concentrated, affecting the normal operation of other tasks, and at this time the calculation resource adjustment range needs to be limited within the maximum adjustment threshold.

[0079] Step S1337: When the calculation resource adjustment range, the storage resource adjustment range, or the network resource adjustment range is less than the minimum adjustment threshold, it is limited to the minimum adjustment threshold.

[0080] Similarly, to ensure that the resource adjustment can achieve a certain effect, a minimum adjustment threshold needs to be set. If the calculation resource adjustment range, the storage resource adjustment range, or the network resource adjustment range is less than the minimum adjustment threshold, it is limited to the minimum adjustment threshold. For example, if the storage resource adjustment range is too small, it may not be able to meet the actual demand of model training for storage resources, and at this time limiting it to the minimum adjustment threshold can ensure a certain adjustment effect.

[0081] Step S134: Set the selection condition of the resource scheduling timing selection unit, which includes triggering the resource scheduling operation when the model parameter update frequency is in the stable interval, and triggering the resource scheduling operation when the training data batch processing progress completes the set proportion of the current batch.

[0082] The role of the resource scheduling timing selection unit is to determine when to perform the resource scheduling operation. The set selection condition can ensure that the resource scheduling is performed at the right time to improve the efficiency of resource allocation.

[0083] Triggering the resource scheduling operation when the model parameter update frequency is in the stable interval is because when the model parameter update frequency is stable, it means that the model training is in a relatively stable state, and at this time performing resource scheduling can avoid causing too much interference to the training process. For example, if resource scheduling is performed when the model parameter update frequency fluctuates greatly, it may cause instability in training, affecting the convergence speed and performance of the model.

[0084] Triggering the resource scheduling operation when the training data batch processing progress completes the set proportion of the current batch is because after completing a certain proportion of batch processing, the demand for resources may change. For example, after completing most of the processing of the current batch, more calculation resources may be needed for the final result aggregation and parameter update, and at this time the resource scheduling operation can be triggered to allocate resources reasonably.

[0085] Step S135: The resource allocation priority judgment unit, the resource adjustment range calculation unit, and the resource scheduling time selection unit are connected in series to form a computing power resource dynamic allocation decision model, wherein the output of the resource allocation priority judgment unit is the input of the resource adjustment range calculation unit, and the output of the resource adjustment range calculation unit is the input of the resource scheduling time selection unit.

[0086] The resource allocation priority judgment unit, the resource adjustment range calculation unit, and the resource scheduling time selection unit are connected in series to form a complete computing power resource dynamic allocation decision model. The resource allocation priority judgment unit first determines the allocation priority of different types of resources according to the input computing power resource demand correlation characteristics, and the output result is the input of the resource adjustment range calculation unit.

[0087] The resource adjustment range calculation unit calculates the adjustment range of different types of resources according to the output of the resource allocation priority judgment unit and the preset calculation logic. Then, the output of the resource adjustment range calculation unit is the input of the resource scheduling time selection unit.

[0088] The resource scheduling time selection unit determines when to perform resource scheduling operation according to the resource adjustment range and the set selection conditions. Through the above series connection mode, each unit cooperates with each other to complete the dynamic allocation decision of computing power resources.

[0089] Step S140: Output a computing power resource allocation scheme through the computing power resource dynamic allocation decision model, wherein the computing power resource allocation scheme includes a computing node number adjustment instruction, a memory capacity allocation instruction, and a data transmission bandwidth adjustment instruction.

[0090] Step S141: Input the computing power resource demand correlation characteristic set into the resource allocation priority judgment unit of the computing power resource dynamic allocation decision model, and determine the current computing power resource allocation priority order through the judgment rule of the resource allocation priority judgment unit.

[0091] After inputting the computing power resource demand correlation characteristic set into the resource allocation priority judgment unit of the computing power resource dynamic allocation decision model, the unit compares the computing resource demand correlation characteristics, the storage resource demand correlation characteristics, and the network resource demand correlation characteristics with the corresponding preset threshold according to the previously set judgment rule.

[0092] If the correlation characteristic of computing resource demand is higher than the first preset threshold, then computing resources are allocated with the highest priority; if the correlation characteristic of storage resource demand is higher than the second preset threshold, then storage resources are allocated with the highest priority; if the correlation characteristic of network resource demand is higher than the third preset threshold, then network resources are allocated with the highest priority. Through the above comparisons and judgments, the current priority order of computing resource allocation is determined. For example, at a certain moment, if the correlation characteristic of computing resource demand is higher than the first preset threshold, while the correlation characteristics of storage resource demand and network resource demand are both lower than their respective preset thresholds, then the priority order of computing resource allocation at this time is computing resources first, followed by storage resources and then network resources.

[0093] Step S142: Based on the current priority order of computing power resource allocation, input the corresponding computing power resource demand association features into the resource adjustment range calculation unit, and calculate the adjustment range value of each type of computing power resource through the calculation logic of the resource adjustment range calculation unit.

[0094] After determining the current priority order of computing power resource allocation, the corresponding computing power resource demand correlation characteristics are input into the resource adjustment range calculation unit. This unit determines the resource adjustment range based on the previously configured calculation logic, that is, the ratio of the difference between the computing power resource demand correlation characteristics and the corresponding preset threshold.

[0095] For computing resources, the computational resource demand correlation characteristics are compared with a first preset threshold, the difference ratio is calculated, and then multiplied by a benchmark adjustment coefficient to obtain the computational resource adjustment range value. For storage and network resources, the adjustment range value is calculated using their respective methods. For example, if the current computational resource allocation priority is the highest, the computational resource demand correlation characteristics are input into the resource adjustment range calculation unit to calculate the computational resource adjustment range value.

[0096] Step S143: Input the adjustment range value into the resource scheduling timing selection unit, and determine the scheduling execution time point of each type of computing power resource in combination with the selection conditions of the resource scheduling timing selection unit.

[0097] The calculated adjustment values ​​for each type of computing power resource are input into the resource scheduling timing selection unit. This unit, combined with previously set selection conditions—that is, triggering resource scheduling operations when the model parameter update frequency is within a stable range, and triggering resource scheduling operations when the training data batch processing has completed the set proportion of the current batch—determines the scheduling execution time points for each type of computing power resource.

[0098] For example, if the current model parameter update frequency is in the stable interval, and the calculation resource adjustment amplitude value is large, it can be determined that the scheduling operation of the calculation resource is performed at this time. Similarly, for storage resources and network resources, the scheduling execution time point is also determined according to the corresponding conditions and adjustment amplitude values.

[0099] Step S144: Based on the current computing resource allocation priority order, the adjustment amplitude value and the scheduling execution time point, generate a calculation node quantity adjustment instruction, a memory capacity allocation instruction and a data transmission bandwidth adjustment instruction, wherein the calculation node quantity adjustment instruction contains a target calculation node quantity and a node increase / decrease direction, the memory capacity allocation instruction contains a target capacity value and a capacity adjustment rate, and the data transmission bandwidth adjustment instruction contains a target bandwidth proportion and a bandwidth switching time.

[0100] Step S1441: Based on the current computing resource allocation priority order, determine the instruction generation order of the calculation resource, the storage resource and the network resource, and the instruction corresponding to the resource type with the highest priority is generated first.

[0101] According to the current computing resource allocation priority order, the order of instruction generation is determined. The instruction corresponding to the resource type with the highest priority is generated first, thereby ensuring that the demand for this type of resource is met first. For example, if the current calculation resource allocation priority is the highest, the calculation node quantity adjustment instruction is generated first.

[0102] Step S1442: According to the calculation resource adjustment amplitude value in the adjustment amplitude value and the current calculation node quantity, calculate the target calculation node quantity, and the calculation method of the target calculation node quantity is the current calculation node quantity plus the product of the current calculation node quantity and the calculation resource adjustment amplitude value.

[0103] The target calculation node quantity is calculated according to the calculation resource adjustment amplitude value and the current calculation node quantity. By multiplying the current calculation node quantity and the calculation resource adjustment amplitude value, the calculation node quantity that needs to be adjusted is obtained, and then the current calculation node quantity is added to obtain the target calculation node quantity. For example, if the current calculation node quantity is a certain value and the calculation resource adjustment amplitude value is a positive value, it means that the calculation node quantity needs to be increased, and the product of the two values is the node quantity that needs to be increased. Then, the current node quantity is added to obtain the target calculation node quantity.

[0104] Step S1443: Determine the calculation node increase / decrease direction in combination with the calculation resource scheduling time point in the scheduling execution time point. When the target calculation node quantity is greater than the current calculation node quantity, the node increase / decrease direction is increase and the node startup preparation needs to be completed before the calculation resource scheduling time point. When the target calculation node quantity is less than the current calculation node quantity, the node increase / decrease direction is decrease and the node task migration needs to be completed before the calculation resource scheduling time point.

[0105] In combination with the computing resource scheduling time point, the node increase / decrease direction is determined according to the comparison result of the target computing node quantity and the current computing node quantity. If the target computing node quantity is greater than the current computing node quantity, it indicates that the computing node needs to be increased, and the node increase / decrease direction is increase. In order to ensure that the newly added computing node can be normally used at the computing resource scheduling time point, the node startup preparation work needs to be completed before the time point, including operating system loading, driver initialization and training environment deployment, etc.

[0106] If the target computing node quantity is less than the current computing node quantity, it indicates that the computing node needs to be reduced, and the node increase / decrease direction is decrease. Before the computing resource scheduling time point, the node task migration work needs to be completed, that is, the training task process being executed on the node to be closed is marked with a state, and the unfinished task data and intermediate results are migrated to other nodes with moderate load in the cluster to ensure the continuity of training.

[0107] Step S1444: generating a computing node quantity adjustment instruction, the computing node quantity adjustment instruction containing an instruction serial number determined based on the current computing power resource allocation priority order, a target computing node quantity calculated based on the adjustment amplitude value, a node increase / decrease direction determined based on the scheduling execution time point, and an execution deadline.

[0108] The computing node quantity adjustment instruction contains multiple key information. The instruction serial number is determined according to the current computing power resource allocation priority order, and the instruction serial number corresponding to the resource type with the highest priority is the most forward. The target computing node quantity is calculated according to the computing resource adjustment amplitude value and the current computing node quantity. The node increase / decrease direction is determined according to the comparison result of the target computing node quantity and the current computing node quantity and the computing resource scheduling time point. The execution deadline is the computing resource scheduling time point, which ensures that the node increase / decrease operation is completed before the time point.

[0109] Step S1445: calculating a target memory capacity according to the storage resource adjustment amplitude value in the adjustment amplitude value and the current memory capacity, and the calculation method of the target memory capacity is the current memory capacity plus the product of the current memory capacity and the storage resource adjustment amplitude value.

[0110] The target memory capacity is calculated according to the storage resource adjustment amplitude value and the current memory capacity. The memory capacity to be adjusted is obtained by multiplying the current memory capacity and the storage resource adjustment amplitude value, and then the current memory capacity is added to obtain the target memory capacity. For example, if the storage resource adjustment amplitude value is positive, it indicates that the memory capacity needs to be increased, and the product of the two is the memory capacity to be increased, and then the current memory capacity is added to obtain the target memory capacity.

[0111] Step S1446: Set the memory capacity adjustment rate according to the priority ranking of the storage resource in the current computing resource allocation priority order. The earlier the priority ranking of the storage resource, the faster the adjustment rate.

[0112] The memory capacity adjustment rate is set according to the ranking of the storage resource in the current computing resource allocation priority order. The earlier the priority ranking of the storage resource, the more urgent the current demand for storage resources, and the faster the memory capacity needs to be adjusted. For example, if the storage resource allocation priority is the highest, the memory capacity adjustment rate can be set to a faster value to meet the demand for storage resources as soon as possible.

[0113] Step S1447: Determine the starting time of memory capacity adjustment based on the storage resource scheduling time point in the scheduling execution time point. The starting time must be earlier than the storage resource scheduling time point and reserve a data migration buffer period.

[0114] The starting time of memory capacity adjustment is determined based on the storage resource scheduling time point. In order to ensure that the memory capacity adjustment can be completed at the storage resource scheduling time point, the starting time needs to be earlier than the time point and a certain data migration buffer period needs to be reserved, because during the adjustment of memory capacity, data migration and reallocation may be needed, and the reserved buffer period can ensure the smooth progress of data migration and avoid data loss or damage.

[0115] Step S1448: Generate memory capacity allocation instructions, which include instruction sequence number determined based on the current computing resource allocation priority order, target memory capacity calculated based on the adjustment amplitude value, capacity adjustment rate set based on the priority ranking, and starting time determined based on the scheduling execution time point.

[0116] The memory capacity allocation instructions contain important information. The instruction sequence number is determined according to the current computing resource allocation priority order, and the higher the priority of the storage resource, the earlier the instruction sequence number. The target memory capacity is calculated according to the storage resource adjustment amplitude value and the current memory capacity. The capacity adjustment rate is set according to the ranking of the storage resource in the priority order. The starting time is determined according to the storage resource scheduling time point, and a data migration buffer period is reserved.

[0117] Step S1449: Calculate the target bandwidth ratio according to the network resource adjustment amplitude value in the adjustment amplitude value and the current data transmission bandwidth. The calculation method of the target bandwidth ratio is to add the current data transmission bandwidth to the product of the current data transmission bandwidth and the network resource adjustment amplitude value.

[0118] The target bandwidth ratio is calculated according to the network resource adjustment amplitude value and the current data transmission bandwidth. The target bandwidth ratio is obtained by multiplying the current data transmission bandwidth by the network resource adjustment amplitude value, and then adding the current data transmission bandwidth. For example, if the network resource adjustment amplitude value is positive, it means that the data transmission bandwidth needs to be increased. The target bandwidth ratio is obtained by multiplying the current data transmission bandwidth by the network resource adjustment amplitude value, and then adding the current data transmission bandwidth.

[0119] Step S14410: Determine the pre-detection duration before bandwidth switching in combination with the priority ranking of the network resource in the current computing resource allocation priority order. The earlier the priority ranking of the network resource, the longer the pre-detection duration.

[0120] In combination with the ranking of the network resource in the current computing resource allocation priority order, the pre-detection duration before bandwidth switching is determined. The earlier the priority ranking of the network resource, the more important the current demand for the network resource, and more thorough pre-detection is needed to ensure smooth bandwidth switching. For example, if the network resource allocation priority is the highest, the pre-detection duration can be set to a longer value to detect the network condition more comprehensively.

[0121] Step S14411: Based on the network resource scheduling time point in the scheduling execution time point, set the bandwidth switching time, which needs to be aligned with the network resource scheduling time point and executed immediately after the pre-detection duration ends.

[0122] Based on the network resource scheduling time point, set the bandwidth switching time. The bandwidth switching time needs to be aligned with the network resource scheduling time point and executed immediately after the pre-detection duration ends, which can ensure that the bandwidth switching is performed at the right time and guarantee the stability of the switching. For example, if the network resource scheduling time point is a certain specific time and the pre-detection duration is a certain duration, the bandwidth switching is performed at that time immediately after the pre-detection duration ends.

[0123] Step S14412: Generate a data transmission bandwidth adjustment instruction, which includes an instruction number determined based on the current computing resource allocation priority order, a target bandwidth ratio calculated based on the adjustment amplitude value, a pre-detection duration determined based on the priority ranking, and a bandwidth switching time set based on the scheduling execution time point.

[0124] The data transmission bandwidth adjustment instruction contains multiple key information. The instruction sequence number is determined according to the current computing resource allocation priority order. The higher the network resource priority, the earlier the instruction sequence number. The target bandwidth ratio is calculated according to the network resource adjustment amplitude value and the current data transmission bandwidth. The pre-detection duration is determined according to the ranking of the network resource in the priority order. The bandwidth switching time is set according to the network resource scheduling time point, and the switching is performed immediately after the pre-detection duration ends.

[0125] Step S145: The computing node quantity adjustment instruction, the memory capacity allocation instruction and the data transmission bandwidth adjustment instruction are sorted according to the order of the scheduling execution time points, and combined to form a computing resource allocation scheme.

[0126] The generated computing node quantity adjustment instruction, memory capacity allocation instruction and data transmission bandwidth adjustment instruction are sorted according to the order of the scheduling execution time points, so that the resource scheduling operation can be performed in a reasonable time sequence, avoiding resource conflicts or scheduling chaos. The sorted instructions are combined together to form a complete computing resource allocation scheme. For example, if the computing resource scheduling time point is the earliest, the computing node quantity adjustment instruction is placed at the front, followed by the memory capacity allocation instruction and the data transmission bandwidth adjustment instruction.

[0127] Step S150: According to the computing resource allocation scheme, the computing resource scheduling operation is performed, and the resource scheduled computing resource usage data and model training progress data are collected as new real-time computing resource usage data and new model training state data, which are used to update the computing resource demand associated feature set.

[0128] Step S151: Analyze the scheduling execution time points in the computing resource allocation scheme, and read the computing node quantity adjustment instruction, memory capacity allocation instruction and data transmission bandwidth adjustment instruction in order of time.

[0129] After obtaining the computing resource allocation scheme, the scheduling execution time points in it are first analyzed. Through the analysis of the scheduling execution time points, the execution order of each instruction can be determined. According to the time sequence, the computing node quantity adjustment instruction, the memory capacity allocation instruction and the data transmission bandwidth adjustment instruction are read in order. For example, if the scheduling execution time point of the computing node quantity adjustment instruction is the earliest, the instruction is read first, and then the memory capacity allocation instruction and the data transmission bandwidth adjustment instruction are read in order.

[0130] Step S152: According to the target computing node quantity and the node increase / decrease direction in the computing node quantity adjustment instruction, the start or shutdown operation of the computing node is performed to adjust the actual running quantity of the computing node.

[0131] Step S1521: Analyze the target node quantity and node increase / decrease direction in the computing node quantity adjustment instruction to determine the current node quantity target to be achieved and the operation type.

[0132] The computing node quantity adjustment instruction is analyzed to extract the target node quantity and node increase / decrease direction. Through these two pieces of information, the node quantity target to be achieved currently and whether to perform an increase operation or a decrease operation are determined. For example, if the target node quantity is greater than the current computing node quantity and the node increase / decrease direction is increase, then a node increase operation needs to be performed, and the target is to increase the computing node quantity to the target node quantity.

[0133] Step S1522: When the node increase / decrease direction is increase, query the idle computing node list in the computing power resource pool to filter out idle computing nodes with the same hardware configuration and software environment as the currently running computing nodes.

[0134] When the node increase / decrease direction is increase, suitable idle computing nodes need to be found from the computing power resource pool. First, query the idle computing node list in the computing power resource pool, and then filter out idle computing nodes with the same hardware configuration and software environment as the currently running computing nodes. This is done to ensure that the newly added computing nodes can seamlessly integrate into the current computing node cluster and avoid problems caused by hardware or software incompatibility. For example, if the currently running computing nodes use a certain specific model of GPU and a certain version of operating system, then idle computing nodes with the same GPU model and operating system version need to be filtered out.

[0135] Step S1523: According to the difference between the target node quantity and the current computing node quantity, select a corresponding number of computing nodes from the filtered idle computing node list as to-be-started nodes, so that the total performance of the to-be-started nodes meets the computing resource demand in the computing resource demand correlation characteristic set.

[0136] According to the difference between the target node quantity and the current computing node quantity, the number of computing nodes that need to be started is determined. A corresponding number of computing nodes are selected from the filtered idle computing node list as to-be-started nodes. The principle of selection is to make the total performance of the to-be-started nodes meet the computing resource demand in the computing resource demand correlation characteristic set. For example, if the computing resource demand correlation characteristic shows that a certain amount of computing power needs to be increased, then idle computing nodes with corresponding computing performance need to be selected as to-be-started nodes.

[0137] Step S1524: Send a start instruction to the selected to-be-started nodes, the start instruction containing access configuration information and resource allocation quota of the AI model training task, and monitor the start progress of the to-be-started nodes.

[0138] The selected to-be-started node is sent a start instruction, which contains access configuration information and resource allocation quota of the AI model training task. The access configuration information allows the to-be-started node to know how to access the current training task, and the resource allocation quota specifies the amount of resources that the node can use in the training task. At the same time, the startup progress of the to-be-started node is monitored to ensure that the node can start normally. For example, through monitoring, the system can real-time understand the operating system loading situation and driver initialization progress of the to-be-started node.

[0139] Step S1525: Wait for the to-be-started node to complete operating system loading, driver initialization and training environment deployment, until the node state becomes ready, add the ready node to the computing node cluster of the AI model training task, and update the cluster node list and load balancing configuration.

[0140] Wait for the to-be-started node to complete operating system loading, driver initialization and training environment deployment, and a series of operations, until the node state becomes ready. When the node is ready, it is added to the computing node cluster of the AI model training task. At the same time, the cluster node list is updated to ensure that the new node is correctly recorded in the list. In addition, the load balancing configuration also needs to be updated to ensure that the new node can reasonably share the computing task and avoid load imbalance.

[0141] Step S1526: When the node increase / decrease direction is decrease, collect the real-time load situation of all running nodes in the current computing node cluster, and sort the nodes in order from low to high according to the load situation.

[0142] When the node increase / decrease direction is decrease, the real-time load situation of all running nodes in the current computing node cluster needs to be collected. Through the collection of node load situation, the working state of each node can be understood. The nodes are sorted in order from low to high according to the load situation, so that the nodes with lower load can be selected for closing operation in priority, to reduce the impact on the training task. For example, if the load of a node is very low, it means that the node currently undertakes less computing task, and closing the node has relatively small impact on the training task.

[0143] Step S1527: According to the difference between the current number of computing nodes and the target number of nodes, select the corresponding number of nodes with the lowest load from the sorted node list as to-be-closed nodes, so that the total load capacity of the remaining nodes is not less than the computing resource requirement of the current training stage.

[0144] According to the difference between the current number of computing nodes and the target number of nodes, the number of nodes to be closed is determined. The corresponding number of nodes with the lowest load is selected from the sorted node list as the nodes to be closed. The principle of selection is to ensure that the total load capacity of the remaining nodes is not lower than the computing resource requirement of the current training stage. For example, if a certain computing capacity is required to complete the task in the current training stage, the total load capacity of the remaining nodes after closing part of the nodes should be able to meet the computing resource requirement to ensure the normal progress of the training.

[0145] Step S1528: The training task processes being executed on the nodes to be closed are marked with a state, and the unfinished task data and intermediate results are migrated to other nodes with moderate load in the cluster, and the continuity of the task processes is maintained during the migration process.

[0146] The training task processes being executed on the nodes to be closed are marked with a state, so that the execution of the task can be accurately understood subsequently. The unfinished task data and intermediate results are migrated to other nodes with moderate load in the cluster, and the continuity of the task processes is maintained during the migration process. For example, the data backup and recovery technology can be used to backup the task data and intermediate results on the nodes to be closed to other nodes, and then restore the task processes on the other nodes to ensure that the task can continue to be executed.

[0147] Step S1529: After confirming that all training task processes on the nodes to be closed have been migrated and the processes on the new nodes are running normally, a closing instruction is sent to the nodes to be closed, which contains a resource release confirmation requirement.

[0148] After confirming that all training task processes on the nodes to be closed have been migrated and the processes on the new nodes are running normally, a closing instruction is sent to the nodes to be closed. The closing instruction contains a resource release confirmation requirement, which requires the nodes to be closed to release all resources occupied by them, such as CPU, GPU, memory, etc. For example, after receiving the closing instruction, the nodes to be closed need to release the memory space occupied by them so that other nodes can use it.

[0149] Step S15210: Monitor the resource release process of the nodes to be closed until the nodes are completely closed and removed from the computing node cluster, record the actual number of running nodes in the adjusted computing node cluster as the new current number of computing nodes.

[0150] The resource release process of the nodes to be closed is monitored to ensure that the nodes can release resources as required. After the nodes are completely closed, they are removed from the computing node cluster. The actual number of running nodes in the adjusted computing node cluster is recorded, which will be used as the new current number of computing nodes for subsequent resource allocation decisions.

[0151] Step S153: According to the target capacity value and the capacity adjustment rate in the memory capacity allocation instruction, perform the dynamic allocation operation of the memory resource, and adjust the actual allocation proportion of the memory space.

[0152] According to the target capacity value and the capacity adjustment rate in the memory capacity allocation instruction, perform the dynamic allocation operation of the memory resource. According to the capacity adjustment rate, gradually adjust the actual allocation proportion of the memory space to reach the target capacity value. For example, if the target capacity value is to increase a certain memory capacity, and the capacity adjustment rate is a certain value, then gradually increase the allocation amount of the memory according to the rate until the target capacity value is reached.

[0153] Step S154: According to the target bandwidth proportion and the bandwidth switching time in the data transmission bandwidth adjustment instruction, perform the allocation adjustment operation of the network bandwidth, and adjust the actual bandwidth proportion of the data transmission channel.

[0154] According to the target bandwidth proportion and the bandwidth switching time in the data transmission bandwidth adjustment instruction, perform the allocation adjustment operation of the network bandwidth. At the bandwidth switching time, adjust the actual bandwidth proportion of the data transmission channel according to the target bandwidth proportion. For example, if the target bandwidth proportion is to increase a certain bandwidth, then at the bandwidth switching time, increase the bandwidth of the data transmission channel to the target bandwidth proportion.

[0155] Step S155: After completing all the computing power resource scheduling operations, collect the adjusted computing node load, memory occupation level and data transmission delay conditions through the resource monitoring interface as the computing power resource usage data after resource scheduling.

[0156] After completing all the computing power resource scheduling operations, collect the relevant data through the resource monitoring interface. Through the resource monitoring interface, the adjusted computing node load, memory occupation level and data transmission delay conditions can be obtained in real time. These data will be used as the computing power resource usage data after resource scheduling for subsequent analysis and decision-making. For example, through the resource monitoring interface, the CPU usage rate, memory usage and data transmission delay time of each computing node can be obtained.

[0157] Step S156: Collect the adjusted model parameter update frequency and training data batch processing progress through the model training monitoring interface as the model training progress data after resource scheduling.

[0158] The model training monitoring interface can interact with the model training framework to obtain the training state information of the model in real time. These data will be used as the model training progress data after resource scheduling to understand the impact of resource scheduling on model training. For example, the model training monitoring interface can obtain the frequency of model parameter updates and the processing progress of the current training data batch.

[0159] Step S157: Determine the resource-scheduled computing resource usage data as new real-time computing resource usage data, and determine the resource-scheduled model training progress data as new model training state data.

[0160] The collected resource-scheduled computing resource usage data is determined as new real-time computing resource usage data, and the resource-scheduled model training progress data is determined as new model training state data. These new data will be used to update the computing resource demand associated feature set, so as to more accurately reflect the actual demand of model training in the next round of resource allocation decision. For example, the new computing node load, memory occupation level and data transmission delay condition are taken as the new real-time computing resource usage data, and the new model parameter update frequency and training data batch processing progress are taken as the new model training state data, and the associated feature extraction and resource allocation decision are performed again.

[0161] Figure 2 An exemplary schematic diagram of hardware and software components of the computing power service dynamic resource allocation system 100 for AI model training provided by some embodiments of the present application is shown, which can implement the idea of the present application. For example, the processor 120 can be used in the computing power service dynamic resource allocation system 100 for AI model training, and used to execute the functions in the present application.

[0162] The computing power service dynamic resource allocation system 100 for AI model training can be a general server or a special-purpose server, both of which can be used to implement the computing power service dynamic resource allocation method for AI model training of the present application. Although only one server is shown in the present application, for the sake of convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0163] For example, the computing power service dynamic resource allocation system 100 applied to AI model training can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as a disk, a ROM, or a RAM, or any combination thereof. Exemplarily, the computing power service dynamic resource allocation system 100 applied to AI model training can also include program instructions stored in a ROM, a RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The computing power service dynamic resource allocation system 100 applied to AI model training also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0164] For ease of illustration, only one processor is described in the computing power service dynamic resource allocation system 100 applied to AI model training. However, it should be noted that the computing power service dynamic resource allocation system 100 applied to AI model training in the present application can also include multiple processors, so the steps described in the present application performed by one processor can also be jointly performed or separately performed by multiple processors. For example, if the processor of the computing power service dynamic resource allocation system 100 applied to AI model training performs steps A and B, it should be understood that steps A and B can also be jointly performed by two different processors or separately performed in one processor. For example, a first processor performs step A, a second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0165] In addition, the present application also provides a readable storage medium, wherein computer executable instructions are pre-set in the readable storage medium, and when a processor executes the computer executable instructions, the method for dynamically allocating resources of a computing power service applied to AI model training is implemented.

[0166] It should be noted that, in order to simplify the description of the present application and to help understand one or more embodiments of the present application, in the foregoing description of the embodiments of the present application, various features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for dynamic resource allocation of computing power services applied to AI model training, characterized in that, The method comprises: Collecting real-time computing resource usage data and model training state data during AI model training, the real-time computing resource usage data including computing node load, memory occupation level and data transmission delay condition, and the model training state data including training task stage identifier, model parameter update frequency and training data batch processing progress; Performing associated feature extraction on the real-time computing resource usage data and the model training state data to generate a computing resource demand associated feature set, the computing resource demand associated feature set including computing resource demand associated features, storage resource demand associated features and network resource demand associated features; Building a computing resource dynamic allocation decision model based on the computing resource demand associated feature set, the computing resource dynamic allocation decision model including a resource allocation priority judgment unit, a resource adjustment amplitude calculation unit and a resource scheduling timing selection unit; Outputting a computing resource allocation scheme through the computing resource dynamic allocation decision model, the computing resource allocation scheme including a computing node number adjustment instruction, a memory capacity allocation instruction and a data transmission bandwidth adjustment instruction; Performing computing resource scheduling operations according to the computing resource allocation scheme, and collecting resource-scheduled computing resource usage data and model training progress data as new real-time computing resource usage data and new model training state data for updating the computing resource demand associated feature set. 2.The method for dynamic resource allocation of computing power service applied to AI model training according to claim 1, wherein, The associated feature extraction on the real-time computing resource usage data and the model training state data to generate a computing resource demand associated feature set comprises: Dividing the real-time computing resource usage data into time series, and dividing the real-time computing resource usage data into a plurality of continuous computing data windows according to fixed time intervals, each computing data window including a computing node load change sequence, a memory occupation level change sequence and a data transmission delay condition change sequence in the corresponding time interval; Performing stage feature labeling on the model training state data, and dividing the model training state data into a plurality of training stage data units according to the training task stage identifier, each training stage data unit including the model parameter update frequency and the training data batch processing progress of the corresponding training stage; Performing time alignment processing on the computing data window and the training stage data unit, so that each training stage data unit corresponds to at least one computing data window, and the time coverage ranges of the two are consistent; Calculating a first correlation coefficient of the training stage data unit and the computing node load in the computing data window after time alignment, a second correlation coefficient of the training stage data unit and the memory occupation level in the computing data window after time alignment, and a third correlation coefficient of the training stage data unit and the data transmission delay condition in the computing data window after time alignment; The first correlation coefficient is determined as a computing resource demand correlation feature, the second correlation coefficient is determined as a storage resource demand correlation feature, and the third correlation coefficient is determined as a network resource demand correlation feature, which are combined to form a computing resource demand correlation feature set. 3.The method of claim 2, wherein, The first correlation coefficient between the training phase data unit after time alignment and the computing node load condition in the computing power data window includes: The model parameter update frequency in the training phase data unit after time alignment is extracted, the average value of the model parameter update frequency in the corresponding training phase is calculated, and a parameter update frequency average value is obtained; The computing node load condition in the computing power data window after time alignment is extracted, the maximum value of the computing node load condition in the corresponding computing power data window is calculated, and a node load rate peak value is obtained; The parameter update frequency average value and the node load rate peak value are subjected to data standardization processing, so that they are in the same data dimension range; The product of the standardized parameter update frequency average value and the standardized node load rate peak value is calculated to obtain a preliminary correlation value; A correlation coefficient correction factor is set, and the correlation coefficient correction factor is positively correlated with the duration length of the training phase data unit; The preliminary correlation value is multiplied by the correlation coefficient correction factor to obtain a first correlation coefficient. 4.The method for dynamic resource allocation of computing power service applied to AI model training according to claim 1, wherein, The computing power resource dynamic allocation decision model is constructed based on the computing power resource demand correlation feature set, which includes: The input layer parameters of the computing power resource dynamic allocation decision model are determined as the computing resource demand correlation feature, the storage resource demand correlation feature, and the network resource demand correlation feature in the computing power resource demand correlation feature set; The judgment rule of the resource allocation priority judgment unit is set, and the judgment rule includes that when the computing resource demand correlation feature is higher than a first preset threshold, the computing resource allocation priority is the highest, when the storage resource demand correlation feature is higher than a second preset threshold, the storage resource allocation priority is the highest, and when the network resource demand correlation feature is higher than a third preset threshold, the network resource allocation priority is the highest; The calculation logic of the resource adjustment amplitude calculation unit is configured, and the calculation logic is to determine the resource adjustment amplitude according to the difference value proportion of the computing power resource demand correlation feature and the corresponding preset threshold, and the greater the difference value proportion, the greater the resource adjustment amplitude; The selection condition of the resource scheduling timing selection unit is set, and the selection condition includes triggering the resource scheduling operation when the model parameter update frequency is in a stable interval and triggering the resource scheduling operation when the training data batch processing progress completes a set proportion of the current batch; The resource allocation priority judgment unit, the resource adjustment amplitude calculation unit, and the resource scheduling timing selection unit are connected in series to construct the computing power resource dynamic allocation decision model, wherein the output of the resource allocation priority judgment unit is used as the input of the resource adjustment amplitude calculation unit, and the output of the resource adjustment amplitude calculation unit is used as the input of the resource scheduling timing selection unit. 5.The method for dynamic resource allocation of computing power service applied to AI model training according to claim 4, wherein, The calculation logic of the resource adjustment amplitude calculation unit includes: The calculation manner for determining the computing resource adjustment range is that the difference between the computing resource demand correlation feature and the first preset threshold is divided by the first preset threshold, and then multiplied by a reference adjustment coefficient to obtain the computing resource adjustment range; The calculation manner for determining the storage resource adjustment range is that the difference between the storage resource demand correlation feature and the second preset threshold is divided by the second preset threshold, and then multiplied by the reference adjustment coefficient to obtain the storage resource adjustment range; The calculation manner for determining the network resource adjustment range is that the difference between the network resource demand correlation feature and the third preset threshold is divided by the third preset threshold, and then multiplied by the reference adjustment coefficient to obtain the network resource adjustment range; The value range of the reference adjustment coefficient is determined according to the total resource quota of the AI model training task; When the computing power resource demand correlation feature is less than or equal to the corresponding preset threshold, the corresponding resource adjustment range is set to zero; When the computing resource adjustment range, the storage resource adjustment range, or the network resource adjustment range is greater than the maximum adjustment threshold, it is limited to the maximum adjustment threshold; When the computing resource adjustment range, the storage resource adjustment range, or the network resource adjustment range is less than the minimum adjustment threshold, it is limited to the minimum adjustment threshold. 6.The method for dynamic resource allocation of computing power service applied to AI model training according to claim 1, wherein, The outputting of the computing power resource allocation scheme by the computing power resource dynamic allocation decision model comprises: inputting the set of computing power resource demand correlation features into a resource allocation priority judgment unit of the computing power resource dynamic allocation decision model, and determining a current computing power resource allocation priority order through a judgment rule of the resource allocation priority judgment unit; according to the current computing power resource allocation priority order, inputting the corresponding computing power resource demand correlation features into a resource adjustment range calculation unit, and calculating the adjustment range values of various types of computing power resources through the calculation logic of the resource adjustment range calculation unit; inputting the adjustment range values into a resource scheduling time point selection unit, and determining the scheduling execution time points of various types of computing power resources in combination with the selection conditions of the resource scheduling time point selection unit; based on the current computing power resource allocation priority order, the adjustment range values, and the scheduling execution time points, generating a computing node quantity adjustment instruction, a memory capacity allocation instruction, and a data transmission bandwidth adjustment instruction, wherein the computing node quantity adjustment instruction contains a target computing node quantity and a node increase / decrease direction, the memory capacity allocation instruction contains a target capacity value and a capacity adjustment rate, and the data transmission bandwidth adjustment instruction contains a target bandwidth proportion and a bandwidth switching time; sorting the computing node quantity adjustment instruction, the memory capacity allocation instruction, and the data transmission bandwidth adjustment instruction according to the order of the scheduling execution time points in sequence to form a computing power resource allocation scheme.

7. The dynamic resource allocation method for computing power services applied to AI model training according to claim 6, characterized in that, The generation of the computing node quantity adjustment instruction, the memory capacity allocation instruction, and the data transmission bandwidth adjustment instruction based on the current computing power resource allocation priority order, the adjustment range values, and the scheduling execution time points comprises: based on the current computing power resource allocation priority order, determining the instruction generation order of the computing resource, the storage resource, and the network resource, and the instruction corresponding to the resource type with the highest priority is generated first; According to the calculation resource adjustment amplitude value in the adjustment amplitude value and the current calculation node quantity, the target calculation node quantity is calculated, and the calculation method of the target calculation node quantity is the current calculation node quantity plus the product of the current calculation node quantity and the calculation resource adjustment amplitude value; In combination with the calculation resource scheduling time point in the scheduling execution time point, the calculation node increase or decrease direction is determined, when the target calculation node quantity is greater than the current calculation node quantity, the node increase or decrease direction is increase and the node start preparation before the calculation resource scheduling time point needs to be completed, when the target calculation node quantity is less than the current calculation node quantity, the node increase or decrease direction is decrease and the node task migration before the calculation resource scheduling time point needs to be completed; A calculation node quantity adjustment instruction is generated, the calculation node quantity adjustment instruction includes an instruction serial number determined based on the current computing resource allocation priority order, a target calculation node quantity calculated based on the adjustment amplitude value, a node increase or decrease direction determined based on the scheduling execution time point and an execution deadline; According to the storage resource adjustment amplitude value in the adjustment amplitude value and the current memory capacity, the target memory capacity is calculated, and the calculation method of the target memory capacity is the current memory capacity plus the product of the current memory capacity and the storage resource adjustment amplitude value; In combination with the priority ranking of the storage resource in the current computing resource allocation priority order, the memory capacity adjustment rate is set, wherein the earlier the priority ranking of the storage resource is, the faster the adjustment rate is; Based on the storage resource scheduling time point in the scheduling execution time point, the starting time of the memory capacity adjustment is determined, and the starting time needs to be earlier than the storage resource scheduling time point and reserve a data migration buffer period; A memory capacity allocation instruction is generated, the memory capacity allocation instruction includes an instruction serial number determined based on the current computing resource allocation priority order, a target memory capacity calculated based on the adjustment amplitude value, a capacity adjustment rate set based on the priority ranking and a starting time determined based on the scheduling execution time point; According to the network resource adjustment amplitude value in the adjustment amplitude value and the current data transmission bandwidth, the target bandwidth ratio is calculated, and the calculation method of the target bandwidth ratio is the current data transmission bandwidth plus the product of the current data transmission bandwidth and the network resource adjustment amplitude value; In combination with the priority ranking of the network resource in the current computing resource allocation priority order, the pre-detection duration before bandwidth switching is determined, and the earlier the priority ranking of the network resource is, the longer the pre-detection duration is; Based on the network resource scheduling time point in the scheduling execution time point, the bandwidth switching time is set, and the bandwidth switching time needs to be aligned with the network resource scheduling time point and execute the switching immediately after the pre-detection duration ends; A data transmission bandwidth adjustment instruction is generated, the data transmission bandwidth adjustment instruction includes an instruction serial number determined based on the current computing resource allocation priority order, a target bandwidth ratio calculated based on the adjustment amplitude value, a pre-detection duration determined based on the priority ranking and a bandwidth switching time set based on the scheduling execution time point. 8.The method for dynamic resource allocation of computing power service applied to AI model training according to claim 1, wherein, The computing resource scheduling operation is performed according to the computing resource allocation scheme, and the computing resource usage data and the model training progress data after resource scheduling are collected as new real-time computing resource usage data and new model training state data, including: The scheduling execution time point in the computing resource allocation scheme is analyzed, and the computing node quantity adjustment instruction, the memory capacity allocation instruction and the data transmission bandwidth adjustment instruction are read in time sequence; According to the target computing node quantity and the node increase or decrease direction in the computing node quantity adjustment instruction, the starting or closing operation of the computing node is performed to adjust the actual running quantity of the computing node; According to the target capacity value and the capacity adjustment rate in the memory capacity allocation instruction, the dynamic allocation operation of the memory resource is performed to adjust the actual allocation proportion of the memory space; According to the target bandwidth proportion and the bandwidth switching time in the data transmission bandwidth adjustment instruction, the allocation adjustment operation of the network bandwidth is performed to adjust the actual bandwidth proportion of the data transmission channel; After completing all computing resource scheduling operations, the adjusted computing node load, memory occupation level and data transmission delay condition are collected through the resource monitoring interface as the computing resource usage data after resource scheduling; The model parameter update frequency and the training data batch processing progress after adjustment are collected through the model training monitoring interface as the model training progress data after resource scheduling; The computing resource usage data after resource scheduling is determined as the new real-time computing resource usage data, and the model training progress data after resource scheduling is determined as the new model training state data. 9.The method for dynamic resource allocation of computing power service applied to AI model training according to claim 8, characterized in that, According to the target computing node quantity and the node increase or decrease direction in the computing node quantity adjustment instruction, the starting or closing operation of the computing node is performed to adjust the actual running quantity of the computing node, including: The target node quantity and the node increase or decrease direction in the computing node quantity adjustment instruction are analyzed to determine the node quantity target to be achieved and the operation type; When the node increase or decrease direction is increase, the idle computing node list in the computing resource pool is queried, and the idle computing nodes with the same hardware configuration and software environment as the current running computing nodes are screened out; According to the difference between the target node quantity and the current computing node quantity, a corresponding number of computing nodes are selected from the screened idle computing node list as the to-be-started nodes, so that the total performance of the to-be-started nodes meets the computing resource demand associated feature set; The starting instruction containing the access configuration information and the resource allocation quota of the AI model training task is sent to the selected to-be-started nodes, and the starting progress of the to-be-started nodes is monitored; After waiting for the to-be-started nodes to complete the operating system loading, driver initialization and training environment deployment, the node state becomes ready, the ready node is added to the computing node cluster of the AI model training task, and the cluster node list and the load balancing configuration are updated; When the node increase or decrease direction is decrease, the real-time load conditions of all running nodes in the current computing node cluster are collected, and the nodes are sorted in order from low to high according to the load conditions; According to the difference between the current number of computing nodes and the target number of nodes, a corresponding number of nodes with the lowest load are selected from the sorted node list as to-be-closed nodes, so that the total load capacity of the remaining nodes is not less than the computing resource requirement of the current training stage; The state of the training task process being executed on the to-be-closed node is marked, the unfinished task data and intermediate results are migrated to other nodes with moderate load in the cluster, and the continuity of the task process is maintained during the migration process; After confirming that all training task processes on the to-be-closed node have been migrated and the processes on the new node are running normally, a closing instruction is sent to the to-be-closed node, and the closing instruction contains a resource release confirmation requirement; The resource release process of the to-be-closed node is monitored until the node is completely closed and removed from the computing node cluster, and the actual number of running nodes in the adjusted computing node cluster is recorded as the new current number of computing nodes. 10.A computing power service dynamic resource allocation system applied to AI model training, characterized in that, The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method applied to AI model training. The application discloses a computing power service dynamic resource allocation method

Images