A resource prediction method, device, equipment and readable storage medium

Abstract

The application discloses a resource prediction method and device, equipment and a readable storage medium, which can be applied to the field of artificial intelligence technology, and the method comprises the steps of obtaining metadata corresponding to a target task; based on the metadata, a preset task classification tree is used to classify the target task, and a target classification result is obtained; a corresponding relationship between different types of tasks and different resource prediction models is stored in the preset task classification tree; a resource prediction model corresponding to the target classification result is called to predict the resources of the target task, and a prediction target data is obtained; the prediction target data comprises the number of CPUs, the number of GPUs, the size of memory and the disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resources are predicted, thereby improving the accuracy of resource prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a resource prediction method and device, equipment and a readable storage medium. BACKGROUND

[0002] Artificial intelligence tasks include natural language processing, image recognition, data analysis, machine learning, and human-computer interaction, etc. In the artificial intelligence tasks, various algorithms and technologies are needed to process and analyze a large amount of data to realize automated and intelligent applications.

[0003] In order to successfully perform the task, it is often necessary to accurately evaluate the computing resources and time required before the task is executed, and then to reasonably allocate resources. The existing resource prediction method often uses the same prediction model for prediction when facing various tasks in the environment, and the data used in the training of the prediction model is not accurate, so that the final prediction result is also not accurate.

[0004] Therefore, how to improve the accuracy of resource prediction is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Based on the above problems, the present application provides a resource prediction method, device, equipment and readable storage medium, which determines a resource prediction model corresponding to a target task based on a preset task classification tree, and then performs resource prediction, thereby improving the accuracy of resource prediction.

[0006] In a first aspect, an embodiment of the present application provides a resource prediction method, comprising:

[0007] Obtaining metadata corresponding to a target task;

[0008] Classifying the target task based on the metadata using a preset task classification tree, and obtaining a target classification result; the preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models;

[0009] Calling a resource prediction model corresponding to the target classification result to perform resource prediction on the target task, and obtaining prediction target data; the prediction target data includes: CPU number, GPU number, memory size and disk space.

[0010] Optionally, the resource prediction model is obtained by training the following method:

[0011] Obtaining historical task data and corresponding historical resource usage data of each artificial intelligence task from a data center; the historical task data includes: task type, task size, task configuration and time constraint;

[0012] The computing resource architecture based on task requests classifies each artificial intelligence task and obtains a classification result;

[0013] The XGBoost prediction model is trained using historical task data and corresponding historical resource usage data of the artificial intelligence task corresponding to each classification result, and a resource prediction model for different categories of artificial intelligence tasks is obtained.

[0014] Optionally, the historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism;

[0015] The period of the first window of the double-layer sliding window is set to 7 days, and the step length is set to 1 day;

[0016] The period of the second window of the double-layer sliding window is set to 1 day, and the step length is set to 1 hour.

[0017] Optionally, before the XGBoost prediction model is trained using the historical task data and corresponding historical resource usage data of the artificial intelligence task corresponding to each classification result, the method further comprises:

[0018] The historical task data and the historical resource usage data are data cleaned and data preprocessed;

[0019] The data cleaning includes missing value processing and outlier processing;

[0020] The data preprocessing includes feature encoding and normalization processing.

[0021] Optionally, the method further comprises:

[0022] Based on the computing resource architecture based on task requests, a preset task classification tree corresponding to all artificial intelligence tasks is constructed;

[0023] Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

[0024] Optionally, the XGBoost prediction model is trained using the historical task data and corresponding historical resource usage data of the artificial intelligence task corresponding to each classification result, comprising:

[0025] Based on the historical task data and corresponding historical resource usage data of the artificial intelligence task corresponding to each classification result, a task feature importance evaluation is performed using a LightGBM model, and an evaluation result is obtained;

[0026] The evaluation result is used to adjust feature weights for an XGBoost prediction model.

[0027] Optionally, the method further comprises:

[0028] The prediction target data corresponding to the prediction value is displayed to the user in the form of a table.

[0029] In a second aspect, the embodiments of the present application provide a resource prediction device, comprising:

[0030] An acquisition module is configured to acquire metadata corresponding to a target task;

[0031] A classification module is configured to classify the target task based on the metadata by using a preset task classification tree, and obtain a target classification result; the preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models;

[0032] A prediction module is configured to call a resource prediction model corresponding to the target classification result to perform resource prediction on the target task, and obtain prediction target data; the prediction target data includes a CPU number, a GPU number, a memory size, and a disk space.

[0033] In a third aspect, the embodiments of the present application provide a resource prediction device, comprising:

[0034] A memory is configured to store a computer program;

[0035] A processor is configured to implement the steps of the resource prediction method when the computer program is executed.

[0036] In a fourth aspect, the embodiments of the present application provide a readable storage medium, which stores a computer program; the computer program is executed by a processor to implement the steps of the resource prediction method.

[0037] As can be seen from the above technical solutions, compared with the prior art, the present application has the following advantages:

[0038] The present application first acquires metadata corresponding to a target task. Then, based on the metadata, the target task is classified by using a preset task classification tree, and a target classification result is obtained. The preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models. Finally, a resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task, and prediction target data is obtained; the prediction target data includes a CPU number, a GPU number, a memory size, and a disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then resource prediction is performed, thereby improving the accuracy of resource prediction. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 a flowchart of a resource prediction method provided by an embodiment of the present application;

[0040] Figure 2 a schematic diagram of a preset task classification tree provided by an embodiment of the present application;

[0041] Figure 3 a structural schematic diagram of a resource prediction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] As described above, the existing resource prediction method has the problem of insufficient accuracy. Specifically, the existing resource prediction method often uses a single prediction model to process all types of tasks, ignoring the differences between different tasks. In addition, the existing technology cannot fully consider the dynamic changes of tasks during training, resulting in inaccurate model training results. Combined with the above two, the existing resource prediction method is difficult to ensure the accuracy of resource prediction.

[0043] To solve the above problems, an embodiment of the present application provides a resource prediction method, which comprises: first, obtaining the metadata corresponding to the target task. Then, based on the metadata, the target task is classified by using a preset task classification tree, and a target classification result is obtained. The preset task classification tree stores the correspondence between different types of tasks and different resource prediction models. Finally, the resource prediction model corresponding to the target classification result is called to predict the resources of the target task, and the prediction target data is obtained; the prediction target data includes: the number of CPUs, the number of GPUs, the size of memory and the disk space.

[0044] In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resource prediction is performed, thereby improving the accuracy of resource prediction.

[0045] It should be noted that the resource prediction method, device, equipment and readable storage medium provided by the present application can be applied to the field of artificial intelligence technology. The above are only examples and do not limit the application of the resource prediction method, device, equipment and readable storage medium provided by the present application.

[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Figure 1 A flowchart of a resource prediction method provided by an embodiment of the present application is shown. The resource prediction method provided by an embodiment of the present application can include the following steps. Figure 1 The resource prediction method provided by an embodiment of the present application can include the following steps.

[0048] S101: Obtain metadata corresponding to a target task.

[0049] In actual applications, the metadata is information for describing data attributes of the target task, which is used to support functions such as indicating storage locations, historical data, resource searching, and file recording, and can be regarded as an electronic catalog. The metadata describes the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval. There are many types of artificial intelligence tasks, and among the many artificial intelligence tasks, the artificial intelligence task input into the resource prediction device is defined as the target task, and then the metadata information of the target task is obtained.

[0050] S102: Based on the metadata, classify the target task by using a preset task classification tree, and obtain a target classification result; the preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models.

[0051] In actual applications, the preset task classification tree is a hierarchical tree structure, and the preset task classification tree pre-stores the types corresponding to each task and the corresponding relationship between each task and each resource prediction model. It is worth noting that tasks of the same type correspond to the same resource prediction model, and tasks of different types generally correspond to different resource prediction models. The resource prediction device can quickly determine the type corresponding to the target task and the corresponding resource prediction model according to the pre-stored information on the preset task classification tree based on the metadata of the target task.

[0052] S103: Call a resource prediction model corresponding to the target classification result to perform resource prediction on the target task, and obtain predicted target data; the predicted target data includes the number of CPUs, the number of GPUs, the size of memory, and the disk space.

[0053] In actual applications, the resource prediction model is pre-trained and is a model for predicting resources of different types of artificial intelligence tasks. After determining the type corresponding to the target task (the target classification result), the resource prediction device automatically calls the corresponding resource prediction model and performs resource prediction on the target task. The predicted target data includes the number of CPUs, the number of GPUs, the size of memory, the disk space, and the time required to complete the task.

[0054] In addition, since the ways of displaying the predicted target data are different, the present embodiment can describe one possible display method.

[0055] In one case, the method further comprises:

[0056] The predicted value corresponding to the predicted target data is displayed to the user in the form of a table.

[0057] In actual application, the predicted target data obtained by predicting the target task exists in the form of data, for example, the required CPU quantity is 1 four-core eight-thread, the required GPU quantity is RTX3060, and the required memory is 16GB, etc. It is worth noting that the "quantity" in the CPU quantity and the GPU quantity does not only refer to the Arabic numeral, but more refers to the model of the CPU and the GPU. These predicted values such as "1 four-core eight-thread CPU", "RTX3060 GPU" and "16GB memory" are displayed to the user in the form of a table.

[0058] In addition, since the training methods of the resource prediction model are different, the embodiments of the present application can illustrate one possible training method.

[0059] In one case, the resource prediction model is obtained by the following method:

[0060] Obtain historical task data and corresponding historical resource usage data corresponding to each artificial intelligence task from the data center; the historical task data includes: task type, task size, task configuration and time constraint;

[0061] Classify each artificial intelligence task based on the computing resource architecture of the task request, and obtain a classification result;

[0062] Use the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence task indicated by each classification result to train the XGBoost prediction model, and obtain a resource prediction model for different categories of artificial intelligence tasks.

[0063] In practical applications, the XGBoost prediction model is used as a basic model, and a resource prediction model corresponding to each type is obtained through a large amount of training. Specifically, first, a large amount of historical task data and historical resource usage data corresponding to artificial intelligence tasks in the data center are collected, and they are divided into a training set and a test set according to a 7:3 ratio. In the training set, the historical task data includes task type, task size, task configuration, and time constraint. The task type is used to represent the type of the task; the task size includes the number of model parameters, the number of dataset samples, and the dataset dimension; the task configuration includes the number of data samples used in each iteration and the number of times the entire dataset is traversed during training; and the time constraint is used to record the latest completion time of the task, which has an impact on resource scheduling. The prediction target data used for training, i.e., the historical resource usage data corresponding to the artificial intelligence tasks in the training set, includes the number of CPUs (cores), the number of GPUs, the size of memory, and the size of disk space, etc. Then, the collected data is classified according to the computing resource architecture based on task requests to obtain the classification results of each artificial intelligence task in the training set, such as task A and task B belonging to the large model training class, task C belonging to the large model inference class, and task D and task E belonging to the machine learning class, etc. Finally, the historical task data and the corresponding historical resource usage data corresponding to each category of artificial intelligence tasks are used to train the XGBoost prediction model, and resource prediction models for different categories of artificial intelligence tasks are obtained. Continuing the above example, it can be understood that after training, when predicting the task of the large model training class, the parameter of the called resource prediction model is X; when predicting the task of the large model inference class, the parameter of the called resource prediction model is Y; and when predicting the task of the machine learning class, the parameter of the called resource prediction model is Z. It is worth noting that there may be other categories of tasks, which are not described one by one here, and the values of X, Y, and Z are obtained through actual training and are generally different.

[0064] In one case, the historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism;

[0065] The period of the first window of the double-layer sliding window is set to 7 days, and the step is set to 1 day;

[0066] The period of the second window of the double-layer sliding window is set to 1 day, and the step is set to 1 hour.

[0067] In practical applications, the embodiment of the present application pre-sets a real-time data stream pipeline, which can store task resource usage data in chronological order (for artificial intelligence tasks, it corresponds to historical resource usage data). These data are collected according to fixed time to ensure that the latest task data is included. The sliding window includes two layers, the window size of the first layer window is 7 days, and the step is 1 day, which is used to capture the medium-term trend of task resource usage; the window size of the second layer window is 1 day, and the step is 1 hour, which is used to capture the short-term fluctuations of task resource usage. Among them, capturing short-term fluctuations is to update the prediction model when fluctuations are found. It can be understood that the historical resource usage data is recorded in the data center based on the double-layer sliding window mechanism. Therefore, when obtaining the historical resource usage data corresponding to the artificial intelligence task, in order to support the sliding window mechanism, it is necessary to collect the time stamp first.

[0068] In addition, since the data preprocessing methods before model training are not the same, the embodiment of the present application can be described for a possible preprocessing method.

[0069] In one case, before the historical task data corresponding to the artificial intelligence task indicated by each classification result and the corresponding historical resource usage data are used to train the XGBoost prediction model, the method further comprises:

[0070] performing data cleaning and data preprocessing on the historical task data and the historical resource usage data;

[0071] The data cleaning includes missing value processing and outlier processing.

[0072] The data preprocessing includes feature encoding and normalization processing.

[0073] In practical applications, in order to provide higher quality data support for training resource prediction models, the collected data (historical task data and historical resource usage data) need to be data cleaned and data preprocessed. Specifically, in the data cleaning process, the isnull() function of pandas can be used to check whether the data has missing values. If the number of missing data samples is small, it can be directly deleted, and if it is large, the mean value of the feature can be used to fill the missing data. Padding is performed to maintain the integrity of the data. Then, outlier processing is performed, including outlier detection and processing. Among them, the outlier detection is to sort the data by using the interquartile range method, calculate the IQR, calculate the limit and mark the outliers; the outlier processing includes: deleting the outlier samples that obviously deviate from the normal range and correcting the confirmed outliers to the upper and lower limit values to ensure the rationality and accuracy of the data. In addition, for category data such as Task_Type, one-hot encoding is adopted to convert each category into a binary vector so that the model can effectively process these data. For example, training tasks are marked as 1, inference tasks are marked as 0, and so on. For numerical information such as Model_Parameters, Batch_Size, etc., normalization is performed where X min and X max are the maximum and minimum values of the feature values, respectively, to eliminate the influence between different dimensions and improve the efficiency and accuracy of model training.

[0074] In addition, since the ways of constructing the preset task classification tree are different, the embodiments of the present application can explain one possible construction method.

[0075] In one case, the method further comprises:

[0076] Based on the computing resource architecture of the task request, a preset task classification tree corresponding to all artificial intelligence tasks is constructed;

[0077] Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

[0078] In actual application, the collected data can be classified to construct a tree structure, which is used for fast classification of target tasks subsequently. Figure 2 A schematic diagram of a preset task classification tree provided by the embodiments of the present application is shown. In combination with Figure 2 shown, the computing resource architecture of the task request is taken as the core, and all artificial intelligence tasks are classified based on the collected historical task data of artificial intelligence tasks, to obtain a preset task classification tree. Specifically, the artificial intelligence tasks can be divided into large-scale computing power center, mixed computing power center and edge computing power center according to the computing resources required. Under the computing power center, it is further divided into three categories of general computing, supercomputing and intelligent computing according to the computing power type. Then, under the computing power type, it is further divided into Volta architecture, Ampere architecture, Hopper architecture and Ascend architecture according to the architecture. It should be noted that, Figure 2The Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture are only displayed under the intelligence calculation branch of the large-scale computing center. It can be understood that the general calculation and super calculation branches of the large-scale computing center, the general calculation, super calculation, and intelligence calculation branches of the mixed computing center, and the general calculation, super calculation, and intelligence calculation branches of the edge computing center can be divided into the Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture. Finally, all artificial intelligence tasks are divided into large-scale training tasks, large-scale inference tasks, and machine learning tasks according to the task type, that is, the Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture are all divided into large-scale training tasks, large-scale inference tasks, and machine learning tasks. It should be noted that, Figure 2 Only one of the divisions under the Hopper architecture is displayed. In this way, a four-layer preset task classification tree is formed, which helps to more accurately identify and manage the computing resource requirements of different tasks when facing target task classification, thereby optimizing resource allocation and scheduling efficiency. It can be understood that in the preset task classification tree, each leaf node corresponds to a class of artificial intelligence tasks, and the resource prediction model needs to be trained separately for each class of artificial intelligence tasks.

[0079] In addition, since the training methods of the XGBoost prediction model are not the same, the embodiments of the present application can illustrate one possible training method.

[0080] In one case, the XGBoost prediction model is trained using the historical task data and corresponding historical resource usage data of the artificial intelligence tasks indicated by each classification result, respectively, including:

[0081] Based on the historical task data and corresponding historical resource usage data of the artificial intelligence tasks indicated by each classification result, a task feature importance evaluation is performed using a LightGBM model, and an evaluation result is obtained.

[0082] The evaluation result is used to adjust the feature weight of the XGBoost prediction model.

[0083] In practical applications, the LightGBM model is introduced to perform deep importance assessment on the features of each type of artificial intelligence task (combined with the corresponding historical resource usage data, the features in the historical task data corresponding to each type of artificial intelligence task), and these assessment results are applied to the XGBoost model to adjust the feature weights in real time, thereby obtaining the resource prediction model corresponding to each type of artificial intelligence task. Specifically, in the model training process, first, feature calculation is performed, including using the LightGBM model to assess feature importance and calculating the influence of each feature on the prediction result. For the importance value of each feature, denoted as important (data importance score), where Gain i is the contribution of feature i to the model performance, and Total Gain is the total contribution of all features. Among them, the feature weight matrix is set to give higher weight to the features with high importance during model training. In combination with the above, the XGBoost prediction model is trained using the artificial intelligence tasks and corresponding data in the training set, and the trained model is tested using the artificial intelligence tasks and corresponding data in the test set. During training, first, the hyperparameters of the XGBoost prediction model are selected, including the number of trees (which can be set to an initial value of 300), the maximum depth of the tree (which can be set to an initial value of 10), the learning rate (which can be set to an initial value of 0.05), the sample sampling ratio (which can be set to an initial value of 0.9), and the random state can be set to 42. Then define the hyperparameter network and search on it, train the model for each hyperparameter combination and evaluate its performance using cross-validation, and finally after the grid search is completed, the best hyperparameter combination is obtained and the model performance in cross-validation is viewed. During validation, the trained model is used to predict the preprocessed test data to obtain the resources required for each task completion: Number_of_CPUs; Number_of_GPUs; Memory_Size; Disk_Space. Compare the predicted data with the actual data. We use mean squared error (MSE) and mean absolute error (MAE) to understand the accuracy of the model and analyze whether the model is within a reasonable range. If it is within the range, the validation is passed and the resource prediction model for each category of artificial intelligence task is obtained. In addition, the trained model can also be incrementally trained using the 'updata()' method in XGBoost. It is worth noting that during training, the learning rate and other hyperparameters need to be adjusted according to the new data to ensure stable updating of the model. At the same time, to prevent the model from relying too much on historical data and causing prediction errors, a model weight decay mechanism can be introduced to gradually reduce the weight of earlier data over time to ensure that the model pays more attention to recent task patterns.

[0084] To sum up, the application first acquires metadata corresponding to a target task. Then, based on the metadata, the target task is classified using a preset task classification tree, and a target classification result is obtained. The preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models. Finally, a resource prediction model corresponding to the target classification result is called to predict the resources of the target task, and predicted target data is obtained; the predicted target data includes the number of CPUs, the number of GPUs, the size of memory, and the disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resources are predicted, improving the accuracy of resource prediction.

[0085] Figure 3 The structure of a resource prediction device provided by an embodiment of the application is shown in the structural schematic diagram. Figure 3 The resource prediction device 300 provided by the embodiment of the application includes:

[0086] The acquisition module 301 is configured to acquire metadata corresponding to a target task.

[0087] The classification module 302 is configured to classify the target task based on the metadata using a preset task classification tree, and obtain a target classification result; the preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models.

[0088] The prediction module 303 is configured to call a resource prediction model corresponding to the target classification result to predict the resources of the target task, and obtain predicted target data; the predicted target data includes the number of CPUs, the number of GPUs, the size of memory, and the disk space.

[0089] As an implementation manner, for how to train the resource prediction model, the above-mentioned resource prediction device 300 further includes a collection module, a classification sub-module, and a training module.

[0090] The collection module is configured to acquire historical task data and corresponding historical resource usage data of each artificial intelligence task from a data center; the historical task data includes task type, task size, task configuration, and time constraints.

[0091] The classification sub-module is configured to classify each artificial intelligence task based on the computing resource architecture of the task request, and obtain a classification result.

[0092] The training module is configured to train an XGBoost prediction model using the historical task data and corresponding historical resource usage data of the artificial intelligence task indicated by each classification result, and obtain a resource prediction model for different categories of artificial intelligence tasks.

[0093] The historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism;

[0094] The period of the first window of the double-layer sliding window is set to 7 days, and the step is set to 1 day;

[0095] The period of the second window of the double-layer sliding window is set to 1 day, and the step is set to 1 hour.

[0096] As an implementation, for how to perform data preprocessing before training, the resource prediction device 300 further includes a data preprocessing module.

[0097] The data preprocessing module is configured to perform data cleaning and data preprocessing on the historical task data and the historical resource usage data.

[0098] The data cleaning includes missing value processing and outlier processing.

[0099] The data preprocessing includes feature encoding and normalization processing.

[0100] As an implementation, for how to construct the preset task classification tree, the resource prediction device 300 further includes a construction module.

[0101] The construction module is configured to construct, based on a computing resource architecture of a task request, a preset task classification tree corresponding to all artificial intelligence tasks.

[0102] Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

[0103] As an implementation, for how to train the resource prediction model, the training module is specifically configured to:

[0104] Based on historical task data and corresponding historical resource usage data of the artificial intelligence task indicated by each classification result, perform task feature importance evaluation using a LightGBM model, and obtain an evaluation result.

[0105] Adjust feature weights for an XGBoost prediction model using the evaluation result.

[0106] As an implementation, for how to perform result display, the resource prediction device 300 further includes a display module.

[0107] The display module is configured to adjust feature weights for an XGBoost prediction model using the evaluation result.

[0108] To sum up, the application first acquires metadata corresponding to a target task. Then, based on the metadata, the target task is classified by using a preset task classification tree, and a target classification result is obtained. The preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models. Finally, a resource prediction model corresponding to the target classification result is called to predict the resources of the target task, and a prediction target data is obtained. The prediction target data includes the number of CPUs, the number of GPUs, the size of memory, and the disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and the resources are predicted, thereby improving the accuracy of resource prediction.

Claims

1. A resource prediction method characterized by, The method comprises: obtaining metadata corresponding to a target task; based on the metadata, using a preset task classification tree to classify the target task, and obtaining a target classification result; the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models; calling a resource prediction model corresponding to the target classification result to perform resource prediction on the target task, and obtaining predicted target data; the predicted target data includes: CPU number, GPU number, memory size and disk space; The resource prediction model is obtained by the following method: obtaining historical task data and corresponding historical resource usage data corresponding to each artificial intelligence task from a data center; the historical task data includes: task type, task size, task configuration and time constraint; Classify each artificial intelligence task based on the computing resource architecture of the task request, and obtain the classification result; Use the historical task data and the corresponding historical resource usage data of the artificial intelligence task indicated by each classification result to train the XGBoost prediction model respectively, and obtain the resource prediction model for different categories of artificial intelligence tasks; The use of historical task data and corresponding historical resource usage data of artificial intelligence tasks indicated by each classification result to train the XGBoost prediction model respectively, comprising: Based on the historical task data and the corresponding historical resource usage data of the artificial intelligence task indicated by each classification result, use the LightGBM model to evaluate the task feature importance, and obtain the evaluation result; Use the evaluation result to adjust the feature weight of the XGBoost prediction model.

2. The method of claim 1, wherein, The historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism; The period of the first window of the double-layer sliding window is set to 7 days, and the step is set to 1 day; The period of the second window of the double-layer sliding window is set to 1 day, and the step is set to 1 hour.

3. The method of claim 1, wherein, Before the use of historical task data and corresponding historical resource usage data of artificial intelligence tasks indicated by each classification result to train the XGBoost prediction model respectively, further comprising: Data cleaning and data preprocessing are performed on the historical task data and the historical resource usage data; The data cleaning includes: missing value processing and outlier processing; The data preprocessing includes: feature encoding and normalization processing.

4. The method of claim 1, wherein, The method further comprises: Based on the computing resource architecture of the task request, a preset task classification tree corresponding to all artificial intelligence tasks is constructed; Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

5. The method of claim 1, wherein, The method further comprises: The predicted value corresponding to the predicted target data is displayed to the user in the form of a table.

6. A resource prediction apparatus characterized by comprising: Comprise: An acquisition module is configured to obtain metadata corresponding to a target task; A classification module is configured to classify the target task based on the metadata using a preset task classification tree, and obtain a target classification result; The preset task classification tree stores a corresponding relationship between different types of tasks and different resource prediction models; The prediction module is configured to perform resource prediction on the target task by calling a resource prediction model corresponding to the target classification result, and obtain prediction target data; The prediction target data includes a number of CPUs, a number of GPUs, a memory size, and a disk space; The collection module is configured to obtain historical task data and corresponding historical resource usage data of each artificial intelligence task from the data center; the historical task data includes a task type, a task size, a task configuration, and a time constraint; The classification submodule is configured to classify each artificial intelligence task based on a computing resource architecture of the task request, and obtain a classification result; The training module is configured to train an XGBoost prediction model by using historical task data and corresponding historical resource usage data of an artificial intelligence task corresponding to each classification result, and obtain a resource prediction model for different types of artificial intelligence tasks; The training module is specifically configured to: perform task feature importance evaluation by using a LightGBM model based on historical task data and corresponding historical resource usage data of an artificial intelligence task corresponding to each classification result, and obtain an evaluation result; adjust feature weights of the XGBoost prediction model by using the evaluation result.

7. A resource prediction device, characterized by, The computer program is stored in the memory and executed by the processor to implement the steps of the resource prediction method according to any one of claims 1 to 5. The computer program is stored in the memory and executed by the processor to implement the steps of the resource prediction method according to any one of claims 1 to 5. ​ 8. A readable storage medium, characterized by, ​

Images