Resource prediction method, device and equipment and readable storage medium

Abstract

The invention discloses a resource prediction method, device and equipment and a readable storage medium, which can be applied to the technical field of artificial intelligence, and the method comprises the following steps: obtaining metadata corresponding to a target task; based on the metadata, utilizing a preset task classification tree to classify the target task, and obtaining a target classification result; corresponding relations between different types of tasks and different resource prediction models are stored in the preset task classification tree; calling a resource prediction model corresponding to the target classification result to perform resource prediction on the target task, and obtaining predicted target data; the predicted target data comprises the number of CPUs, the number of GPUs, the memory size and the disk space. Thus, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then resource prediction is performed, so that the accuracy of resource prediction is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a resource prediction method, device, equipment and readable storage medium. Background Art

[0002] Artificial intelligence tasks include natural language processing, image recognition, data analysis, machine learning, and human-computer interaction. In artificial intelligence tasks, various algorithms and technologies are needed to process and analyze large amounts of data to achieve automated and intelligent applications.

[0003] In order to ensure the smooth progress of tasks, it is often necessary to accurately evaluate the computing resources and time required before executing the task, and then make reasonable resource allocation. When facing various tasks in a large environment, existing resource prediction methods often use the same prediction model for prediction, and the data used in the training of the prediction model is inaccurate, which leads to inaccurate prediction results.

[0004] Therefore, how to improve the accuracy of resource prediction is an issue that technical personnel in this field need to solve urgently. Summary of the invention

[0005] Based on the above problems, the present application provides a resource prediction method, apparatus, device and readable storage medium, which determines the resource prediction model corresponding to the target task based on a preset task classification tree, and then predicts the resources, thereby improving the accuracy of resource prediction.

[0006] In a first aspect, an embodiment of the present application provides a resource prediction method, comprising:

[0007] Get the metadata corresponding to the target task;

[0008] Based on the metadata, the target task is classified using a preset task classification tree to obtain a target classification result; the preset task classification tree stores the corresponding relationship between different types of tasks and different resource prediction models;

[0009] The resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task and obtain predicted target data; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size and the disk space.

[0010] Optionally, the resource prediction model is trained by the following method:

[0011] Obtain historical task data and corresponding historical resource usage data corresponding to each artificial intelligence task from the data center; the historical task data includes: task type, task scale, task configuration, and time constraint;

[0012] Classify each AI task based on the computing resource architecture requested by the task and obtain the classification results;

[0013] The XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, and a resource prediction model for different categories of artificial intelligence tasks is obtained.

[0014] Optionally, the historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism;

[0015] The period of the first window of the double sliding window is set to 7 days, and the step size is set to 1 day;

[0016] The period of the second window of the double sliding window is set to 1 day, and the step size is set to 1 hour.

[0017] Optionally, before respectively training the XGBoost prediction model using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks indicated by each classification result, the method further includes:

[0018] Performing data cleaning and data preprocessing on the historical task data and the historical resource usage data;

[0019] The data cleaning includes: missing value processing and outlier processing;

[0020] The data preprocessing includes: feature encoding and normalization processing.

[0021] Optionally, the method further includes:

[0022] Based on the computing resource architecture of the task request, a preset task classification tree corresponding to all artificial intelligence tasks is constructed;

[0023] Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

[0024] Optionally, the using of the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks indicated by each classification result to train the XGBoost prediction model respectively includes:

[0025] Based on the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, the LightGBM model is used to evaluate the importance of task features and obtain the evaluation results;

[0026] The evaluation results are used to adjust feature weights for the XGBoost prediction model.

[0027] Optionally, the method further includes:

[0028] The predicted values ​​corresponding to the predicted target data are displayed to the user in the form of a table.

[0029] In a second aspect, an embodiment of the present application provides a resource prediction device, including:

[0030] The acquisition module is used to obtain the metadata corresponding to the target task;

[0031] A classification module, used to classify the target task based on the metadata using a preset task classification tree and obtain a target classification result; the preset task classification tree stores the corresponding relationship between different types of tasks and different resource prediction models;

[0032] The prediction module is used to call the resource prediction model corresponding to the target classification result to perform resource prediction on the target task and obtain predicted target data; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size and the disk space.

[0033] In a third aspect, an embodiment of the present application provides a resource prediction device, including:

[0034] Memory for storing computer programs;

[0035] A processor is used to implement the steps of the resource prediction method described above when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the resource prediction method described above are implemented.

[0037] It can be seen from the above technical solutions that compared with the prior art, the present application has the following advantages:

[0038] This application first obtains the metadata corresponding to the target task. Then, based on the metadata, the target task is classified using the preset task classification tree, and the target classification result is obtained. Among them, the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models. Finally, the resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task, and the predicted target data is obtained; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size, and the disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resources are predicted, which improves the accuracy of resource prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a resource prediction method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a preset task classification tree provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a resource prediction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] As mentioned above, existing resource prediction methods have the problem of insufficient accuracy. Specifically, existing resource prediction methods often use a single prediction model to handle all types of tasks, ignoring the differences between different tasks. In addition, the existing technology cannot fully consider the dynamic changes of tasks during training, resulting in inaccurate model training results. The combination of the two makes it difficult for existing resource prediction methods to guarantee the accuracy of resource prediction.

[0043] To solve the above problems, an embodiment of the present application provides a resource prediction method, comprising: first obtaining metadata corresponding to a target task. Then, based on the metadata, the target task is classified using a preset task classification tree, and a target classification result is obtained. Among them, the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models. Finally, the resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task, and the predicted target data is obtained; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size, and the disk space.

[0044] In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resources are predicted, which improves the accuracy of resource prediction.

[0045] It should be noted that the resource prediction method, device, equipment and readable storage medium provided by the present application can be applied to the field of artificial intelligence technology. The above is only an example and does not limit the application field of the resource prediction method, device, equipment and readable storage medium provided by the present application.

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Figure 1 A flowchart of a resource prediction method provided in an embodiment of the present application. Figure 1 As shown, a resource prediction method provided in an embodiment of the present application may include:

[0048] S101: Obtain metadata corresponding to a target task.

[0049] In practical applications, metadata is information used to describe the data attributes of a target task. It is used to support functions such as indicating storage locations, historical data, resource searches, and file records, and can be considered an electronic directory. It describes the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval. There are many types of artificial intelligence tasks. Among the many artificial intelligence tasks, the artificial intelligence tasks input into the resource prediction device are defined as target tasks, and then the metadata information of the target tasks is obtained.

[0050] S102: Based on the metadata, the target task is classified using a preset task classification tree to obtain a target classification result; the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models.

[0051] In practical applications, the preset task classification tree is a hierarchical tree structure, which pre-stores the types corresponding to each task and the corresponding relationship between each task and each resource prediction model. It is worth noting that tasks of the same type correspond to the same resource prediction model, and tasks of different types generally correspond to different resource prediction models. The resource prediction device can quickly determine the type corresponding to the target task and the corresponding resource prediction model based on the metadata of the target task and the pre-stored information on the preset task classification tree.

[0052] S103: Calling a resource prediction model corresponding to the target classification result to perform resource prediction on the target task and obtain prediction target data; the prediction target data includes: number of CPUs, number of GPUs, memory size, and disk space.

[0053] In practical applications, the resource prediction model is pre-trained and is used to predict resources for different types of artificial intelligence tasks. After determining the type of target task (target classification result), the resource prediction device automatically calls the corresponding resource prediction model and performs resource prediction for the target task. The predicted target data includes the number of CPUs, the number of GPUs, the memory size, the disk space, and the estimated time required to complete the task.

[0054] In addition, since the methods of displaying predicted target data are different, the embodiments of the present application can illustrate a possible display method.

[0055] In one embodiment, the method further comprises:

[0056] The predicted values ​​corresponding to the predicted target data are displayed to the user in the form of a table.

[0057] In actual applications, the predicted target data obtained by predicting the target task exists in the form of data, for example, the required number of CPUs is 1 quad-core eight-thread, the required number of GPUs is RTX3060, the required memory is 16GB, etc. It is worth noting that the "number" in the number of CPUs and GPUs does not only refer to Arabic numerals, but more to the models of CPUs and GPUs. The predicted values ​​obtained by these predictions, such as "1 quad-core eight-thread CPU", "RTX3060 GPU", and "16GB memory", etc., will be displayed to the user in a table.

[0058] In addition, since the methods of training resource prediction models are different, the embodiments of the present application can illustrate a possible training method.

[0059] In one case, the resource prediction model is trained by the following method:

[0060] Obtaining historical task data and corresponding historical resource usage data corresponding to each artificial intelligence task from the data center; the historical task data includes: task type, task scale, task configuration, and time constraint;

[0061] Classify each AI task based on the computing resource architecture requested by the task and obtain the classification results;

[0062] The XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, and a resource prediction model for different categories of artificial intelligence tasks is obtained.

[0063] In practical applications, the XGBoost prediction model is used as the basic model, and a resource prediction model corresponding to each type is obtained through a large number of training. Specifically, it is necessary to first collect the historical task data and historical resource usage data corresponding to a large number of artificial intelligence tasks in the data center, and divide them into training sets and test sets in a ratio of 7:3. In the training set, the historical task data includes task type, task scale, task configuration and time constraint. Among them, the task type is used to indicate the type of task; the task scale includes the number of model parameters, the number of data set samples and the data set dimension; the task configuration includes the number of data samples used in each iteration and the number of times the entire data set is traversed during training; the time constraint is used to record the latest completion time of the task, which has an impact on resource scheduling. The prediction target data used for training is the historical resource usage data corresponding to the artificial intelligence tasks in the training set, including the number of CPUs (cores), the number of GPUs, the memory size and the disk space. Then, the collected data is classified according to the computing resource architecture based on the task request, and the classification results of each artificial intelligence task in the training set are obtained, such as task A and task B belong to the large model training class, task C belongs to the large model reasoning class, and task D and task E belong to the machine learning class. Finally, the XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to each category of artificial intelligence tasks, and the resource prediction model for different categories of artificial intelligence tasks is obtained. Continuing with the above example, it can be understood that after the training is completed, when it is used to predict the tasks of the large model training class, the parameter of the resource prediction model called is X; when predicting the tasks of the large model reasoning class, the parameter of the resource prediction model called is Y; when predicting the tasks of the machine learning class, the parameter of the resource prediction model called is Z. It is worth noting that there may be other categories of tasks that are not listed here one by one. The values ​​of X, Y, and Z need to be obtained through actual training, and their values ​​are generally different.

[0064] In one embodiment, the historical resource usage data is recorded in the data center based on a double sliding window mechanism;

[0065] The period of the first window of the double sliding window is set to 7 days, and the step size is set to 1 day;

[0066] The period of the second window of the double sliding window is set to 1 day, and the step size is set to 1 hour.

[0067] In practical applications, the embodiment of the present application pre-sets a real-time data stream pipeline, which can store task resource usage data in chronological order (for training artificial intelligence tasks, the corresponding data is historical resource usage data). These data are collected at fixed times to ensure that the latest task data is included. Its sliding window includes two layers. The window size of the first layer window is 7 days and the step size is 1 day, which is used to capture the medium-term trend of task resource usage; the window size of the second layer window is 1 day and the step size is 1 hour, which is used to capture short-term fluctuations in task resource usage. Among them, capturing short-term fluctuations is to update the prediction model when it is found that the data fluctuates. It can be understood that the historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism. Therefore, when obtaining the historical resource usage data corresponding to each artificial intelligence task, in order to support the sliding window mechanism, it is necessary to first collect a more accurate timestamp.

[0068] In addition, since the methods of data preprocessing before model training are not the same, the embodiments of the present application can illustrate a possible preprocessing method.

[0069] In one case, before the XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks indicated by each classification result, the method further includes:

[0070] Performing data cleaning and data preprocessing on the historical task data and the historical resource usage data;

[0071] The data cleaning includes: missing value processing and outlier processing;

[0072] The data preprocessing includes: feature encoding and normalization processing.

[0073] In practical applications, in order to provide higher quality data support for training resource prediction models, it is necessary to clean and preprocess the collected data (historical task data and the historical resource usage data). Specifically, during the data cleaning process, you can first use the isnull() function of pandas to check whether there are missing values ​​in the data. If the number of missing data samples is small, you can directly delete them. If the number is large, you can use the mean of the feature. Fill in to maintain the integrity of the data. Then perform outlier processing, including outlier detection and processing. Among them, outlier detection uses the interquartile range method to sort the data, calculate IQR, calculate the boundaries and mark outliers; outlier processing includes: deleting outlier samples that are obviously deviated from the normal range and correcting the confirmed outliers to the upper and lower boundary values ​​to ensure the rationality and accuracy of the data. In addition, for category data such as Task_Type, unique hot encoding is used to convert each category into a binary vector so that the model can effectively process this data. For example, mark the training task as 1, the inference task as 0, and so on. Normalize numerical information such as Model_Parameters, Batch_Size, etc. Where X min and X max are the maximum and minimum values ​​of the eigenvalues, respectively, to eliminate the influence of different dimensions and improve the efficiency and accuracy of model training.

[0074] In addition, since there are different ways to construct the preset task classification tree, the embodiment of the present application can illustrate a possible construction method.

[0075] In one embodiment, the method further comprises:

[0076] Based on the computing resource architecture of the task request, a preset task classification tree corresponding to all artificial intelligence tasks is constructed;

[0077] Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

[0078] In practical applications, the collected data can be classified to build a tree structure, which is then used to quickly classify the target tasks. Figure 2 A schematic diagram of a preset task classification tree provided in an embodiment of the present application. Figure 2 As shown, taking the computing resource architecture of the task request as the core, all artificial intelligence tasks are classified in combination with the collected historical task data of artificial intelligence tasks, and then a preset task classification tree is obtained. Specifically, artificial intelligence tasks can be divided into large-scale computing centers, hybrid computing centers, and edge computing centers according to the computing resources they require. Under the computing center, it is further divided into three categories according to the computing power type: general computing, super computing, and intelligent computing. Then, under the computing power type, it is subdivided according to the architecture into Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture. It should be noted that Figure 2Only the Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture divided under the intelligent computing branch of the large-scale computing power center are shown. It can be understood that the general computing and supercomputing branches of the large-scale computing power center, the general computing, supercomputing, and intelligent computing branches of the hybrid computing power center, and the general computing, supercomputing, and intelligent computing branches of the edge computing power center can all be divided into four architectures: Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture. Finally, all artificial intelligence tasks are divided into large-scale training tasks, large-scale reasoning tasks, and machine learning tasks according to the task type, that is, the Volta architecture, Ampere architecture, Hopper architecture, and Ascend architecture are divided into three types: large-scale training tasks, large-scale reasoning tasks, and machine learning tasks. It should be noted that Figure 2 Only one of the divisions under the Hopper architecture is shown. In this way, a four-layer preset task classification tree is formed, which helps to more accurately identify and manage the computing resource requirements of different tasks when facing the target task classification, thereby optimizing resource allocation and scheduling efficiency. It can be understood that in the preset task classification tree, each leaf node corresponds to a category of artificial intelligence tasks, and the resource prediction model needs to be trained separately for each category of artificial intelligence tasks.

[0079] In addition, since the methods for training the XGBoost prediction model are different, the embodiments of the present application can illustrate a possible training method.

[0080] In one case, the XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks indicated by each classification result, including:

[0081] Based on the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, the LightGBM model is used to evaluate the importance of task features and obtain the evaluation results;

[0082] The evaluation results are used to adjust feature weights for the XGBoost prediction model.

[0083] In practical applications, the LightGBM model is introduced to conduct in-depth importance evaluation of the features of each type of artificial intelligence task (combined with the corresponding historical resource usage data to evaluate the features in the historical task data corresponding to each type of artificial intelligence task), and these evaluation results are applied to the XGBoost model to adjust the feature weights in real time, thereby obtaining the resource prediction model corresponding to each type of artificial intelligence task. Specifically, during the model training process, feature calculation is first performed, including using the LightGBM model to evaluate the importance of features and calculating the degree of influence of each feature on the prediction results. The importance value of each feature is recorded as important (data importance score). Gain i is the contribution of feature i to the model performance, and Total Gain is the total contribution of all features. Among them, the feature weight matrix is ​​set to give higher weights to features with high importance during model training. Combined with the above, the XGBoost prediction model is trained using the artificial intelligence tasks and corresponding data in the training set, and the trained model is tested using the artificial intelligence tasks and corresponding data in the test set. During the training process, the hyperparameters of the XGBoost prediction model are first selected, including the number of trees (the initial value can be set to 300), the maximum depth of the tree (the initial value can be set to 10), the learning rate (the initial value can be set to 0.05), the sample sampling ratio (the initial value can be set to 0.9), and the random state can be set to 42. Then define the hyperparameter network and search on it, train the model for each hyperparameter combination, and use cross-validation to evaluate its performance. Finally, after the grid search is completed, obtain the best hyperparameter combination and check the performance of the model in cross-validation. During the verification process, the preprocessed test data is predicted with the trained model to obtain the resources required to complete each task: Number_of_CPUs; Number_of_GPUs; Memory_Size; Disk_Space. Compare the predicted data with the actual data. We calculate the mean square error (MSE) and mean absolute error (MAE) to understand the accuracy of the model and analyze whether the model is within a reasonable range. If it is within the range, the verification is passed, and a resource prediction model for each category of artificial intelligence tasks is obtained. In addition, incremental training can be used to perform incremental training on the trained model using the 'updata()' method in XGBoost. It is worth noting that during the training process, hyperparameters such as the learning rate need to be adjusted according to the new data to ensure the stable update of the model. At the same time, in order to prevent the model from being inaccurate due to over-reliance on historical data, a model weight decay mechanism can be introduced to gradually reduce the weight of older data over time to ensure that the model pays more attention to recent task patterns.

[0084] In summary, the present application first obtains the metadata corresponding to the target task. Then, based on the metadata, the target task is classified using a preset task classification tree, and the target classification result is obtained. Among them, the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models. Finally, the resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task, and the predicted target data is obtained; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size, and the disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resources are predicted, thereby improving the accuracy of resource prediction.

[0085] Figure 3 A schematic diagram of the structure of a resource prediction device provided in an embodiment of the present application. Figure 3 As shown, the resource prediction device 300 provided in the embodiment of the present application includes:

[0086] An acquisition module 301 is used to acquire metadata corresponding to a target task;

[0087] A classification module 302 is used to classify the target task based on the metadata using a preset task classification tree and obtain a target classification result; the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models;

[0088] The prediction module 303 is used to call the resource prediction model corresponding to the target classification result to perform resource prediction on the target task and obtain predicted target data; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size and the disk space.

[0089] As an implementation method, regarding how to train a resource prediction model, the resource prediction device 300 further includes: a collection module, a classification submodule, and a training module;

[0090] A collection module, used to obtain historical task data and corresponding historical resource usage data corresponding to each artificial intelligence task from the data center; the historical task data includes: task type, task scale, task configuration and time constraint;

[0091] The classification submodule is used to classify each artificial intelligence task based on the computing resource architecture of the task request and obtain the classification results;

[0092] The training module is used to train the XGBoost prediction model using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, and obtain resource prediction models for different categories of artificial intelligence tasks.

[0093] Wherein, historical resource usage data is recorded in the data center based on a double-layer sliding window mechanism;

[0094] The period of the first window of the double sliding window is set to 7 days, and the step size is set to 1 day;

[0095] The period of the second window of the double sliding window is set to 1 day, and the step size is set to 1 hour.

[0096] As an implementation method, regarding how to perform data preprocessing before training, the resource prediction device 300 further includes: a data preprocessing module;

[0097] A data preprocessing module, used for performing data cleaning and data preprocessing on the historical task data and the historical resource usage data;

[0098] The data cleaning includes: missing value processing and outlier processing;

[0099] The data preprocessing includes: feature encoding and normalization processing.

[0100] As an implementation method, with respect to how to construct a preset task classification tree, the resource prediction device 300 further includes: a construction module;

[0101] A construction module, used to construct a preset task classification tree corresponding to all artificial intelligence tasks based on the computing resource architecture requested by the task;

[0102] Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

[0103] As an implementation method, regarding how to train a resource prediction model, the above training module is specifically used for:

[0104] Based on the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, the LightGBM model is used to evaluate the importance of task features and obtain the evaluation results;

[0105] The evaluation results are used to adjust feature weights for the XGBoost prediction model.

[0106] As an implementation method, regarding how to display the results, the resource prediction device 300 further includes: a display module;

[0107] A presentation module is used to adjust feature weights for an XGBoost prediction model using the evaluation results.

[0108] In summary, the present application first obtains the metadata corresponding to the target task. Then, based on the metadata, the target task is classified using a preset task classification tree, and the target classification result is obtained. Among them, the preset task classification tree stores the correspondence between different types of tasks and different resource prediction models. Finally, the resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task, and the predicted target data is obtained; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size, and the disk space. In this way, the resource prediction model corresponding to the target task is determined based on the preset task classification tree, and then the resources are predicted, thereby improving the accuracy of resource prediction.

Claims

1. A resource prediction method, characterized in that: The method comprises: Get the metadata corresponding to the target task; Based on the metadata, the target task is classified using a preset task classification tree, and a target classification result is obtained; the preset task classification tree stores the corresponding relationship between different types of tasks and different resource prediction models; The resource prediction model corresponding to the target classification result is called to perform resource prediction on the target task and obtain predicted target data; the predicted target data includes: the number of CPUs, the number of GPUs, the memory size and the disk space.

2. The method according to claim 1, characterized in that The resource prediction model is trained by the following method: Obtaining historical task data and corresponding historical resource usage data corresponding to each artificial intelligence task from the data center; the historical task data includes: task type, task scale, task configuration, and time constraint; Classify each AI task based on the computing resource architecture requested by the task and obtain the classification results; The XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, and a resource prediction model for different categories of artificial intelligence tasks is obtained.

3. The method according to claim 2, characterized in that The historical resource usage data is recorded in the data center based on a double sliding window mechanism; The period of the first window of the double sliding window is set to 7 days, and the step size is set to 1 day; The period of the second window of the double sliding window is set to 1 day, and the step size is set to 1 hour.

4. The method according to claim 2, characterized in that: Before the XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks indicated by each classification result, the method further includes: Performing data cleaning and data preprocessing on the historical task data and the historical resource usage data; The data cleaning includes: missing value processing and outlier processing; The data preprocessing includes: feature encoding and normalization processing.

5. The method according to claim 2, characterized in that: The method further comprises: Based on the computing resource architecture of the task request, a preset task classification tree corresponding to all artificial intelligence tasks is constructed; Each leaf node of the preset task classification tree indicates an artificial intelligence task of a classification result and a corresponding resource prediction model.

6. The method according to claim 2, characterized in that The XGBoost prediction model is trained using the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks indicated by each classification result, including: Based on the historical task data and the corresponding historical resource usage data corresponding to the artificial intelligence tasks referred to by each classification result, the LightGBM model is used to evaluate the importance of task features and obtain the evaluation results; The evaluation results are used to adjust feature weights for the XGBoost prediction model.

7. The method according to claim 1, characterized in that The method further comprises: The predicted values ​​corresponding to the predicted target data are displayed to the user in the form of a table.

8. A resource prediction device, characterized in that: include: The acquisition module is used to obtain the metadata corresponding to the target task; A classification module, used to classify the target task based on the metadata using a preset task classification tree and obtain a target classification result; The preset task classification tree stores the correspondence between different types of tasks and different resource prediction models; A prediction module, used to call a resource prediction model corresponding to the target classification result to perform resource prediction on the target task and obtain predicted target data; The predicted target data includes: number of CPUs, number of GPUs, memory size, and disk space.

9. A resource prediction device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the resource prediction method according to any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the resource prediction method according to any one of claims 1 to 7 are implemented.

Images