Task execution duration prediction model training method and device and electronic equipment

Abstract

The embodiment of the invention provides a task execution time length prediction model training method and device and electronic equipment, and is applied to the technical field of information. And calculating relevancy and importance scores of various types of data in the first task data and the task execution duration through different calculation modes, and screening the first task data to obtain second task data with relatively large influence on the task execution duration. Therefore, the data with the same type as the second task data is used for training and prediction, the data types with low weight and low influence on task execution duration prediction are screened out, and the calculation power waste can be reduced while the task execution duration prediction accuracy can be ensured.

Description

Technical Field

[0001] This application relates to the field of big data analytics technology, and in particular to a method, apparatus, and electronic device for training a task execution duration prediction model. Background Technology

[0002] In the era of big data, the operation of a business may depend on the execution of multiple tasks. However, in practical applications, due to limitations in hardware resources or business requirements, it is necessary to schedule multiple tasks to improve resource utilization. Task execution duration is a crucial factor in task scheduling; it refers to the time required from the start to the end of a task. Therefore, predicting task execution duration helps in effective task orchestration while allocating resources rationally.

[0003] In related technologies, those skilled in the art typically use neural network models to predict task execution time based on input data. Generally, the input data consists of data about a few fixed types of tasks, determined empirically, such as task name and task type. However, not all of this task-specific data input to the neural network model is meaningful for predicting execution time. Therefore, potentially meaningless task-specific data not only fails to help the neural network model predict task execution time but also wastes computational resources. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, and electronic device for training a task execution time prediction model, so as to reduce the waste of computing power. The specific technical solution is as follows:

[0005] In a first aspect of this application, a method for training a task execution duration prediction model is provided, the method comprising:

[0006] Obtain first task data for the sample task, wherein the first task data includes task data and task-related data; the task-related data is data representing the utilization rate of device hardware resources during the execution of the sample task and data representing the execution steps of the sample task.

[0007] Calculate the correlation and importance scores of each data point in the first task data with the task execution time; wherein, the correlation and importance scores are information representing the influence of a certain type of data on the task execution time obtained through different calculation methods;

[0008] Based on the relevance and importance scores of each data point, at least two types of data are selected from the first task data as the second task data.

[0009] The second task data is input into the task execution time prediction model for prediction, and a first prediction result is obtained;

[0010] Calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task;

[0011] Based on the first loss value, the parameters of the task execution time prediction model are adjusted to obtain the trained task execution time prediction model.

[0012] In one possible implementation, calculating the correlation and importance scores between each piece of data in the first task data and the task execution time includes:

[0013] Based on the first grid division and mutual information, the correlation between each data point in the first task data and the task execution time is calculated; the first grid division is a grid specification used to statistically analyze the probability distribution of a certain type of data and the task execution time based on the number of sample tasks; the grid specification includes horizontal and vertical grid quantity information; the mutual information is information representing the degree of mutual dependence between a certain type of data and the task execution time, calculated based on the probability distribution of a certain type of data and the task execution time in the first grid division.

[0014] Based on the contribution value of the first task data to the prediction of the first neural network model, the importance score of each data in the first task data and the task execution time is calculated; the first neural network model is a network model pre-trained using the first task data and the true value of the execution time of the sample task; the contribution value is information representing the contribution of a certain data to the prediction of the first neural network model, calculated based on the split nodes and the gain of the split nodes in the prediction process of each data in the first neural network model.

[0015] In one possible implementation, acquiring the first task data of the sample task includes:

[0016] Obtain the original task data of the sample task; wherein, the original task data is of the same type as the first task data, but the presentation format is different; the presentation format includes at least one of text, numbers, tables or graphics;

[0017] The original task data is represented numerically to obtain the numerically represented original task data.

[0018] The original task data after numerical representation is standardized and normalized to obtain the first task data.

[0019] In one possible implementation, by combining the relevance and importance scores of each data point, at least two types of data are selected from the first task data as the second task data; including:

[0020] Calculate the dependency degree between each pair of data in the first task data and establish a dependency redundancy matrix; wherein, the matrix elements of the dependency redundancy matrix are the numerical values ​​of the dependency degree; the dependency degree is information representing the degree of mutual dependence between the two types of data, calculated based on the probability distribution of the two types of data in the second grid partition; the second grid partition is a grid specification used to statistically analyze the probability distribution of the two types of data, divided according to the number of sample tasks.

[0021] Select the data corresponding to the maximum relevance value from the first task data as the data in the first task sub-data set;

[0022] According to the following formula, select the data with the largest first value from the data other than the first task sub-data set as the data in the first task sub-data set, until the data types in the first task sub-data set are equal to the first preset value;

[0023]

[0024] Wherein, the Score(f) j ) represents the first value, and the f j This represents the j-th type of data among all data other than the first task sub-data set; the MIC(f) j (T) represents the correlation between the j-th type of data and the task execution time; T represents the task execution time; S represents the first task sub-data set, and |S| represents the number of selected data types in the first task sub-data set; f s This represents the s-th type of data in the first task sub-data set; the MIC(f) j ,f s ) represents the dependency degree between the j-th type of data in the data other than the first task sub-data set retrieved according to the dependency redundancy matrix and the s-th type of data in the first task sub-data set;

[0025] Based on the importance scores in descending order, at least two types of data are selected from the first task data as the second task sub-data.

[0026] The data in the first task sub-data set and the second task sub-data are used as the second task data.

[0027] In one possible implementation, the step of calculating the correlation between each piece of data in the first task data and the task execution time based on the first grid division and mutual information includes:

[0028] For each type of data, the grid division for that type of data is determined based on the number of sample tasks.

[0029] For each of the grid partitions, calculate the probability distribution of each data point in the task execution time of the sample task within that grid partition;

[0030] Based on the probability distribution, calculate the mutual information between this type of data and the task execution time in the grid division;

[0031] The mutual information is normalized, and the maximum value of the normalized mutual information in each grid division is selected as the correlation between this type of data and the task execution time.

[0032] In one possible implementation, calculating the importance score of each data point in the first task data relative to the task execution time includes:

[0033] The first neural network model is trained by using the first task data and the true value of the execution time of the sample task to obtain the trained first neural network model.

[0034] For each sample task, the contribution value of each data point is calculated based on the split nodes and the gains of the split nodes in the prediction process of the first neural network model. The contribution value represents the degree of contribution of a certain data point to the prediction of the first neural network model.

[0035] For each category of data in the first task data, calculate the weighted average of the absolute values ​​of each contribution value of that category of data to obtain the importance score of that category of data.

[0036] In a second aspect of this application, a method for predicting task execution time is provided, the method comprising:

[0037] Obtain the third task data for the task to be predicted;

[0038] The third task data is input into the trained task execution time prediction model for prediction to obtain a second prediction result. The task execution time prediction model is trained by any of the methods described in the first aspect of the present application. The second task data consists of at least two types of data input when training the task execution time prediction model. The third task data is of the same type as the second task data.

[0039] In a third aspect of this application, a training apparatus for a task execution duration prediction model is provided, the apparatus comprising:

[0040] The first task data acquisition module is used to acquire the first task data of the sample task, wherein the first task data includes task data and task-related data; the task-related data is data representing the utilization rate of device hardware resources when the sample task is executed, and data representing the execution steps of the sample task.

[0041] The relevance and importance score calculation module is used to calculate the relevance and importance scores of each data point in the first task data with the task execution time; wherein, the relevance and importance scores are information representing the influence of a certain type of data on the task execution time obtained by different calculation methods;

[0042] The data filtering module is used to combine the relevance and importance scores of each data point to select at least two types of data from the first task data as the second task data.

[0043] The first prediction module is used to input the second task data into the task execution time prediction model for prediction and obtain the first prediction result.

[0044] The first loss value calculation module is used to calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task.

[0045] The parameter adjustment module is used to adjust the parameters of the task execution time prediction model according to the first loss value, so as to obtain the trained task execution time prediction model.

[0046] In one possible implementation, the relevance and importance score calculation module includes:

[0047] The correlation calculation submodule is specifically used to calculate the correlation between each data point in the first task data and the task execution time based on the first grid division and mutual information. The first grid division is a grid specification used to statistically distribute the probability distribution of a certain type of data and the task execution time based on the number of sample tasks. The grid specification includes horizontal and vertical grid quantity information. The mutual information is information representing the degree of mutual dependence between a certain type of data and the task execution time, calculated based on the probability distribution of a certain type of data and the task execution time in the first grid division.

[0048] The importance score calculation submodule is specifically used to calculate the importance score of each data point in the first task data relative to the task execution time based on the contribution value of the first task data to the prediction of the first neural network model; the first neural network model is a network model pre-trained using the first task data and the ground truth execution time of the sample task; the contribution value is calculated based on the split nodes and the gain of the split nodes during the prediction process of each data point in the first neural network model, to obtain information representing the contribution of a certain data point to the prediction of the first neural network model.

[0049] In one possible implementation, the first task data acquisition module includes:

[0050] The original task data acquisition submodule is specifically used to acquire the original task data of the sample task; wherein, the original task data is of the same type as the first task data, but the presentation format is different; the presentation format includes at least one of text, numbers, tables or graphics;

[0051] The numericalization submodule is specifically used to represent each data point of the original task data with numerical values ​​to obtain the numerically represented original task data.

[0052] The formatting processing submodule is specifically used to standardize and normalize the original task data after the numerical representation to obtain the first task data.

[0053] In one possible implementation, the data filtering module includes:

[0054] The dependency redundancy matrix establishment submodule is specifically used to calculate the dependency degree between every two types of data in the first task data and establish a dependency redundancy matrix; wherein, the matrix elements of the dependency redundancy matrix are the numerical values ​​of the dependency degree; the dependency degree is information representing the degree of mutual dependence between the two types of data, calculated based on the probability distribution of the two types of data in the second grid partition; the second grid partition is a grid specification used to statistically analyze the probability distribution of the two types of data, divided according to the number of sample tasks;

[0055] The first task sub-data first selection sub-module is specifically used to select the data corresponding to the maximum relevance value from the first task data as the data in the first task sub-data set;

[0056] The first task sub-data second selection sub-module is specifically used to select the data with the largest first value from other data besides the first task sub-data set as the data in the first task sub-data set according to the following formula, until the data type in the first task sub-data set is a first preset value;

[0057]

[0058] Wherein, the Score(f) j ) represents the first value, and the f j This represents the j-th type of data among all data other than the first task sub-data set; the MIC(f) j (T) represents the correlation between the j-th type of data and the task execution time; T represents the task execution time; S represents the first task sub-data set, and |S| represents the number of selected data types in the first task sub-data set; f s This represents the s-th type of data in the first task sub-data set; the MIC(f) j ,f s ) represents the dependency degree between the j-th type of data in the data other than the first task sub-data set retrieved according to the dependency redundancy matrix and the s-th type of data in the first task sub-data set;

[0059] The second task sub-data selection submodule is specifically used to select at least two types of data from the first task data as the second task sub-data according to the order of importance scores from largest to smallest.

[0060] The second task data acquisition submodule is specifically used to take the data in the first task sub-data set and the second task sub-data as the second task data.

[0061] In one possible implementation, the relevance calculation submodule includes:

[0062] The grid partitioning determination unit is specifically used to determine the grid partitioning of each type of data based on the number of sample tasks.

[0063] The probability distribution calculation unit is specifically used to calculate the probability distribution of each data point in the task execution time of the sample task within each grid partition;

[0064] The mutual information calculation unit is specifically used to calculate the mutual information between the data and the task execution time in the grid division according to the probability distribution.

[0065] The normalization processing unit is specifically used to normalize the mutual information and select the maximum value of the normalized mutual information in each grid division as the correlation between this type of data and the task execution time.

[0066] In one possible implementation, the importance score calculation submodule includes:

[0067] The first neural network model training unit is specifically used to train the first neural network model using the first task data and the execution time truth value of the sample task to obtain the trained first neural network model.

[0068] The contribution value calculation unit is specifically used to calculate the contribution value of each data point for each sample task based on the split nodes and the gains of the split nodes in the prediction process of the first neural network model. The contribution value represents the degree of contribution of a certain data point to the prediction of the first neural network model.

[0069] The weighted average unit is specifically used to calculate the weighted average of the absolute values ​​of each contribution value of each data category in the first task data to obtain the importance score of that data category.

[0070] In a fourth aspect of this application, a task execution duration prediction apparatus is provided, the apparatus comprising:

[0071] The third task data acquisition module is used to acquire the third task data of the task to be predicted.

[0072] The second prediction module is used to input the third task data into the trained task execution time prediction model for prediction to obtain a second prediction result. The task execution time prediction model is trained by any of the methods described in the first aspect of the embodiments of this application. The second task data consists of at least two types of data input when training the task execution time prediction model. The third task data is of the same type as the second task data.

[0073] In a fifth aspect of the embodiments of this application, an electronic device is provided, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0074] Memory, used to store computer programs;

[0075] When a processor executes a program stored in memory, it implements the steps of the method described in either the first or second aspect of the embodiments of this application.

[0076] In another aspect of this application, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the above-described task execution duration model training methods and / or task execution duration prediction methods.

[0077] In another aspect of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the above-described task execution duration model training methods and / or task execution duration prediction methods.

[0078] This application provides a method, apparatus, and electronic device for training a task execution time prediction model. The method involves acquiring first task data for a sample task, whereby the first task data includes task data and task-related data. The task-related data represents the utilization rate of device hardware resources during the execution of the sample task, as well as data representing the execution steps of the sample task. The method calculates the correlation and importance score between each data point in the first task data and the task execution time. The correlation and importance score are information representing the influence of a certain type of data on the task execution time, calculated using different methods. Combining the correlation and importance scores of each data point, at least two types of data are selected from the first task data as second task data. The second task data is input into the task execution time prediction model for prediction to obtain a first prediction result. Based on the first prediction result and the true value of the execution time of the sample task, a first loss value is calculated. Based on the first loss value, the parameters of the task execution time prediction model are adjusted to obtain the trained task execution time prediction model.

[0079] By applying the method of this application embodiment, the correlation and importance scores between each data point in the first task data and the task execution time are calculated using different calculation methods. The first task data is then filtered to obtain the second task data that has a greater impact on the task execution time. Therefore, training and prediction are performed using data of the same type as the second task data, while filtering out data types with a lower impact on task execution time prediction. This approach can reduce computational waste while ensuring the accuracy of task execution time prediction. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0081] Figure 1 A flowchart illustrating a method for training a task execution duration prediction model provided in this application embodiment;

[0082] Figure 2 A flowchart illustrating data collection provided in an embodiment of this application;

[0083] Figure 3 A flowchart illustrating data processing and filtering provided in this application embodiment;

[0084] Figure 4A flowchart for training the task execution time prediction model provided in the embodiments of this application;

[0085] Figure 5 A flowchart of a task execution time prediction method provided in an embodiment of this application;

[0086] Figure 6 Another flowchart for the task execution time prediction method provided in the embodiments of this application;

[0087] Figure 7 A flowchart for visualizing the prediction results provided in the embodiments of this application;

[0088] Figure 8 A schematic diagram of a training device for a task execution duration prediction model provided in an embodiment of this application;

[0089] Figure 9 A schematic diagram of a task execution time prediction device provided in an embodiment of this application;

[0090] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0091] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0092] In the field of big data analytics, predicting task execution time is a crucial component for improving system performance. A business operation may rely on the execution of multiple tasks. Predicting the execution time of each task allows technical personnel to schedule tasks accordingly, thereby improving the utilization of system resources. Furthermore, predicting the execution time of each task in advance allows for the early detection of performance bottlenecks during task execution, facilitating optimization of task scheduling and better management of operating equipment costs.

[0093] However, in related technologies, the prediction of task execution time relies solely on the task's own data, resulting in a limited variety of data and significant errors in the predicted execution time. Furthermore, the prediction process often employs basic static algorithms, lacking dynamism and thus failing to anticipate unforeseen circumstances during task execution. Consequently, the actual execution time may exceed the predicted time, leading to task delays.

[0094] Furthermore, in related technologies, when using a fixed number of task data types to predict task execution time based on experience, not all of these task data are meaningful for predicting task execution time. Meaningless task data not only does not help in predicting task execution time, but also wastes computing power.

[0095] To address at least one of the aforementioned problems, a first aspect of this application provides a method for training a task execution duration prediction model, the method comprising: Figure 1 The steps shown are as follows:

[0096] Step S101: Obtain the first task data of the sample task.

[0097] The first task data includes task data and task-related data. Task-related data represents the utilization rate of device hardware resources during sample task execution, as well as data representing the execution steps of the sample task. Sample tasks are multiple historical tasks that have already been executed. Task data includes the task itself and contextual information related to the task. In one example, task data includes, but is not limited to, task execution metadata and SQL (Structured Query Language) operation information for the task.

[0098] The task execution metadata includes, but is not limited to, information such as task ID (Identity Document), task name, the SQL statement of the task, the task submission time, start time, and end time. SQL operation information includes, but is not limited to, the type of SQL statement, such as SELECT (an instruction to query data in a table field), JOIN (join), AGGREGATE (an aggregate function), etc.; and information about the tables and fields involved in the SQL statement.

[0099] Data representing the utilization of device hardware resources during the execution of the sample task includes, but is not limited to, information on the utilization of the running device's CPU (Central Processing Unit), memory, disk I / O (Input / Output), and network I / O. Data representing the execution steps of the sample task can include information such as the logical plan and physical plan of the sample task. The logical plan is an abstract representation of the sample task's execution steps, describing the operations to be performed, the data processing content, and the business logic objectives, without involving the specific execution environment and resource allocation. The physical plan refers to the specific execution scheme of the task, describing how the logical plan is implemented in the actual environment, including information on resource allocation (such as CPU and memory), parallelism settings, data distribution strategies, and specific execution operations.

[0100] In practical applications, the method of this application embodiment can be implemented by a terminal device. Specifically, the terminal device can be an electronic device such as a computer or server that can support network model training.

[0101] Step S102: Calculate the correlation and importance scores of each data point in the first task data with the task execution time.

[0102] To more accurately filter out data that significantly impacts task execution time, we collect as much task-related data as possible when acquiring the first set of task data. Therefore, the first set of task data includes multiple categories of task data. For each category, we calculate the correlation and importance score between that category of task data and the task execution time. For example, if the collected first set of task data includes task name, task submission time, SQL statement type, CPU utilization, memory utilization, physical plan, and logical plan data, we calculate the correlation and importance score between each category and the task execution time.

[0103] In another example, if the first task data includes three types of data: task name, device memory usage during the execution of the sample task, and device CPU usage during the execution of the sample task, and the number of sample tasks is 3; then the collected first task data could be (task name of sample task 1, device memory usage during the execution of sample task 1, device CPU usage during the execution of sample task 2; task name of sample task 2, device memory usage during the execution of sample task 2, device CPU usage during the execution of sample task 2; task name of sample task 3, device memory usage during the execution of sample task 3, device CPU usage during the execution of sample task 3).

[0104] During the calculation, the correlation and importance scores between the task name data (task name of sample task 1, task name of sample task 2, and task name of sample task 3) and the task execution time can be calculated. Similarly, the correlation and importance scores between the device memory usage rate and the task execution time during sample task execution, and the correlation and importance scores between the device CPU usage rate and the task execution time during sample task execution can be calculated.

[0105] Both relevance and importance scores characterize the impact of this type of data on task execution time, but they are calculated differently. For example, when calculating relevance, it can be done by calculating cosine similarity, Euclidean distance, Pearson correlation coefficient, Manhattan distance, etc., to determine the relevance between various task data types and task execution time in the first task data. Alternatively, it can be calculated using the mRMR (Maximum Relevance Minimum Redundancy) algorithm based on the Maximum Information Coefficient (MIC). Taking the calculation of the Pearson correlation coefficient as an example, the calculated relevance is within the range of [-1, 1]. The larger the absolute value of the relevance, the stronger the correlation between this type of task data and task execution time.

[0106] Calculating the importance score of each data point in the first task data can be achieved using a neural network model. For example, the importance of each type of task data can be evaluated through the feature splitting process of a decision tree model; or the importance of that type of task data can be evaluated by randomly deleting task data of a certain type and retraining the neural network model; or the importance of each type of task data can be evaluated by performing feature importance analysis using the gradient boosting algorithm.

[0107] Step S103: Combining the relevance and importance scores of each data point, select at least two types of data from the first task data as the second task data.

[0108] After calculating the correlation between each data point in the first task data and the task execution time, the data in the first task data can be sorted according to the degree of correlation, resulting in a first sorted list. Similarly, after calculating the importance score of each data point in the first task data relative to the task execution time, the data in the first task data can also be sorted according to the magnitude of this importance score, resulting in a second sorted list. When selecting the second task data, one or more types of task data can be selected from the above two sorted lists according to actual business needs, and these will be used as the types of task data in the second task data.

[0109] In practical applications, human experience can be incorporated to assign different weights to various data types in the first task data. This allows for the selection of second task data by combining the relevance, importance scores, and weights of each data type in the first task data. For example, different weights can be assigned to relevance, importance scores, and human experience. By combining these weights with the corresponding values ​​for each data type, the data in the first task can be sorted, and a certain number of categories can be selected as the second task data.

[0110] In one example, the top three relevance tasks are selected from the first set of task data, ranked from highest to lowest relevance. Then, the top three importance tasks are selected, ranked from highest to lowest importance score. These selected task data categories are then used together as the second set of task data. The task data selected based on relevance and the task data selected based on importance score may or may not include the same types of task data. Furthermore, in this embodiment, different selection rules for the second set of task data can be determined based on the calculation method of relevance and importance scores, as detailed in the following embodiments.

[0111] Step S104: Input the second task data into the task execution time prediction model for prediction to obtain the first prediction result.

[0112] Step S105: Calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task.

[0113] Step S106: Adjust the parameters of the task execution time prediction model based on the first loss value to obtain the trained task execution time prediction model.

[0114] In this embodiment, the task execution time prediction model is a deep learning model. In one example, a learning algorithm for predicting task execution time can be selected from Spark MLlib (a machine learning library) based on actual business needs. For instance, any algorithm such as linear regression, random forest, or gradient boosting tree can be chosen to predict task execution time.

[0115] When training the task execution time prediction model, the second task data is selected as the input feature, and the task execution time is output as the label. In one example, the selection and training of the task execution time prediction model are implemented using the following code:

[0116] from pyspark.ml.feature import VectorAssembler

[0117] from pyspark.ml.regression import RandomForestRegressor

[0118] from pyspark.ml.evaluation import RegressionEvaluator

[0119] from pyspark.ml.tuning import ParamGridBuilder, CrossValidator

[0120] # Label and feature selection

[0121] assembler = VectorAssembler(inputCols=["cpu_usage", "memory_usage", "disk_io", "network_io"], outputCol="features")

[0122] data = assembler.transform(df_clean)

[0123] (training_data, test_data) = data.randomSplit([0.8, 0.2])

[0124] # Model selection and training

[0125] rf = RandomForestRegressor(labelCol="execution_time", featuresCol="features")

[0126] paramGrid = ParamGridBuilder().addGrid(rf.numTrees, [10, 50, 100]).build()

[0127] crossval = CrossValidator(estimator=rf, estimatorParamMaps=paramGrid, evaluator=RegressionEvaluator(labelCol="execution_time"), numFolds=5)

[0128] cvModel = crossval.fit(training_data)

[0129] When training the model, the optimal parameters for the task execution time prediction model can be selected using methods such as cross-validation, regularization, and network / random search. In one example, cross-validation is used to select the optimal parameters for the task execution time prediction model. For instance, the following code can be added to the code above for model evaluation and parameter selection:

[0130] #Model Evaluation

[0131] predictions=cvModel.transform(test_data)

[0132] evaluator=RegressionEvaluator(labelCol="execution_time")

[0133] rmse=evaluator.evaluate(predictions)

[0134] print(f"Root Mean Squared Error(RMSE)on test data={rmse}").

[0135] By applying the method of this application embodiment, the correlation and importance scores between each data point in the first task data and the task execution time are calculated using different calculation methods. The first task data is then filtered to obtain the second task data that has a greater impact on the task execution time. Therefore, training and prediction are performed using data of the same type as the second task data, while filtering out data types with lower weights and less impact on task execution time prediction. This approach can reduce computational waste while ensuring the accuracy of task execution time prediction.

[0136] In one possible implementation, the relevance and importance scores can be calculated in step S102 according to the following steps:

[0137] Step 1: Based on the first grid division and mutual information, calculate the correlation between each data point in the first task data and the task execution time.

[0138] The first grid is a grid specification used to statistically analyze the probability distribution of a certain type of data and the execution time of a task, based on the number of sample tasks. The grid specification includes horizontal and vertical grid quantity information. Mutual information is information that characterizes the degree of interdependence between a certain type of data and the execution time of a task, calculated based on the probability distribution of a certain type of data and the execution time of a task in the first grid.

[0139] In one example, the first grid partition is a grid representing the probability distribution of statistical data and task execution time, based on the number of sample tasks. The horizontal and vertical number of grid elements in the grid specification cannot exceed n. 0.6 Here, n represents the number of sample tasks. For example, if the number of sample tasks is 6, then the number of rows and columns in the grid cannot exceed 6. 0.6 ≈2.93; therefore, the grid can be divided into 1*1 grid, 1*2 grid, 2*1 grid, and 2*2 grid; then, for the data in the first task data that needs to be relevance calculated, the probability distribution of the data and the execution time of each sample task in the above grid specifications can be statistically analyzed.

[0140] In practical applications, before calculating the correlation between each data point in the first task data and the task execution time, each data point can be encoded. For example, numerical features can be standardized using Z-score (standardized variable) or Min-Max (Min-Max Normalization); nominal features (e.g., task name, username) can be one-hot encoded; and ordinal features (e.g., the priority of sample tasks) can be integer encoded.

[0141] Step 2: Based on the contribution value of the first task data to the prediction of the first neural network model, calculate the importance score of each data point in the first task data in relation to the task execution time.

[0142] The first neural network model is a network model pre-trained using the first task data and the ground truth execution time of the sample task. The contribution value is calculated based on the split nodes and their gains during the prediction process of each data point in the first neural network model, representing the contribution of a particular data point to the prediction of the first neural network model. In one example, this first neural network model is a gradient boosting model, such as XGBoost (ExtremeGradient Boosting) or LightGBM (Light Gradient Boosting Machine).

[0143] By applying the method of this application embodiment, the correlation and importance scores between various types of data in the first task data and the task execution time are calculated using different calculation methods. The first task data is then filtered to obtain the second task data that has a greater impact on the task execution time. Therefore, training and prediction are performed using data of the same type as the second task data, while filtering out data types with lower weights and less impact on task execution time prediction. This approach can reduce computational waste while ensuring the accuracy of task execution time prediction.

[0144] In one possible implementation, step one above can be achieved by calculating the relevance through the following steps:

[0145] Step 1: For each type of data, determine the grid division of the data based on the number of sample tasks.

[0146] Step 2: For each grid partition, calculate the probability distribution of each data point in the task execution time of that data type and the sample task within that grid partition.

[0147] Step 3: Based on the probability distribution, calculate the mutual information between this type of data and the task execution time in the grid division.

[0148] Step 4: Normalize the mutual information and select the maximum value of the normalized mutual information in each grid division as the correlation between this type of data and the task execution time.

[0149] Among them, the horizontal and vertical number of grids in the grid specification cannot exceed n. 0.6 Let n represent the number of sample tasks. Taking the example of n=6 sample tasks, for a certain type of data in the first task data for which relevance needs to be calculated, if the value of this data is X and the value of the task execution time is Y, then assume that the (X, Y) collected from each sample task are (1,3), (2,6), (3,9), (4,12), (5,15), and (6,18) respectively. Then, for each grid size, the probability distribution of (X, Y) in that grid size is statistically analyzed.

[0150] Taking a 2x2 grid as an example, the X space can be divided into X<3.5 and X>3.5, and the Y space can be divided into Y<10.5 and Y>10.5. The probability distribution of each (X,Y) value in the 2x2 grid is shown in Table 1 below:

[0151] Table 1

[0152] Y<10.5 Y>10.5 X<3.5 3 / 6 0 X>3.5 0 3 / 6

[0153] When calculating the probability distribution of each (X,Y) value, we can calculate its joint probability and marginal probability. The joint probability refers to the probability that both events occur simultaneously, for example, P(X<3.5, Y<10.5) = 0.5; the marginal probability refers to the probability that a single event occurs, for example, P(X<3.5) = 0.5. After calculating the joint and marginal probabilities, mutual information can be calculated using the following formula:

[0154]

[0155] Where I(X,Y) represents the mutual information between X and Y, that is, the mutual information between a certain type of data in the first task data in this application embodiment and the task execution time, X i Y represents the i-th X event; for example, in the example above, X events include X<3.5 and X>3.5 events. j This represents the i-th Y event. For example, in the example above, Y events include Y<10.5 and Y>10.5 events.

[0156] In a 2x2 grid, the mutual information between this type of data and the task execution time is 1. In practical applications, to better compare the mutual information, the mutual information of each grid can be normalized to fall within the (0, 1) interval. From the normalized mutual information corresponding to each first grid division (i.e., each divided grid division), the maximum value is selected as the correlation between this type of data and the task execution time.

[0157] By applying the method of this application embodiment, the correlation strength between each data point and the task execution time can be measured by calculating the maximum information coefficient (MIC) (i.e., the relevance in this application embodiment). This calculation method can capture a wide range of relationship types, including linear, nonlinear, and non-functional relationships. It can ensure that different types of relationships (such as linear and sine waves) are given similar scores when the noise levels are similar, thereby ensuring its universality and fairness, making the calculated relevance more accurate. Furthermore, when filtering the data in the first task data based on the relevance, it can more accurately filter out the data that has a greater impact on the task execution time.

[0158] In one possible implementation, step two above can be achieved by calculating the importance score through the following steps:

[0159] Step A: Train the first neural network model using the first task data and the true values ​​of the execution time of the sample task to obtain the trained first neural network model.

[0160] Step B: For each sample task, calculate the contribution value of each data point based on the split nodes and the gains of the split nodes during the prediction process of the first neural network model in the first task data.

[0161] Step C: For each data category in the first task data, calculate the weighted average of the absolute values ​​of each contribution value of that data category to obtain the importance score of that data category.

[0162] The contribution value represents the degree to which a particular data point contributes to the prediction of the first neural network model. In practical applications, the first neural network model can be a gradient boosting tree model. An interpretation tool is used to interpret the first neural network model, obtaining the splitting path for each data point in the first task data. This interpretation tool can be a TreeExplainer tool specifically designed for tree models. In the first neural network model, the first task data is split at splitting nodes according to preset splitting rules. The task execution time is predicted based on the gain of each splitting node. The gain of a splitting node refers to the magnitude of the gain that splitting node brings to the prediction baseline. In one example, the prediction baseline can be calculated by averaging the predicted values ​​of each sample task.

[0163] In another example, the baseline value for the task execution time prediction of the first neural network model is 15. In this model, if the preset splitting rule is: if the root node data A ≤ 1.5, then enter the left child node, and the predicted value of the left child node is 10; otherwise, enter the right child node. If data B ≤ 0.8, then the predicted value is 20; otherwise, the predicted value is 30. The average predicted value of the right subtree is 25. In this example, if the first task data includes data A and data B, and A = 2, B = 0.5; then when the first neural network model makes its prediction, since data A = 2 > 1.5, it enters the right subtree, and data B = 0.5 ≤ 0.8, resulting in a final predicted value of 20.

[0164] In this example, data A pushes the prediction from the baseline of 15 to the mean of the right subtree of 25, with a gain of +10 at the split node; data B reduces the prediction from 25 to 20 in the right subtree, so the gain of data B at the split node is -5. Therefore, it is calculated that in the sample corresponding to this example, the contribution value of data A is 10, and the contribution value of data B is -5.

[0165] For data A in the first task data, the importance score of data A can be obtained by calculating the weighted average of the absolute values ​​of the contributions of data A in the first task data of each sample.

[0166] For example, the importance score can be calculated using the following formula:

[0167]

[0168] Where GlobalSHAPImportancej represents the importance score; m represents the number of sample tasks; and shapvaluesi,j represents the contribution value of the j-th data in the first task data corresponding to the i-th sample task.

[0169] In practical applications, after the first neural network model has been trained, the importance score can be obtained directly by calling the relevant functions. For example, the following code can be used to analyze the importance of various task data in the first task data by examining the `feature_importances_` attribute.

[0170]

[0171] By applying the method of this application embodiment, the importance score of various task data in the first task data relative to the task execution time can be calculated through different calculation methods. This facilitates subsequent filtering of the first task data based on the importance score, obtaining task data with higher relevance for model training and prediction, thus reducing computational waste while ensuring accuracy. Furthermore, this calculation method can interpret the data in global or local predictions, clearly demonstrating how each data point pushes the predicted value from the baseline to the final predicted value. Therefore, it ensures the reliability of the calculated importance score, and compared to other methods, the confidence level of the importance score calculated by this method is higher.

[0172] In one possible implementation, the original task data of the sample task can be obtained first, and the original task data can be further processed to obtain the first task data. For example, the first task data can be obtained through the following steps:

[0173] Step (1): Obtain the original task data of the sample task.

[0174] Step (2): Represent each data point of the original task data with numerical values ​​to obtain the original task data after numerical representation.

[0175] Step (3): Standardize and normalize the original task data after numerical representation to obtain the first task data.

[0176] The original task data and the first task data have the same data types but different presentation formats; the presentation formats include at least one of text, numbers, tables, or graphics. In one example, the original task data includes task metadata, SQL operation information, resource usage information, and execution plan information. In another example, the original task data may also include task execution frequency characteristics (e.g., taskLaunchNum, taskFailedNum), data reading characteristics (e.g., inputBytes, shuffleReadBytes), memory usage characteristics (e.g., usedMemoryBytes, totalMemoryBytes), and network transmission characteristics (e.g., shuffleWriteBytes, inputRecords).

[0177] In practical applications, raw task data can be obtained through Spark (a fast and general-purpose cluster computing platform) listeners, Spark's REST API (an interface for interacting with the Spark cluster via HTTP requests), or log files. For example, the following code retrieves raw task data:

[0178] #Example data loading and transformation

[0179] spark=SparkSession.builder.appName("DataProcessing").getOrCreate()

[0180] df=spark.read.json("hdfs: / / / path / to / json / data").

[0181] In practical applications, not all raw task data is represented numerically. Therefore, it is necessary to process the non-numerical data into numerical values ​​so that it can be recognized by machines. In one example, different numbers can be used to represent the field types of the task data. For instance, "1" can be used to represent character data, and "2" can be used to represent integer data.

[0182] After representing the raw task data as numerical task data that can be recognized by machines, data cleaning can be performed. This involves standardizing and normalizing the numerical raw task data to remove outliers, missing data, and duplicate data. In one example, the character length of each type of task data can be identified, and data with inconsistent lengths can be removed. In practical applications, data cleaning can be implemented using the following code:

[0183] #Data Cleaning

[0184] df_clean=df.na.drop("any").

[0185] The method described in this application can be used to clean and format the raw data. The raw data can be processed by numericalization, standardization and normalization to transform it into first task data that can be recognized by the machine, thereby facilitating the subsequent screening of the first task data.

[0186] In one possible implementation, after calculating the relevance and importance scores of each data point, the following steps can be used to combine the relevance and importance scores of each data point to select at least two types of data from the first task data as the second task data:

[0187] Step a: Calculate the dependency between every two types of data in the first task data and establish a dependency redundancy matrix.

[0188] The matrix elements of the dependency redundancy matrix are the values ​​of the dependency; the dependency is information representing the degree of mutual dependence between the two types of data, calculated based on the probability distribution of the two types of data in the second grid partition; the second grid partition is a grid size used to statistically analyze the probability distribution of the two types of data, based on the number of sample tasks.

[0189] In this embodiment of the application, since the number of sample tasks is fixed, the quantity of each type of data in the first task data is also consistent with the number of sample tasks. Therefore, the grid specification corresponding to the second grid division is the same as the grid specification corresponding to the first grid division. The only difference between the two is the statistical probability distribution data.

[0190] In the first task data, the dependency between any two categories can be calculated using the method described above for calculating the correlation between a certain category of data and the task execution time. For example, for each second grid division, the probability distribution of the two categories of data under that second grid division can be statistically analyzed, and the correlation coefficient between the two categories of data can be calculated. The correlation coefficients are then normalized, and the maximum value among the normalized correlation coefficients is selected as the dependency between the two categories of data. The calculation method for the correlation coefficient is the same as the calculation method for mutual information in the above embodiment, and will not be repeated here.

[0191] A dependency redundancy matrix is ​​constructed using the calculated dependencies. In one example, the first task data contains three data types: a1, a2, and a3. The dependency between a1 and a2 is 0.92, the dependency between a2 and a3 is 0.78, and the dependency between a1 and a3 is 0.45. The resulting dependency redundancy matrix is:

[0192]

[0193] Step b: Select the data corresponding to the maximum relevance from the first task data as the data in the first task sub-data set.

[0194] Step c: According to the following formula, select the data with the largest first value from the data other than the first task sub-data set as the data in the first task sub-data set, until the data types in the first task sub-data set are equal to the first preset value.

[0195]

[0196] Among them, Score(f j ) represents the first value, f j f represents the j-th type of data in the data other than the first task sub-data set; MIC(fj,T) represents the correlation between the j-th type of data and the task execution time; T represents the task execution time; S represents the first task sub-data set, and |S| represents the number of selected data types in the first task sub-data set; f s Represents the s-th type of data in the first task sub-data set; MIC(f j ,f s ) represents the dependency between the j-th type of data and the s-th type of data in the first task sub-data set, among the other data retrieved according to the dependency redundancy matrix.

[0197] In practical applications, a greedy algorithm based on the principle of Minimum Redundancy Maximum Relevance (mRMR) can be used to iteratively select K types of data to obtain the first task sub-data set, where K is a first preset value. This first preset value can be set according to actual needs, for example, it can be a value such as 3, 5, or 8.

[0198] Step d: Select at least two types of data from the first task data as the second task sub-data, in descending order of importance score.

[0199] Step e: Combine the data from the first task sub-data set with the second task sub-data set to form the second task data.

[0200] In practical applications, the data can be sorted in descending order based on the importance scores of each type of data in the first task data, and the top N types of data can be selected as the sub-data for the second task.

[0201] By applying the method of this application embodiment, the correlation and importance scores between various types of data in the first task data and the task execution time are calculated using different calculation methods. This allows for the filtering of the first task data, resulting in second task data that has a greater impact on the task execution time. Therefore, training and prediction are performed using data of the same types as the second task data, filtering out data types with lower weights and less impact on task execution time prediction. This ensures the accuracy of task execution time prediction while reducing computational waste. Furthermore, when filtering based on correlation, the dependency redundancy matrix between data points can be considered. Data with high dependency in the dependency redundancy matrix is ​​eliminated, ensuring rich information representation while avoiding information overlap. This avoids multiple calculations of overlapping information and reduces computational waste.

[0202] In one possible implementation, the method of this application embodiment is applied to a distributed system, and the original task data, the second task data, and the trained task execution time prediction model can all be stored in a first distributed storage system.

[0203] In one example, the first task data, the second task data, and the trained task execution time prediction model are stored in HDFS (Hadoop Distributed File System). When retrieving this data, it can be obtained directly from the storage system without having to collect it sequentially according to data type.

[0204] The following example illustrates the task execution time model training method of this application embodiment, such as... Figure 2 As shown, data collection is performed first to obtain the raw task data for each sample task. Task metadata, SQL operation information, resource usage information, and execution plan information are collected separately, and the collected raw task data is stored in HDFS. There is no specific order in which the raw task data is collected; it can be collected sequentially or synchronously. Figure 2 This is just one example.

[0205] Figure 3The diagram illustrates the process of data processing and filtering to obtain the second task data in this embodiment. The raw task data of the collected sample tasks is loaded from HDFS. The raw task data is then cleaned to remove and repair null values, outliers, and duplicate data. The raw task data is then transformed, including numericalization, standardization, normalization, and time format conversion, into machine-readable first task data. Feature engineering is performed on the first task data, calculating the correlation and importance scores between various task data types and task execution times. Based on these scores, the first task data is filtered to obtain the second task data. The second task data is then converted into a vector format and stored in HDFS for subsequent training or prediction by the task execution time prediction model.

[0206] Figure 4 The diagram illustrates the training process for the task execution time prediction model. Processed second task data is loaded directly from HDFS, serving as the model's input features. The task execution time of the sample tasks is used as the model's label. A machine learning algorithm (e.g., gradient boosting tree) is selected to train the task execution time prediction model. Optimal parameters are selected through cross-validation, and the model is evaluated to obtain the trained task execution time prediction model. Similarly, the trained task execution time prediction model is stored in HDFS for subsequent direct retrieval for task execution time prediction.

[0207] By applying the method of this application embodiment, the original task data, the second task data, and the trained task execution time prediction model can all be stored in the first distributed storage system. This allows data or models to be directly obtained from the first distributed storage system, avoiding repeated data collection or model training processes and simplifying the training of the task execution time prediction model and subsequent prediction operations.

[0208] In a second aspect of this application, a method for predicting task execution time is provided, the method comprising: Figure 5 The steps shown are as follows:

[0209] Step S501: Obtain the third task data of the task to be predicted.

[0210] Step S502: Input the third task data into the trained task execution time prediction model for prediction to obtain the second prediction result.

[0211] The task execution time prediction model is trained using any method of the first aspect of this application and can be directly loaded from the first distributed storage system. The second task data consists of at least two types of data input when training the task execution time prediction model; the third task data has the same data types as the second task data. When collecting the third task data, it can be collected only according to the data types contained in the second task data, and the data can be processed to transform it into a feature vector that can be recognized and processed by a machine.

[0212] like Figure 6 The diagram shows another flowchart of the task execution time prediction method provided in this application embodiment. A pre-trained task execution time prediction model is loaded from HDFS. The collected third task data (i.e., new task data) is converted into a feature vector and input into the task execution time prediction model for prediction, resulting in a second prediction result. In practical applications, the second prediction result is also stored in HDFS so that tasks can be scheduled according to the prediction result.

[0213] In one example, the following code transforms and predicts feature vectors from third-task data:

[0214] #New Task Data Conversion

[0215] new_task_data=assembler.transform(new_task_df)

[0216] #predict

[0217] predictions=cvModel.transform(new_task_data)

[0218] predictions.select("task_id","prediction").show().

[0219] After obtaining the second prediction result, it can be displayed using visualization tools to facilitate better task scheduling or system optimization. Tools such as Grafana (a cross-platform open-source data visualization tool) and Tableau (a visualization analytics platform) can be used to display the second prediction result. Figure 7 As shown, the second prediction result is loaded from HDFS, displayed using Grafana, and the tasks are scheduled based on the second prediction result.

[0220] By applying the method of the embodiments of this application, a well-trained task execution time prediction model can be used to directly collect task data that is meaningful for predicting task execution time, thereby ensuring the accuracy of task execution time prediction while reducing the waste of computing power.

[0221] In a third aspect of this application, a training apparatus for a task execution duration prediction model is provided, the apparatus comprising: Figure 8 The structure shown is as follows:

[0222] The first task data acquisition module 801 is used to acquire the first task data of the sample task. The first task data includes task data and task-related data. The task-related data is data representing the utilization rate of device hardware resources when the sample task is executed, and data representing the execution steps of the sample task.

[0223] The relevance and importance score calculation module 802 is used to calculate the relevance and importance scores of each data in the first task data and the task execution time; wherein, the relevance and importance scores are information representing the influence of a certain type of data on the task execution time obtained by different calculation methods.

[0224] The data filtering module 803 is used to select at least two types of data from the first task data as the second task data by combining the relevance and importance scores of each data.

[0225] The first prediction module 804 is used to input the second task data into the task execution time prediction model for prediction and obtain the first prediction result.

[0226] The first loss value calculation module 805 is used to calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task.

[0227] The parameter adjustment module 806 is used to adjust the parameters of the task execution time prediction model based on the first loss value, so as to obtain the trained task execution time prediction model.

[0228] In one possible implementation, the relevance and importance score calculation module includes:

[0229] The correlation calculation submodule is specifically used to calculate the correlation between each data point in the first task data and the task execution time based on the first grid division and mutual information. The first grid division is a grid specification used to statistically analyze the probability distribution of a certain type of data and the task execution time based on the number of sample tasks. The grid specification includes horizontal and vertical grid quantity information. The mutual information is information that characterizes the degree of interdependence between a certain type of data and the task execution time, calculated based on the probability distribution of a certain type of data and the task execution time in the first grid division.

[0230] The importance score calculation submodule is specifically used to calculate the importance score of each data point in the first task data relative to the task execution time based on the contribution value of the first task data to the prediction of the first neural network model. The first neural network model is a network model pre-trained using the first task data and the true values ​​of the execution time of the sample tasks. The contribution value is calculated based on the split nodes and the gain of the split nodes during the prediction process of each data point in the first neural network model, and the information representing the contribution of a certain data point to the prediction of the first neural network model is obtained.

[0231] In one possible implementation, the first task data acquisition module includes:

[0232] The original task data acquisition submodule is specifically used to acquire the original task data of the sample task; wherein, the original task data and the first task data have the same data type but different presentation formats; the presentation formats include at least one of text, numbers, tables or graphics.

[0233] The numerical representation submodule is specifically used to represent each data point of the original task data with numerical values, resulting in the numerically represented original task data.

[0234] The formatting processing submodule is specifically used to standardize and normalize the raw task data after numerical representation to obtain the first task data.

[0235] In one possible implementation, the data filtering module includes:

[0236] The dependency redundancy matrix establishment submodule is specifically used to calculate the dependency degree between each pair of data in the first task data and establish a dependency redundancy matrix. The matrix elements of the dependency redundancy matrix are the numerical values ​​of the dependency degree. The dependency degree is information representing the degree of mutual dependence between the two data types, calculated based on the probability distribution of the two data types in the second grid partition. The second grid partition is a grid specification used to statistically analyze the probability distribution of the two data types based on the number of sample tasks.

[0237] The first task sub-data first selection sub-module is specifically used to select the data corresponding to the maximum relevance from the first task data as the data in the first task sub-data set.

[0238] The first task sub-data second selection sub-module is specifically used to select the data with the largest first value from other data besides the first task sub-data set as the data in the first task sub-data set according to the following formula, until the data type in the first task sub-data set is the first preset value;

[0239]

[0240] Among them, Score(fj ) represents the first value, f j This represents the j-th type of data in the data other than the first task sub-data set; MIC(f j (T) represents the correlation between the j-th type of data and the task execution time; T represents the task execution time; S represents the first task sub-data set, and |S| represents the number of selected data types in the first task sub-data set; f s Represents the s-th type of data in the first task sub-data set; MIC(f j ,f s ) represents the dependency between the j-th type of data and the s-th type of data in the first task sub-data set, among the other data retrieved according to the dependency redundancy matrix.

[0241] The second task sub-data selection submodule is specifically used to select at least two types of data from the first task data as the second task sub-data in descending order of importance score.

[0242] The second task data acquisition submodule is specifically used to take the data in the first task subdata set and the second task subdata as the second task data.

[0243] In one possible implementation, the relevance calculation submodule includes:

[0244] The grid division determines the unit, specifically for each type of data, based on the number of sample tasks, to determine the grid division for that type of data.

[0245] The probability distribution calculation unit is specifically used to calculate the probability distribution of each data point in the task execution time of the sample task within each grid partition.

[0246] The mutual information calculation unit is specifically used to calculate the mutual information between this type of data and the task execution time in the grid division based on the probability distribution.

[0247] The normalization processing unit is specifically used to normalize the mutual information and select the maximum value of the normalized mutual information in each grid division as the correlation between this type of data and the task execution time.

[0248] In one possible implementation, the importance score calculation submodule includes:

[0249] The first neural network model training unit is specifically used to train the first neural network model using the first task data and the true values ​​of the execution time of the sample task, so as to obtain the trained first neural network model.

[0250] The contribution value calculation unit is specifically used to calculate the contribution value of each data point for each sample task, based on the split nodes and the gains of the split nodes in the prediction process of the first neural network model. The contribution value represents the degree of contribution of a certain data point to the prediction of the first neural network model.

[0251] The weighted average unit is specifically used to calculate the weighted average of the absolute values ​​of each contribution value of each data category in the first task data, and obtain the importance score of that data category.

[0252] The apparatus of this application calculates the correlation and importance scores between various types of data in the first task data and the task execution time using different calculation methods, filters the first task data, and obtains the second task data that has a greater impact on the task execution time. Therefore, training and prediction are performed using data of the same type as the second task data, while filtering out data types with lower weights and less impact on task execution time prediction. This ensures the accuracy of task execution time prediction while reducing computational waste.

[0253] In a fourth aspect of this application, a task execution duration prediction apparatus is provided, the apparatus comprising: Figure 9 The structure shown is as follows:

[0254] The third task data acquisition module 901 is used to acquire the third task data of the task to be predicted.

[0255] The second prediction module 902 is used to input the third task data into the trained task execution time prediction model for prediction and obtain a second prediction result. The task execution time prediction model is trained by any of the methods in the first aspect of the embodiments of this application. The second task data consists of at least two types of data input when training the task execution time prediction model. The third task data has the same data type as the second task data.

[0256] The apparatus of this application embodiment can directly collect task data that is meaningful for predicting task execution time by using a trained task execution time prediction model, thereby ensuring the accuracy of task execution time prediction while reducing the waste of computing power.

[0257] This application also provides an electronic device, such as... Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0258] Memory 1003 is used to store computer programs;

[0259] When processor 1001 executes a program stored in memory 1003, it performs the following steps:

[0260] Obtain the first task data of the sample task, wherein the first task data includes task data and task-related data; the task-related data is the data representing the utilization rate of device hardware resources when the sample task is executed, and the data representing the execution steps of the sample task.

[0261] Calculate the correlation and importance scores of each data point in the first task data with the task execution time; where the correlation and importance scores are information representing the influence of a certain type of data on the task execution time, obtained through different calculation methods.

[0262] Based on the relevance and importance scores of each data point, at least two types of data are selected from the data of the first task as the data of the second task.

[0263] The second task data is input into the task execution time prediction model for prediction, and the first prediction result is obtained.

[0264] Calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task;

[0265] Based on the first loss value, the parameters of the task execution time prediction model are adjusted to obtain the trained task execution time prediction model.

[0266] or,

[0267] Obtain the third task data for the task to be predicted;

[0268] The third task data is input into the trained task execution time prediction model for prediction to obtain a second prediction result. The task execution time prediction model is trained by any of the methods in the first aspect of the embodiments of this application. The second task data consists of at least two types of data input when training the task execution time prediction model. The third task data has the same data type as the second task data.

[0269] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0270] The communication interface is used for communication between the aforementioned terminal and other devices.

[0271] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0272] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0273] In another embodiment provided in this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, it implements the task execution time prediction model training method or the task execution time prediction method described in any of the above embodiments.

[0274] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the task execution time prediction model training methods or task execution time prediction methods described in the above embodiments.

[0275] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0276] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0277] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0278] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for training a task execution duration prediction model, characterized in that, The method includes: Obtain first task data for the sample task, wherein the first task data includes task data and task-related data; the task-related data is data representing the utilization rate of device hardware resources during the execution of the sample task and data representing the execution steps of the sample task. Calculate the correlation and importance scores of each data point in the first task data with the task execution time; wherein, the correlation and importance scores are information representing the influence of a certain type of data on the task execution time obtained through different calculation methods; Based on the relevance and importance scores of each data point, at least two types of data are selected from the first task data as the second task data. The second task data is input into the task execution time prediction model for prediction, and a first prediction result is obtained; Calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task; Based on the first loss value, the parameters of the task execution time prediction model are adjusted to obtain the trained task execution time prediction model.

2. The method according to claim 1, characterized in that, The calculation of the correlation and importance scores between each data point in the first task data and the task execution time includes: Based on the first grid division and mutual information, the correlation between each data point in the first task data and the task execution time is calculated; the first grid division is a grid specification used to statistically analyze the probability distribution of a certain type of data and the task execution time based on the number of sample tasks; the grid specification includes horizontal and vertical grid quantity information; the mutual information is information representing the degree of mutual dependence between the data type and the task execution time, calculated based on the probability distribution of a certain type of data and the task execution time in the first grid division. Based on the contribution value of the first task data to the prediction of the first neural network model, the importance score of each data in the first task data and the task execution time is calculated; the first neural network model is a network model pre-trained using the first task data and the true value of the execution time of the sample task; the contribution value is information representing the contribution of a certain data to the prediction of the first neural network model, calculated based on the split nodes and the gain of the split nodes in the prediction process of each data in the first neural network model.

3. The method according to claim 1, characterized in that, The first task data for obtaining the sample task includes: Obtain the original task data of the sample task; wherein, the original task data is of the same type as the first task data, but the presentation format is different; the presentation format includes at least one of text, numbers, tables or graphics; The original task data is represented numerically to obtain the numerically represented original task data. The original task data after numerical representation is standardized and normalized to obtain the first task data.

4. The method according to claim 1, characterized in that, The method combines the relevance and importance scores of each data point to select at least two types of data from the first task data as the second task data; including: Calculate the dependency degree between each pair of data in the first task data and establish a dependency redundancy matrix; wherein, the matrix elements of the dependency redundancy matrix are the numerical values ​​of the dependency degree; the dependency degree is information representing the degree of mutual dependence between the two types of data, calculated based on the probability distribution of the two types of data in the second grid partition; the second grid partition is a grid specification used to statistically analyze the probability distribution of the two types of data, divided according to the number of sample tasks. Select the data corresponding to the maximum relevance value from the first task data as the data in the first task sub-data set; According to the following formula, select the data with the largest first value from the data other than the first task sub-data set as the data in the first task sub-data set, until the data types in the first task sub-data set are equal to the first preset value; Wherein, the Score(f) j ) represents the first value, and the f j This represents the j-th type of data among all data other than the first task sub-data set; the MIC(f) j (T) represents the correlation between the j-th type of data and the task execution time; T represents the task execution time; S represents the first task sub-data set, and |S| represents the number of selected data types in the first task sub-data set; f s This represents the s-th type of data in the first task sub-data set; the MIC(f) j ,f s ) represents the dependency degree between the j-th type of data in the data other than the first task sub-data set retrieved according to the dependency redundancy matrix and the s-th type of data in the first task sub-data set; Based on the importance scores in descending order, at least two types of data are selected from the first task data as the second task sub-data. The data in the first task sub-data set and the second task sub-data are used as the second task data.

5. The method according to claim 2, characterized in that, The step of calculating the correlation between each data point in the first task data and the task execution time based on the first grid division and mutual information includes: For each type of data, the grid division for that type of data is determined based on the number of sample tasks. For each of the grid partitions, calculate the probability distribution of each data point in the task execution time of the sample task within that grid partition; Based on the probability distribution, calculate the mutual information between this type of data and the task execution time in the grid division; The mutual information is normalized, and the maximum value of the normalized mutual information in each grid division is selected as the correlation between this type of data and the task execution time.

6. The method according to claim 2, characterized in that, The calculation of the importance score of each data point in the first task data relative to the task execution time includes: The first neural network model is trained by using the first task data and the true value of the execution time of the sample task to obtain the trained first neural network model. For each sample task, the contribution value of each data point is calculated based on the split nodes and the gains of the split nodes in the prediction process of the first neural network model. The contribution value represents the degree of contribution of a certain data point to the prediction of the first neural network model. For each category of data in the first task data, calculate the weighted average of the absolute values ​​of each contribution value of that category of data to obtain the importance score of that category of data.

7. A method for predicting task execution time, characterized in that, The method includes: Obtain the third task data for the task to be predicted; The third task data is input into the trained task execution time prediction model for prediction to obtain a second prediction result. The task execution time prediction model is trained by the method described in any one of claims 1-6. The second task data consists of at least two types of data input when training the task execution time prediction model. The third task data is of the same type as the second task data.

8. A training device for a task execution duration prediction model, characterized in that, The device includes: The first task data acquisition module is used to acquire the first task data of the sample task, wherein the first task data includes task data and task-related data; the task-related data is data representing the utilization rate of device hardware resources when the sample task is executed, and data representing the execution steps of the sample task. The relevance and importance score calculation module is used to calculate the relevance and importance scores of each data point in the first task data with the task execution time; wherein, the relevance and importance scores are information representing the influence of a certain type of data on the task execution time obtained by different calculation methods; The data filtering module is used to combine the relevance and importance scores of each data point to select at least two types of data from the first task data as the second task data. The first prediction module is used to input the second task data into the task execution time prediction model for prediction and obtain the first prediction result. The first loss value calculation module is used to calculate the first loss value based on the first prediction result and the true value of the execution time of the sample task. The parameter adjustment module is used to adjust the parameters of the task execution time prediction model according to the first loss value, so as to obtain the trained task execution time prediction model.

9. A task execution time prediction device, characterized in that, The device includes: The third task data acquisition module is used to acquire the third task data of the task to be predicted. The third prediction module is used to input the third task data into the trained task execution time prediction model for prediction to obtain a third prediction result. The task execution time prediction model is trained by the method described in any one of claims 1-6. The second task data consists of at least two types of data input when training the task execution time prediction model. The third task data has the same data type as the second task data.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images