Task execution method, device and electronic equipment

Abstract

The embodiment of the present invention provides a task execution method, device and electronic device. The method includes: obtaining the syntax parsing features of each syntax element in the code of the task to be executed and the metadata features of the data to be processed by the task to be executed; inputting the syntax parsing features and the metadata features into a pre-trained parameter prediction model to obtain the target operating parameters output by the parameter prediction model; calling the operating resources represented by the target operating parameters to execute the task to be executed; wherein the parameter prediction model is pre-trained with a parameter prediction training set, and the parameter prediction training set includes the syntax parsing features of each syntax element in the code of the parameter prediction sample task, the metadata features of the data to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task. Reasonable operating resources can be automatically allocated to tasks.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a task execution method, device and electronic equipment. Background Art

[0002] To complete a big data processing task, a certain amount of operating resources must be allocated to it. If too few operating resources are allocated to a big data processing task, it will take a long time to complete. If too many operating resources are allocated to a big data processing task, it will lead to a waste of system resources.

[0003] Therefore, how to reasonably allocate operating resources for big data processing tasks has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a task execution method, device, and electronic device to achieve reasonable allocation of operating resources for tasks. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present invention, a task execution method is provided, the method comprising:

[0006] Acquire grammatical parsing features of each grammatical element in the code of the task to be executed and metadata features of data to be processed by the task to be executed;

[0007] Inputting the syntax parsing features and metadata features of the task to be executed into a pre-trained parameter prediction model to obtain the target operating parameters output by the parameter prediction model;

[0008] Calling the operating resources represented by the target operating parameters to execute the task to be executed;

[0009] Among them, the parameter prediction model has been pre-trained with a parameter prediction training set, and the parameter prediction training set includes the grammatical parsing features of each grammatical element in the code of the parameter prediction sample task, the metadata features of the data required to be processed by the parameter prediction sample task, and the operating parameters in the parameter prediction training set optimized for the parameter prediction sample task.

[0010] In a possible embodiment, the parameter prediction model is pre-trained in the following manner:

[0011] Inputting the grammatical parsing features and metadata features in the parameter prediction training set into the parameter initialization model to obtain the predicted operating parameters output by the parameter initialization model;

[0012] The predicted operating parameters and the operating parameters in the parameter prediction training set are fitted to obtain a parameter prediction model.

[0013] In a possible embodiment, there are multiple tasks to be executed, and the method further includes:

[0014] For each of the tasks to be executed, inputting the syntax parsing features, metadata features, and target running parameters of the task to be executed into a pre-trained duration prediction model to obtain a predicted running duration output by the duration prediction model;

[0015] The calling of the operating resource represented by the target operating parameter to execute the task to be executed includes:

[0016] Determining a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters;

[0017] Calling the running resource represented by the target running parameter of the target task to be executed to execute the target task to be executed;

[0018] Among them, the duration prediction model has been pre-trained with a duration prediction training set, and the duration prediction training set includes the grammatical parsing features of each grammatical element in the code of the duration prediction sample task, the metadata features of the sample data required to be processed by the duration prediction sample task, the operating parameters, and the actual operating time required to call the operating resources represented by the operating parameters in the duration prediction training set to execute the duration prediction sample task on the duration prediction training set.

[0019] In a possible embodiment, the duration prediction model is pre-trained in the following manner:

[0020] Inputting the grammatical parsing features and metadata features in the duration prediction training set and the operation parameters in the duration prediction training set into the duration initialization model to obtain the output running time output by the duration initialization model;

[0021] According to the difference between the predicted running time and the actual running time, the model parameters of the initial running time model are adjusted to obtain a running time prediction model.

[0022] In a possible embodiment, the method further includes:

[0023] Determining upstream and downstream relationships between the tasks to be executed based on the input data and output data of each task to be executed;

[0024] According to the determined upstream and downstream relationships, all the tasks to be executed are divided into a plurality of task sequences to be executed, wherein an upstream and downstream relationship exists between every two adjacent tasks to be executed in each of the task sequences to be executed;

[0025] The determining of the target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters includes:

[0026] For each of the to-be-executed task sequences, determining the sequence runtime and sequence operating resources required to complete the to-be-executed task sequence based on the predicted runtime of each of the to-be-executed tasks in the to-be-executed task sequence and the operating resources represented by the target operating parameters;

[0027] Determine a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources;

[0028] Each of the to-be-executed tasks in the target to-be-executed task sequence is sequentially determined as a target to-be-executed task.

[0029] In a possible embodiment, determining the target sequence of tasks to be executed based on currently available running resources, the sequence running duration of each of the tasks to be executed, and the sequence running resources includes:

[0030] In descending order of the total predicted running time, for each of the task sequences to be executed, if the sequence running resources of the task sequence to be executed are less than the currently available running resources, the task sequence to be executed is determined as the target task sequence to be executed.

[0031] In a second aspect of an embodiment of the present invention, a task execution device is provided, the device comprising:

[0032] An acquisition module, configured to acquire grammatical parsing features of each grammatical element in the code of the task to be executed and metadata features of the data to be processed by the task to be executed;

[0033] A parameter prediction module, configured to input the grammatical parsing features and the metadata features into a pre-trained parameter prediction model to obtain target operating parameters output by the parameter prediction model;

[0034] A scheduling module, configured to call the operating resources represented by the target operating parameters to execute the task to be executed;

[0035] Among them, the parameter prediction model is pre-trained with a parameter prediction training set, and the parameter prediction training set includes the grammatical parsing features of each grammatical element in the code of the parameter prediction sample task, the metadata features of the data required to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task.

[0036] In a possible embodiment, the apparatus further includes a parameter prediction model training module, configured to pre-train the parameter prediction model in the following manner:

[0037] Inputting the grammatical parsing features and the metadata features in the parameter prediction training set into the parameter initialization model to obtain the predicted operating parameters output by the parameter initialization model;

[0038] The predicted operating parameters and the operating parameters in the parameter prediction training set are fitted to obtain a parameter prediction model.

[0039] In a possible embodiment, there are multiple tasks to be performed, and the device further includes:

[0040] a duration prediction module for inputting the grammatical parsing features, the metadata features, and the target running parameters of each task to be executed into a pre-trained duration prediction model to obtain a predicted running duration output by the duration prediction model;

[0041] The duration prediction model is pre-trained with a duration prediction training set, which includes grammatical parsing features of each grammatical element in the code of a duration prediction sample task, metadata features of sample data required to be processed by the duration prediction sample task, operating parameters, and actual operating time required to execute the duration prediction sample task on the duration prediction training set by calling operating resources represented by the operating parameters in the duration prediction training set;

[0042] The scheduling module calls the operating resources represented by the target operating parameters to execute the task to be executed, including:

[0043] Determining a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters;

[0044] The operating resource represented by the target operating parameter of the target task to be executed is called to execute the target task to be executed.

[0045] In a possible embodiment, the apparatus further includes a duration prediction model training module, configured to pre-train the duration prediction model in the following manner:

[0046] Inputting the grammatical parsing features in the duration prediction training set, the metadata features in the duration prediction training set, and the operating parameters in the duration prediction training set into the duration initialization model to obtain the output running time output by the duration initialization model;

[0047] The predicted running time and the actual running time are fitted to obtain a time prediction model.

[0048] In a possible embodiment, the device further includes:

[0049] A lineage analysis module, configured to determine the upstream and downstream relationships between the tasks to be executed based on the input data and output data of each task to be executed;

[0050] A task division module is used to divide all the tasks to be executed into a plurality of task sequences to be executed according to the determined upstream and downstream relationships, wherein an upstream and downstream relationship exists between every two adjacent tasks to be executed in each of the task sequences to be executed;

[0051] The scheduling module determines a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters, including:

[0052] For each of the to-be-executed task sequences, determining the sequence runtime and sequence operating resources required to complete the to-be-executed task sequence based on the predicted runtime of each of the to-be-executed tasks in the to-be-executed task sequence and the operating resources represented by the target operating parameters;

[0053] Determine a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources;

[0054] Each of the to-be-executed tasks in the target to-be-executed task sequence is sequentially determined as a target to-be-executed task.

[0055] In a possible embodiment, the scheduling module determines the target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources, including:

[0056] In descending order of the total predicted running time, for each of the task sequences to be executed, if the sequence running resources of the task sequence to be executed are less than the currently available running resources, the task sequence to be executed is determined as the target task sequence to be executed.

[0057] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0058] Memory for storing computer programs;

[0059] The processor is configured to implement any of the method steps described in the first aspect when executing a program stored in the memory.

[0060] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the method steps described in the first aspect is implemented.

[0061] Beneficial effects of the embodiments of the present invention:

[0062] The task execution method, device and electronic device provided by the embodiments of the present invention can construct the characteristics of the task to be executed through grammatical parsing features and metadata features, and use a pre-trained parameter prediction model to analyze the constructed characteristics of the task to be executed to obtain target operating parameters, so as to execute the task to be executed according to the operating resources represented by the target operating parameters. Since the parameter prediction model can learn how to predict optimized operating parameters based on grammatical parsing features and metadata features during the training process, the present invention can automatically optimize the operating resources according to the task to be executed, thereby realizing reasonable scheduling of operating resources, and effectively avoiding resource waste while ensuring the execution efficiency of the task to be executed.

[0063] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0065] Figure 1 A schematic diagram of a process flow of a task execution method provided by an embodiment of the present invention;

[0066] Figure 2 A flowchart of a method for training a parameter prediction model applied to a task execution method is provided for an embodiment of the present invention;

[0067] Figure 3 Another flowchart of a task execution method provided by an embodiment of the present invention;

[0068] Figure 4 A flowchart of a method for training a duration prediction model for a task execution method is provided for an embodiment of the present invention;

[0069] Figure 5A schematic diagram of the structure of a task execution device provided by an embodiment of the present invention;

[0070] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0072] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a task execution method according to an embodiment of the present invention. The task execution method according to an embodiment of the present invention can be applied to any electronic device with task execution capability. The task execution method according to the present invention may include:

[0073] S101, obtaining syntax parsing features of each syntax element in the code of the task to be executed and metadata features of data required to be processed by the task to be executed.

[0074] The syntax parsing features are the features of each syntax element in the code of the task to be executed. The syntax elements in the code of the task to be executed can reflect the calculations involved in the task to be executed to a certain extent. Therefore, the syntax parsing features can reflect the calculations involved in the task to be executed to a certain extent.

[0075] Syntax parsing features can be represented in various forms, including but not limited to numerical values, arrays, and feature vectors. In one possible embodiment, the syntax parsing feature is a feature vector, where each feature dimension in the feature space to which the feature vector belongs corresponds to a syntax element, and the value of the feature vector in each feature dimension represents the number of syntax elements corresponding to that feature dimension in the code of the task to be executed.

[0076] Exemplarily, the feature space to which the feature vector belongs has a total of five feature dimensions, wherein the first feature dimension corresponds to the syntax element select (a syntax element for selection), the second feature dimension corresponds to the syntax element from (a syntax element for copying), the third feature dimension corresponds to the syntax element group by (a syntax element for specifying the field on which data is grouped), the fourth feature dimension corresponds to the syntax element order by (a syntax element for sorting), and the fifth feature dimension corresponds to the syntax element insert into (a syntax element for inserting a new row into a table). If the value of the feature vector in the first feature dimension is 1, it means that the code of the task to be executed contains one syntax element select, and if the value of the feature vector in the second feature dimension is 1, it means that the code of the task to be executed contains one syntax element from. The syntax element corresponding to each feature dimension in the feature space is set by the user based on actual needs or experience, and different feature dimensions correspond to different syntax elements.

[0077] Assume that the code of the task to be executed is as follows:

[0078] INSERT INTO TABLE TARGET_TABLE

[0079] SELECT *

[0080] FROM SOURCE_TABLE

[0081] GROUP BY COL1,COL2

[0082] The feature vector of the grammatical parsing feature can be expressed as {1, 1, 1, 0, 1}, where the first digit of the feature vector is the value of the first feature dimension, the second digit is the value of the second feature dimension, and so on.

[0083] The metadata characteristics of data are characteristics of the metadata of the data, and the metadata of a piece of data is used to describe the data, so the metadata characteristics can describe the data that needs to be processed by the task to be executed to a certain extent.

[0084] Metadata features can be represented in various forms, including but not limited to numerical values, arrays, and feature vectors. In one possible embodiment, the metadata feature is a feature vector, where each feature dimension in the feature space to which the feature vector belongs corresponds to a piece of metadata, and the value of the feature vector in each feature dimension represents the value of the metadata of the data.

[0085] Exemplarily, the feature space to which the feature vector belongs has a total of five feature dimensions, of which the first feature dimension corresponds to the metadata storage size, the second feature dimension corresponds to the number of metadata partitions, the third feature dimension corresponds to the metadata storage size, the fourth feature dimension corresponds to the number of metadata integer data, and the fifth feature dimension corresponds to the number of metadata string data. If the value of the feature vector in the first feature dimension is 10, it means that the storage size occupied by the data required to be processed for the task to be executed is 10 units, and if the value of the feature vector in the second feature dimension is 1, it means that the number of partitions of the data required to be processed for the task to be executed is 1. The metadata corresponding to each feature dimension in the feature space is set by the user based on actual needs or experience, and different feature dimensions correspond to different metadata.

[0086] S102: Inputting the syntax parsing features and metadata features of the task to be executed into a pre-trained parameter prediction model to obtain target operating parameters output by the parameter prediction model.

[0087] Among them, the parameter prediction model is pre-trained with a parameter prediction training set, which includes the grammatical parsing features of each grammatical element in the code of the parameter prediction sample task, the data features required to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task.

[0088] The way an operating parameter represents an operating resource can vary depending on the application scenario. For example, an operating parameter can be a set of tag values, where each tag corresponds to an operating resource, and the tag value of each tag represents the number of operating resources associated with that tag. For example, if a tag in the operating parameter corresponds to a cache, and the tag value of that tag is 4, then the operating resource represented by the operating parameter includes 4 units of cache.

[0089] The grammatical parsing features in the parameter prediction training set and the grammatical parsing features of the task to be executed have the same feature dimensions, and the only difference is the feature values in each feature dimension. Furthermore, the metadata features in the parameter prediction training set and the metadata features of the task to be executed have the same feature dimensions, and the only difference is the feature values in each feature dimension.

[0090] Similarly, the target operating parameters and the operating parameters in the parameter prediction training set, as well as the predicted operating parameters and the operating parameters in the duration prediction training set described below, represent operating resources in the same way. For example, if the target operating parameters are a set of label values, each label corresponds to an operating resource, then the operating parameters in the parameter prediction training set are also a set of label values, and each label value corresponds to the same operating resource as the corresponding label value in the target operating parameters.

[0091] S103: Call the running resource indicated by the target running parameter to execute the task to be executed.

[0092] As analyzed above, the syntax parsing features can reflect the calculations involved in the task to a certain extent, while the metadata features can describe the tasks that need to be processed to a certain extent. Therefore, the syntax parsing features and metadata features can reflect what kind of calculations need to be performed on what kind of data in the process of executing the task, that is, the syntax parsing features and metadata features can reflect the algorithmic complexity of the task. Therefore, in theory, reasonable operating resources can be allocated to the task based on the syntax parsing features and metadata features.

[0093] Since the parameter prediction model has been pre-trained with the parameter prediction training set, which also includes the grammatical parsing features, metadata features, and optimized operating parameters in the parameter prediction training set, the parameter prediction model can theoretically learn how to predict optimized operating parameters based on the grammatical parsing features and metadata features during the training process. Therefore, the target operating parameters output by the parameter prediction model can be considered as target operating parameters optimized for the task to be executed. Therefore, executing the task to be executed according to the operating resources represented by the target operating parameters can effectively avoid resource waste while ensuring the execution efficiency of the task to be executed.

[0094] By selecting this embodiment, the features of the task to be executed can be constructed through grammatical parsing features and metadata features, and the constructed features of the task to be executed can be analyzed using a pre-trained parameter prediction model to obtain target operating parameters, thereby executing the task to be executed according to the operating resources represented by the target operating parameters. Since the parameter prediction model can learn how to predict optimized operating parameters based on grammatical parsing features and metadata features during the training process, the present invention can automatically optimize the operating resources according to the task to be executed, thereby realizing reasonable scheduling of operating resources, and effectively avoiding resource waste while ensuring the execution efficiency of the task to be executed.

[0095] In order to more clearly illustrate the task execution method provided by the embodiment of the present invention, the training method of the parameter prediction model will be described below:

[0096] The parameter prediction model provided in the embodiments of the present invention may be any model, including but not limited to a neural network model trained based on deep learning, and an algorithmic model trained based on a traditional machine learning algorithm, such as a multivariate linear regression model. The parameter prediction model provided in the embodiments of the present invention may be pre-trained by the execution subject of the task execution method provided in the embodiments of the present invention, or may be trained by an electronic device other than the execution subject.

[0097] The training method of the parameter prediction model can be found in Figure 2 ,include:

[0098] S201 , inputting the syntax parsing features in the parameter prediction training set and the metadata features in the parameter prediction training set into the parameter initialization model to obtain the predicted operating parameters output by the parameter initialization model.

[0099] The parameter initialization model is a model used to predict operating parameters based on syntax parsing features and metadata features, and the model parameters of the parameter initialization model can be obtained through training or set by the user based on actual experience or needs.

[0100] S202 , fitting the predicted operating parameters and the operating parameters in the parameter prediction training set to obtain a parameter prediction model.

[0101] The fitting method may vary depending on the application scenario and the type of parameter prediction model. For example, a loss function may be constructed based on the difference between the predicted operating parameters and the operating parameters, and the model parameters of the initial parameter model may be adjusted in the direction of the gradient descent of the loss function until the model parameters converge, or the number of adjustments reaches a preset threshold, and the model at this time is used as the parameter prediction model.

[0102] In some application scenarios, there are multiple pending tasks. Factors affecting the efficiency of these tasks include not only the resources allocated to each task but also the order in which they are executed. Therefore, in these application scenarios, determining the proper order in which tasks are executed becomes a pressing technical challenge.

[0103] Based on this, in a possible embodiment, in the aforementioned Figure 1 Based on the embodiment shown, it also includes:

[0104] S104: For each task to be executed, the syntax parsing features, metadata features, and target operation parameters of the task to be executed are input into a pre-trained duration prediction model.

[0105] Among them, the duration prediction model has been pre-trained with a duration prediction training set, which includes the grammatical parsing features in the duration prediction training set of each grammatical element in the code of the duration prediction sample task, the metadata features in the duration prediction training set of the sample data required to be processed by the duration prediction sample task, the operating parameters, and the actual operating time required for the operating resources represented by the operating parameters in the duration prediction training set to execute the duration prediction sample task on the duration prediction training set.

[0106] The grammatical parsing features in the duration prediction training set and the grammatical parsing features of the task to be executed have the same feature dimensions, differing only in the feature values within each feature dimension. Furthermore, the metadata features in the duration prediction training set and the metadata features of the task to be executed have the same feature dimensions, differing only in the feature values within each feature dimension.

[0107] As analyzed above, the syntax parsing features and metadata features can reflect the algorithmic complexity of the task, while the operating parameters can reflect the resources called to execute the task. When the algorithmic complexity and the resources called to execute the task are constant, the time required to complete the task is constant (or similar).

[0108] Therefore, in theory, the time required to complete a task can be predicted based on grammatical parsing features, metadata features, and operating parameters.

[0109] The duration prediction model is pre-trained on a duration prediction training set, which includes parsing features, metadata features, runtime parameters, and actual runtime. Therefore, theoretically, the duration prediction model can learn during training how to predict the runtime required to complete a task based on these features. Therefore, the predicted runtime output by the duration prediction model can be considered the runtime required to complete the task.

[0110] The duration prediction sample task and the aforementioned parameter prediction sample task may be the same task or different tasks, and this embodiment does not impose any limitation on this.

[0111] The training method of the duration prediction model can be as follows Figure 3 Shown, including:

[0112] S301, inputting the grammatical parsing features and metadata features and running parameters in the duration prediction training set into the duration initialization model to obtain the output running time output by the duration initialization model;

[0113] The duration initialization model is a model used to predict operating parameters based on syntax parsing features and metadata features, and the model parameters of the duration initialization model can be obtained through training or set by the user based on actual experience or needs.

[0114] S302: Fit the output running time with the actual running time to obtain a time prediction model.

[0115] The fitting method may vary depending on the application scenario and the type of parameter prediction model. For example, a loss function may be constructed based on the difference between the output running time and the actual running time, and the model parameters of the initial duration model may be adjusted in the direction of the gradient descent of the loss function until the model parameters converge, or the number of adjustments reaches a preset threshold, and the model at this time is used as the duration prediction model.

[0116] In this embodiment, Figure 4 As shown, the aforementioned S103 includes:

[0117] S1031 , determining a target task to be executed based on currently available execution resources, the predicted execution time of each task to be executed, and the execution resources represented by the target execution parameters.

[0118] S1032: Call the running resource indicated by the target running parameter of the target to-be-executed task to execute the target to-be-executed task.

[0119] By selecting this embodiment, the predicted running time required to complete the task to be executed can be further predicted based on the running resources represented by the predicted first parameter, so that the task to be executed can be reasonably scheduled based on the predicted running time.

[0120] The scheduling method can vary depending on the application scenario, such as scheduling based on the highest execution efficiency, scheduling based on the minimum total waiting time, scheduling based on the minimum system resource usage, etc. Users can choose different scheduling methods based on actual needs.

[0121] For example, taking scheduling in the most efficient manner as an example, assume that there are three tasks to be executed, namely the first task to be executed, the second task to be executed, and the third task to be executed, and the first task to be executed requires 3 units of system resources, the second task to be executed requires 4 units of system resources, and the third task to be executed requires 2 units of system resources, and the currently available running resources are 5 units of system resources. At this time, if the first task to be executed and the third task to be executed are used as the target tasks to be executed, the remaining 5 units of system resources can be fully utilized to maximize the execution efficiency. If the second task to be executed is used as the target task to be executed, one unit of system resources will still be in space, and the execution efficiency cannot be maximized.

[0122] For scenarios where there are multiple tasks to be executed, there are certain dependencies between different tasks to be executed. For example, the execution of one task to be executed depends on the execution result of another task to be executed. Therefore, before executing the first task to be executed, the other task to be executed needs to be executed first.

[0123] Based on this, in a possible embodiment, the method further includes:

[0124] S105 , determining the upstream and downstream relationships between the tasks to be executed based on the input data and output data of each task to be executed.

[0125] For example, assuming that the output data of the first task to be executed is the first data table, and the input data of the second task to be executed is the second data table, it can be determined that there is an upstream and downstream relationship between the first task to be executed and the second task to be executed, and the first task to be executed is located upstream of the second task to be executed.

[0126] The input data and output data of each task to be executed are obtained by parsing the syntax elements of the task to be executed. For example, assuming that the task to be executed contains the syntax elements Insertinto and from, and the data table corresponding to the syntax element Insertinto is the second data table, and the data table corresponding to the syntax element from is the first data table, then the input data of the task to be executed is the first data table, and the output data is the second data table.

[0127] S106 , dividing all to-be-executed tasks into a plurality of to-be-executed task sequences according to the determined upstream and downstream relationships, wherein an upstream and downstream relationship exists between every two adjacent to-be-executed tasks in each to-be-executed task sequence.

[0128] Every two pending tasks that have an upstream-downstream relationship are grouped into the same pending task sequence.

[0129] In a possible embodiment, the aforementioned S1031 includes:

[0130] S1301a , for each task sequence to be executed, determine the sequence execution time and sequence execution resources required to complete the task sequence to be executed based on the predicted execution time of each task to be executed in the task sequence to be executed and the execution resources represented by the target execution parameters.

[0131] S1031b: Determine the target task sequence to be executed based on the currently available running resources, sequence running time, and sequence running resources.

[0132] S1031c, sequentially determining each of the to-be-executed tasks in the target to-be-executed task sequence as a target to-be-executed task

[0133] In a possible embodiment, the aforementioned S1031a may be implemented in the following manner:

[0134] According to the order of the total predicted running time from large to small, for each of the task sequences to be executed, if the running resources of the task sequence to be executed are less than the currently available running resources, the task sequence to be executed is determined as the target task sequence to be executed.

[0135] See also Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a structure of a task execution device provided by an embodiment of the present invention, which may include:

[0136] An acquisition module 501 is configured to acquire syntax parsing features of each syntax element in the code of a task to be executed and metadata features of data to be processed by the task to be executed;

[0137] A parameter prediction module 502 is configured to input the syntax parsing features and the metadata features into a pre-trained parameter prediction model to obtain target operating parameters output by the parameter prediction model;

[0138] The scheduling module 503 is used to call the operating resources represented by the target operating parameters to execute the task to be executed;

[0139] Among them, the parameter prediction model has been pre-trained with a parameter prediction training set, and the parameter prediction training set includes the grammatical parsing features in the parameter prediction training set of each grammatical element in the code of the parameter prediction sample task, the metadata features in the parameter prediction training set of the data to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task.

[0140] In a possible embodiment, the apparatus further includes a parameter prediction model training module, configured to pre-train the parameter prediction model in the following manner:

[0141] Inputting the grammatical parsing features and the metadata features in the parameter prediction training set into the parameter initialization model to obtain the predicted operating parameters output by the parameter initialization model;

[0142] The predicted operating parameters and the operating parameters in the parameter prediction training set are fitted to obtain a parameter prediction model.

[0143] In a possible embodiment, there are multiple tasks to be performed, and the device further includes:

[0144] a duration prediction module for inputting the grammatical parsing features, the metadata features, and the target running parameters of each task to be executed into a pre-trained duration prediction model to obtain a predicted running duration output by the duration prediction model;

[0145] The duration prediction model is pre-trained with a duration prediction training set, which includes grammatical parsing features of each grammatical element in the code of a duration prediction sample task, metadata features of sample data required to be processed by the duration prediction sample task, operating parameters, and actual operating time required to execute the duration prediction sample task on the duration prediction training set by calling operating resources represented by the operating parameters in the duration prediction training set;

[0146] The scheduling module calls the operating resources represented by the target operating parameters to execute the task to be executed, including:

[0147] Determining a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters;

[0148] The operating resource represented by the target operating parameter of the target task to be executed is called to execute the target task to be executed.

[0149] In a possible embodiment, the apparatus further includes a duration prediction model training module, configured to pre-train the duration prediction model in the following manner:

[0150] Inputting the grammatical parsing features in the duration prediction training set, the metadata features in the duration prediction training set, and the operating parameters in the duration prediction training set into the duration initialization model to obtain the output running time output by the duration initialization model;

[0151] The predicted running time and the actual running time are fitted to obtain a time prediction model.

[0152] In a possible embodiment, the device further includes:

[0153] A lineage analysis module, configured to determine the upstream and downstream relationships between the tasks to be executed based on the input data and output data of each task to be executed;

[0154] A task division module is used to divide all the tasks to be executed into a plurality of task sequences to be executed according to the determined upstream and downstream relationships, wherein an upstream and downstream relationship exists between every two adjacent tasks to be executed in each of the task sequences to be executed;

[0155] The scheduling module determines a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters, including:

[0156] For each of the to-be-executed task sequences, determining the sequence runtime and sequence operating resources required to complete the to-be-executed task sequence based on the predicted runtime of each of the to-be-executed tasks in the to-be-executed task sequence and the operating resources represented by the target operating parameters;

[0157] Determine a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources;

[0158] Each of the to-be-executed tasks in the target to-be-executed task sequence is sequentially determined as a target to-be-executed task.

[0159] In a possible embodiment, the scheduling module determines the target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources, including:

[0160] In descending order of the total predicted running time, for each of the task sequences to be executed, if the sequence running resources of the task sequence to be executed are less than the currently available running resources, the task sequence to be executed is determined as the target task sequence to be executed.

[0161] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0162] Memory 603, used for storing computer programs;

[0163] The processor 601 is configured to execute the program stored in the memory 603 by performing the following steps:

[0164] Acquire grammatical parsing features of each grammatical element in the code of the task to be executed and metadata features of data to be processed by the task to be executed;

[0165] Inputting the grammatical parsing features and the metadata features into a pre-trained parameter prediction model to obtain target operating parameters output by the parameter prediction model;

[0166] Calling the operating resources represented by the target operating parameters to execute the task to be executed;

[0167] Among them, the parameter prediction model is pre-trained with a parameter prediction training set, and the parameter prediction training set includes the grammatical parsing features of each grammatical element in the code of the parameter prediction sample task, the metadata features of the data required to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task.

[0168] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0169] The communication interface is used for communication between the above electronic device and other devices.

[0170] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0172] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned task execution methods are implemented.

[0173] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any one of the task execution methods in the above embodiments.

[0174] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0175] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0176] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the embodiments of the apparatus, electronic device, computer-readable storage medium, and computer program product are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, reference can be made to the descriptions of the method embodiments.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A task execution method, characterized in that: The method comprises: Obtaining syntax parsing features of each syntax element in the code of the task to be executed and metadata features of the data to be processed by the task to be executed; wherein the syntax parsing feature is a feature vector, each feature dimension in the feature space to which the feature vector belongs corresponds to a syntax element, and the value of the feature vector in each feature dimension represents the number of syntax elements corresponding to the feature dimension in the code of the task to be executed; Inputting the syntax parsing features and metadata features of the task to be executed into a pre-trained parameter prediction model to obtain the target operating parameters output by the parameter prediction model; Calling the operating resources represented by the target operating parameters to execute the task to be executed; Among them, the parameter prediction model is pre-trained with a parameter prediction training set, and the parameter prediction training set includes the grammatical parsing features of each grammatical element in the code of the parameter prediction sample task, the metadata features of the data required to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task.

2. The method according to claim 1, characterized in that The parameter prediction model is pre-trained in the following way: Inputting the grammatical parsing features and metadata features in the parameter prediction training set into the parameter initialization model to obtain the predicted operating parameters output by the parameter initialization model; The predicted operating parameters and the operating parameters in the parameter prediction training set are fitted to obtain a parameter prediction model.

3. The method according to claim 1, characterized in that There are multiple tasks to be performed, and the method further includes: For each of the tasks to be executed, inputting the syntax parsing features, metadata features, and target running parameters of the task to be executed into a pre-trained duration prediction model to obtain a predicted running duration output by the duration prediction model; The calling of the operating resource represented by the target operating parameter to execute the task to be executed includes: Determining a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters; Calling the operating resource represented by the target operating parameter of the target task to be executed to execute the target task to be executed; Among them, the duration prediction model has been pre-trained with a duration prediction training set, and the duration prediction training set includes the grammatical parsing features of each grammatical element in the code of the duration prediction sample task, the metadata features of the sample data required to be processed by the duration prediction sample task, the operating parameters, and the actual operating time required to call the operating resources represented by the operating parameters in the duration prediction training set to execute the duration prediction sample task on the duration prediction training set.

4. The method according to claim 3, characterized in that The duration prediction model is pre-trained in the following way: Inputting the grammatical parsing features and metadata features in the duration prediction training set and the operation parameters in the duration prediction training set into the duration initialization model to obtain the output running time output by the duration initialization model; According to the difference between the predicted running time and the actual running time, the model parameters of the initial running time model are adjusted to obtain a running time prediction model.

5. The method according to claim 3, characterized in that The method further comprises: Determining upstream and downstream relationships between the tasks to be executed based on the input data and output data of each task to be executed; According to the determined upstream and downstream relationships, all the tasks to be executed are divided into a plurality of task sequences to be executed, wherein an upstream and downstream relationship exists between every two adjacent tasks to be executed in each of the task sequences to be executed; The determining of the target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters includes: For each of the to-be-executed task sequences, determining the sequence runtime and sequence operating resources required to complete the to-be-executed task sequence based on the predicted runtime of each of the to-be-executed tasks in the to-be-executed task sequence and the operating resources represented by the target operating parameters; Determine a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources; Each of the to-be-executed tasks in the target to-be-executed task sequence is sequentially determined as a target to-be-executed task.

6. The method according to claim 5, characterized in that The step of determining a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources includes: In descending order of the total predicted running time, for each of the task sequences to be executed, if the sequence running resources of the task sequence to be executed are less than the currently available running resources, the task sequence to be executed is determined as the target task sequence to be executed.

7. A task execution device, characterized in that: The device comprises: an acquisition module, configured to acquire grammatical parsing features of each grammatical element in the code of the task to be executed and metadata features of the data to be processed by the task to be executed; wherein the grammatical parsing features are a feature vector, each feature dimension in the feature space to which the feature vector belongs corresponds to a grammatical element, and the value of the feature vector in each feature dimension represents the number of grammatical elements in the code of the task to be executed corresponding to the feature dimension; A parameter prediction module, configured to input the grammatical parsing features and the metadata features into a pre-trained parameter prediction model to obtain target operating parameters output by the parameter prediction model; A scheduling module, configured to call the operating resources represented by the target operating parameters to execute the task to be executed; Among them, the parameter prediction model is pre-trained with a parameter prediction training set, and the parameter prediction training set includes the grammatical parsing features of each grammatical element in the code of the parameter prediction sample task, the metadata features of the data required to be processed by the parameter prediction sample task, and the operating parameters optimized for the parameter prediction sample task.

8. The device according to claim 7, characterized in that The device also includes a parameter prediction model training module, which is used to train the parameter prediction model in advance by the following methods: Inputting the grammatical parsing features and the metadata features in the parameter prediction training set into the parameter initialization model to obtain the predicted operating parameters output by the parameter initialization model; The predicted operating parameters and the operating parameters in the parameter prediction training set are fitted to obtain a parameter prediction model.

9. The device according to claim 7, characterized in that The number of the tasks to be executed is multiple, and the device further includes: a duration prediction module for inputting the grammatical parsing features, the metadata features, and the target running parameters of each task to be executed into a pre-trained duration prediction model to obtain a predicted running duration output by the duration prediction model; The duration prediction model is pre-trained with a duration prediction training set, which includes grammatical parsing features of each grammatical element in the code of a duration prediction sample task, metadata features of sample data required to be processed by the duration prediction sample task, operating parameters, and actual operating time required to execute the duration prediction sample task on the duration prediction training set by calling operating resources represented by the operating parameters in the duration prediction training set; The scheduling module calls the operating resources represented by the target operating parameters to execute the task to be executed, including: Determining a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters; The operating resource represented by the target operating parameter of the target task to be executed is called to execute the target task to be executed.

10. The device according to claim 9, characterized in that The device further includes a duration prediction model training module, which is used to train the duration prediction model in advance by: Inputting the grammatical parsing features in the duration prediction training set, the metadata features in the duration prediction training set, and the operating parameters in the duration prediction training set into the duration initialization model to obtain the output running time output by the duration initialization model; The predicted running time and the actual running time are fitted to obtain a time prediction model.

11. The device according to claim 9, characterized in that The device further comprises: A lineage analysis module, configured to determine the upstream and downstream relationships between the tasks to be executed based on the input data and output data of each task to be executed; A task division module is used to divide all the tasks to be executed into a plurality of task sequences to be executed according to the determined upstream and downstream relationships, wherein an upstream and downstream relationship exists between every two adjacent tasks to be executed in each of the task sequences to be executed; The scheduling module determines a target task to be executed based on currently available operating resources, the predicted operating time of each task to be executed, and the operating resources represented by the target operating parameters, including: For each of the to-be-executed task sequences, determining the sequence runtime and sequence operating resources required to complete the to-be-executed task sequence based on the predicted runtime of each of the to-be-executed tasks in the to-be-executed task sequence and the operating resources represented by the target operating parameters; Determine a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources; Each of the to-be-executed tasks in the target to-be-executed task sequence is sequentially determined as a target to-be-executed task.

12. The device according to claim 11, characterized in that The scheduling module determines a target sequence of tasks to be executed based on currently available running resources, the sequence running time of each of the tasks to be executed, and the sequence running resources, including: In descending order of the total predicted running time, for each of the task sequences to be executed, if the sequence running resources of the task sequence to be executed are less than the currently available running resources, the task sequence to be executed is determined as the target task sequence to be executed.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 6 are implemented.

Images