Task execution method and device

Abstract

The present invention discloses a task execution method, device, computing equipment and storage medium. The method includes: during the operation of a main task, determining a subtask to be processed from a plurality of subtasks; wherein the plurality of subtasks are a plurality of parallel executed subtasks obtained by dividing the main task, and the number of repeated executions of the subtask to be processed is greater than or equal to a threshold; predicting the execution result of the subtask to be processed through a preset prediction model to obtain the predicted execution result of the subtask to be processed; completing the main task according to the execution results of other subtasks in the plurality of subtasks and the predicted execution results. By predicting the predicted execution result of the subtask to be processed and combining it with the execution results of other subtasks to complete the main task, the failure of the entire main task can be avoided, and the fault tolerance mechanism of task execution is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task execution method, device, computing equipment and computer storage medium. Background Art

[0002] Big data parallel computing usually uses computing frameworks such as MapReduce, Hive, Tez, and Spark. These computing frameworks divide a large main task into multiple subtasks for computing, and then merge the computing results of multiple subtasks. When a subtask calculation is abnormal, the computing framework usually retries the subtask. Although there is a sharding task retry mechanism, when the retry limit is reached, the failure of the subtask will cause the calculation of the entire main task to fail, lacking a better fault tolerance mechanism. In addition, the waiting time for retrying the subtask is very long. Although only one subtask is retried, even if other subtasks have been completed, the entire main task is in a waiting state. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a task execution method, apparatus, computing device, and computer storage medium that overcome the above problems or at least partially solve the above problems.

[0004] According to one aspect of an embodiment of the present invention, there is provided a task execution method, comprising:

[0005] During the running of the main task, a subtask to be processed is determined from the multiple subtasks; wherein the multiple subtasks are multiple parallel-executed subtasks obtained by dividing the main task, and the number of repeated executions of the subtask to be processed is greater than or equal to a threshold;

[0006] The execution result of the subtask to be processed is predicted by using a preset prediction model to obtain the predicted execution result of the subtask to be processed;

[0007] The main task is completed based on the execution results of other subtasks in multiple subtasks and the predicted execution results.

[0008] Optionally, the preset prediction model is trained based on execution parameters of subtasks that have been successfully executed, and the execution parameters include execution time, execution completion degree and / or data processing volume of the subtask.

[0009] Optionally, the method further comprises:

[0010] Determine the execution time of the subtask according to the running time of the subtask to be processed;

[0011] Count the data obtained from executing the subtasks to be processed and determine the data processing volume of the subtasks;

[0012] The execution completion degree of the subtasks is calculated based on the total data processing volume and data processing volume of the subtasks to be processed; the total data processing volume of the subtasks to be processed is determined based on the division of the main tasks.

[0013] Optionally, the subtask that has been successfully executed specifically refers to: a subtask that has been successfully executed in other main tasks of the same type as the main task.

[0014] Optionally, the preset prediction model is a regression model.

[0015] Optionally, predicting the execution result of the subtask to be processed by a preset prediction model, and obtaining the predicted execution result of the subtask to be processed further includes:

[0016] The total execution time of the subtasks to be processed is calculated based on the execution time and execution completion degree of the subtasks to be processed;

[0017] The total time consumed to complete the execution of the subtask to be processed and the amount of data processed are input into a preset prediction model to obtain the predicted execution result of the subtask to be processed.

[0018] Optionally, the method further comprises:

[0019] If it is determined that the number of repeated executions of the subtask is less than the threshold, the subtask is executed again.

[0020] According to another aspect of an embodiment of the present invention, there is provided a task execution device, comprising:

[0021] A determination module, adapted to determine a subtask to be processed from a plurality of subtasks during the running of a main task; wherein the plurality of subtasks are a plurality of subtasks executed in parallel obtained by dividing the main task, and the number of repetitions of the subtask to be processed is greater than or equal to a threshold;

[0022] A prediction module, adapted to predict the execution result of the subtask to be processed by using a preset prediction model, and obtain the predicted execution result of the subtask to be processed;

[0023] The completion module is suitable for completing the main task according to the execution results of other subtasks in the multiple subtasks and the predicted execution results.

[0024] According to another aspect of an embodiment of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0025] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the above-mentioned task execution method.

[0026] According to another aspect of an embodiment of the present invention, a computer storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned task execution method.

[0027] According to the solution provided by the above embodiment of the present invention, by predicting the predicted execution results of the subtasks to be processed and combining them with the execution results of other subtasks to complete the main task, the failure of the entire main task can be avoided and the fault tolerance mechanism of task execution is improved.

[0028] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation method of the embodiment of the present invention is specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the embodiments of the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0030] Figure 1 A flowchart of a task execution method provided by an embodiment of the present invention is shown;

[0031] Figure 2 A schematic diagram showing the structure of a task execution device provided by an embodiment of the present invention is shown;

[0032] Figure 3 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0034] Figure 1 FIG. 1 is a flowchart of a task execution method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0035] Step S101, during the execution of the main task, a subtask to be processed is determined from a plurality of subtasks.

[0036] In this embodiment, the main task is divided into multiple subtasks that can be executed in parallel. For example, the main task is an execution statement of a MySQL database, which involves large data volume operations. It is divided into multiple small subtasks, each subtask is a separate execution statement, and the execution result of the main task can be finally obtained based on the execution results of multiple subtasks.

[0037] In the prior art, when a subtask fails to execute, the main task will continue to wait for the subtask to execute successfully before obtaining the final execution result, which takes a long time and is prone to cause the failure of the entire main task to execute due to the failure of the subtask. Considering the above problems, this embodiment records the number of repeated executions of the subtask according to the execution of the subtask, and the number of repeated executions is used to mark the number of failed executions of the subtask. Specifically, the number of repeated executions can be set to 0 when the subtask fails to execute and needs to be re-executed, and when the subtask fails to execute multiple times, the number of repeated executions is added by 1 each time it is re-executed. In order to avoid the failure of the main task due to too many subtask execution failures and too long time, this embodiment judges the number of repeated executions of the subtask. According to the number of repeated executions of the subtask, when it is judged that the number of repeated executions of the subtask is greater than or equal to the threshold, the subtask is determined to be a subtask to be processed. The threshold can be set according to the implementation situation, such as 3 times. When it is judged that the number of repeated executions of the subtask is greater than or equal to 3, the subtask is determined to be a subtask to be processed, and the subtask is no longer re-executed.

[0038] Furthermore, if it is determined that the number of repeated executions of the subtask is less than the threshold, the subtask can be re-executed to obtain the accurate execution result of the subtask and ensure the accuracy of the main task. If the subtask is successfully executed this time, the main task is completed according to the execution result of the successfully executed subtask; if the subtask fails this time, the number of repeated executions is increased by 1, and based on the updated number of repeated executions, it is determined whether the subtask is a pending subtask.

[0039] Step S102: predicting the execution result of the subtask to be processed by using a preset prediction model to obtain the predicted execution result of the subtask to be processed.

[0040] The preset prediction model in this embodiment can be trained based on the execution parameters of the subtasks that have been successfully executed, so as to predict the predicted execution results of the subtasks to be processed based on the subtasks that have been successfully executed, and ensure the accuracy of the prediction. Preferably, the subtasks that have been successfully executed are the subtasks that have been successfully executed in this main task, and the prediction is made based on the correlation between the execution results of the subtasks that are located in the same execution environment and the tasks of the same type. Furthermore, the subtasks that have been successfully executed can also include subtasks that have been successfully executed in other main tasks of the same type as the main task, so as to make predictions based on the same type of tasks, increase training samples, and improve training accuracy.

[0041] The execution parameters include the execution time t, execution completion p, data processing volume s, etc. of the subtask. The execution time of the subtask is the running time of the subtask. The data processing volume is obtained by counting the data obtained by the execution of the subtask. The execution completion degree is 100% for the subtask that has been successfully executed. For the subtask to be processed, it is necessary to calculate the execution completion degree of the subtask based on the total data processing volume and data processing volume of the subtask to be processed. The total data processing volume of the subtask to be processed is determined based on the division of the main task. For example, when the main task is divided, the total data processing volume of each subtask is determined in advance, such as 128m. After the subtask to be processed is determined, the execution completion degree of the subtask to be processed is obtained by using the data processing volume / total data processing volume according to the data processing volume of the subtask to be processed obtained at the current time point. The execution parameters of each task (the subtask that has been successfully executed and the subtask to be processed) are obtained after the subtask to be processed is determined. The prediction model is trained based on the execution parameters of the subtask that has been successfully executed. The training starts after the subtask to be processed is determined to ensure that the prediction model is more suitable for the current execution of the main task.

[0042] The preset prediction model can adopt different models according to the implementation situation. In this embodiment, the regression model is used as an example for explanation, but it is not limited to this model. The regression model is trained based on the execution parameters of the subtasks that have been successfully executed in the same main task. Through multivariate regression analysis, the accuracy of the prediction of each subtask in the entire main task is guaranteed. Specifically, the execution parameters of the subtasks that have been successfully executed are input into the regression model to be trained for training, and the output results are compared with the execution results of the subtasks that have been successfully executed. According to the comparison results, the training parameters of the regression model are adjusted to obtain the trained regression model. If the constructed regression model is:

[0043] y=b1x1+b2x2+c

[0044] Among them, y is the output result, corresponding to the execution result of the subtask; b1, b2, c are training parameters, x1 is the execution time of the subtask, and x2 is the data processing volume of the subtask. According to the execution parameters of the subtasks that have been successfully executed, the above regression model is trained, and the training parameters are adjusted to obtain a trained regression model suitable for the main task. After the regression model is trained, the total execution time of the subtask to be processed is calculated according to the execution time t and the execution completion degree p of the subtask to be processed, and the total execution time and data processing volume of the subtask to be processed are input into the preset prediction model (regression model) to obtain the predicted execution result of the subtask to be processed. That is, y=b1t / p+b2s+c.

[0045] Step S103, completing the main task according to the execution results of other subtasks in the multiple subtasks and the predicted execution results.

[0046] After obtaining the predicted execution results of the subtasks to be processed, the above execution results are merged and integrated according to the predicted execution results and the execution results of the subtasks that have been successfully executed to complete the execution of the main task.

[0047] In practical applications, the calculation of the main task often only requires an approximate value. For example, the main task of calculating the monthly active users requires an approximate value and does not need to be accurate to the unit. Through this embodiment, a prediction method is used to obtain the execution result of a subtask to be processed, which can avoid the failure of the entire main task due to the failure of a subtask to be executed.

[0048] According to the method provided by the above embodiment of the present invention, by predicting the predicted execution result of the subtask to be processed and combining it with the execution results of other subtasks to complete the main task, the failure of the entire main task can be avoided, and the fault tolerance mechanism of task execution is improved. Furthermore, based on the execution parameters of the subtask, the prediction of the execution result is more accurate.

[0049] Figure 2 FIG. 4 shows a functional block diagram of a task execution device according to an embodiment of the present invention. Figure 2 As shown, the task execution device includes the following modules:

[0050] The determination module 210 is adapted to determine a subtask to be processed from a plurality of subtasks during the running process of the main task; wherein the plurality of subtasks are a plurality of parallel-executed subtasks obtained by dividing the main task, and the number of repetitions of the subtask to be processed is greater than or equal to a threshold;

[0051] The prediction module 220 is adapted to predict the execution result of the subtask to be processed by using a preset prediction model to obtain the predicted execution result of the subtask to be processed;

[0052] The completion module 230 is adapted to complete the main task according to the execution results of other subtasks in the plurality of subtasks and the predicted execution results.

[0053] Optionally, the preset prediction model is trained based on execution parameters of subtasks that have been successfully executed, and the execution parameters include execution time, execution completion degree and / or data processing volume of the subtask.

[0054] Optionally, the device also includes: a parameter acquisition module 240, suitable for determining the execution time of the subtask according to the running time of the subtask to be processed; counting the data obtained from the execution of the subtask to be processed to determine the data processing volume of the subtask; calculating the execution completion degree of the subtask according to the total data processing volume and data processing volume of the subtask to be processed; the total data processing volume of the subtask to be processed is determined based on the division of the main task.

[0055] Optionally, the subtask that has been successfully executed specifically refers to: a subtask that has been successfully executed in other main tasks of the same type as the main task.

[0056] Optionally, the preset prediction model is a regression model.

[0057] Optionally, the prediction module 220 is further adapted to:

[0058] The total execution time of the subtasks to be processed is calculated based on the execution time and execution completion degree of the subtasks to be processed;

[0059] The total time consumed to complete the execution of the subtask to be processed and the amount of data processed are input into a preset prediction model to obtain the predicted execution result of the subtask to be processed.

[0060] Optionally, the device further comprises: a re-execution module 250, adapted to re-execute the sub-task if it is determined that the number of repeated executions of the sub-task is less than a threshold value.

[0061] The description of each module above refers to the corresponding description in the method embodiment and will not be repeated here.

[0062] An embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the task execution method in any of the above method embodiments.

[0063] Figure 3 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention is shown, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0064] like Figure 3As shown, the computing device may include: a processor, a communications interface, a memory, and a communication bus.

[0065] The processor, the communication interface, and the memory communicate with each other via a communication bus. The communication interface is used to communicate with other devices such as a client or other server network elements. The processor is used to execute a program, and specifically can execute the relevant steps in the above-mentioned task execution method embodiment for a computing device.

[0066] Specifically, the program may include program codes including computer operation instructions.

[0067] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0068] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0069] The program can be specifically used to enable the processor to execute the task execution method in any of the above method embodiments. The specific implementation of each step in the program can refer to the corresponding descriptions in the corresponding steps and units in the above task execution embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiments, which will not be repeated here.

[0070] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the embodiment of the present invention described herein, and the description of the above specific language is to disclose the best mode of the embodiment of the present invention.

[0071] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0072] Similarly, it should be understood that in order to streamline the embodiments of the present invention and to aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: that the claimed embodiments of the present invention require more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0073] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0074] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0075] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The embodiments of the present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing an embodiment of the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0076] It should be noted that the above embodiments illustrate rather than limit the present invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim listing a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A task execution method, comprising: During the running of the main task, a subtask to be processed is determined from a plurality of subtasks; wherein the plurality of subtasks are a plurality of subtasks executed in parallel obtained by dividing the main task, and the number of repeated executions of the subtask to be processed is greater than or equal to a threshold; The total execution time of the subtask to be processed is calculated according to the execution time and execution completion degree of the subtask to be processed; the total execution time of the subtask to be processed and the data processing amount of the subtask to be processed are input into a preset prediction model to obtain the predicted execution result of the subtask to be processed; the prediction model is a regression model; The main task is completed according to the execution results of other subtasks in the multiple subtasks and the predicted execution result.

2. The method according to claim 1, characterized in that: The preset prediction model is trained based on execution parameters of subtasks that have been successfully executed, and the execution parameters include execution time, execution completion degree and / or data processing volume of the subtask.

3. The method according to claim 2, characterized in that The method further comprises: Determining the execution time of the subtask to be processed according to the running time of the subtask to be processed; Counting the data obtained from executing the subtask to be processed, and determining the data processing amount of the subtask to be processed; The execution completion degree of the subtask to be processed is calculated based on the total data processing volume and the data processing volume of the subtask to be processed; the total data processing volume of the subtask to be processed is determined based on the division of the main task.

4. The method according to claim 2 or 3, characterized in that: The subtask that has been successfully executed specifically refers to: a subtask that has been successfully executed in other main tasks of the same type as the main task.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If it is determined that the number of repeated executions of the subtask is less than a threshold, the subtask whose number of repeated executions is less than the threshold is re-executed.

6. A task execution device, comprising: A determination module, adapted to determine a subtask to be processed from a plurality of subtasks during the running of a main task; wherein the plurality of subtasks are a plurality of subtasks executed in parallel obtained by dividing the main task, and the number of repeated executions of the subtask to be processed is greater than or equal to a threshold; The prediction module is adapted to calculate the total execution time of the subtask to be processed according to the execution time and execution completion degree of the subtask to be processed; the total execution time of the subtask to be processed and the data processing amount of the subtask to be processed are input into a preset prediction model to obtain the predicted execution result of the subtask to be processed; the prediction model is a regression model; The completion module is adapted to complete the main task according to the execution results of other subtasks in the multiple subtasks and the predicted execution result.

7. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the task execution method according to any one of claims 1 to 5.

8. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform an operation corresponding to the task execution method according to any one of claims 1 to 5.

Images