Task processing method and device

Abstract

The embodiments of this specification provide a task processing method and device, wherein the task processing method includes: determining at least one execution operator based on execution information of a pending task in an execution state; selecting a target execution operator from the at least one execution operator according to operator information associated with a preset operator selection dimension; reading the task information of the pending task when it is monitored that the target execution operator is in a running state; reoptimizing the pending task according to a reoptimization strategy corresponding to the task information, obtaining a target task and executing it, thereby realizing the collection of task information of the pending task through the execution operator contained in the pending task, reducing the cost investment in hardware storage devices, and eliminating the need to collect data at a specified time point for the pending task, thereby advancing the reoptimization and effectively improving the reoptimization effect of the pending task.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a task processing method. Background Art

[0002] With the rapid development of information technology, data has exploded, and the scenario of processing large amounts of data has become more and more common. As a result, a method of decomposing a task into multiple subtasks and processing the subtasks in parallel has emerged. This method effectively improves the efficiency of task processing. Furthermore, in the process of executing a task according to a preset plan, a method of re-optimizing the plan that has not been executed and obtaining a better execution logic for the unexecuted part has been widely used. In the existing technology, the implementation of re-optimization needs to meet two prerequisites. One is that the current plan has not been sent to the computing layer for execution, and the other is to collect relevant information on the nodes of the current plan. The former ensures that the re-optimization does not lag behind the task execution, which is beneficial to the performance of the task. For a plan that has already been executed, it is meaningless to reoptimize the execution plan. The latter ensures that the specific strategy of reoptimization is executed according to the collected relevant information. In order to meet these two prerequisites, the reoptimization in the existing technology is carried out in the BSP model. Because each node in the BSP model has a corresponding storage medium, the node data can be easily collected, and at the time point of collecting the node data, the subsequent processing tasks have not been executed, which meets the two prerequisites of reoptimization and can be performed. However, for the MPP model where there is no storage medium between nodes, the existing reoptimization scheme cannot collect the node data on the MPP model, and thus cannot carry out task reoptimization. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a task processing method. One or more embodiments of this specification also relate to a task processing apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a task processing method is provided, including:

[0005] Determining at least one execution operator based on execution information of the pending tasks in the execution state;

[0006] Selecting a target execution operator from the at least one execution operator according to operator information associated with a preset operator selection dimension;

[0007] When it is detected that the target execution operator is in a running state, reading the task information of the task to be processed;

[0008] The task to be processed is re-optimized according to the re-optimization strategy corresponding to the task information to obtain the target task and execute it.

[0009] According to a second aspect of the embodiments of this specification, there is provided a task processing device, including:

[0010] a determination module configured to determine at least one execution operator based on execution information of the pending task in the execution state;

[0011] a selection module configured to select a target execution operator from the at least one execution operator according to operator information associated with a preset operator selection dimension;

[0012] A monitoring module is configured to read the task information of the task to be processed when detecting that the target execution operator is in a running state;

[0013] The processing module is configured to reoptimize the task to be processed according to the reoptimization strategy corresponding to the task information, obtain the target task and execute it.

[0014] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0015] memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned task processing method are implemented.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned task processing method are implemented.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned task processing method.

[0019] An embodiment of the present specification implements a query on the data to be processed, determines the execution operators contained therein, and selects a target execution operator among the execution operators based on the operator information associated with the preset operator selection dimension. When the target execution operator is in a running state, the execution information of the task to be processed is obtained, and the corresponding re-optimization strategy is selected according to the different execution information. The task to be processed is updated to the target task and executed, thereby realizing the collection of task information of the task to be processed through the execution operator contained in the task to be processed, reducing the cost investment in hardware storage devices, and eliminating the need for data collection at a specified time point for the task to be processed, so that re-optimization is advanced, and the re-optimization effect of the task to be processed is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of a task processing method provided by one embodiment of this specification;

[0021] Figure 2 This is a schematic diagram of a first scheduling sequence of a task processing method provided by an embodiment of this specification;

[0022] Figure 3 This is a schematic diagram of a second scheduling sequence of a task processing method provided by an embodiment of this specification;

[0023] Figure 4 This is a third scheduling sequence diagram of a task processing method provided by an embodiment of this specification;

[0024] Figure 5 This is a fourth scheduling sequence diagram of a task processing method provided by an embodiment of this specification;

[0025] Figure 6 This is a process flow chart of a task processing method provided by one embodiment of this specification;

[0026] Figure 7 This is a structural diagram of a task processing device provided by one embodiment of this specification;

[0027] Figure 8 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0028] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0029] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0031] First, the terms involved in one or more embodiments of this specification are explained.

[0032] MPP (Massively Parallel Processor) is a parallel computing system consisting of multiple nodes consisting of microprocessors, local memory, and network interface circuits; the nodes are interconnected by a customized high-speed network. A massively parallel processor is an asynchronous multiple instruction stream, multiple data stream (MISD) system because its programs have multiple processes distributed across the microprocessors. Each process has its own independent address space, and processes communicate with each other through message passing.

[0033] Global blocking point operators: Operators that can collect all data before performing calculations, such as the Build end of the Join operator, the Aggregation operator, the Sort operator, etc., have the ability to collect all information.

[0034] Local blocking point operator: An operator that can partially block data collection to wait for local information, such as the buffer in the Shuffle operator. This buffer may be 64MB, and this space can be used to collect local data information.

[0035] Scheduling: After parsing the task execution plan, it derives stage-level execution units. These interdependent execution units form the DAG graph. Because of these interdependencies, these stages must be executed one by one or concurrently according to a certain scheduling order.

[0036] Reoptimization: Reoptimization can solve problems with the accuracy of statistical information, collect information while the task is running, and reoptimize the parts of the task that have not been executed to obtain a better execution plan for the unexecuted parts.

[0037] In this specification, a task processing method is provided. This specification also relates to a task processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0038] In the prior art, if a task involves more than one execution step during execution, an execution plan must be developed and then standardized according to the execution plan. However, statistical information issues often lead to suboptimal or even disastrous plans. Among existing solutions, collecting information statistics during task execution is the most real-time approach. This approach, which collects information during task execution and then reoptimizes the execution plan based on the statistical information, is highly effective in the context of the BSP model. This is due to the inherent characteristics of the BSP model, which stores the relevant data generated during each stage of task processing in a storage space. This task processing method means that the BSP model has a natural blocking point for collecting information generated during task execution and provides a time point at which the current task plan ends and the subsequent task plan has not yet begun. This means that data collection for the current plan is complete but has not yet been sent to the computing layer, allowing for effective plan reoptimization. The data obtained from the current task's completion determines whether the original task plan has room for improvement. If so, a new plan is implemented through reoptimization. Subsequent tasks can abandon the original planned task processing method and adopt the new plan instead. In summary, by collecting information at the completion of a child stage and adjusting the parent stage's plan, a better execution plan is obtained. This plan reoptimization utilizes the natural blocking points identified by the BSP model's inherent structural characteristics.

[0039] However, in the MPP model, since the model structure is different from that of the BSP model, there is no feature of storing the task processing data of each stage in the task processing process in the storage space. This situation leads to the absence of natural blocking points in the MPP model. Therefore, the task reoptimization solution implemented in the BSP model cannot achieve similar effects in the MPP model.

[0040] Furthermore, although the MPP model and the BSP model share the same prerequisites for plan reoptimization—the first is the collection of data during task execution, and the second is that tasks are not continued when data is collected, leaving time for plan reoptimization—they differ in structure, making it impossible to apply the task reoptimization scheme in the BSP model to the MPP model. Therefore, in this embodiment, a task processing method is proposed to implement plan reoptimization for tasks in the MPP model.

[0041] In a task processing method provided in this embodiment, the operator in the task execution process is used as a blocking point for plan re-optimization. It should be understood that the operator of the task implements the task specified by the relevant algorithm statement in the code layer, and in some algorithm statements, it is specified that the data must be fully collected. For example, in the algorithm statement for determining the maximum value, minimum value, etc. of the data, it is necessary to collect the data obtained by the task processing in the previous step, and then select the data that meets the requirements. In this case, the operator corresponding to the algorithm statement can be used as a blocking point for data collection. The task processing method provided in this embodiment is based on this idea, and determines the specific operator in the task to be processed, and then collects data based on the specific operator, and finally realizes plan re-optimization of the task to be processed. It solves the problem that, for example, in the MPP model, the data generated in one stage during the task execution process will not be written into the storage space, and thus plan re-optimization cannot be carried out. And because the task processing method uses a specific operator as a blocking point, it is more flexible in task processing time. For example, in the process of plan re-optimization of the BSP model, it is necessary to collect relevant data after a stage of execution is completed. In the task processing method provided by this embodiment, the target execution operator appears in a task stage, and the collection of task information is realized in this task stage. In terms of time, it is earlier than the BSP model in which task information is collected at the end of the task stage, which reserves more processing time for plan re-optimization.

[0042] See also Figure 1 , Figure 1 A flowchart of a task processing method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0043] Step S102: Determine at least one execution operator based on the execution information of the pending task in the execution state.

[0044] Specifically, the task processing method in this embodiment is applied to re-optimize the tasks of parallel processing in the computing system, to reshape the execution logic of the parallel processing tasks in the computing system, to optimize the task execution logic, and to improve the efficiency of task processing. The computing nodes contained in the parallel processing computing system do not need to have related hardware storage devices, and are suitable for BSP models, MPP models, etc.

[0045] Among them, the pending task can be understood as the task that needs to be re-optimized; the execution information can be understood as the relevant information of the execution process during the execution of the pending task, such as the number of computing nodes included in the pending task, the input and output of each computing node, the association relationship between each computing node, etc.; the execution operator can be understood as the relevant information of each computing node in the execution information of the pending task.

[0046] Based on this, during the execution of pending tasks, queries are performed on the compute nodes included in the pending tasks, the inputs and outputs of each compute node, and the relationships between the compute nodes. The query results then reveal the relevant information for each compute node involved in the pending tasks. This clarifies the overall execution process of pending tasks. This holistic understanding of the execution process allows for the subsequent development of re-optimization strategies, reshaping the execution logic of pending tasks, optimizing the execution process, and improving their efficiency.

[0047] Furthermore, when querying the pending tasks and obtaining the execution operators, if a real-time query is performed, the query result is obtained when the pending tasks are completed. In this case, the pending tasks have already been completed, and re-optimization is undoubtedly delayed. In order to ensure that the query of the pending tasks is ahead of time, in this embodiment, the specific implementation method is as follows:

[0048] When the pending task is in the execution state, source program information of the pending task is queried; the source program information is used as the execution information, and the execution information is parsed to obtain at least one execution operator.

[0049] Among them, the source program information can be understood as the relevant code data of the task to be processed at the code layer; and when the source program information is used as the execution information of the task to be executed, the execution operator can be understood as the subject that executes the data processing statements contained in the relevant programming code to process the task data.

[0050] Based on this, during the execution of the pending task, the code layer of the pending task is read to obtain code information related to the pending task, and then the code information is parsed to determine the statement for processing the task data of the pending task.

[0051] For example, in a school's campus system, there are two tables, Table A and Table B. Table A records the names and student IDs of the school's middle school students, while Table B contains the student IDs and grades. At this time, the two tables need to be combined. Therefore, a task is created to combine Table A and Table B. The code layer in the combination task is queried to determine the source program information of the combination task. Then, the source program information is parsed to determine that it contains J1, J2, J3, and J4 statements, which correspond to the J1, J2, J3, and J4 operators.

[0052] In summary, by querying the source program information of the pending tasks, the execution process of the pending tasks is clarified, and this query method can take precedence over the execution process of the pending tasks. There is no need to read the relevant task information after the current processing link of the task is completed, ensuring that the re-optimization process of the pending tasks has more sufficient time and improving the re-optimization quality.

[0053] Step S104: selecting a target execution operator from the at least one execution operator according to the operator information associated with the preset operator selection dimension.

[0054] Specifically, after determining the execution operators in the pending task, not all of the obtained execution operators can read the data during the execution of the pending task, that is, act as the blocking point of the pending task to collect data; only certain specific execution operators can achieve the above functions, so it is necessary to select a target execution operator from the obtained execution operators as the blocking point in the re-optimization process of the pending task to collect data during the execution of the pending task.

[0055] Among them, the operator selection dimension can be understood as the dimension related to the target execution operator, which serves as the selection criterion for the target execution operator; the operator information can be understood as the relevant information of the target execution operator, and the target execution operator can be selected from more than one execution operator through this operator information; the target execution operator can be understood as an operator that can collect data generated during the execution of the task to be processed, that is, an operator that can serve as a blocking point in the re-optimization process of the task to be processed and store information.

[0056] Based on this, the operator information for selecting the target execution operator is determined according to the preset target execution operator selection criteria. Then, based on the operator information, a selection is made from at least one execution operator, selecting the execution operator that meets the content described in the operator information as the target execution operator. After obtaining the target execution operator in this way, the data generated during the execution of the pending task can be stored. Furthermore, the data collected by the target execution operator can be used to formulate a re-optimization strategy for the pending task, thus implementing the re-optimization process.

[0057] Furthermore, in the process of screening the execution operators to obtain the target execution operator, a clear target execution operator selection standard is required to ensure the accuracy of the target execution operator selection. In this embodiment, the specific implementation method is as follows:

[0058] Determine the operator type corresponding to each execution operator, and determine the operator information associated with the preset operator selection dimension; filter the operator type corresponding to each execution operator according to the operator information to obtain a target operator type; and select a target execution operator from the at least one execution operator according to the target operator type.

[0059] The operator type can be understood as the type of execution operator, such as the join operator, aggregation operator, sort operator, and shuffle operator, which have different operator types. The target operator type can be understood as the operator type corresponding to the target execution operator.

[0060] Based on this, the operator type of each execution operator is determined, and then the information containing the operator type of the target execution operator is determined according to the preset operator selection dimension; then the operator type of each execution operator is screened according to the information to obtain the target operator type corresponding to the target execution operator, and finally the execution operator whose operator type is consistent with the target operator type in the execution operator is selected as the target execution operator. It should be noted that the operator type contained in the operator information determined according to the operator selection dimension can be one or more, and accordingly, the target operator type determined according to the operator information can be one or more. Similarly, the number of target execution operators selected in the execution operator through the target operator type can be one or more. The specific number of operator types contained in the operator information, the number of target operator types and the number of target execution operators are all determined by the actual usage scenario, and this embodiment does not limit it.

[0061] Continuing with the above example, the operator types of operators J1, J2, J3, and J4 are determined to be j1, j2, j3, and j4, respectively. Then, the operator information is determined based on the preset operator selection dimension, where the operator information includes information that the operator type is j3. Then, the target operator information can be determined to include relevant information of type j3. Therefore, the J3 operator is determined as the target execution operator based on the target operator information.

[0062] In summary, by using the operator type to determine the target execution operator, the target execution operator can be accurately selected from the execution operators. Through this selection process, the optimization effect and optimization efficiency of the subsequent re-optimization of the task to be processed are guaranteed.

[0063] Furthermore, the target execution operator is used to collect data generated during the execution of the task to be processed. According to the different data collection situations of the target execution operator, the target execution operator can be divided into two different types. In this embodiment, the target operators include a global blocking point operator and a local blocking point operator; wherein, the global blocking point operator reads the global task information of the task to be processed in the execution state, and the local blocking point operator reads the local task information of the task to be processed in the execution state.

[0064] Among them, the global blocking point operator can collect all the data generated during the execution of the task to be processed, and execute the subsequent calculation process of the task to be processed after the collection, such as the Join operator, Aggregation operator, Sort operator, Window operator, etc.; Correspondingly, the local blocking point operator can only collect part of the data generated during the execution of the task to be processed, and execute the subsequent calculation process of the task to be processed after the collection, such as the Shuffle operator. The buffer in one case is 64MB, and this part of the space can be used to collect local data information.

[0065] Step S106: when it is detected that the target execution operator is in a running state, the task information of the task to be processed is read.

[0066] Specifically, after the target execution operator is determined, due to the characteristics of the target execution operator, it needs to collect the data generated in the task to be processed, and then perform subsequent calculation steps after the collection is completed. According to this characteristic, the target execution operator serves as a blocking point for the re-optimization of the task to be processed, and is used to collect the data generated by the task to be processed and provide a time node for the task to be processed to carry out the re-optimization steps.

[0067] Among them, task information can be understood as the data generated during the execution of the task to be processed. This part of the data will be processed by the target execution operator, that is, this part of the data will be collected by the target execution operator.

[0068] Based on this, the target execution operator is in the running state, indicating that the execution link of the pending task has reached the node of the target execution operator. When this situation is detected, the target execution operator can collect the data generated by the pending task and further read the collected data. Based on the collected data information related to the pending task, the re-optimization strategy can be formulated.

[0069] Furthermore, during the execution of the pending task, there is a situation where the node where the current operator is located directly sends the received data to the next node for processing after receiving the relevant data. In this case, no processing time is reserved for the formulation of the reoptimization strategy, making it impossible to carry out the reoptimization strategy formulation of the pending task normally. To solve this problem, in this embodiment, the specific implementation method is as follows:

[0070] Determine the operator running type of each execution operator; create an operator calling strategy for the task to be processed according to the operator running type of each execution operator; update the node execution order of the task to be processed based on the operator calling strategy to obtain a task to be processed, wherein the task to be processed includes at least two task processing nodes.

[0071] Among them, the operator running type can be understood as the type of execution operator, such as the join operator, aggregation operator, sort operator and shuffle operator, etc., which have different operator types. It should be noted that in actual use, the operator running type and the operator type can be the same concept, both corresponding to the type of execution operator; the operator calling strategy can be understood as a strategy that indicates the execution order of the nodes corresponding to each execution operator in the pending task; the node execution order can be understood as the order in which the nodes corresponding to each execution operator in the pending task start running; the task processing node can be understood as parsing the pending task into stage-level execution units. These execution units are interdependent to form a DAG graph, where the task processing node corresponds to the node in the DAG graph.

[0072] Based on this, the types of execution operators included in the pending task are determined, and then an operator call strategy is created for the nodes corresponding to the execution operators in the pending task according to the types, indicating the execution order of these nodes; based on the strategy, the execution order of each execution operator in the pending task is updated, and after the update, the pending adjustment task is obtained.

[0073] Continuing with the above example, after executing the join task of Table A and Table B, the joined table is joined with Table C. Here, Table C is the bill table of students' shopping at the campus supermarket. The process of joining Table A with Table B and then joining with Table C is as follows: Figure 2The diagram shows a first scheduling sequence diagram of a task processing method, where Table A corresponds to ScanA in the diagram, Table B corresponds to ScanB in the diagram, and Table C corresponds to ScanC in the diagram. JoinA corresponds to the task of joining Table A with Table B, and JoinB corresponds to the task of joining the table obtained by combining Table A and Table B with Table C. The query execution operator has the operator operation type of join. Based on the operator operation type found in the query, an operator call strategy is created based on the idea of ​​preventing each link of the task from directly deriving processing data. The execution order of each node in the task to be processed is determined based on this operator call strategy. The resulting execution order is: ScanB and ScanC are scheduled first, followed by JoinA, JoinB, and ScanA. This implements one of the preconditions for re-optimization: the current node data has not been sent to the computing layer for execution.

[0074] In summary, the scheduling layer in AnalyticDB's MPP computing engine differs from that of MPP engines like Impala and Presto. Because Impala and Presto both adopt an all-at-once scheduling strategy, all generated plans are directly sent to the computing layer for execution, making the "plan not yet sent to the computing layer for execution" condition unattainable. By adjusting the node execution order of pending tasks, the current node data of pending tasks executed according to the DAG topological order is not sent to the computing layer for execution. This reserves time for the formulation of re-optimization strategies, ensuring their completeness and achieving certain optimizations for CPU, memory, and network performance.

[0075] Furthermore, after adjusting the node execution order of the pending tasks, the pending adjusted tasks are obtained. In the subsequent process of formulating the re-optimization strategy, the strategy is formulated along the original pending tasks, which will affect the efficiency of formulating the re-optimization strategy. To solve this problem, in this embodiment, the specific implementation method is as follows:

[0076] Executing the pending adjustment task; and reading the adjustment task information of the pending adjustment task when it is detected that the target execution operator in the pending adjustment task is in a running state.

[0077] The adjustment task information can be understood as the data generated during the execution of the adjustment task to be processed. This part of the data will be processed by the target execution operator, that is, this part of the data will be collected by the target execution operator.

[0078] Based on this, the process of processing the pending adjustment task in this part is similar to the process of processing the pending task described above, and is not described in detail in this embodiment.

[0079] Using the above example, Figure 2As shown in the first scheduling sequence diagram of a task processing method, during the scheduling of ScanB and ScanC, data of Table B and Table C can be collected, and then when JoinA, JoinB, and ScanA are scheduled, data of Table A is collected.

[0080] In summary, since the data processed between the pending tasks and the pending adjustment tasks and the execution operators contained therein have not changed, and because the reoptimization strategy adjusts the order of task execution and pursues the maximization of task processing efficiency, in this case, the reoptimization strategy is formulated for the pending tasks and the pending adjustment tasks, and the final reoptimization results are consistent, but the pending adjustment tasks can be simpler in the process of implementing the reoptimization strategy formulation.

[0081] Furthermore, in a computing system similar to the BSP model structure, when running a pending task, the data obtained after the calculation of each node is completed will be stored in the relevant disk. In a computing system similar to the MPP model structure, the MPP engine generally transmits information in a pipeline stream manner. Unlike the BSP model, which has obvious blocking points, the MPP model does not include storage space associated with the node, so the data of the pending task cannot be captured through the relevant disk storage space. In order to realize the collection of data for the pending task, in this embodiment, the specific implementation method is as follows:

[0082] Determine a temporary storage space for task data associated with the task to be processed; and read task information from the temporary storage space for task data.

[0083] Among them, the task data temporary storage space can be understood as a storage space used to cache the data of the execution process during the execution of the pending task. In actual usage scenarios, it is mostly the memory space of the relevant computing system.

[0084] Based on this, the memory occupied by the relevant computing devices during the execution of the pending tasks is determined, and then the task information is read from the occupied memory to implement the subsequent formulation of the weight optimization strategy.

[0085] Continuing with the above example, taking the scheduling of ScanB as an example, after ScanB is scheduled, table B is read from the memory of the computing device that executes the task of scheduling ScanB.

[0086] In summary, by reading information from the task data temporary storage space, the relevant data generated during the execution of the task to be processed can be read, ensuring the smooth implementation of the subsequent re-optimization strategy formulation steps.

[0087] Step S108: re-optimize the task to be processed according to the re-optimization strategy corresponding to the task information, obtain the target task and execute it.

[0088] Specifically, after the task information is determined, the dimensions that can be adjusted during the execution of the task to be processed can be determined according to the instructions of the task information, thereby optimizing the execution process of the task to be processed.

[0089] Among them, the reoptimization strategy can be understood as corresponding to the task information of the pending task. When the task information in the pending task presents different situations, different reoptimization strategies will be adopted to adjust the pending strategy; the target task can be understood as the task with a new node execution order obtained after reoptimization of the pending task. Compared with the pending task, the target task can not only achieve the same task goal as the pending task, but also have higher execution efficiency.

[0090] Based on this, according to the obtained task information, the corresponding reoptimization strategy is selected, and then the task to be processed is reoptimized according to the selected reoptimization strategy to obtain the updated target task, and the obtained target task is executed. It should be noted that there is more than one type of reoptimization strategy, and different reoptimization strategies can be used to adjust the task to be processed in different application scenarios. The specific reoptimization strategy to be adopted is determined by the actual usage scenario, and this embodiment does not limit it.

[0091] Furthermore, when the pending tasks are pre-processed to obtain pending adjustment tasks, in order to achieve the purpose of re-optimization, in this embodiment, the specific implementation method is as follows:

[0092] The pending adjustment task is reoptimized according to the reoptimization strategy corresponding to the task adjustment information to obtain a target adjustment task and execute it.

[0093] Among them, the target adjustment task can be understood as the task obtained after re-optimizing the adjustment task to be processed. It should be noted that in actual application scenarios, the target task corresponding to the task to be processed is consistent with the target adjustment task corresponding to the adjustment task to be processed.

[0094] Based on this, the process of re-optimizing the adjustment task to be processed is similar to the process of re-optimizing the task to be processed, and this implementation example does not limit this.

[0095] Furthermore, during the execution of the pending task, the target execution operator processes the data of the two links. If the amount of data on the two links is different from the estimated value, for example, the difference between the data amounts on the two links is large, the target execution operator will cache the data of the link with the larger amount of data. This will seriously occupy storage space. To solve this problem, in this embodiment, the specific implementation method is as follows:

[0096] When it is determined according to the task information that the task to be processed is in the first task state, a first re-optimization strategy corresponding to the first task state is selected; according to the first re-optimization strategy, the task nodes of the task to be processed are task exchanged to obtain the target task and execute it.

[0097] Among them, the first task state can be understood as a situation where the target execution operator in the pending task has an inaccurate estimate of the amount of data on the link related to it. For example, the target execution operator has two links, and the target execution operator stipulates that the amount of data on one link is much larger than the amount of data on the other link. However, the actual situation is contrary to the provisions of the target execution operator. In this case, the pending task is in the first task state; the first re-optimization strategy can be understood as the re-optimization strategy adopted for re-optimizing the pending task when the pending task is in the first task state; the task node can be understood as a node that provides calculation data for the target execution operator, such as Figure 2 In the first scheduling sequence diagram of a task processing method shown, it is assumed that JoinA is the target execution operator, and ScanA and ScanB are task nodes.

[0098] Based on this, the data volume of the task node associated with the target execution operator is determined according to the task information. When the data volume is different from the data volume specified by the target execution operator, the task to be processed is in the first task state. At this time, the first re-optimization strategy corresponding to the first task state is selected, and the tasks executed by the task node associated with the target execution operator of the task to be processed are exchanged to obtain the target task and execute it.

[0099] Continuing with the previous example, we can total the student purchase bills in Table C and calculate the total purchase amount for each student. We can then combine the calculated amount table with Table A, as shown in the following example: Figure 3 The second scheduling sequence diagram of a task processing method shown in the figure shows that when reading data from Table C, it is found that the data volume of Table C is large. In this case, because the aggreagtion operator collects the data from Table C and does not directly send the data from Table C to the join operator, the aggreagtion operator blocks the data of Table C, pausing the task process. The join operator stipulates that the data volume transmitted by the left link is large and the data volume is small. It is determined that the data volume of Table A is much smaller than that of Table C. In this case, the corresponding first optimization strategy is selected, converting the data on the left and right links of the join operator. That is, the data of Table C after the aggreagtion operator is transmitted by the left link of the join operator, and the data of Table A is transmitted by the right link of the join operator. The updated target data is obtained and executed.

[0100] In summary, by re-optimizing the pending tasks in the first task state using the first re-optimization strategy, the storage resources consumed during the task execution process are effectively saved.

[0101] Furthermore, when the target execution operator processes the data of the two links, the links at this time are network transmission. In this case, if the data volume of one link is small and the data volume of the other link is large, and the task to be performed by the target execution operator is to match the data on the two links, there is a lot of data in the link with the larger data volume that cannot be matched with the data of the other link. At this time, this data is invalid data relative to the target execution operator. Transmitting this invalid data through the network is a waste of network resources. To solve this problem, in this embodiment, the specific implementation method is as follows:

[0102] When it is determined according to the task information that the task to be processed is in the second task state, the second optimization strategy corresponding to the second task state is selected; according to the second optimization strategy, a dynamic node is added to the task execution link of the task to be processed, and the target task is obtained and executed.

[0103] Among them, the second task state can be understood as the task state of the data to be processed when the target execution operator in the task to be processed matches the data on the link related to it, and the data volume on one link is much larger than the data volume on the other link; the second re-optimization strategy can be understood as the re-optimization strategy adopted for re-optimizing the task to be processed when the task to be processed is in the second task state; the task execution link can be understood as, Figure 3 In the second scheduling sequence diagram of a task processing method shown, ScanA is connected to the execution path of the Aggregation node and the join operator.

[0104] Based on this, it is determined that the target execution operator in the task to be processed matches the data on the associated link. However, when the data volume on the associated link is very different, the task state of the task to be processed is determined to be the second execution state, and the second optimization strategy corresponding to the second task state is selected. A new dynamic node is added to the task execution link associated with the target execution operator of the task to be processed to obtain the target task and execute it.

[0105] Continuing with the above example, when processing Table A and Table C, it is necessary to combine the student name and student ID data in Table A with the bill data of consumption in the campus supermarket in Table C, such as Figure 4As shown in the third scheduling sequence diagram of a task processing method, if the amount of data in Table A is much larger than that in Table C, the corresponding second optimization strategy is selected in this case, and a dynamic node is added between Table A and Join. The runtimefilter node is used to filter the data, filtering out the data in Table A that is irrelevant to the data in Table C, and then transmitting the filtered Table A to the join node, saving transmission resources.

[0106] In summary, through the above method, dynamic nodes are added to the task execution link, and the data on the link is preprocessed, which saves transmission resources and increases transmission efficiency.

[0107] Furthermore, when the target execution operator processes data from two links, the links at this time are network transmission methods. In this case, if the data volume of one link is much smaller than that of the other link, the process of transmitting the data on the link with the larger data volume to the node where the target execution operator is located will consume a large amount of transmission resources and a lot of time. To solve this problem, in this embodiment, the specific implementation method is as follows:

[0108] When it is determined according to the task information that the task to be processed is in the third task state, the third optimization strategy corresponding to the third task state is selected; the task nodes of the task to be processed are merged according to the third optimization strategy to obtain the target task and execute it.

[0109] Among them, the third task state can be understood as the task state of the data to be processed when the target execution operator in the task to be processed processes the data on the link related to it, where the amount of data on one link is very small and the amount of data on the other link is very large; the third re-optimization strategy can be understood as the re-optimization strategy adopted for re-optimizing the task to be processed when the task to be processed is in the third task state.

[0110] Based on this, in the task to be processed, the target execution operator processes the data on the link related to it. When the amount of data on one link is small and the amount of data on the other link is large, the third optimization strategy is selected to update the task to be processed, broadcast the data with a particularly small amount of data in the link to the data with a particularly large amount of data in the link, merge the two nodes, obtain the target task, and execute it.

[0111] Continuing with the above example, when processing Table A and Table C, it is detected that the amount of data in Table A is particularly large, and the amount of data in Table C is particularly small. In this case, the data in Table C is broadcast to the node where Table A is located, such as Figure 5The fourth scheduling sequence diagram of a task processing method shown is shown, and then the join task is executed on the device where table A is located, avoiding data movement of table A with a large amount of data.

[0112] In summary, the above approach allows for the preservation of particularly large amounts of data in the link associated with the target execution operator, saving transmission resources and accelerating task processing efficiency. It should be noted that the above embodiments provide three reoptimization schemes, illustrating the diverse types and real-time approaches to reoptimization. Therefore, the specific reoptimization strategy used to update the pending task is determined by the actual usage scenario and is not limited in this embodiment.

[0113] An embodiment of the present specification implements a query on the data to be processed, determines the execution operators contained therein, and selects a target execution operator among the execution operators based on the operator information associated with the preset operator selection dimension. When the target execution operator is in a running state, the execution information of the task to be processed is obtained, and the corresponding re-optimization strategy is selected according to the different execution information. The task to be processed is updated to the target task and executed, thereby realizing the collection of task information of the task to be processed through the execution operator contained in the task to be processed, reducing the cost investment in hardware storage devices, and eliminating the need for data collection at a specified time point for the task to be processed, so that re-optimization is advanced, and the re-optimization effect of the task to be processed is effectively improved.

[0114] The following combined Figure 6 , taking the application of the task processing method provided in this specification in demographic statistics as an example, the task processing method is further explained. Figure 6 A flowchart of a task processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0115] Step S602: When the task to be processed is in the execution state, query the source program information of the task to be processed.

[0116] Specifically, during the population census, there are three tables: the original population statistics table A, the population increase table B, and the population decrease table C. The original population statistics table A is the table obtained from the previous population census; the population increase table B is a statistical table of data such as newborns from the time the previous population census was completed until the current population census; the population decrease table C starts at the same time as the population increase table and records data such as deaths and missing persons during this period.

[0117] It should be noted that the three tables are stored in the computing system of the MPP model. Therefore, in the MPP model environment, a demographic task is created that combines the three tables to achieve the purpose of demographic counting. To achieve this goal, the first step required is to query the code layer of the demographic task.

[0118] Step S604: Use the source program information as execution information, and parse the execution information to obtain at least one execution operator.

[0119] Specifically, the code of the demographic task is parsed to obtain the algorithm statements A1 and A2 used therein. According to the algorithm statements A1 and A2, the corresponding operators a1 and a2 can be determined.

[0120] Step S606: Determine the operator type corresponding to each execution operator, and determine the operator information associated with the preset operator selection dimension.

[0121] Specifically, it is determined that the types x1 and x2 of operators a1 and a2 are both join types, and it is determined that the preset operator selection dimension is a dimension that can be used to collect data from the table information included in the demographic task.

[0122] Then, it is determined that the operator type x1 recorded in the operator information associated with this operator selection dimension is a join type, x3 is an aggregation type, and x7 is a sort type.

[0123] Step S608: Filter the operator types corresponding to the various execution operators according to the operator information to obtain the target operator type.

[0124] Specifically, according to the operator types x1, x3, and x7 recorded in the operator information, the corresponding operator types in operators a1 and a2 can be joined, and types x1 and x2 can be selected as the target operator types.

[0125] Step S610: Select a target execution operator from at least one execution operator according to the target operator type.

[0126] Specifically, operators a1 and a2 corresponding to target operator types x1 and x2 are selected as target execution operators. The target execution operators can serve as blocking points in the execution process of the demographic task, and collect relevant task data generated by the demographic task during the execution of the demographic task. Furthermore, by querying the demographic task, a DAG graph of the demographic task is obtained. Then, based on the DAG graph, it is determined whether the execution plan indicating the scheduling order of each stage in the demographic task is reasonable. If the execution plan is unreasonable, the execution plan of the demographic task is re-optimized so that the demographic task can be executed more efficiently.

[0127] Step S612: Determine the operator operation type of each execution operator.

[0128] Specifically, in the demographic task, the operator operation type is the same as the operator type. It can be known that the operator operation type of a1 and a2 is join.

[0129] Step S614: Create an operator calling strategy for the task to be processed according to the operator running type of each execution operator.

[0130] Specifically, after querying the code layer for the demographic task, it was determined that operator a1 merges the original population table A with the population increase table B to obtain table E, and operator a2 merges table E with the population decrease table C. Based on this relationship, the demographic task was parsed into stage-level execution units. A DAG graph was further constructed based on the dependencies between each execution unit. Through analysis of this DAG graph, the operator call order was determined to be: population decrease table C, population increase table B, the merge task between tables A and B, the merge task between tables E and C, and finally the original population table A. This adjustment prevents data obtained at each stage from being directly sent to the next stage for processing, achieving data blocking.

[0131] Step S616: Update the node execution order of the tasks to be processed based on the operator call strategy to obtain the adjustment tasks to be processed.

[0132] Step S618: Determine the temporary storage space for task data associated with the pending adjustment task.

[0133] Specifically, the memory space on the server side is determined during the execution of the demographic task on the server side.

[0134] Step S620: Read task information from the task data temporary storage space.

[0135] Specifically, during the execution of operators a1 and a2, task information is read from the server's memory space. This step uses a1 and a2 as blocking points for the demographic task, achieving the purpose of collecting data generated during the execution of the demographic task. Furthermore, based on the collected data generated during the execution of the demographic task, the execution effect of the original execution plan during the execution of the demographic task is clarified, thereby determining whether the population collection plan needs to be optimized to enhance its execution efficiency.

[0136] Step S622: re-optimize the pending adjustment task according to the re-optimization strategy corresponding to the task information, obtain the target task and execute it.

[0137] Specifically, according to the above method, the target execution operator corresponding to the demographic task is determined, and the target execution operator is used as a blocking point to collect task information of the demographic task, thereby realizing the prerequisite for planned reoptimization of the demographic task. Subsequently, a reoptimization strategy is formulated based on the task information collection results of the population collection task by the target execution operator. It should be noted that the reoptimization strategy is artificially set, and there is more than one type. The specific reoptimization strategy adopted is determined by the actual usage scenario, and this embodiment does not limit it.

[0138] An embodiment of the present specification implements a query on the data to be processed, determines the execution operators contained therein, and selects a target execution operator among the execution operators based on the operator information associated with the preset operator selection dimension. When the target execution operator is in a running state, the execution information of the task to be processed is obtained, and the corresponding re-optimization strategy is selected according to the different execution information. The task to be processed is updated to the target task and executed, thereby realizing the collection of task information of the task to be processed through the execution operator contained in the task to be processed, reducing the cost investment in hardware storage devices, and eliminating the need for data collection at a specified time point for the task to be processed, so that re-optimization is advanced, and the re-optimization effect of the task to be processed is effectively improved.

[0139] Corresponding to the above method embodiment, this specification also provides a task processing device embodiment, Figure 7 FIG1 shows a schematic diagram of the structure of a task processing device provided by an embodiment of this specification. Figure 7 As shown, the device includes:

[0140] The determination module 702 is configured to determine at least one execution operator based on the execution information of the pending task in the execution state;

[0141] A selection module 704 is configured to select a target execution operator from the at least one execution operator according to operator information associated with a preset operator selection dimension;

[0142] The monitoring module 706 is configured to read the task information of the pending task when detecting that the target execution operator is in a running state;

[0143] The processing module 708 is configured to perform re-optimization processing on the task to be processed according to the re-optimization strategy corresponding to the task information, obtain the target task, and execute it.

[0144] In an optional embodiment, the determining module 702 may also be configured to:

[0145] When the pending task is in the execution state, source program information of the pending task is queried; the source program information is used as the execution information, and the execution information is parsed to obtain at least one execution operator.

[0146] In an optional embodiment, the selection module 704 may also be configured to:

[0147] Determine the operator type corresponding to each execution operator, and determine the operator information associated with the preset operator selection dimension;

[0148] Filter the operator types corresponding to each execution operator according to the operator information to obtain the target operator type;

[0149] A target execution operator is selected from the at least one execution operator according to the target operator type.

[0150] In an optional embodiment, the task processing device further includes:

[0151] The calling module is configured to determine the operator running type of each execution operator; create an operator calling strategy for the task to be processed according to the operator running type of each execution operator; update the node execution order of the task to be processed based on the operator calling strategy to obtain a task to be processed, wherein the task to be processed includes at least two task processing nodes.

[0152] In an optional embodiment, the monitoring module 706 may also be configured to:

[0153] executing the pending adjustment task; and reading the adjustment task information of the pending adjustment task when it is detected that the target execution operator in the pending adjustment task is in a running state;

[0154] Correspondingly, the task to be processed is re-optimized according to the re-optimization strategy corresponding to the task information, and the target task is obtained and executed, including: the task to be processed is re-optimized according to the re-optimization strategy corresponding to the task adjustment information, and the target adjustment task is obtained and executed.

[0155] In an optional embodiment, the processing module 708 may also be configured to:

[0156] When it is determined according to the task information that the task to be processed is in the first task state, a first re-optimization strategy corresponding to the first task state is selected; according to the first re-optimization strategy, the task nodes of the task to be processed are task exchanged to obtain the target task and execute it.

[0157] In an optional embodiment, the processing module 708 may also be configured to:

[0158] When it is determined according to the task information that the task to be processed is in the second task state, the second optimization strategy corresponding to the second task state is selected; according to the second optimization strategy, a dynamic node is added to the task execution link of the task to be processed, and the target task is obtained and executed.

[0159] In an optional embodiment, the processing module 708 may also be configured to:

[0160] When it is determined according to the task information that the task to be processed is in the third task state, the third optimization strategy corresponding to the third task state is selected; the task nodes of the task to be processed are merged according to the third optimization strategy to obtain the target task and execute it.

[0161] In an optional embodiment, the monitoring module 706 may also be configured to:

[0162] Determine a temporary storage space for task data associated with the task to be processed; and read task information from the temporary storage space for task data.

[0163] The task processing device provided in one embodiment of the present specification implements the execution steps of the task processing method provided in one embodiment of the present specification, collects task information of the task to be processed through the execution operator contained in the task to be processed, reduces the cost investment of hardware storage devices, and eliminates the need to collect data at the specified time point of the task to be processed, thereby advancing re-optimization and effectively improving the re-optimization effect of the task to be processed.

[0164] The above is a schematic scheme of a task processing device of this embodiment. It should be noted that the technical scheme of the task processing device and the technical scheme of the task processing method described above are of the same concept. For details not described in detail in the technical scheme of the task processing device, please refer to the description of the technical scheme of the task processing method described above.

[0165] Figure 8 8 shows a block diagram of a computing device 800 according to one embodiment of the present disclosure. Components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0166] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)), whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0167] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 8 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0168] The computing device 800 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 800 can also be a mobile or stationary server.

[0169] The processor 820 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned task processing method when executed by the processor.

[0170] The above is a schematic solution of a computing device of this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned task processing method are of the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned task processing method.

[0171] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned task processing method when executed by a processor.

[0172] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the task processing method described above are of the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the task processing method described above.

[0173] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned task processing method.

[0174] The above is a schematic solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the task processing method described above are of the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the task processing method described above.

[0175] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0176] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0177] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0178] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0179] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A task processing method, comprising: Determining at least one execution operator based on execution information of the pending tasks in the execution state; Determine the operator type corresponding to each execution operator, and determine the operator information associated with the preset operator selection dimension; Filter the operator types corresponding to each execution operator according to the operator information to obtain the target operator type; Selecting a target execution operator from the at least one execution operator according to the target operator type, the target execution operator being a blocking point operator for reoptimization of the task to be processed, and being used to collect data generated during the execution of the task to be processed, the target execution operator including a global blocking point operator and a local blocking point operator, the global blocking point operator reading global task information of the task to be processed in an execution state, and the local blocking point operator reading local task information of the task to be processed in an execution state; When it is detected that the target execution operator is in a running state, reading the task information of the task to be processed; Reoptimize the unexecuted part of the pending task according to the reoptimization strategy corresponding to the task information, obtain the target task and execute it.

2. The method according to claim 1, wherein determining at least one execution operator based on the execution information of the pending task in the execution state comprises: When the pending task is in execution state, querying source program information of the pending task; The source program information is used as the execution information, and the execution information is parsed to obtain at least one execution operator.

3. The method according to claim 1, wherein before reading the task information of the pending task when the target execution operator is detected to be in a running state, the method further comprises: Determine the operator operation type of each execution operator; Creating an operator call strategy for the task to be processed based on the operator running type of each execution operator; The node execution order of the task to be processed is updated based on the operator call strategy to obtain an adjustment task to be processed, wherein the task to be processed includes at least two task processing nodes.

4. The method according to claim 3, wherein when the target execution operator is detected to be in a running state, reading the task information of the pending task comprises: Executing the pending adjustment task; When it is detected that the target execution operator in the pending adjustment task is in a running state, reading the adjustment task information of the pending adjustment task; Accordingly, reoptimizing the task to be processed according to the reoptimization strategy corresponding to the task information to obtain the target task and execute it includes: The pending adjustment task is reoptimized according to the reoptimization strategy corresponding to the task adjustment information to obtain a target adjustment task and execute it.

5. The method according to claim 1, wherein the re-optimization process is performed on the unexecuted portion of the pending task according to the re-optimization strategy corresponding to the task information to obtain the target task and execute it, comprising: When it is determined according to the task information that the task to be processed is in a first task state, selecting a first re-optimization strategy corresponding to the first task state; According to the first optimization strategy, the task nodes of the task to be processed are exchanged to obtain the target task and execute it.

6. The method according to claim 1, wherein the re-optimization process is performed on the unexecuted portion of the pending task according to the re-optimization strategy corresponding to the task information, comprising: When it is determined according to the task information that the task to be processed is in the second task state, selecting a second optimization strategy corresponding to the second task state; According to the second optimization strategy, a dynamic node is added to the task execution link of the task to be processed, and the target task is obtained and executed.

7. The method according to claim 1, wherein reoptimizing the unexecuted portion of the pending task according to the reoptimization strategy corresponding to the task information comprises: When it is determined according to the task information that the task to be processed is in the third task state, selecting a third optimization strategy corresponding to the third task state; The task nodes of the tasks to be processed are merged according to the third optimization strategy to obtain the target task and execute it.

8. The method according to claim 1, wherein the step of reading the task information of the task to be processed comprises: Determine a temporary storage space for task data associated with the pending task; The task information is read from the task data temporary storage space.

9. A task processing device comprising: a determination module configured to determine at least one execution operator based on execution information of the pending task in the execution state; A selection module is configured to determine the operator type corresponding to each execution operator and determine the operator information associated with the preset operator selection dimension; The operator types corresponding to the various execution operators are screened according to the operator information to obtain a target operator type; a target execution operator is selected from the at least one execution operator according to the target operator type, wherein the target execution operator is a blocking point operator for re-optimization of the task to be processed and is used to collect data generated during the execution of the task to be processed, and the target execution operator includes a global blocking point operator and a local blocking point operator, wherein the global blocking point operator reads global task information of the task to be processed in the execution state, and the local blocking point operator reads local task information of the task to be processed in the execution state; A monitoring module is configured to read the task information of the task to be processed when detecting that the target execution operator is in a running state; The processing module is configured to reoptimize the unexecuted part of the task to be processed according to the reoptimization strategy corresponding to the task information, obtain the target task and execute it.

10. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the task processing method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the task processing method according to any one of claims 1 to 8.

Images