Calculation task scheduling execution method and device and readable storage medium

Abstract

The invention relates to the field of big data sorting, in particular to a calculation task scheduling execution method and device and a readable storage medium, and the method comprises the steps: obtaining target calculation task information; a target calculation task is analyzed and processed through an execution plan generator, whether the execution plan generator and a resource scheduler are interacted or not is determined according to an analysis and processing result, operator resources of preset operators are evaluated and configured, a corresponding execution plan is generated, and multiple operators in the execution plan are transversely arranged in a layered mode according to types. Wherein the computing platform comprises a cloud computing platform, and a plurality of operators are preset in the computing platform; and according to the execution plan, calling and executing in sequence through a task scheduler until all operators are executed. From the aspect of the resource utilization rate, the average utilization rate of computing resources is remarkably broken through by innovating a scheduling mechanism, and the overall level stably reaches or exceeds 80%.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data sorting, and in particular to a method and device for scheduling and executing a computing task and a readable storage medium. BACKGROUND

[0002] It is currently a fact standard that databases, data computing platforms, big data, and the like are run on the cloud. Meanwhile, there are many database or data warehouse products in the market that provide cross-multi-cloud platforms. In addition, there are many open source projects of data warehouse and SQL engine types. The core components of these databases, data warehouse commercial products, and open source products are computing engines. To handle large-scale computing tasks, these computing engines usually adopt a distributed architecture. Computing tasks are distributedly executed by a batch of machines (or virtual machines, cloud servers, cloud service containers (Docker or Kubernetes, etc.). From the perspective of whether a service node is visible to a user, the computing model can be divided into Serverless (serverless) and non-Serverless (traditional server). Serverless or non-Serverless does not affect the type of computing model.

[0003] There are several classifications of computing models for distributed execution, which can be broadly categorized into four different computing architectures and parallel computing models: SMP (Symmetric Multi-Processing), MPP (Massively Parallel Processing), NUMA (Non-Uniform Memory Access), and BSP (Bulk Synchronous Parallel). The main differences between them are in the way processors, memory, and communication are handled, and the corresponding disadvantages of each model are as follows: 1) SMP (Symmetric Multi-Processing): SMP is a multi-processor computing architecture where all processors share the same memory and I / O resources. In an SMP system, communication between processors is done through shared memory, so there is no latency in data transfer. The advantage of an SMP system is simplicity and ease of use, but the disadvantage is limited scalability, as the number of processors increases, the competition for memory and I / O resources can cause performance degradation. 2) NUMA (Non-Uniform Memory Access): NUMA is a multi-processor computing architecture that is optimized over SMP to address the issue of memory and I / O resource competition. In a NUMA system, processors are divided into multiple nodes, each with its own local memory. Processors can access local memory and the memory of other nodes, but accessing local memory is faster than accessing the memory of other nodes. The advantage of a NUMA system is better scalability, but the disadvantage is higher programming complexity, as data locality and memory access performance need to be considered. 3) MPP (Massively Parallel Processing): MPP is a distributed computing architecture that connects a large number of processors (usually independent computing nodes) together to process large amounts of data simultaneously. In an MPP system, each processor has its own memory and disk storage, and processors communicate with each other through a high-speed network. MPP systems can handle large-scale data sets because they can distribute data and computing tasks across multiple nodes. The advantage of an MPP system is scalability and high performance, but the disadvantage is higher complexity and cost. 4) BSP (Bulk Synchronous Parallel): BSP is a parallel computing model that divides computing tasks into a series of supersteps. In each superstep, processors perform local computations and then exchange data between processors. All processors synchronize at the end of each superstep to ensure that all processors have completed the current superstep's computation and communication before moving on to the next superstep. The advantage of the BSP model is simplicity and ease of implementation, while the disadvantage is the synchronization overhead that can affect performance.

[0004] In summary, SMP and NUMA focus on memory and I / O resource management of a multi-processor system, MPP focuses on large-scale data processing and high-performance distributed computing, and BSP focuses on a simple and understandable parallel computing model. At present, in the field of big data and data warehouse, the computing model of mainstream products is MPP or BSP. In the mode of placing and scheduling in the unit of work nodes, there is a space for improving the utilization rate from the perspective of resource utilization, whether it is MPP or BSP. SUMMARY

[0005] (I) Invention purposes

[0006] The purpose of the present application is to provide a computing task scheduling execution method, device and readable storage medium. From the perspective of resource utilization, the present application realizes a significant breakthrough in the average utilization rate of computing resources by innovative scheduling mechanism, and the overall level has stabilized at or above 80%.

[0007] (II) Technical solutions

[0008] To solve the above problems, the first aspect of the present application provides a computing task scheduling execution method, which is used for a scheduling execution server in a computing system, the computing system further comprising an execution plan generator, a resource scheduler, a task scheduler and a computing platform, and the method comprising:

[0009] Obtaining target computing task information, the target computing task including or not including any one or more of cost targets and performance targets;

[0010] Analyzing and processing the target computing task by the execution plan generator, determining whether to interact the execution plan generator with the resource scheduler according to the analysis and processing result, evaluating and configuring operator resources of preset operators, and generating a corresponding execution plan, wherein multiple operators in the execution plan are horizontally layered according to types; wherein the computing platform includes but is not limited to a cloud computing platform, and a plurality of operators are preset in the computing platform. Horizontal layering means that one or more operators can be placed in the same layer, and the determination is made according to the analysis result. Meanwhile, horizontal layering can be understood as one level (one horizontal) at the same time or in the same step in terms of runtime sequence or process.

[0011] According to the execution plan, sequentially calling and executing by the task scheduler until all operators are executed.

[0012] Further, the determination whether to interact the execution generator with the resource scheduler according to the analysis and processing result, and the evaluation and configuration of operator resources of preset operators include:

[0013] If the target computing task does not include a cost target and a performance target, then based on the target computing task, the execution plan generator evaluates and configures the operator resources of the corresponding operator according to the corresponding historical execution information;

[0014] If the target computing task includes both a cost target and a performance target, the execution generator interacts with the resource scheduler, and the resource scheduler evaluates and configures the operator resources of the operator according to the interaction result.

[0015] Furthermore, the types of the operators include: at least one of: a decryption operator DECRYPT, a decompression operator DECOMPRESS, a scanning and filtering operator SCAN&FILTER, a data merging operator MERGE COMPUTE, an association operator JOIN, a data redistribution operator SHUFFLE, an aggregation operator AGGREGATE, a projection operator PROJECT, and an operator INDEXING for automatically building a data index.

[0016] Furthermore, the execution plan includes the number of CPU cores and memory allocated to similar operators in each layer.

[0017] Furthermore, the calling and executing in sequence by the task scheduler until all operators are executed includes:

[0018] Hierarchical execution starts from the data access layer through the task scheduler;

[0019] If the current operator layer is completed, the computing power resources occupied by the current computing task will be released and the next operator layer will be executed;

[0020] The task ends when all operator layers are executed.

[0021] Furthermore, the manner of configuring the operator resources of the preset operator includes: on-demand allocation and delayed allocation.

[0022] In addition, a second aspect of the present invention provides a scheduling and execution device for computing tasks, the device being located in a scheduling and execution server in a computing system, the computing system further comprising an execution plan generator, a resource scheduler, a task scheduler, and a computing platform, the device comprising:

[0023] An information acquisition module is used to acquire target computing task information, where the target computing task may include or exclude any one or more of a cost target and a performance target;

[0024] An analysis processing module is configured to analyze and process the target computing task by the execution plan generator, determine whether to interact the execution plan generator with a resource scheduler according to an analysis processing result, evaluate and configure operator resources of preset operators, and generate a corresponding execution plan, in which multiple operators are horizontally layered according to types.

[0025] An execution module is configured to sequentially call and execute the target computing task by the task scheduler according to the execution plan until all operators are executed.

[0026] Further, the analysis processing module is configured to:

[0027] If the target computing task does not include a cost target and a performance target, evaluate and configure operator resources of corresponding operators according to historical execution information by the execution plan generator based on the target computing task;

[0028] If the target computing task includes both the cost target and the performance target, interact the execution generator with the resource scheduler, and evaluate and configure operator resources of operators by the resource scheduler according to an interaction result.

[0029] Further, the execution module is configured to:

[0030] Start layered execution from a data access layer by the task scheduler;

[0031] If a current operator layer is executed, release computing power resources occupied by a current computing task, and continue to execute a next operator layer;

[0032] Until all operator layers are executed, the task is ended.

[0033] A third aspect of the present application provides a computer readable storage medium storing a computer program, which is executed to implement the method of any one of the above descriptions.

[0034] (Three) beneficial effects

[0035] The technical scheme of the present application has the following beneficial technical effects: the present application provides a computing task scheduling execution method, device and readable storage medium, and the average utilization rate of computing resources is significantly improved through an innovative scheduling mechanism. The target computing task is analyzed and processed by the execution plan generator, and then it is determined whether to interact with the resource scheduler, and the operator resources of the preset operator are evaluated and configured to generate a corresponding execution plan, and multiple operators in the execution plan are horizontally layered according to types. Multiple operators are layered and aggregated into an operator pool according to categories, the present application pre-constructs or constructs an operator computing power pool according to various types of operators, the operator pool provides computing power services of the same type of operator, and the computing platform includes but is not limited to a cloud computing platform. Due to the development of cloud computing basic capabilities (virtualization, computing, software and hardware cooperation, network, memory, etc.), cloud computing is the preferred platform for implementing the operator-level service of the present application, and the present application can well solve the problem of uneven resource utilization on the time slice of the working node caused by the barrier (a synchronization mechanism) synchronization waiting due to the data dependency of the upstream and downstream logic in the operator tree in the mode of operator-level computing power resource scheduling. The present application has technical innovation and core optimization in execution and operator placement, resource allocation of the computing engine, and focuses on improving the utilization rate of computing resources. Compared with MPP, BSP or other single machine, distributed parallel computing models, the average utilization rate of computing resources is greatly improved, reaching or exceeding 80%. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a computing task scheduling execution method flowchart of the present application;

[0037] Figure 2 is a computing task scheduling execution device schematic diagram of the present application;

[0038] Figure 3 is an architecture schematic diagram of the prior art MPP model;

[0039] Figure 4 is an architecture schematic diagram of the prior art BSP model;

[0040] Figure 5 is a comparison diagram of the architecture of the present application and the prior art;

[0041] Figure 6 is a scheduling execution method framework schematic diagram of the embodiment of the present application;

[0042] Figure 7 is an execution plan operator tree schematic diagram in the scheduling execution method of the embodiment of the present application;

[0043] Figure 8is the execution step schematic diagram of the computing task of the specific embodiment of the present application;

[0044] Figure 9 is the evaluation data comparison graph collected and observed by the specific embodiment of the present application:

[0045] Figure 10 is the computing power scale and delay comparison graph of a single computing task of the specific embodiment of the present application;

[0046] Figure 11 is the operator execution form schematic diagram of SQL in the technical implementation of the specific embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application is further described in detail below in combination with specific embodiments and with reference to the drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present application. In addition, in the following description, the description of the known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present application.

[0048] At present, in the field of big data and data warehouse, the computing model of the mainstream product is MPP or BSP, such as Figure 3 As shown, the MPP model can distribute data and computing tasks on multiple nodes to achieve scalability and high performance, and the parallelism of the task is improved by increasing one node, but the flexibility is poor, and the data needs to be redistributed and scattered on the node by increasing one node, which will affect the online business to some extent. In addition, the "sub-shard task" is composed of a group of operators, which is usually allocated and scheduled to the same working node (such as Figure 3The shown node 1: Node 1, node 2: Node2), is usually a physical machine, a virtual machine, a cloud computing server, a cloud service container (Docker or Kubernetes Pod), etc., that is, the "sub-slice task" is the minimum scheduling and allocation unit of MPP. Due to the dependence of the computing logic, there is a barrier between the operator tree levels in the worker node, and the execution between the upstream and downstream operators has a dependence relationship, which causes the resource utilization rate of the worker node to be unstable in different time slices, and there is a difference in time utilization. For example, in the Scan Filter operator execution phase, the CPU has a utilization rate of 80%, and the memory utilization rate is 10%; in the Agg phase, the CPU utilization rate is 40%, and the memory utilization rate is 90%. In this way, the fixed worker node resources (CPU, memory, etc.) still have room for improvement in utilization. Agg (AGGREGATE) represents an aggregation operation, mainly used for grouping and aggregating data. In the database execution plan, the Agg stage is usually used with sorting (Sort), filtering (Filter), and other operations to form a complete data processing flow. The BSP model has better flexibility in operator placement than MPP. A Stage composed of a group of operators is the minimum scheduling and allocation unit of BSP, similar to MPP, as shown in Figure 4 As shown, there may be a barrier between the operator levels within or between Stages in the worker node, and the execution between the upstream and downstream operators has a dependence relationship, which causes the resource utilization rate of the worker node to be unstable in different time slices, and there is a difference in time utilization. In the big data processing scenario, Stage refers to the execution phase of a job (Job). The Stage refers to the multi-stage processing process of the model from the original framework to the hardware executable format, including model conversion, optimization, quantization, and hardware adaptation, etc. core steps, and the Local Store is a local storage. Figure 4In the context of RDD (Resilient Distributed Datasets), it is a resilient distributed dataset. In computing engines, "barrier" usually refers to a synchronization mechanism that ensures that all parallel tasks or threads have completed their current tasks or operations before proceeding to the next calculation or operation. Barrier works as follows: when a task or thread reaches the barrier, it stops executing until all tasks or threads reach the barrier. Once all tasks or threads reach the barrier, they can start executing the next task or operation at the same time. This synchronization mechanism is very important in parallel computing because it ensures the consistency and correctness of the data. For example, when performing parallel computing, you may need to calculate some intermediate results first, and then perform the next calculation based on these intermediate results. By using barriers, you can ensure that all tasks or threads have completed the calculation of the intermediate results before starting the next calculation.

[0049] In some parallel computing models, such as the Bulk Synchronous Parallel (BSP) model mentioned above, barriers are a core component. In the BSP model, computing tasks are divided into a series of supersteps. At the end of each superstep, all tasks or threads need to reach a barrier to synchronize before entering the next superstep. In some cases, computing tasks in an MPP system may need to be synchronized. For example, if a computing task needs to wait for the results of other tasks, or needs to ensure that all tasks have completed a certain stage of calculation, then a barrier or similar synchronization mechanism may be needed.

[0050] Overall, whether it is the "sub-shard task" of MPP or the operator combination stage of BSP, there is room for improvement in resource utilization from the perspective of resource utilization under the mode of placement and scheduling based on working nodes.

[0051] Combined with the description of the above technical solution, Figure 5 This paper describes the computing engine mode of this application, from the mainstream MPP / BSP operator combination allocation on a group of worker nodes to the operator-level granularity cloud native computing resource scheduling described in this application. Figure 5 As can be seen in the figure, before this application was proposed, whether it was MPP sharding or BSP stage by stage, the hybrid operator combination was "vertically" arranged and distributed among the working nodes. In this application, operators are "horizontally" aggregated into operator pools according to their types. The operator pool provides computing power services for operators of the same type (Figure 5 Operator Serverless in the Kubernetes cluster is a computing power pool of similar operators on the cloud, and Storage refers to storage.

[0052] The present invention is described in detail below with reference to the embodiments.

[0053] like Figure 1 and Figure 6 As shown, a first aspect of the present invention provides a method for scheduling and executing computing tasks, the method being used for a scheduling and execution server in a computing system, the computing system further comprising an execution plan generator, a resource scheduler, a task scheduler, and a computing platform, the method comprising:

[0054] S1, obtain the target computing task information, which may or may not include any one or more of the cost target and performance target. The computing task (SQL) is sent to the system, and the computing task can carry the user-specified cost and performance target (for example, willing to complete the computing task at a cost of no more than 100 yuan, or try to complete the computing task with a delay of 5 seconds, etc.). Figure 8 for Figure 6 Specific implementation steps.

[0055] S2, analyzes and processes the target computing task through the execution plan generator (syntactic analysis, logical analysis, time, data volume, etc.), determines whether to interact with the execution plan generator and the resource scheduler based on the analysis and processing results, and evaluates and configures the operator resources of the preset operator to generate a corresponding execution plan. Multiple operators in the execution plan are arranged in a horizontal layer according to type, wherein the computing platform presets multiple operators, and the computing platform includes but is not limited to the cloud computing platform. Due to the development of cloud computing basic capabilities (virtualization, computing, software and hardware collaboration, network, memory, etc.), cloud computing is the preferred platform for implementing the operator-level service of this proposal (but not limited to cloud computing). Taking cloud computing as an example, the implementation of OperatorServerless uses cloud servers and containers (Docker or Kubernetes Pod, etc.) as computing power platforms, and schedules operator computing power pools specifically for the same type of operators (which can be threads or processes running specific operators). In this invention, the operator-level computing resource scheduling and execution of computing tasks involves an execution plan generator combined with operator-level computing resource scheduling estimation and allocation. This process includes: analyzing the operator's situation: predicting the computing resources required to complete the operator's calculation based on the operator's data volume, computational complexity, and computational market estimates. It also analyzes the operator's main consumption types (e.g., data consumption, computational consumption, and time consumption), as well as the computing task attributes (e.g., video generation, text generation, and multimodal generation), and prioritizes the operator calculation order.

[0056] In addition, the user specifies the cost and performance targets for the computing task, and the execution plan generator and the resource scheduler analyze and estimate the computing resource at the operator level. The process includes: if the computing resource consumption is huge, the task is decomposed, such as data segmentation, parallel computing, etc., to reduce the computing cost consumption. More complex operator coordination can use multi-task parallel computing on multi-core GPU and multi-card GPU. According to the actual GPU running conditions, such as computing resource occupancy rate, data cache occupancy rate, etc., the operator task is decomposed and distributed to the specified core or card in different multi-core GPU or multi-card GPU to achieve the maximum load of GPU computing efficiency.

[0057] The computing resource pool of each operator maintains a certain water level to ensure the timeliness of on-demand operator execution. Multiple operators are horizontally layered according to types. The horizontal layering is in the runtime sequence or the process. In different times or different steps, at the same time or in the same step, a level is determined according to the analysis results. The computing order, scheduling order, operator splitting, and operator combination of the operator in a time sequence or a step are arranged. For example, under a certain operator, both video processing and text processing are required. At a certain time point or a certain time sequence (understood as a horizontal layer), the corresponding computing tasks of all videos are processed simultaneously by scheduling the operator. In another horizontal layer, another operator is scheduled to process all texts simultaneously.

[0058] The way of configuring the operator resource of the preset operator includes on-demand allocation and delayed allocation. The resource scheduler pre-allocates the operator computing resource for the resource plan of the computing task. The actual operator computing resource allocation in this step can also be delayed to the actual allocation during the operator runtime.

[0059] In the S2 step, the method of determining whether to interact with the execution generator and the resource scheduler according to the analysis processing result, and evaluating and configuring the operator resource of the preset operator includes:

[0060] 1) If the target computing task does not include cost targets and performance targets, the execution plan generator evaluates and configures the operator resource of the corresponding operator based on the corresponding historical execution information according to the target computing task;

[0061] 2) If the target computing task includes both cost targets and performance targets, the execution generator interacts with the resource scheduler, and the resource scheduler evaluates and configures the operator resource according to the interaction result.

[0062] For example, the operator execution form of SQL implemented in the present application is as follows: Figure 11As shown, the products and services implementing the technology of the present application use cloud servers, containers (Docker or Kubernetes Pod, etc.) as computing power platforms on a cloud computing platform (not limited to cloud computing), and pre-construct or construct on-demand operator computing power pools for various types of operators, such as Figure 7 and Figure 11 As shown, such as decryption operator DECRYPT, decompression operator DECOMPRESS, scanning and filtering operator SCAN&FILTER, data merging operator MERGE COMPUTE, join operator JOIN, data redistribution operator SHUFFLE, aggregation operator AGGREGATE, projection operator PROJECT, etc., and also include some background data processing operators, such as the operator INDEXING for automatically constructing data index, etc. From Figure 11 As can be seen, in the mode of operator-level computing power resource scheduling, the problem of uneven resource utilization on the time slice of the working node caused by barrier synchronization waiting due to data dependency in the upstream and downstream logic in the operator tree can be well solved.

[0063] S3, according to the execution plan, sequentially calling and executing through the task scheduler until all operators are executed. The sequentially calling and executing through the task scheduler until all operators are executed include:

[0064] 1) starting from the data access layer, executing layer by layer through the task scheduler;

[0065] 2) if the current operator layer is executed, the computing power resources occupied by the current computing task are released, and the next operator layer is executed;

[0066] 3) until each operator layer is executed, the task is ended, and in this step, the task scheduler starts to schedule the execution plan operator tree; the operator tree starts from the data access layer and executes layer by layer, executes the operator task on the operator computing power resources allocated by the resource plan for the current layer, and releases the computing power resources occupied by the current computing task after execution. Execute one layer after another until all operators are executed. In step S3, the execution plan generator generates an execution plan operator tree corresponding to the computing task, and the task scheduler schedules the execution plan operator tree to the computing power resource pool of each layer of operator to execute. The execution plan includes the number of CPU cores and memory allocated to each layer of the same type of operator. Whether the computing task has a cost and performance target, the execution plan operator tree generated by the execution plan generation will contain the computing power scale requirement of the operator execution (such as total CPU core number, memory number, etc., not limited to these computing power resource indicators) in each layer of the same type of operator.

[0067] As Figure 2As shown, the second aspect of the present application provides a scheduling execution device of a computing task, the device is located in a scheduling execution server in a computing system, the computing system further comprises an execution plan generator, a resource scheduler, a task scheduler and a computing platform, the device comprises:

[0068] An acquisition information module 21 is configured to acquire target computing task information, the target computing task includes or does not include any one or more of a cost target and a performance target.

[0069] 1) If the target computing task does not include a cost target and a performance target, the execution plan generator is used to evaluate and configure operator resources of a corresponding operator based on corresponding historical execution information based on the target computing task.

[0070] 2) If the target computing task includes a cost target and a performance target, the execution generator is interacted with the resource scheduler, and the resource scheduler is used to evaluate and configure operator resources of an operator according to an interaction result.

[0071] An analysis processing module 22 is configured to analyze and process the target computing task by using the execution plan generator, determine whether to interact the execution plan generator with the resource scheduler according to an analysis processing result, evaluate and configure operator resources of a preset operator, and generate a corresponding execution plan, wherein a plurality of operators are arranged in a horizontal layer according to types in the execution plan, and the computing platform includes a cloud computing platform, and a plurality of operators are preset in the computing platform.

[0072] An execution module 23 is configured to sequentially call and execute the target computing task by using the task scheduler according to the execution plan until all operators are executed.

[0073] (1) The task scheduler is used to start layer-by-layer execution from a data access layer;

[0074] (2) If a current operator layer is executed, computing power resources occupied by the current computing task are released, and a next operator layer is executed;

[0075] (3) Until all operator layers are executed, the task is ended.

[0076] The third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed as the above-mentioned method.

[0077] In the S2 step, it is determined whether the execution plan generator interacts with the resource scheduler according to the analysis processing result, and the operator resources of the preset operator are evaluated and configured to generate a detailed step of the corresponding execution plan, which is described through the following specific embodiments:

[0078] 1) Analysis processing result determination

[0079] The execution plan generator analyzes the cost target (such as budget constraint, resource utilization) and performance target (such as delay, throughput) of the target computing task, and determines whether dynamic resource scheduling is needed: if no interaction is needed (for example, the task has no explicit target or the resource demand is fixed): directly use the default resource configuration to generate a static execution plan. If interaction is needed (for example, the task needs elastic resources or the optimization target is complex): trigger the interaction process with the resource scheduler.

[0080] 2) Resource demand evaluation request

[0081] The execution plan generator sends an operator resource evaluation request to the resource scheduler, including: operator type (such as CPU-intensive, GPU-accelerated), data size (input / output volume), target constraint (such as "cost priority" or "performance priority").

[0082] 3) Resource scheduler dynamic evaluation

[0083] The resource scheduler performs the following operations according to the current system state (such as node load, available resource pool) and task demand: operator resource matching: allocate resource types (such as vCPU, memory, GPU card number) for each preset operator. Elastic configuration suggestion: provide optional resource configuration schemes (for example: scheme A: low-priority task, allocate elastic resources on demand; scheme B: high-priority task, reserve exclusive resources).

[0084] 4) Negotiation and confirmation

[0085] The execution plan generator accepts the configuration and generates an execution plan with resource constraints (such as "operator X needs 2 GPUs, timeout threshold 5 minutes") according to the feedback of the resource scheduler: if the resources are sufficient / scheme is feasible. If the resources are insufficient: trigger the degradation strategy (such as adjust the task sharding number) or return an error.

[0086] 5) Execution plan generation

[0087] The final generated execution plan includes: operator dependency graph, resource binding configuration of each operator (such as container specification, elastic scaling rule), monitoring callback interface (for dynamic adjustment at runtime).

[0088] The present application has technical innovation and core optimization on the execution and operator placement of computing engine, resource allocation, focuses on improving the utilization rate of computing resources. Compared with the prior art, whether it is MPP or BSP, or other single machine, distributed parallel computing model, the resource utilization level of computing task on physical machine server (Server), virtual machine (VM or cloud server), container (Container, including Docker and Kubernetes Pod, etc.) is gradually improved, and the results are shown in Table 1 and Figure 9 As shown in the table, the present application has a substantial improvement in the average utilization level of computing resources, reaching or exceeding 80%.

[0089] Table 1 Comparison of computing resource utilization

[0090]

[0091] In addition, due to the fine-grained operator scheduling capability of the present application (operator-level cloud native), from the user's perspective, the scheduling flexibility of computing power resources can be greatly improved. For a single computing task, such as Figure 10 As shown in the figure, with the same money (same cost, the area of the rectangle with diagonal shadow represents the cost of the computing task), a large amount of computing power resources can be scheduled in a very short delay, with greater concurrency and computing power utilization, to complete the computing task in a shorter time, so that the user's TCO is lower and the experience of computing service is better. In the figure, for example, the computing task that takes 1000 seconds to complete with 1000 core computing power can be completed in 10 seconds (lower delay) with 100000 core (larger computing power scale) under the system capability of the present application.

Claims

1. A method for scheduling and executing a computing task, characterized in that: The method is used for a scheduling execution server in a computing system, wherein the computing system further includes an execution plan generator, a resource scheduler, a task scheduler, and a computing platform. The method includes: Get target computing task information; The target computing task is analyzed and processed by the execution plan generator, and based on the analysis and processing results, it is determined whether the execution plan generator is to interact with a resource scheduler, and operator resources of preset operators are evaluated and configured to generate a corresponding execution plan, wherein multiple operators in the execution plan are arranged horizontally and hierarchically according to type; wherein the computing platform includes a cloud computing platform, and multiple operators are preset in the computing platform; According to the execution plan, the tasks are called and executed in sequence by the task scheduler until all operators are executed.

2. The method for scheduling and executing computing tasks according to claim 1, wherein: The determining, based on the analysis and processing results, whether to interact the execution plan generator with the resource scheduler and evaluating and configuring the operator resources of the preset operator includes: If the target computing task does not include a cost target and a performance target, then based on the target computing task, the execution plan generator evaluates and configures the operator resources of the corresponding operator according to the corresponding historical execution information; If the target computing task includes both a cost target and a performance target, the execution plan generator interacts with the resource scheduler, and the resource scheduler evaluates and configures the operator resources of the operator according to the interaction result.

3. The method for scheduling and executing computing tasks according to claim 1, wherein: The types of operators include: at least one of a decryption operator, a decompression operator, a scanning and filtering operator, a data merging operator, an association operator, a data redistribution operator, an aggregation operator, a projection operator, and an automatic data index construction operator.

4. The method for scheduling and executing computing tasks according to claim 1, wherein: The execution plan includes the number of CPU cores and memory allocated to similar operators at each layer.

5. The method for scheduling and executing computing tasks according to claim 1, wherein: The calling and executing of the tasks in sequence by the task scheduler until all operators are executed includes: Hierarchical execution starts from the data access layer through the task scheduler; If the current operator layer is completed, the computing resources occupied by the current computing task will be released and the next operator layer will be executed; The task ends when all operator layers are executed.

6. The method for scheduling and executing computing tasks according to claim 1, wherein: The manner of configuring the operator resources of the preset operator includes: on-demand allocation and delayed allocation.

7. A scheduling and execution device for computing tasks, characterized in that: The device is located in a scheduling execution server in a computing system, the computing system further comprising an execution plan generator, a resource scheduler, a task scheduler and a computing platform, and the device comprises: The information acquisition module is used to obtain target computing task information; an analysis and processing module, configured to analyze and process the target computing task through the execution plan generator, determine whether to interact with the resource scheduler based on the analysis and processing results, evaluate and configure operator resources of preset operators, and generate a corresponding execution plan, wherein multiple operators in the execution plan are arranged horizontally and hierarchically according to type; wherein the computing platform includes a cloud computing platform, and multiple operators are preset in the computing platform; The execution module is used to call and execute in sequence according to the execution plan through the task scheduler until all operators are executed.

8. The computing task scheduling and execution device according to claim 7, characterized in that: The analysis and processing module is used for: If the target computing task does not include a cost target and a performance target, then based on the target computing task, the execution plan generator evaluates and configures the operator resources of the corresponding operator according to the corresponding historical execution information; If the target computing task includes both a cost target and a performance target, the execution plan generator interacts with the resource scheduler, and the resource scheduler evaluates and configures the operator resources of the operator according to the interaction result.

9. The computing task scheduling and execution device according to claim 7, characterized in that: The execution module is used to: Hierarchical execution starts from the data access layer through the task scheduler; If the current operator layer is completed, the computing power resources occupied by the current computing task will be released and the next operator layer will be executed; The task ends when all operator layers are executed.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images