Query plan processing method and apparatus

Abstract

Provided in one or more embodiments of the present disclosure are a query plan processing method and apparatus. The method comprises: in response to a query plan to be executed, dividing, into a plurality of query execution groups, query execution steps which are included in said query plan; and on the basis of a scheduling sequence of the plurality of query execution groups, sequentially scheduling into an execution queue query execution groups which meet a scheduling starting condition, wherein the scheduling starting condition corresponding to any of the plurality of query execution groups comprises the previous query execution group of any query execution group starting to generate execution result data.

Description

Query plan processing method and device

[0001] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on September 4, 2023, with application number 202311137689.8 and application name “Query Plan Processing Method and Device”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] One or more embodiments of the present disclosure relate to the field of big data, and more particularly, to a method and apparatus for processing a query plan. Background Art

[0003] A real-time data warehouse is a concept that stands in contrast to offline data warehouses. It further addresses the timeliness requirements of offline data warehouses, resolving the issue of low data timeliness in offline data warehouses. A real-time data warehouse requires the service platform to provide both online query and offline computing services. This means the service platform must be able to quickly respond to online tasks while also handling complex offline tasks.

[0004] In related technologies, service platforms meet these requirements by integrating two computing engines: an interactive analysis engine for online tasks and a big data engine for offline tasks. However, this approach not only requires the simultaneous operation and maintenance of both computing engines, increasing operational costs, but also requires users to select between the two computing engines after issuing tasks, increasing the complexity of user operations.

[0005] Summary of the Invention

[0006] In view of this, one or more embodiments of the present disclosure provide a method and apparatus for processing a query plan, which can address the deficiencies in related technologies.

[0007] To achieve the above objectives, one or more embodiments of the present disclosure provide the following technical solutions:

[0008] According to a first aspect of one or more embodiments of the present disclosure, a method for processing a query plan is proposed, the method comprising:

[0009] In response to a query plan to be executed, dividing the query execution steps contained in the query plan into a plurality of query execution groups;

[0010] The query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence according to the scheduling order of the multiple query execution groups; wherein the scheduling start condition corresponding to any query execution group among the multiple query execution groups includes: the previous query execution group of any query execution group starts to generate execution result data.

[0011] According to a second aspect of one or more embodiments of the present disclosure, a query plan processing device is provided, the device comprising:

[0012] A division unit: in response to a query plan to be executed, dividing the query execution steps contained in the query plan into a plurality of query execution groups;

[0013] Scheduling unit: Schedules query execution groups that meet the scheduling start conditions to the execution queue in sequence according to the scheduling order of the multiple query execution groups; wherein the scheduling start conditions corresponding to any query execution group among the multiple query execution groups include: the previous query execution group of any query execution group starts to generate execution result data.

[0014] According to a third aspect of one or more embodiments of the present disclosure, an electronic device is provided, including:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor implements the steps of the method described in the first aspect by running the executable instructions.

[0018] According to a fourth aspect of one or more embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] According to a fifth aspect of one or more embodiments of the present disclosure, a computer program is proposed, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0020] It can be seen from the above technical solutions that the query plan processing method and device provided by one or more embodiments of the present disclosure, on the one hand, by dividing the query execution steps contained in the query plan into multiple query execution groups, so that the multiple query execution groups can be scheduled to the execution queue in sequence according to the scheduling order, even in the case of a complex query plan, the query plan can be split into multiple query execution groups and scheduled in sequence, thereby realizing the processing of complex offline tasks; on the other hand, by setting the scheduling start condition of the multiple query execution groups to the start of the previous query execution group to generate execution result data, any query execution group can be scheduled to the execution queue when the previous query execution group starts to generate execution result data, and does not need to be scheduled to the execution queue when the previous query execution group is completed, thereby improving the scheduling efficiency of the query execution group, thereby speeding up the response speed to online tasks. In addition, since the above method can both process complex offline tasks and quickly respond to online tasks, the above method only needs to operate and maintain one computing engine to meet the requirements of real-time data warehouses, thereby reducing operation and maintenance costs and reducing the complexity of user operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a schematic diagram of a query plan processing framework provided by an exemplary embodiment.

[0022] FIG2 is a flowchart of a method for processing a query plan provided by an exemplary embodiment.

[0023] FIG3 is a schematic diagram of a query plan with a tree structure provided by an exemplary embodiment.

[0024] FIG4 is a schematic diagram of a time comparison provided by an exemplary embodiment.

[0025] FIG5 is a schematic diagram of dividing query execution groups provided by an exemplary embodiment.

[0026] FIG6 is a flowchart of a data distribution method provided by an exemplary embodiment.

[0027] FIG7 is a schematic structural diagram of a device provided by an exemplary embodiment.

[0028] FIG8 is a block diagram of a query plan processing apparatus provided by an exemplary embodiment. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible implementations consistent with one or more embodiments of the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of the present disclosure, as detailed in the appended claims.

[0030] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this disclosure. In some other embodiments, the method may include more or fewer steps than those described in this disclosure. In addition, a single step described in this disclosure may be broken down into multiple steps for description in other embodiments; and multiple steps described in this disclosure may be combined into a single step for description in other embodiments.

[0031] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] To further illustrate one or more embodiments of the present disclosure, the following embodiments are provided:

[0033] A real-time data warehouse is a concept that stands in contrast to offline data warehouses. It further addresses the timeliness requirements of offline data warehouses, resolving the issue of low data timeliness in offline data warehouses. A real-time data warehouse requires the service platform to provide both online query and offline computing services. This means the service platform must be able to quickly respond to online tasks while also handling complex offline tasks.

[0034] In related technologies, the service platform meets the above requirements by integrating two computing engines. The two computing engines include an interactive analysis engine, such as the MPP engine (Massively Parallel Processing), and a big data engine, such as the BSP engine (Bulk Synchronous Parallel). When processing query tasks issued by users, both computing engines will generate corresponding query plans, which contain a set of query execution steps to complete the query tasks. As shown in Figure 1, query plan 10 contains four query execution steps: query execution step 101, query execution step 102, query execution step 103, and query execution step 104. The execution order of these four query execution steps is the above-mentioned display order. The computing engine can schedule the four query execution steps to the execution queue. Being in the execution queue does not mean that the query execution step can be executed immediately. The next query execution step will only start executing when the previous query execution step is completed. For example, even if query execution step 101 and query execution step 102 are simultaneously scheduled to the execution queue, the execution queue will execute query execution step 101 first, and then start executing query execution step 102 after query execution step 101 is completed. However, even if the query execution step in the execution queue has not started executing, it will still occupy some resources.

[0035] Scheduling, in computing, is the method of allocating resources required for work. Resources can refer to virtual computing resources, such as threads, processes, or data streams, or hardware resources, such as processors, network connections, or expansion cards. The program that performs scheduling is called a scheduler. Its typical implementation is to keep all computing resources busy, allowing multiple users to effectively share system resources simultaneously or achieve a specified quality of service. In a distributed database, the scheduler enables multiple query tasks issued by users, or multiple query execution steps split from a single query task, to be processed in parallel during execution, making it possible to fully utilize the resources of the entire distributed system.

[0036] As mentioned previously, related technologies integrate the features of two computing engines to meet the requirements of real-time data warehouses, and the Presto engine is precisely this type of engine. The Presto engine implements a scheduler that supports both MPP and BSP engines. When processing query plans corresponding to online tasks, this scheduler uses an all-at-once scheduling algorithm, scheduling all query execution steps simultaneously to the execution queue to achieve rapid response to online tasks. However, in the case of relatively complex query plans, this method schedules a large number of query execution steps to the execution queue at once, leading to resource contention and waste.

[0037] When processing query plans for offline tasks, this scheduler uses a phased scheduling algorithm. This algorithm divides query execution steps into multiple query execution phases and schedules the next query execution phase to the execution queue only after the previous query execution phase has completed. This approach addresses the resource competition and waste issues inherent in the all-at-once scheduling algorithm. However, since the next query execution phase can only be scheduled to the execution queue after the previous query execution phase has completed, this method limits scheduling efficiency because resource scheduling takes time. This, in turn, limits query task processing efficiency and reduces response speed.

[0038] In addition, integrating two computing engines requires operating and maintaining both computing engines simultaneously, which increases operation and maintenance costs. It also requires users to select between the two computing engines after issuing tasks, which increases the complexity of user operations.

[0039] To address the deficiencies in the related art, the present disclosure proposes a method for processing a query plan.

[0040] FIG2 is a flowchart of a query plan processing method provided by an exemplary embodiment. As shown in FIG2 , the method includes at least the following steps:

[0041] Step 202 : In response to a query plan to be executed, divide the query execution steps contained in the query plan into a plurality of query execution groups.

[0042] When a user issues a query task, the service platform generates a corresponding query plan. This query plan is a tree structure connected by physical operators, where each physical operator can be considered an iterator. A query plan can contain multiple query execution steps, each of which can be considered a node in the tree structure. The computing engine can be considered a virtual machine that processes this tree structure, processing the nodes from the bottom up.

[0043] The following is a detailed description of the query plan with a tree structure in conjunction with Figure 3. Figure 3 is a schematic diagram of a query plan provided by an exemplary embodiment. As shown in Figure 3, the query plan is a tree structure, and the three query execution steps included in the query plan can be regarded as three nodes in the tree structure, including query execution step 301-query execution step 303. According to the bottom-up processing order (the arrows in the tree structure are used to identify the dependency relationship between the query execution steps, not representing the processing order), the execution order of these three query execution steps is query execution step 301 to query execution step 303. The operator corresponding to query execution step 301 is the Scan operator, and the Scan operator is used to read the data in the table; the operator corresponding to query execution step 302 is the Selection operator, and the Selection operator is used to select the data that meets the conditions in the table; the operator corresponding to query execution step 303 is the Projection operator, and the Projection operator is used to output the selected data. Therefore, the function of the query plan composed of query execution step 301-query execution step 303 is to read the data in a specific table and select the data that meets the conditions from the table for output.

[0044] In this embodiment, by generating a query plan with a tree structure, the computing engine can process the query execution steps in the query plan, thereby achieving a response to the query task issued by the user.

[0045] Query execution steps can be divided into multiple query execution groups, and a query execution group can contain one or more query execution steps. Taking Figure 3 as an example, query execution steps 301 and 302 can be divided into the same query execution group, while query execution steps 303, 304, and 305 are divided into different query execution groups. Therefore, all query execution steps are divided into four query execution groups. The specific process of dividing query execution groups will be explained in detail later and will not be repeated here.

[0046] Step 204: Schedule the query execution groups that meet the scheduling start conditions to the execution queue in sequence according to the scheduling order of the multiple query execution groups; wherein the scheduling start condition corresponding to any query execution group among the multiple query execution groups includes: the previous query execution group of any query execution group starts to generate execution result data.

[0047] As mentioned above, the BSP engine can only schedule the next query execution stage to the execution queue after the previous query execution stage is completed. This scheduling method has limited efficiency. Taking Figure 3 as an example, if query execution steps 301 and 302 belong to query execution stage A, and query execution step 303 belongs to query execution stage B, then for the BSP engine, query execution stage B will only be scheduled to the execution queue for execution if query execution stage A outputs all the field values ​​of fields A and B. As shown in Figure 4, the time taken by the BSP engine to execute query execution stage A and query execution stage B is: the execution time of query execution stage A + the scheduling time of query execution stage B + the execution time of query execution stage B.

[0048] The query plan processing method proposed in the embodiment of Figure 2 sets the scheduling start condition corresponding to any query execution group among the multiple query execution groups to: the previous query execution group of any query execution group starts to generate execution result data. Taking Figure 3 as an example, if query execution step 301 and query execution step 302 belong to query execution stage A, and query execution step 303 belongs to query execution stage B, then for this embodiment, when query execution stage A outputs the first field value of field A or the first field value of field B, query execution stage B can be scheduled to the execution queue for execution. At this time, the execution task for query execution stage A and the scheduling task for query execution stage B are processed in parallel, so the scheduling efficiency can be improved. As shown in Figure 4, the time taken by this embodiment to execute query execution stage A and query execution stage B is less than the time taken by the BSP engine.

[0049] As for the method for determining the scheduling order of the query execution group, this disclosure will be described in detail later and will not be repeated here.

[0050] The query plan processing method proposed in the present disclosure does not schedule all query execution steps to the execution queue at the same time like the MPP engine, but divides the query execution steps contained in the query plan into multiple query execution groups, so that the multiple query execution groups can be scheduled to the execution queue in sequence according to the scheduling order, so it can handle complex offline tasks. The query plan processing method proposed in the present disclosure does not schedule the next query execution stage to the execution queue only after the previous query execution stage is completed, like the BSP engine, but sets the scheduling start condition of the query execution group to the previous query execution group to start generating execution result data, which speeds up the task processing rate, so it can handle online tasks. It can be seen that according to the query plan processing method proposed in the present disclosure, there is no need to maintain two computing engines, and only one computing engine needs to be maintained to handle both offline tasks and online tasks.

[0051] In this embodiment, on the one hand, by dividing the query execution steps contained in the query plan into multiple query execution groups, the multiple query execution groups can be scheduled to the execution queue in sequence according to the scheduling order. Even in the case of a complex query plan, the query plan can be split into multiple query execution groups and scheduled in sequence, thereby realizing the processing of complex offline tasks; on the other hand, by setting the scheduling start condition of the multiple query execution groups to the start of the previous query execution group to generate execution result data, any query execution group can be scheduled to the execution queue when the previous query execution group starts to generate execution result data, and does not need to be scheduled to the execution queue when the previous query execution group is completed. This improves the scheduling efficiency of the query execution group, thereby speeding up the response speed to online tasks. In addition, since the above method can both process complex offline tasks and quickly respond to online tasks, the above method only needs to operate and maintain one computing engine to meet the requirements of real-time data warehouses, thereby reducing operation and maintenance costs and reducing the complexity of user operations.

[0052] In one embodiment, dividing the query execution steps contained in the query plan into multiple query execution groups includes: constructing a dependency topology graph with the query execution steps as points and the dependency relationships between the query execution steps as edges, and dividing the query execution steps corresponding to the ring dependencies in the dependency topology graph into the same query execution group.

[0053] There are dependencies between scheduling execution steps. For example, the output of scheduling execution step A is the input of scheduling execution step B. In this case, scheduling execution step B can be said to be dependent on scheduling execution step A. Of course, there are also cases where two scheduling execution steps are dependent on each other. For example, the first value x output by scheduling execution step A is the first value input by scheduling execution step B, and scheduling execution step B can calculate value y based on x. Value y can be the first value output by scheduling execution step B and the second value input by scheduling execution step A. The second value output by scheduling execution step A is value z calculated based on value y, and this value z is the second value input by scheduling execution step B. Therefore, scheduling execution step A and scheduling execution step B are each other's outputs and inputs, that is, scheduling execution step A and scheduling execution step B are dependent on each other.

[0054] Of course, interdependence can occur between more than two scheduling execution steps. Three scheduling execution steps can also be interdependent. For example, the output of scheduling execution step A is the input of scheduling execution step B, the output of scheduling execution step B is the input of scheduling execution step C, and the output of scheduling execution step C is the input of scheduling execution step A. In this case, scheduling execution step A, scheduling execution step B, and scheduling execution step C are interdependent. Similarly, a larger number of scheduling execution steps can also be interdependent.

[0055] When constructing a dependency topology graph using query execution steps as nodes and the dependencies between query execution steps as edges, a circular dependency relationship will form between interdependent scheduling execution steps. As shown in Figure 5, if scheduling execution step 502 depends on scheduling execution step 501, an arrow pointing from scheduling execution step 502 to scheduling execution step 501 can be constructed between scheduling execution step 501 and scheduling execution step 502. Scheduling execution step 502 and scheduling execution step 503 are interdependent, thus forming a circular dependency relationship. Since interdependent scheduling execution steps need to be executed in parallel in the execution queue, the scheduling execution steps corresponding to the circular dependency relationship need to be scheduled into the execution queue simultaneously, otherwise deadlock is likely to occur. Therefore, the scheduling execution steps corresponding to the circular dependency relationship need to be assigned to the same query execution group. As shown in Figure 5, scheduling execution step 501 is assigned to query execution group 1, scheduling execution step 502 and scheduling execution step 503 are assigned to query execution group 2, and scheduling execution step 501 is assigned to query execution group 3. The order in which the query execution groups are dispatched to the execution queue is: query execution group 1, query execution group 2, and query execution group 3.

[0056] In this embodiment, a dependency topology graph is constructed by the dependency relationship between the scheduling execution steps, so that the computing engine can divide the mutually dependent scheduling execution steps into the same scheduling execution group to avoid the occurrence of deadlock problems.

[0057] In one embodiment, the query execution step includes a strong blocking step, and the strong blocking step includes at least one strong blocking operator, which is an operator that starts execution only after all data is collected; dividing the query execution steps contained in the query plan into multiple query execution groups includes: dividing different strong blocking steps into different query execution groups.

[0058] Each query execution step contains one or more database operators. Database operators can be understood as the specific actions of each query execution step in the query plan. For example, the common SELECT, FROM, and JOIN in SQL statements are all operators.

[0059] Database operators can be divided into two categories based on their semantics and physical implementation: streaming operators that do not require data collection and strongly blocking operators that do. Strongly blocking operators include hash joins, sorts, and hash aggregations. These operators require all input data to be collected before they begin outputting the computed data. Streaming operators include filter, project, and scan operators. These operators only need to receive a portion of the data before they begin computing and outputting the data.

[0060] If a query execution group includes two strongly blocking steps, they will affect each other, slowing down execution. For example, if the previous strongly blocking step cannot output calculated data until it has collected data from all previous steps, then the next strongly blocking step cannot complete data collection and output calculated data until the previous step has finished outputting data. Therefore, grouping different strongly blocking steps into different query execution groups can improve step execution efficiency.

[0061] In this embodiment, by dividing different strongly blocking steps into different query execution groups, a finer-grained division of query execution groups is achieved, thereby improving the efficiency of scheduling query execution groups to query queues, and further improving query efficiency.

[0062] In one embodiment, the method further includes: determining dependencies between the multiple query execution groups according to the dependencies between the query execution steps, and determining the scheduling order according to the dependencies between the multiple query execution groups.

[0063] As mentioned above, there are dependencies between query execution steps. When query execution step B depends on query execution step A, query execution step A needs to be executed before query execution step B. If the execution order of the two is swapped, a deadlock problem may occur. Query execution groups containing query execution steps also have dependencies. The dependencies of query execution groups can be determined based on the dependencies of query execution steps. Taking Figure 5 as an example, since query execution step 502 depends on query execution step 501, query execution group 2 depends on query execution group 1; since query execution step 504 depends on query execution step 503, query execution group 3 depends on query execution group 2. Therefore, it can be determined that the scheduling order of query execution groups from front to back is: query execution group 1, query execution group 2, query execution group 3.

[0064] In this embodiment, the dependencies between query execution groups are determined by the dependencies between query execution steps, and the scheduling order is determined according to the dependencies between query execution groups, thereby avoiding deadlock problems caused by query execution groups being in a disordered scheduling order.

[0065] In one embodiment, the multiple query execution groups include a first query execution group and a second query execution group, and the output data of the first query execution group includes at least a portion of the input data of the second query execution group; the method further includes: determining a distributed data distribution method between the first query execution group and the second query execution group based on the data volume of the output data of the first query execution group; processing the output data of the first query execution group according to the determined distributed data distribution method, and distributing the processed data to the second query execution group.

[0066] Distributed data shuffle is the cross-node and cross-distance data distribution within a cluster. In query plan processing scenarios, shuffle is required between the previous query execution step and the next query execution step, and shuffle is also required between query execution groups.

[0067] Furthermore, the distributed data distribution method includes: a memory-based distributed data distribution method and a read-write disk distributed data distribution method; the processing of the output data of the first query execution group according to the determined distributed data distribution method includes: when the data volume of the data is less than a preset threshold, processing the output data according to the memory-based distributed data distribution method; when the data volume of the data is not less than the preset threshold, processing the output data according to the read-write disk distributed data distribution method.

[0068] Memory-based distributed data distribution places data in memory for distribution. Due to the high read and write speed of memory, this method is highly efficient, but it consumes more memory. Disk-based distributed data distribution places data on disk for distribution. This method has a limited execution rate but consumes less memory. Therefore, the shuffle method can be determined based on the amount of output data from the first query execution group.

[0069] The phased scheduling algorithm used by the BSP engine limits data distribution to a distributed approach involving disk reads and writes. This is because BSP engines are primarily used to handle offline tasks with large data volumes. However, the query plan scheduling method proposed in this disclosure needs to handle both offline tasks with large data volumes and online tasks with smaller data volumes. Therefore, it cannot be limited to a distributed data distribution approach.

[0070] The specific determination method is described in detail below with reference to Figure 6. Figure 6 is a flow chart of a data distribution method provided by an exemplary embodiment. As shown in Figure 6, the method includes at least the following steps:

[0071] Step 601: Execute the first query execution group. Step 602: Store the output data in memory. By default, a memory-based distributed data distribution method may be used for data distribution, that is, the output data is stored in memory.

[0072] Step 603: Whether the output data volume reaches a preset threshold.

[0073] The preset threshold can be 1GB. If the output data volume of the first query execution group does not reach 1GB, data distribution continues using the memory-based distributed data distribution method. That is, step 602 is continued to store the output data in memory. If the output data volume of the first query execution group reaches 1GB, step 604 is performed to load the output data stored in memory to disk. Step 605 is used to distribute data using the read-write disk distributed data distribution method.

[0074] Step 606: Clear the output data stored in the memory.

[0075] After data distribution is performed using a distributed data distribution method that reads and writes disks, the output data previously stored in memory can be saved to avoid memory occupation.

[0076] In this embodiment, the distributed data distribution mode is switched based on the amount of output data of the first query execution group, so that data distribution is not limited to one mode, and real-time adjustment of the data distribution mode is achieved, taking into account the efficiency of data distribution and the occupancy of memory resources.

[0077] Of course, the specific determination method is not limited to this, and can also be determined by the user according to needs, and this disclosure does not limit this.

[0078] In this embodiment, the distributed data distribution mode is switched based on the amount of output data of the first query execution group, so that data distribution is not limited to one mode, and real-time adjustment of the data distribution mode is achieved, taking into account the efficiency of data distribution and the occupancy of memory resources.

[0079] FIG7 is a schematic structural diagram of a device provided by an exemplary embodiment. Referring to FIG7 , at the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 709, and a non-volatile memory 710, and may also include hardware required for other functions. One or more embodiments of the present disclosure may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of the present disclosure do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0080] Referring to FIG8 , a query plan processing device may be applied to the device shown in FIG8 to implement the technical solution of the present disclosure. The device may include:

[0081] A division unit 802 is configured to divide the query execution steps contained in the query plan into a plurality of query execution groups in response to the query plan to be executed;

[0082] The scheduling unit 804 is used to schedule the query execution groups that meet the scheduling start conditions to the execution queue in sequence according to the scheduling order of the multiple query execution groups; wherein the scheduling start conditions corresponding to any query execution group among the multiple query execution groups include: the previous query execution group of any query execution group starts to generate execution result data.

[0083] Optionally, the dividing unit 802 is specifically configured to:

[0084] A dependency topology graph is constructed with the query execution steps as points and the dependency relationships between the query execution steps as edges, and query execution steps corresponding to the ring dependency relationships in the dependency topology graph are divided into the same query execution group.

[0085] Optionally, the query execution step includes a strong blocking step, the strong blocking step includes at least one strong blocking operator, and the strong blocking operator is an operator that starts execution only after all data is collected; the partitioning unit 802 is specifically configured to:

[0086] Divide different strongly blocking steps into different query execution groups.

[0087] Optionally, the method further includes:

[0088] The first determining unit 806 is configured to determine the dependency relationship between the multiple query execution groups according to the dependency relationship between the query execution steps, and determine the scheduling order according to the dependency relationship between the multiple query execution groups.

[0089] Optionally, the multiple query execution groups include a first query execution group and a second query execution group, and output data of the first query execution group includes at least a portion of input data of the second query execution group; the method further includes:

[0090] A second determining unit 808 is configured to determine a distributed data distribution method between the first query execution group and the second query execution group according to the amount of output data of the first query execution group;

[0091] The processing unit 810 is configured to process the output data of the first query execution group according to the determined distributed data distribution method, and distribute the processed data to the second query execution group.

[0092] Optionally, the distributed data distribution method includes: a memory-based distributed data distribution method and a disk-reading and writing distributed data distribution method; the processing unit 810 is specifically configured to:

[0093] When the amount of the data is less than a preset threshold, processing the output data according to the memory-based distributed data distribution method;

[0094] When the amount of the data is not less than a preset threshold, the output data is processed according to the distributed data distribution method of the read-write disk.

[0095] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0096] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0098] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0099] For the computer-readable medium (or computer-readable storage medium) as described above or in any other form, computer instructions may be stored thereon, which, when executed by a processor, implement one or more of the above-mentioned embodiments, thereby realizing the technical solution of the present disclosure.

[0100] The present disclosure also provides a computer program that, when executed by a processor, implements one or more of the above-described embodiments, thereby realizing the technical solution of the present disclosure. The computer program may be recorded on the above-described or any other form of computer-readable medium, and the present disclosure is not limited thereto.

[0101] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0102] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present disclosure. The singular forms "a," "the," and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0104] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0105] The above description is merely a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A method for processing a query plan, comprising: In response to a query plan to be executed, dividing the query execution steps contained in the query plan into a plurality of query execution groups; According to the scheduling order of the multiple query execution groups, the query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence; wherein the scheduling start condition corresponding to any query execution group among the multiple query execution groups includes: the previous query execution group of any query execution group starts to generate execution result data.

2. The method according to claim 1, wherein: The dividing the query execution steps contained in the query plan into a plurality of query execution groups includes: A dependency topology graph is constructed with the query execution steps as points and the dependency relationships between the query execution steps as edges, and query execution steps corresponding to the ring dependency relationships in the dependency topology graph are divided into the same query execution group.

3. The method according to claim 1, wherein: The query execution step includes a strong blocking step, and the strong blocking step includes at least one strong blocking operator, and the strong blocking operator is an operator that starts to execute only after all data are collected; the query execution steps contained in the query plan are divided into multiple query execution groups, including: Divide different strongly blocking steps into different query execution groups.

4. The method according to any one of claims 1 to 3, further comprising: The dependency relationship between the multiple query execution groups is determined according to the dependency relationship between the query execution steps, and the scheduling order is determined according to the dependency relationship between the multiple query execution groups.

5. The method according to any one of claims 1 to 4, wherein: The multiple query execution groups include a first query execution group and a second query execution group, and the output data of the first query execution group includes at least a portion of the input data of the second query execution group; the method further includes: Determining a distributed data distribution method between the first query execution group and the second query execution group according to the data volume of the output data of the first query execution group; The output data of the first query execution group is processed according to the determined distributed data distribution mode, and the processed data is distributed to the second query execution group.

6. The method according to claim 5, wherein: The distributed data distribution method includes: a distributed data distribution method based on memory and a distributed data distribution method based on reading and writing disks; the processing of the output data of the first query execution group according to the determined distributed data distribution method includes: When the amount of the data is less than a preset threshold, processing the output data according to the memory-based distributed data distribution method; When the amount of the data is not less than the preset threshold, the output data is processed according to the distributed data distribution method of the read-write disk.

7. A query plan processing device, comprising: A division unit: in response to a query plan to be executed, dividing the query execution steps contained in the query plan into a plurality of query execution groups; A scheduling unit: schedules query execution groups that meet the scheduling start conditions to the execution queue in sequence according to the scheduling order of the multiple query execution groups; wherein the scheduling start conditions corresponding to any query execution group among the multiple query execution groups include: the previous query execution group of any query execution group starts to generate execution result data.

8. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions. steps.

9. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

10. A computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images