Query plan processing method and apparatus

Abstract

The one or more embodiments of the specification provide a query plan processing method and device, the method comprising: in response to a query plan to be executed, dividing query execution steps contained in the query plan into a plurality of query execution groups; and according to a scheduling sequence of the plurality of query execution groups, sequentially scheduling a query execution group meeting a scheduling start condition into an execution queue; wherein a scheduling start condition corresponding to any query execution group in the plurality of query execution groups comprises: a previous query execution group of the any query execution group starts to generate execution result data.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of big data, and more particularly to a method and apparatus for processing query plans. Background Technology

[0002] A real-time data warehouse is a concept relative to an offline data warehouse. It builds upon the capabilities of an offline data warehouse by further meeting timeliness requirements, thus solving the problem of low data timeliness in offline data warehouses. A real-time data warehouse requires the service platform to provide both online query and offline computation services; that is, the service platform needs to have the ability to quickly respond to online tasks and handle complex offline tasks simultaneously.

[0003] In related technologies, service platforms meet the above requirements by integrating two computing engines: an interactive analysis engine for online tasks and a big data engine for offline tasks. However, this method not only requires the simultaneous operation and maintenance of two computing engines, increasing operational costs, but also requires users to select between the two engines after issuing tasks, increasing the complexity of user operations. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide a method and apparatus for processing query plans, which can solve the deficiencies existing in the related art.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for processing query plans is provided, the method comprising:

[0007] In response to a query plan to be executed, the query execution steps contained in the query plan are divided into multiple query execution groups;

[0008] According to the scheduling order of the plurality of query execution groups, query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence; wherein, the scheduling start condition for any query execution group among the plurality of query execution groups includes: the previous query execution group of any query execution group starts generating execution result data.

[0009] According to a second aspect of one or more embodiments of this specification, a query plan processing apparatus is provided, the apparatus comprising:

[0010] Division Unit: In response to a query plan to be executed, the query execution steps contained in the query plan are divided into multiple query execution groups;

[0011] Scheduling unit: According to the scheduling order of the plurality of query execution groups, the query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence; wherein, the scheduling start condition for any query execution group among the plurality of query execution groups includes: the previous query execution group of any query execution group starts generating execution result data.

[0012] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0013] processor;

[0014] Memory used to store processor-executable instructions;

[0015] The processor implements the steps of the method as described in the first aspect by running the executable instructions.

[0016] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0017] According to a fifth aspect of one or more embodiments of this specification, a computer program is provided that, when executed by a processor, implements the steps of the method as described in the first aspect.

[0018] As can be seen from the above technical solutions, the query plan processing method provided in one or more embodiments of this specification, on the one hand, divides the query execution steps contained in the query plan into multiple query execution groups, so that these multiple query execution groups can be scheduled to the execution queue sequentially according to the scheduling order. Even in the case of complex query plans, the query plan can be split into multiple query execution groups and scheduled sequentially, thereby realizing the processing of complex offline tasks. On the other hand, by setting the scheduling start condition of these multiple query execution groups to the start of the previous query execution group generating execution result data, any query execution group can be scheduled to the execution queue when the previous query execution group starts generating execution result data, instead of waiting until the previous query execution group has finished executing. This improves the scheduling efficiency of the query execution groups, thereby speeding up the response speed to online tasks. In addition, since the above method can process complex offline tasks and respond quickly to online tasks, it only requires the maintenance of one computing engine to meet the requirements of a real-time data warehouse, thereby reducing maintenance costs and the complexity of user operations. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a query plan processing framework provided in an exemplary embodiment.

[0020] Figure 2 This is a flowchart of a query plan processing method provided in an exemplary embodiment.

[0021] Figure 3 This is a schematic diagram of a tree-structured query plan provided in an exemplary embodiment.

[0022] Figure 4 This is a schematic diagram illustrating a time comparison provided in an exemplary embodiment.

[0023] Figure 5 This is a schematic diagram illustrating the division of query execution groups as provided in an exemplary embodiment.

[0024] Figure 6 This is a flowchart of a data distribution method provided in an exemplary embodiment.

[0025] Figure 7 This is a schematic structural diagram of a device provided in an exemplary embodiment.

[0026] Figure 8 This is a block diagram of a query plan processing apparatus provided in an exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0029] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] To further illustrate one or more embodiments of this specification, the following embodiments are provided:

[0031] A real-time data warehouse is a concept relative to an offline data warehouse. It builds upon the capabilities of an offline data warehouse by further meeting timeliness requirements, thus solving the problem of low data timeliness in offline data warehouses. A real-time data warehouse requires the service platform to provide both online query and offline computation services; that is, the service platform needs to have the ability to quickly respond to online tasks and handle complex offline tasks simultaneously.

[0032] In related technologies, the service platform meets the above requirements by integrating two computing engines: an interactive analytics engine, such as an MPP (Massively Parallel Processing) engine, and a big data engine, such as a BSP (Bulk Synchronous Parallel) engine. Both computing engines generate corresponding query plans when processing user-submitted query tasks. These query plans contain a set of query execution steps to complete the query task. For example... Figure 1 As shown, query plan 10 includes four query execution steps: query execution step 101, query execution step 102, query execution step 103, and query execution step 104. The execution order of these four query execution steps is as described above. The computing engine can schedule the four query execution steps into an execution queue. Being in the execution queue does not mean that a query execution step can be executed immediately; the next query execution step will only begin execution after the previous one has finished. For example, even if query execution steps 101 and 102 are simultaneously scheduled into the execution queue, the execution queue will execute query execution step 101 first, and then begin executing query execution step 102 after query execution step 101 has finished. However, even if a query execution step in the execution queue has not yet started execution, it will still consume some resources.

[0033] In computing, scheduling is a method of allocating resources needed for tasks. Resources can refer to virtual computing resources, such as threads, processes, or data streams; or hardware resources, such as processors, network connections, or expansion cards. The program that performs scheduling is called a scheduler. Its typical implementation keeps all computing resources busy, allowing multiple users to effectively share system resources simultaneously or achieve a specified quality of service. In distributed databases, the scheduler enables multiple query tasks issued by users, or multiple query execution steps broken down from a single query task, to be processed in parallel, making it possible to fully utilize the resources of the entire distributed system.

[0034] As mentioned earlier, related technologies integrate the features of two computing engines to meet the requirements of real-time data warehouses, and the Presto engine is one such engine. The Presto engine implements a scheduler that supports both MPP and BSP engines. When processing query plans for online tasks, this scheduler uses an all-at-once scheduling algorithm, which schedules all query execution steps to the execution queue simultaneously to achieve a fast response to online tasks. However, when the query plan is relatively complex, this method, by scheduling a large number of query execution steps to the execution queue at once, can lead to resource contention and waste.

[0035] This scheduler uses a phased scheduling algorithm when processing query plans for offline tasks. This divides the query execution steps into multiple execution phases, and only schedules the next execution phase into the execution queue after the previous phase has completed. This allows for handling complex offline tasks. While this method solves the resource contention and waste problems of the all-at-once scheduling algorithm, its limitation of scheduling time restricts its efficiency, thus limiting the efficiency of query task processing and reducing response speed.

[0036] Furthermore, integrating two computing engines requires the simultaneous operation and maintenance of both engines, increasing operational costs. It also requires users to select between the two computing engines after issuing tasks, increasing the complexity of user operations.

[0037] To address the shortcomings of related technologies, this specification proposes a method for processing query plans.

[0038] Figure 2 This is a flowchart illustrating a query plan processing method provided in an exemplary embodiment, such as... Figure 2 As shown, the method includes at least the following steps:

[0039] Step 202: In response to the query plan to be executed, the query execution steps contained in the query plan are divided into multiple query execution groups.

[0040] When a user issues a query task, the service platform can generate a corresponding query plan. This query plan is a tree structure connected by physical operators, where each physical operator can be considered an iterator. A query plan can contain multiple query execution steps, each of which can be considered a node in the tree structure. The computing engine can be viewed as a virtual machine that processes this tree structure, handling the nodes from bottom to top.

[0041] The following is combined with Figure 3 The query plan for the tree structure is then completed in detail. Figure 3 This is a schematic diagram of a query plan provided in an exemplary embodiment, such as... Figure 3 As shown, the query plan is a tree structure, and the three query execution steps included in the query plan can be regarded as three nodes in the tree structure, including query execution steps 301 to 303. According to the bottom-up processing order (the arrows in the tree structure are used to identify the dependencies between query execution steps and do not represent the processing order), the execution order of these three query execution steps is query execution steps 301 to 303. The operator corresponding to query execution step 301 is the Scan operator, which is used to read data from the table; the operator corresponding to query execution step 302 is the Selection operator, which is used to select data from the table that meets the conditions; the operator corresponding to query execution step 303 is the Projection operator, which is used to output the selected data. Therefore, the function of the query plan consisting of query execution steps 301 to 303 is to read data from a specific table and select data from that table that meets the conditions for output.

[0042] In this embodiment, by generating a tree-structured query plan, the computing engine can process the query execution steps in the query plan, thereby responding to the query tasks issued by the user.

[0043] Query execution steps can be divided into multiple query execution groups, and a query execution group can contain one or more query execution steps.

[0044] Step 204: According to the scheduling order of the plurality of query execution groups, the query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence; wherein, the scheduling start condition for any query execution group among the plurality of query execution groups includes: the previous query execution group of any query execution group starts generating execution result data.

[0045] As mentioned earlier, the BSP engine can only schedule the next query execution phase into the execution queue after the previous query execution phase has been completed, which limits the efficiency of this scheduling method. Figure 3 For example, if query execution steps 301 and 302 belong to query execution phase A, and query execution step 303 belongs to query execution phase B, then for the BSP engine, query execution phase B will only be scheduled into the execution queue for execution if query execution phase A outputs all the values ​​of fields A and B. Figure 4As shown, the time taken by the BSP engine to execute query execution phase A and query execution phase B is: execution time of query execution phase A + scheduling time of query execution phase B + execution time of query execution phase B.

[0046] and Figure 2 The query plan processing method proposed in the embodiment sets the scheduling start condition for any query execution group among multiple query execution groups to: the previous query execution group of that query execution group starts generating execution result data. Still using... Figure 3 For example, if query execution steps 301 and 302 belong to query execution stage A, and query execution step 303 belongs to query execution stage B, then in this embodiment, when query execution stage A outputs the value of the first field A or the value of the first field B, query execution stage B can be scheduled to be executed in the execution queue. At this time, the execution task for query execution stage A and the scheduling task for query execution stage B are processed in parallel, thus improving scheduling efficiency. Figure 4 As shown, in this embodiment, the time taken to execute query execution phase A and query execution phase B is less than that taken by the BSP engine.

[0047] The method for determining the scheduling order of the query execution group will be explained in detail later in this manual, and will not be repeated here.

[0048] The query plan processing method proposed in this specification does not schedule all query execution steps to the execution queue simultaneously, as the MPP engine does. Instead, it divides the query execution steps within the query plan into multiple query execution groups. These groups are then scheduled to the execution queue sequentially according to the scheduling order, thus enabling the handling of complex offline tasks. Furthermore, unlike the BSP engine, which schedules the next query execution stage only after the previous one has completed, the method sets the scheduling start condition for each query execution group to the start of the previous group generating execution results. This accelerates task processing and allows for the handling of online tasks. Therefore, the query plan processing method proposed in this specification eliminates the need to maintain two separate computing engines; maintaining only one engine is sufficient to handle both offline and online tasks.

[0049] In this embodiment, on the one hand, by dividing the query execution steps contained in the query plan into multiple query execution groups, these multiple query execution groups can be scheduled to the execution queue sequentially according to the scheduling order. Even in the case of complex query plans, the query plan can be split into multiple query execution groups and scheduled sequentially, thereby realizing the processing of complex offline tasks. On the other hand, by setting the scheduling start condition of these multiple query execution groups to the start of the previous query execution group generating execution result data, any query execution group can be scheduled to the execution queue as soon as the previous query execution group starts generating execution result data, rather than waiting until the previous query execution group has finished executing. This improves the scheduling efficiency of the query execution groups, thereby speeding up the response speed to online tasks. In addition, since the above method can handle both complex offline tasks and quickly respond to online tasks, it only requires the maintenance of one computing engine to meet the requirements of a real-time data warehouse, thereby reducing maintenance costs and the complexity of user operations.

[0050] In one embodiment, dividing the query execution steps contained in the query plan into multiple query execution groups includes: constructing a dependency topology graph with the query execution steps as points and the dependencies between the query execution steps as edges, and dividing the query execution steps corresponding to the circular dependencies in the dependency topology graph into the same query execution group.

[0051] There are dependencies between scheduling execution steps. For example, the output of scheduling execution step A is the input of scheduling execution step B. In this case, scheduling execution step B can be said to depend on scheduling execution step A. Of course, there are also cases where two scheduling execution steps are mutually dependent. For example, the first value x output by scheduling execution step A is the first value input by scheduling execution step B, and scheduling execution step B can calculate the value y based on x. The value y can be the first value output by scheduling execution step B and the second value input by scheduling execution step A. The second value output by scheduling execution step A is the value z calculated based on the value y, and this value z is the second value input by scheduling execution step B. Therefore, scheduling execution step A and scheduling execution step B are both outputs and inputs to each other; that is, scheduling execution steps A and scheduling execution step B are mutually dependent.

[0052] Of course, interdependence can occur in more than just two scheduling execution steps. Three scheduling execution steps can also be interdependent. For example, the output of scheduling execution step A is the input of scheduling execution step B, the output of scheduling execution step B is the input of scheduling execution step C, and the output of scheduling execution step C is the input of scheduling execution step A. In this case, scheduling execution steps A, B, and C are interdependent. Similarly, a larger number of scheduling execution steps can also be interdependent.

[0053] When constructing a dependency topology graph with query execution steps as nodes and dependencies between query execution steps as edges, mutually dependent scheduling execution steps will form a circular dependency relationship. For example... Figure 5 As shown, when scheduling execution step 502 depends on scheduling execution step 501, an arrow pointing from scheduling execution step 502 to scheduling execution step 501 can be constructed between them. Scheduling execution steps 502 and 503 are mutually dependent, thus forming a circular dependency relationship. Since mutually dependent scheduling execution steps need to be executed in parallel in the execution queue, the scheduling execution steps corresponding to the circular dependency relationship need to be scheduled to the execution queue simultaneously; otherwise, deadlock problems may occur. Therefore, the scheduling execution steps corresponding to the circular dependency relationship need to be grouped into the same query execution group. For example... Figure 5 As shown, scheduling execution step 501 is assigned to query execution group 1, scheduling execution steps 502 and 503 are assigned to query execution group 2, and scheduling execution step 501 is assigned to query execution group 3. The order in which query execution groups are scheduled into the execution queue is: query execution group 1, query execution group 2, and query execution group 3.

[0054] In this embodiment, a dependency topology graph is constructed by the dependencies between scheduling execution steps, so that the computing engine can divide interdependent scheduling execution steps into the same scheduling execution group to avoid deadlock problems.

[0055] In one embodiment, the query execution step includes a strongly blocking step, which includes at least one strongly blocking operator, and the strongly blocking operator is an operator that starts execution only after all data has been collected; the step of dividing the query execution steps contained in the query plan into multiple query execution groups includes: dividing different strongly blocking steps into different query execution groups.

[0056] Each query execution step contains one or more database operators. Database operators can be understood as the specific actions of each query execution step in the query plan. For example, common SQL statements such as SELECT (selection), FROM (location), and JOIN (join) are all operators.

[0057] Database operators can be categorized into two types based on their semantics and physical implementation: streaming operators that do not require data collection and strongly blocking operators that do require data collection. Strongly blocking operators include hash join, sort, and hash aggregation operators; these operators require all input data to be collected before outputting the calculated data. Streaming operators include filter, project, and scan operators; these operators only need to receive a portion of the data to begin computation and output the data.

[0058] If a query execution group includes two strongly blocking steps, these two steps will interfere with each other, thus slowing down the process. For example, if the previous strongly blocking step only starts outputting the calculated data after collecting data from all previous steps, then the next strongly blocking step will also only be able to collect data and start outputting the calculated data after the previous strongly blocking step has finished outputting its data. Therefore, dividing different strongly blocking steps into different query execution groups can improve step execution efficiency.

[0059] In this embodiment, by dividing different strongly blocking steps into different query execution groups, a finer-grained division of query execution groups is achieved, thereby improving the efficiency of scheduling query execution groups to the query queue and thus improving query efficiency.

[0060] In one embodiment, the method further includes: determining the dependencies between the plurality of query execution groups based on the dependencies between the query execution steps, and determining the scheduling order based on the dependencies between the plurality of query execution groups.

[0061] As mentioned earlier, there are dependencies between query execution steps. If query execution step B depends on query execution step A, then query execution step A must be executed before query execution step B. If their execution order is reversed, it may lead to a deadlock. Query execution groups containing query execution steps also have dependencies; the dependencies of the query execution group can be determined based on the dependencies of the query execution steps. Figure 5 For example, since query execution step 502 depends on query execution step 501, query execution group 2 depends on query execution group 1; since query execution step 504 depends on query execution step 503, query execution group 3 depends on query execution group 2. Therefore, the scheduling order of the query execution groups from front to back can be determined as: query execution group 1, query execution group 2, and query execution group 3.

[0062] In this embodiment, the dependencies between query execution groups are determined by the dependencies between query execution steps, and the scheduling order is determined according to the dependencies between query execution groups to avoid deadlock problems caused by disordered scheduling order of query execution groups.

[0063] In one embodiment, the plurality of query execution groups includes a first query execution group and a second query execution group, wherein the output data of the first query execution group includes at least a portion of the input data of the second query execution group; the method further includes: determining a distributed data distribution method between the first query execution group and the second query execution group based on the amount of output data of the first query execution group; processing the output data of the first query execution group according to the determined distributed data distribution method, and distributing the processed data to the second query execution group.

[0064] Distributed data distribution (Shuffle) is the distribution of data across nodes and short distances within a cluster. In the query plan processing scenario, a shuffle is required between the previous query execution step and the next query execution step, and a shuffle is also required between query execution groups.

[0065] Furthermore, the distributed data distribution method includes: a memory-based distributed data distribution method and a disk-based distributed data distribution method; the step of processing the output data of the first query execution group according to the determined distributed data distribution method includes: processing the output data according to the memory-based distributed data distribution method when the data volume is less than a preset threshold; and processing the output data according to the disk-based distributed data distribution method when the data volume is not less than the preset threshold.

[0066] Memory-based distributed data distribution distributes data within memory. Due to the high read / write speed of memory, this method is highly efficient, but it consumes a significant amount of memory. Disk-based distributed data distribution distributes data within disk. This method has a limited execution speed but consumes less memory. Therefore, the Shuffle method can be determined based on the amount of output data from the first query execution group.

[0067] The phased scheduling algorithm used by the BSP engine limits data distribution to a distributed data distribution method involving reading and writing to disk. This is because the BSP engine is primarily used for handling offline tasks with large data volumes. However, the query plan scheduling method proposed in this specification needs to handle both offline tasks with large data volumes and online tasks with smaller data volumes; therefore, it cannot limit the distributed data distribution method.

[0068] The following is combined with Figure 6 The specific determination method will be explained in detail. Figure 6 This is a flowchart of a data distribution method provided in an exemplary embodiment, such as... Figure 6 As shown, the method includes at least the following steps:

[0069] Step 601: Execute the first query execution group. Step 602: Store the output data in memory. It is possible to default to using a memory-based distributed data distribution method first, that is, storing the output data in memory.

[0070] Step 603: Has the output data volume reached the preset threshold?

[0071] The preset threshold can be 1GB. If the output data of the first query execution group does not reach 1GB, the data distribution method based on memory continues to be used, that is, step 602 continues to be executed to store the output data in memory. If the output data of the first query execution group reaches 1GB, step 604 is executed to load the output data stored in memory onto the disk. Step 605, the data distribution method of reading and writing disks is used.

[0072] Step 606: Clear the output data stored in memory.

[0073] After data distribution is performed using a distributed data distribution method that reads and writes to disk, the output data previously stored in memory can be distributed to avoid consuming memory.

[0074] In this embodiment, the distributed data distribution method is switched based on the amount of data output by the first query execution group, so that data distribution is not limited to one method. This enables real-time adjustment of the data distribution method, taking into account both the efficiency of data distribution and the consumption of memory resources.

[0075] Of course, the specific method of determination is not limited to this, and it can also be determined by the user according to their needs. This manual does not impose any restrictions on this.

[0076] In this embodiment, the distributed data distribution method is switched based on the amount of data output by the first query execution group, so that data distribution is not limited to one method. This enables real-time adjustment of the data distribution method, taking into account both the efficiency of data distribution and the consumption of memory resources.

[0077] Figure 7 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 7At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, memory 709, and non-volatile memory 710, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 702 reads the corresponding computer program from the non-volatile memory 710 into memory 708 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0078] Please refer to Figure 8 A query plan processing device can be applied to, for example... Figure 8 The device shown, in order to implement the technical solution of this specification, may include:

[0079] The partitioning unit 802 is used to partition the query execution steps contained in the query plan into multiple query execution groups in response to the query plan to be executed;

[0080] The scheduling unit 804 is used to schedule query execution groups that meet the scheduling start conditions to the execution queue in sequence according to the scheduling order of the plurality of query execution groups; wherein, the scheduling start condition for any query execution group among the plurality of query execution groups includes: the previous query execution group of any query execution group starts generating execution result data.

[0081] Optionally, the partitioning unit 802 is specifically used for:

[0082] A dependency topology graph is constructed with the query execution steps as points and the dependencies between the query execution steps as edges. The query execution steps corresponding to the circular dependencies in the dependency topology graph are assigned to the same query execution group.

[0083] Optionally, the query execution step includes a strongly blocking step, which includes at least one strongly blocking operator, wherein the strongly blocking operator is an operator that starts execution only after all data has been collected; the partitioning unit 802 is specifically used for:

[0084] Different strongly blocking steps are divided into different query execution groups.

[0085] Optionally, the method further includes:

[0086] The first determining unit 806 is used to determine the dependency relationship between the plurality of query execution groups based on the dependency relationship between the query execution steps, and to determine the scheduling order based on the dependency relationship between the plurality of query execution groups.

[0087] Optionally, the plurality of query execution groups includes a first query execution group and a second query execution group, wherein the output data of the first query execution group includes at least a portion of the input data of the second query execution group; the method further includes:

[0088] The second determining unit 808 is used to determine the distributed data distribution method between the first query execution group and the second query execution group based on the amount of data in the output data of the first query execution group;

[0089] The processing unit 810 is used to process the output data of the first query execution group according to the determined distributed data distribution method, and distribute the processed data to the second query execution group.

[0090] Optionally, the distributed data distribution method includes: a memory-based distributed data distribution method and a disk-based distributed data distribution method; the processing unit 810 is specifically used for:

[0091] If the amount of data is less than a preset threshold, the output data is processed according to the memory-based distributed data distribution method.

[0092] If the amount of data is not less than a preset threshold, the output data is processed according to the distributed data distribution method of reading and writing disks.

[0093] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0094] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0095] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0096] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0097] The computer-readable medium (or computer-readable storage medium) described above or in any other form may store computer instructions that, when executed by a processor, implement one or more of the embodiments described above, thereby realizing the technical solutions of this specification.

[0098] This specification also provides a computer program that, when executed by a processor, implements one or more of the embodiments described above, thereby achieving the technical solutions of this specification. This computer program may be specifically recorded on the computer-readable medium described above or in any other form, and this specification does not impose any limitations on this.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0101] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0102] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0103] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for processing query plans, characterized in that, The method includes: In response to a query plan to be executed, the query execution steps contained in the query plan are divided into multiple query execution groups, including a first query execution group and a second query execution group. The output data of the first query execution group includes at least a portion of the input data of the second query execution group, and at least one query execution group contains multiple query execution steps. According to the scheduling order of the plurality of query execution groups, query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence; wherein, the scheduling start condition for any query execution group among the plurality of query execution groups includes: the previous query execution group of any query execution group starts generating execution result data; The method further includes: The distributed data distribution method between the first query execution group and the second query execution group is determined based on the amount of data output by the first query execution group. The distributed data distribution is cross-node and cross-short-range data distribution within the cluster. The output data of the first query execution group is processed according to the determined distributed data distribution method, and the processed data is then distributed to the second query execution group.

2. The method according to claim 1, characterized in that, The step of dividing the query execution steps contained in the query plan into multiple query execution groups includes: A dependency topology graph is constructed with the query execution steps as points and the dependencies between the query execution steps as edges. The query execution steps corresponding to the circular dependencies in the dependency topology graph are assigned to the same query execution group.

3. The method according to claim 1, characterized in that, The query execution step includes a strongly blocking step, which contains at least one strongly blocking operator, wherein the strongly blocking operator is an operator that only begins execution after all data has been collected; the step of dividing the query execution steps contained in the query plan into multiple query execution groups includes: Different strongly blocking steps are divided into different query execution groups.

4. The method according to claim 1, characterized in that, The method further includes: The dependencies between the multiple query execution groups are determined based on the dependencies between the query execution steps, and the scheduling order is determined based on the dependencies between the multiple query execution groups.

5. The method according to claim 1, characterized in that, The distributed data distribution methods include: a memory-based distributed data distribution method and a disk-based distributed data distribution method; the processing of the output data of the first query execution group according to the determined distributed data distribution method includes: If the amount of data is less than a preset threshold, the output data is processed according to the memory-based distributed data distribution method. If the amount of data is not less than the preset threshold, the output data is processed according to the distributed data distribution method of reading and writing disks.

6. A query plan processing apparatus, characterized in that, The device includes: Division Unit: In response to a query plan to be executed, the query execution steps contained in the query plan are divided into multiple query execution groups. The multiple query execution groups include a first query execution group and a second query execution group. The output data of the first query execution group includes at least a portion of the input data of the second query execution group. At least one query execution group contains multiple query execution steps. Scheduling unit: According to the scheduling order of the plurality of query execution groups, the query execution groups that meet the scheduling start conditions are scheduled to the execution queue in sequence; wherein, the scheduling start condition for any query execution group among the plurality of query execution groups includes: the previous query execution group of any query execution group starts generating execution result data; The device further includes: The second determining unit is used to determine the distributed data distribution method between the first query execution group and the second query execution group based on the amount of data output by the first query execution group, wherein the distributed data distribution is cross-node and cross-short-range data distribution within the cluster. The processing unit is used to process the output data of the first query execution group according to the determined distributed data distribution method, and distribute the processed data to the second query execution group.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the steps of the method as described in any one of claims 1-5 by running the executable instructions.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, When the program product is executed by the processor, it implements the steps of the method as described in any one of claims 1-5.

Images