Data Processing Method, Apparatus, Product, and Device

Abstract

The present application discloses a data processing method, apparatus, product, and device. The method includes: obtaining a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; obtaining a data set, where the data set includes resource data consumed by M historical subtrees during respective historical executions, and the historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer; performing a first matching process on the N subtrees in the data set, where the first matching process is used to screen matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees; determining reference resource data for the N subtrees based on the first matching result obtained from the first matching process; and estimating the execution cost corresponding to the candidate plan tree by using the reference resource data of the N subtrees. By using the present application, the accuracy of the estimated execution cost of the candidate plan tree can be improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, apparatus, product, and device. Background Art

[0002] SQL statements (structured statements) are used to query and operate on data in a database. In practical applications, SQL statements are converted into multiple candidate plan trees, and then, based on the execution costs corresponding to each of the multiple candidate plan trees, a candidate plan tree for finally executing the SQL statement is determined from the multiple candidate plan trees. It can be seen that the execution cost corresponding to the candidate plan tree of an SQL statement plays an extremely important role in the execution process of the SQL statement. Therefore, how to more accurately obtain the execution cost corresponding to the candidate plan tree of an SQL statement is a hot issue. Summary of the Invention

[0003] This application provides a data processing method, apparatus, product, and device, which can improve the accuracy of the estimated execution cost of a candidate plan tree.

[0004] On the one hand, this application provides a data processing method, which includes:

[0005] Obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement;

[0006] Obtain a data set, where the data set includes resource data consumed by M historical subtrees during their respective historical executions. The historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer;

[0007] Perform a first matching process on the resource data of the N subtrees in the data set. The first matching process is used to screen out matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees;

[0008] Based on the first matching result obtained from the first matching process, determine the reference resource data of the N subtrees;

[0009] Use the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree.

[0010] On the one hand, this application provides a data processing apparatus, which includes:

[0011] A first acquisition module, configured to obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement;

[0012] A second acquisition module, configured to acquire a dataset, where the dataset includes resource data consumed by each of M historical subtrees during historical execution, and the historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer;

[0013] A matching module, configured to perform a first matching process on resource data for N subtrees in the dataset, where the first matching process is used to screen out matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees;

[0014] A determination module, configured to determine reference resource data for the N subtrees based on a first matching result obtained from the first matching process;

[0015] An estimation module, configured to estimate the execution cost corresponding to a candidate plan tree by using the reference resource data of the N subtrees.

[0016] In one implementation, the manner in which the matching module performs a first matching process on resource data for N subtrees in the dataset includes:

[0017] Performing a normalization representation process on each of the N subtrees respectively to generate standard representation information for each subtree;

[0018] Based on the standard representation information of the N subtrees, performing a first matching process on resource data for the N subtrees in the dataset;

[0019] Wherein, the resource data consumed by each historical subtree in the dataset has a mapping relationship with the standard representation information of each historical subtree respectively; the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees is used to reflect the similarity between the N subtrees and the M historical subtrees.

[0020] In one implementation, any one of the N subtrees is a first subtree, and the dataset includes M standard representation information of M historical subtrees;

[0021] The manner in which the matching module performs a first matching process on resource data for N subtrees in the dataset based on the standard representation information of the N subtrees includes:

[0022] Obtaining the information similarity between the standard representation information of the first subtree and each of the M standard representation information;

[0023] Determining the standard representation information with the largest information similarity to the standard representation information of the first subtree among the M standard representation information as the to-be-determined representation information;

[0024] If the information similarity between the standard representation information of the first subtree and the to-be-determined representation information is greater than or equal to the set similarity threshold, then the resource data in the dataset that has a mapping relationship with the to-be-determined representation information is used as the resource data that is screened out and matches the first subtree;

[0025] Among them, the first matching result includes the resource data that is screened out and matches the first subtree.

[0026] In one implementation, the matching module is further configured to:

[0027] If the information similarity between the standard representation information of the first subtree and the to-be-determined representation information is less than the similarity threshold, then it is determined that no resource data that matches the first subtree is screened out in the dataset;

[0028] Among them, the first matching result includes that no resource data that matches the first subtree is screened out in the dataset.

[0029] In one implementation, each of the N subtrees is composed of one or more operator nodes of the candidate plan tree; the manner in which the determination module determines the reference resource data of the N subtrees based on the first matching result includes:

[0030] If the first matching result includes the resource data that is screened out and matches the first subtree, then the screened resource data that matches the first subtree is used as the reference resource data of the first subtree;

[0031] If the first matching result includes that no resource data that matches the first subtree is screened out, then the reference resource data of the first subtree is determined based on the number of operator nodes included in the first subtree.

[0032] In one implementation, the manner in which the determination module determines the reference resource data of the first subtree based on the number of operator nodes included in the first subtree includes:

[0033] When the first subtree includes one operator node of the candidate plan tree, the resource data consumed by the table data called by the first subtree is used as the reference resource data of the first subtree;

[0034] When the first subtree includes multiple operator nodes of the candidate plan tree, a second matching process for resource data of the first subtree is performed in the dataset, and based on the second matching result obtained from the second matching process, the reference resource data of each operator node in the first subtree is determined; among them, the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, and the second matching process is used to screen out the matching resource data for each operator node in the first subtree through the similarity between each operator node in the first subtree and M historical subtrees.

[0035] In one implementation, any operator node in the first subtree is a target operator node; the way for the determination module to determine the reference resource data of each operator node in the first subtree based on the second matching result includes:

[0036] If the second matching result includes the resource data filtered out and matched with the target operator node, then use the filtered resource data matched with the target operator node as the reference resource data of the target operator node;

[0037] If the second matching result includes no resource data filtered out and matched with the target operator node, then use the resource data consumed by the table data called by the target operator node as the reference resource data of the target operator node.

[0038] In one implementation, any one of the N subtrees is the first subtree; the way for the matching module to perform standardized representation processing on each of the N subtrees and generate the standard representation information of each subtree includes:

[0039] Perform data parsing processing on the first subtree from K data domains to obtain the representation data of the first subtree under the K data domains; K is a positive integer;

[0040] Perform representation processing on the representation data of the first subtree under the K data domains according to the set standard format to generate the standard representation information of the first subtree;

[0041] Among them, the K data domains include at least one of the following: the operator domain corresponding to the operator called by the subtree, the object domain corresponding to the data object in the table data called by the subtree, and the condition domain corresponding to the condition included in the operator called by the subtree.

[0042] In one implementation, the process for the first acquisition module to acquire the N subtrees of the candidate plan tree includes:

[0043] Identify the branch nodes in the candidate plan tree, and divide each branch under the branch nodes into the first type of subtrees of the candidate plan tree;

[0044] Construct the second type of subtrees of the candidate plan tree through each operator node in the candidate plan tree except the first type of subtrees, and one operator node in the candidate plan tree except the first type of subtrees is used to construct one second type of subtree;

[0045] Use the first type of subtrees and the second type of subtrees as the N subtrees obtained by dividing the candidate plan tree.

[0046] In one implementation, there are multiple candidate plan trees for the operation statement, and the above data processing device further includes an execution module, and this execution module is used for:

[0047] Select the candidate plan tree with the minimum corresponding execution cost from multiple candidate plan trees as the target plan tree of the operation statement;

[0048] Execute the operation statement using the target plan tree.

[0049] In one implementation, the resource data consumed by each historical subtree in the dataset has a mapping relationship with the standard representation information of each historical subtree respectively, and any subtree of the target plan tree is the second subtree;

[0050] After executing the target plan tree, the execution module is further configured to:

[0051] If the standard representation information of the second subtree is not included in the dataset, after executing the second subtree, obtain the resource data consumed by the second subtree during the execution process;

[0052] Construct a mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree, and add the constructed mapping relationship to the dataset.

[0053] In one implementation, the method for the execution module to obtain the resource data consumed by the second subtree during the execution process includes:

[0054] Obtain the input and output data of the last operator node in the second subtree, and use the input and output data of the last operator node as the input and output data of the second subtree;

[0055] Obtain the memory data consumed by each operator node in the second subtree, and use the maximum memory data consumed by the operator nodes in the second subtree as the memory data consumed by the second subtree;

[0056] Obtain the processor data consumed by each operator node in the second subtree, and use the maximum processor data consumed by the operator nodes in the second subtree as the processor data consumed by the second subtree;

[0057] Wherein, the resource data consumed by the second subtree includes the input and output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree.

[0058] In one implementation, the process for the first acquisition module to obtain the candidate plan tree of the operation statement includes:

[0059] Perform statement parsing processing on the operation statement to generate the logical plan tree of the operation statement;

[0060] Perform conversion processing on the logical plan tree to generate multiple candidate plan trees of the operation statement.

[0061] In one implementation, the estimation module uses the reference resource data of N subtrees to estimate the execution cost corresponding to the candidate plan tree, and the method includes:

[0062] Input the reference resource data of N subtrees into the cost estimation model;

[0063] Call the cost estimation model to estimate the execution cost corresponding to the candidate plan tree based on the reference resource data of N subtrees.

[0064] On the one hand, the present application provides a computer device, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the method in one aspect of the present application.

[0065] On the one hand, the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor, the processor executes the method in the above-mentioned one aspect.

[0066] According to one aspect of the present application, there is provided a computer program product. The computer program product includes a computer program. The computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the methods provided in the above-mentioned one aspect and various optional manners.

[0067] The present application can obtain the candidate plan tree of the operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; and can obtain a data set, which includes the resource data consumed by each of the M historical subtrees during the historical execution process. The historical subtrees are the subtrees of the historical plan tree that has been historically executed, and M is a positive integer; and can perform a first matching process on the N subtrees in the data set. The first matching process is used to screen the matching resource data for the N subtrees through the similarity between the N subtrees and the M historical subtrees; thus, the reference resource data of the N subtrees can be determined based on the first matching result obtained from the first matching process; and the reference resource data of the N subtrees is used to estimate the execution cost corresponding to the candidate plan tree. It can be seen that the method proposed by the present application can obtain the reference resource data of each subtree in the candidate plan tree at the subtree granularity, and moreover, the reference resource data can be obtained by matching from the resource data actually consumed by the historical subtrees during the actual historical execution process. Therefore, the accuracy and reliability of the reference resource data obtained for each subtree are ensured. Thus, through the accurate and reliable reference resource data of each subtree, the execution cost corresponding to the candidate plan tree can also be accurately estimated. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] To more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0069] Figure 1 It is a schematic structural diagram of a network architecture for data processing provided by an embodiment of the present application;

[0070] Figure 2 It is a schematic diagram of a scenario for obtaining the execution cost corresponding to a candidate plan tree provided by an embodiment of the present application;

[0071] Figure 3 It is a schematic flowchart of a data processing method provided by an embodiment of the present application;

[0072] Figure 4 It is a schematic structural diagram of a candidate plan tree provided by an embodiment of the present application;

[0073] Figure 5 It is a schematic interface diagram of a diagnostic interface for an operator node provided by an embodiment of the present application;

[0074] Figure 6 It is a schematic flowchart of a first matching process for resource data of N subtrees in a data set provided by an embodiment of the present application;

[0075] Figure 7 It is a schematic framework diagram for collecting resource data consumed by historical subtrees provided by an embodiment of the present application;

[0076] Figure 8 It is a schematic framework diagram for executing an operation statement provided by an embodiment of the present application;

[0077] Figure 9 It is a schematic diagram of a scenario for resource data matching provided by an embodiment of the present application;

[0078] Figure 10 It is a schematic flowchart for determining the reference resource data of N subtrees based on the first matching result provided by an embodiment of the present application;

[0079] Figure 11 It is a schematic structural diagram of the subtree structure of a first subtree provided by an embodiment of the present application;

[0080] Figure 12 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application;

[0081] Figure 13It is a schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0082] Next, the technical solutions in the present application will be clearly and completely described in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0083] All data collected in the present application (such as operation statements, candidate plan trees, subtrees of candidate plan trees, historical subtrees, and resource data consumed by historical subtrees, etc.) are collected with the consent and authorization of the owner of the data (such as users, institutions, or enterprises), and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards in relevant regions.

[0084] SQL: Structured Query Language, a standard programming language for managing and operating relational databases. Among them, SQL statements are instructions for performing specific tasks or queries, and these tasks include operations such as data creation, reading, updating, and deletion. These operations are usually abbreviated as CRUD operations (basic operations in database management).

[0085] Relational algebra: A mathematical theory for describing and operating data in a database. This theory abstracts a series of operation operators for data operations, such as selection, projection, union, intersection, difference, Cartesian product, natural join, division, etc. The combined use of these operation operators can form complex data processing operations, providing a theoretical basis for the design of the SQL language.

[0086] Abstract syntax tree: Abbreviated as AST. The lexical analyzer can decompose the SQL source code into individual lexical units (tokens), and then the syntax analyzer organizes these lexical units into a tree structure according to the syntax rules. This tree structure can be called an abstract syntax tree.

[0087] Logical execution plan: A plan generated during the SQL query optimization process, used to describe the logical operation sequence and execution method of the SQL query without considering specific physical implementation details. This part translates the SQL logic into an execution sequence composed of operation operators in relational algebra theory through the AST. Among them, the logical execution plan can also be a tree structure, and the logical execution plan can be called a logical plan tree.

[0088] Physical execution plan: A plan generated during the SQL query optimization process, which is used to describe the specific physical operation sequence and execution method of an SQL query, taking into account details such as the underlying storage structure, indexes, and execution algorithms. The physical execution plan is the code logic for the actual operation of SQL. Among them, the physical execution plan can also be in a tree structure, and the physical execution plan can be called the physical plan tree.

[0089] Execution operator subtree (which can be abbreviated as subtree): Both the logical execution plan and the physical execution plan can be expressed as directed acyclic graphs respectively. The execution operator subtree is the local logic of the directed acyclic graph and can completely express a local fragment of the data processing logic.

[0090] CBO model: A cost-based optimization model (which can also be called a cost model), which is an optimization method that comprehensively considers various factors and selects the optimal execution plan (such as the optimal physical execution plan) through cost calculation. This CBO model can be used to estimate the execution cost corresponding to the physical plan tree.

[0091] Please refer to Figure 1 , Figure 1 is a schematic structural diagram of a network architecture for data processing provided by an embodiment of the present application. As Figure 1 shown, the network architecture may include a server 200 and a cluster of terminal devices. The cluster of terminal devices may include one or more terminal devices, and the number of terminal devices will not be limited here. As Figure 1 shown, the multiple terminal devices may specifically include terminal device 1, terminal device 2, terminal device 3,..., terminal device n; as Figure 1 shown, terminal device 1, terminal device 2, terminal device 3,..., terminal device n can all be network-connected to the server 200, so that each terminal device can perform data interaction with the server 200 through the network connection.

[0092] As Figure 1 shown, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal device can be: intelligent terminals such as smart phones, tablet computers, laptop computers, desktop computers, smart TVs, in-vehicle terminals, and smart homes. Hereinafter, taking the communication between terminal device 1 and the server 200 as an example, the specific description of the embodiment of the present application will be carried out.

[0093] Here, this application takes the SQL statement to be processed as an example of a statement for data query for relevant description. In actual application scenarios, the SQL statement to be processed can not only be a statement for data query, but also a statement for other operations on data (such as deletion, modification, addition, etc.). Among them, the user can trigger an operation for data query in the terminal device 1, and the terminal device 1 can generate a data query request and send the data query request to the server 200. So that the server 200 can generate a data query statement (which is an SQL statement and can be used as the operation statement described in the following process) according to the data query request, and this data query statement is used to query the data requested by the user.

[0094] The server 200 can first generate multiple candidate plan trees (belonging to physical plan trees) of the data query statement and can estimate the execution costs respectively corresponding to each candidate plan tree. Thus, the data query statement can be finally executed through the candidate plan tree with the smallest estimated corresponding execution cost to query and obtain the data requested by the user. The server 200 can return the queried and obtained data to the terminal device 1, so that the terminal device 1 can display the obtained data on the terminal interface for the user to view.

[0095] The process by which the server 200 estimates the execution costs corresponding to each candidate plan tree can be referred to Figure 2 , Figure 2 is a schematic diagram of a scenario for obtaining the execution cost corresponding to a candidate plan tree provided by an embodiment of this application. As Figure 2 shown, the server 200 in the above Figure 1 can obtain a data set, which can include the resource data actually consumed by M historical subtrees during the historical execution process. M is a positive integer. Here, M is assumed to be equal to 5. These M historical subtrees belong to the subtrees of the historical plan tree that has been actually executed historically. And there is a mapping relationship between the resource data consumed by each historical subtree in the data set and the standard representation information of each historical subtree. The standard representation information of the historical subtree is the representation information obtained after performing standardization representation processing on the historical subtree. The principle of this standardization representation processing can be referred to the relevant description in the corresponding embodiment of the following Figure 6 .

[0096] Here, the M historical subtrees may include historical subtree L1, historical subtree L2, historical subtree L3, historical subtree L4, and historical subtree L5. The standard representation information of historical subtree L1 is standard representation information B1, the standard representation information of historical subtree L2 is standard representation information B2, the standard representation information of historical subtree L3 is standard representation information B3, the standard representation information of historical subtree L4 is standard representation information B4, and the standard representation information of historical subtree L5 is standard representation information B5. Therefore, the dataset may include the mapping relationship between the standard representation information B1 and the resource data Z1 actually consumed by historical subtree L1, the mapping relationship between the standard representation information B2 and the resource data Z2 actually consumed by historical subtree L2, the mapping relationship between the standard representation information B3 and the resource data Z3 actually consumed by historical subtree L3, the mapping relationship between the standard representation information B4 and the resource data Z4 actually consumed by historical subtree L4, and the mapping relationship between the standard representation information B5 and the resource data Z5 actually consumed by historical subtree L5.

[0097] The server 200 may also perform subtree partitioning on the candidate plan tree to obtain N subtrees of the candidate plan tree, where N is a positive integer. Here, it is assumed that N is equal to 4. The N subtrees may include subtree H1, subtree H2, subtree H3, and subtree H4. The server 200 may also perform standard representation processing on the N subtrees respectively to obtain the standard representation information of each of the N subtrees, including the standard representation information S1 of subtree H1, the standard representation information S2 of subtree H2, the standard representation information S3 of subtree H3, and the standard representation information S4 of subtree H4.

[0098] The server 200 may use the standard representation information of the above N subtrees to perform a first matching process on the resource data in the dataset. This first matching process is used to screen the resource data that matches the N subtrees through the similarity between the N subtrees and the above M historical subtrees to obtain a first matching result. Here, the first matching result may include the reference resource data C1 matched for subtree H1, the reference resource data C2 matched for subtree H2, the reference resource data C3 matched for subtree H3, and the reference resource data C4 matched for subtree H4. The similarity between the N subtrees and the M historical subtrees can be reflected by the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees. Therefore, it can be understood that the server 200 can screen the resource data that matches the N subtrees in the dataset through the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees to obtain a first matching result. The specific process of obtaining the first matching result through the first matching process can also refer to the detailed description in the following embodiments.

[0099] The server 200 can determine the reference resource data of each of the N subtrees based on the first matching result obtained above. Thus, the execution cost corresponding to the candidate plan tree can be estimated based on the reference resource data of each of the N subtrees. The specific process can also refer to the relevant descriptions in the following embodiments.

[0100] By using the method provided in this application, the reference resource data of subtrees can be obtained at a fine-grained level of subtrees, and the reference resource data can be obtained by matching the resource data actually consumed by historical subtrees in the dataset during actual execution. Therefore, the reference resource data determined for the subtrees of the candidate plan tree in this way can be used to accurately estimate the execution cost corresponding to the candidate plan tree and reduce the overhead caused by executing SQL statements.

[0101] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a data processing method provided by an embodiment of this application. The execution subject in the embodiment of this application can be a data processing device (which can be simply referred to as a processing device). The processing device can be a computer device or a computer device cluster composed of multiple computer devices. The computer device can be a server, a terminal device, or other devices. This application does not make any restrictions on this. As Figure 3 shown, the method can include:

[0102] Step S101: Obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement.

[0103] Specifically, the operation statement can be an SQL statement, which can be used to query or operate on data in a data table (such as insert, delete, modify, etc.). The processing device can obtain the operation statement, and the operation statement can be any SQL statement that needs to be executed currently. What the specific SQL statement is can be determined according to the actual application scenario.

[0104] The processing device can also obtain the candidate plan tree of the operation statement. The candidate plan tree can be the physical plan tree of the operation statement, and there can be multiple candidate plan trees. Among them, the process by which the processing device obtains the candidate plan tree of the operation statement can include: The processing device can perform syntax parsing processing on the operation statement to generate an abstract syntax tree of the operation statement. For example, the processing device can use a configured lexical analyzer and syntax analyzer to perform syntax parsing processing on the operation statement to generate an abstract syntax tree of the operation statement.

[0105] The processing device can perform related processing on the abstract syntax tree of the operation statement to logically optimize it, so as to generate the logical plan tree of the operation statement. For example, the processing device can, according to the configured processing logic, optimize the nodes in the abstract syntax tree and convert them into nodes in the logical plan tree, thereby obtaining the logical plan tree of the operation statement. Thus, the processing device can perform transformation processing on the logical plan tree of the operation statement to obtain multiple physical plan trees of the operation statement, and these multiple physical plan trees can all be used as the candidate plan trees of the operation statement in this application. For example, the processing device can obtain a configured query optimizer, which can have a series of optimization rules and rewriters. Through these optimization rules and rewriters, physical plan trees can be generated and obtained according to the structure and attributes of the logical plan tree, as well as the characteristics of the underlying storage system and execution engine.

[0106] Since the principles of obtaining the execution cost corresponding to each candidate plan tree are independent and the same, the following process will take the process of obtaining the execution cost corresponding to one candidate plan tree as an example for specific description. That is, the number of candidate plan trees is not emphasized temporarily in the following process and is collectively referred to as the candidate plan tree, as described below.

[0107] The processing device can also obtain N subtrees of the candidate plan tree of the operation statement, where N is a positive integer, and the specific value of N can be determined according to the actual application scenario. The number of subtrees divided for different candidate plan trees can be the same or different. That is, the candidate plan tree can be divided into these N subtrees. Among them, the candidate plan tree can contain at least one operator node (which can be simply referred to as a node). The operator node can be the basic component unit of the candidate plan tree, and an operator node can represent a specific execution operation on the data table. Based on this, a subtree of the candidate plan tree can be composed of at least one operator node in the candidate plan tree, that is, a subtree of the candidate plan tree can contain one or more operator nodes in the candidate plan tree.

[0108] In an exemplary implementation manner, the process for the processing device to obtain N subtrees of the candidate plan tree may include: The processing device can identify the branch nodes (which also belong to the operator nodes of the candidate plan tree) in the candidate plan tree. There can be one or more of these branch nodes, or in special cases, no branch nodes may be identified (indicating that there are no branch nodes in the candidate plan tree), and at least 2 branches can be connected under each branch node.

[0109] Therefore, the processing device can divide each branch under the branch node into subtrees of the candidate plan tree respectively, and this subtree can be called a first-type subtree. A branch under a branch node can be divided into a first-type subtree. Therefore, the number of first-type subtrees can be equal to the number of all branches under each branch node.

[0110] In addition, the processing device can also construct subtrees of the candidate plan tree through each operator node in the candidate plan tree except the above-mentioned first-type subtrees, and this subtree can be called a second-type subtree. An operator node in the candidate plan tree except the first-type subtrees can be used to construct a second-type subtree, that is, the number of second-type subtrees can be equal to the number of operator nodes in the candidate plan tree except the first-type subtrees.

[0111] The processing device can use the above-obtained first-type subtrees and second-type subtrees as the N subtrees obtained by dividing the candidate plan tree, and these N subtrees can include each of the above-mentioned divided first-type subtrees and each of the second-type subtrees.

[0112] Among them, the above-described method of dividing the candidate plan tree into subtrees is only an example description. In actual application requirements, the candidate plan tree can also be divided into each subtree by any other feasible method, so as to obtain the reference resource data of the candidate plan tree at the granularity of subtrees (which can include the reference resource data of each subtree of the candidate plan tree). The specific process can be referred to the subsequent description.

[0113] Step S102: Obtain a data set, where the data set includes the resource data consumed by each of the M historical subtrees during the historical execution process. The historical subtrees are subtrees of the historical plan tree that has been historically executed, and M is a positive integer.

[0114] Specifically, the processing device can obtain a data set, which can include the resource data actually consumed by each of the M historical subtrees during the historical execution process. M is a positive integer, and the specific value of M can be determined according to the actual application scenario. Here, M can be the number of resource data included in the current data set when obtaining the execution cost corresponding to the above candidate plan tree. A historical subtree can have one piece of resource data actually consumed.

[0115] The historical subtree can be a subtree of the historical plan tree that has been historically actually executed. The historical plan tree can be the physical plan tree actually executed for the SQL statement that has been historically executed. The division method of the historical subtrees of the historical plan tree can be the same as the division method of the subtrees of the above candidate plan tree.

[0116] From the dataset of this application, it can be seen that when the physical plan tree of each SQL statement is executed in this application, the resource data actually consumed by each subtree in the physical plan tree can be collected according to the granularity of the subtree and stored in the dataset, so as to accurately and reliably obtain the reference resource data of the subsequent physical plan tree to be executed through this dataset, as described below.

[0117] Step S103, perform a first matching process on the resource data of N subtrees in the dataset. The first matching process is used to screen the matching resource data for the N subtrees by the similarity between the N subtrees and M historical subtrees.

[0118] Specifically, the processing device can perform a matching process on the above N subtrees in the dataset, which can be called the first matching process. The first matching process can be used to screen the matching resource data for the N subtrees from the dataset by the similarity between the N subtrees and N historical subtrees.

[0119] In the process of this first matching process, the more similar the historical subtree in the dataset is to the subtree of the candidate plan tree, the more matching the resource data of the historical subtree is to the subtree of the candidate plan tree. Through this first matching process, a first matching result can be obtained. The first matching result can include the resource data screened from the dataset that matches the subtree in the N subtrees, or can also include the resource data that is not screened from the dataset and matches the subtree in the N subtrees. Among them, for the specific implementation process of this first matching process, reference can be made to the description in the corresponding embodiment below. Figure 6 Corresponding description in the embodiment.

[0120] Step S104, determine the reference resource data of the N subtrees based on the first matching result obtained from the first matching process.

[0121] Specifically, the processing device can determine the respective reference resource data of each subtree in the N subtrees through the first matching result obtained from the above first matching process. A subtree can have one reference resource data, and the reference resource data of a subtree is the resource data that the subtree needs to consume during subsequent actual execution. That is to say, the reference resource data of a subtree can be the resource data that is pre-evaluated for the subtree and that the subtree may consume during actual execution.

[0122] Among them, for the specific process of determining the reference resource data of the N subtrees of the candidate plan tree through the first matching result obtained from the first matching process, reference can be made to the specific description in the corresponding embodiment below. Figure 10 Corresponding specific description in the embodiment.

[0123] Step S105, estimate the execution cost corresponding to the candidate plan tree by using the reference resource data of the N subtrees.

[0124] Specifically, the processing device may use the reference resource data of the above-obtained N subtrees to estimate the execution cost corresponding to the candidate plan tree to which the N subtrees belong.

[0125] Exemplarily, the process of estimating the execution cost corresponding to the candidate plan tree to which it belongs through the reference resource data of the N subtrees may include: The processing device may obtain a cost estimation model, which may be simply referred to as a cost model (which may be a CBO model), and the cost estimation model may be a trained model that can be used to estimate the cost of a physical plan tree. The processing device may input the reference resource data of the N subtrees into the cost estimation model to call the cost estimation model to estimate the execution cost corresponding to the candidate plan tree through the reference resource data of the N subtrees.

[0126] In one implementation, the cost estimation model may estimate the execution cost corresponding to the current operator node through the execution cost corresponding to the previous operator node of the current operator node and the reference resource data of the current operator node, and use this principle to estimate the execution cost corresponding to each operator node of the candidate plan tree from bottom to top. Thus, the execution cost estimated corresponding to the topmost operator node (i.e., the root node) in the candidate plan tree may be used as the execution cost estimated for the entire candidate plan tree.

[0127] In this application, since the reference resource data of the subtrees can be obtained at the granularity of the subtrees, therefore, the cost estimation model can also change the original estimation of the execution cost corresponding to the operator node at the granularity of the operator node to obtaining the execution cost corresponding to the subtree at the granularity of the subtree. This principle can also be to estimate the execution cost corresponding to the current subtree through the execution cost corresponding to the previous subtree of the current subtree and the reference resource data of the current subtree. And the last subtree estimated in this way will also be the root node of the candidate plan tree. Thus, the execution cost estimated for this root node can also be used as the execution cost estimated for the entire candidate plan tree.

[0128] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a candidate plan tree provided by an embodiment of this application. As Figure 4As shown, the candidate plan tree may include operator nodes "Project(id, name, subject, score)" (a projection operator, where id (identifier), name (name), subject (subject), and score (score) can all be data objects called, or fields called in a data table), operator node "SortMergeJoin(a.id = b.id)" (a join operator that can be used for table join operations), operator node "Project(id, name)" (a projection operator, where id and name represent called fields or data objects), operator node "Filter(gender = male and age > 18)" (a filtering operator, where "gender = male and age > 18" is the filtering condition, meaning filtering out females over 18 years old), operator node "TableScan(student)" (a data scanning operator, which can be abbreviated as a scanning operator, where student is the data object called, meaning student), operator node "Project(id, subject, score)" (a projection operator, where id, subject, and score represent called fields or data objects), operator node "Filter(subject = english)" (a filtering operator), and operator node "TableScan(score)" (a data scanning operator).

[0129] Among them, the operator node "Project(id, name, subject, score)" is the root node of the candidate plan tree. The operator node "SortMergeJoin(a.id = b.id)" is a branch node of the candidate plan tree, and it has two branches connected below, including branch 1 (i.e., the left branch) and branch 2 (right branch) here. Both branch 1 and branch 2 contain 3 operator nodes.

[0130] For example, the candidate plan tree can be divided into subtree 1 composed of the operator node "Project (id, name, subject, score)", subtree 2 composed of the operator node "SortMergeJoin (a.id = b.id)", subtree 3 composed of branch 1, and subtree 4 composed of branch 2. Among them, if there are more branch nodes connected after branch 1 or branch 2, the branch 1 or branch 2 can be further divided into finer subtrees through the connected branch node. The division principle can be the same, that is, the branches under the branch node can be divided into subtrees. Since the execution cost of the subtree can be evaluated from bottom to top, subtree 3 and subtree 4 both belong to the upper subtree of subtree 2, and subtree 2 can be the upper subtree of subtree 1. Similarly, according to the operator granularity, the operator node "TableScan (student)" is the upper operator node of the operator node "Filter (gender = male and age > 18)", the operator node "Filter (gender = male and age > 18)" is the upper operator node of the operator node "Project (id, name)", the operator nodes "Project (id, name)" and "Project (id, subject, score)" both belong to the upper operator nodes of the operator node "SortMergeJoin (a.id = b.id)", the operator node "SortMergeJoin (a.id = b.id)" is the upper operator node of the operator node "Project (id, name, subject, score)", the operator node "TableScan (score)" is the upper operator node of the operator node "Filter (subject = english)", and the operator node "Filter (subject = english)" is the upper operator node of the operator node "Project (id, subject, score)".

[0131] Here, since no more subtrees are connected after subtree 3 and subtree 4, there is no previous subtree for both subtree 3 and subtree 4. The cost estimation model can be called to estimate the execution cost corresponding to subtree 3 through the reference resource data of subtree 3, and the cost estimation model can be called to estimate the execution cost corresponding to subtree 4 through the reference resource data of subtree 4. Moreover, the execution cost corresponding to subtree 2 can be estimated through the execution cost corresponding to subtree 3, the execution cost corresponding to subtree 4, and the reference resource data of subtree 2. The execution cost corresponding to subtree 1 can also be estimated through the execution cost corresponding to subtree 2 and the reference resource data of subtree 1. Finally, the execution cost corresponding to this subtree 1 (i.e., the root node) can be used as the execution cost estimated for the candidate plan tree.

[0132] Among them, if in the following Figure 6 and Figure 10 corresponding embodiments, no resource data matching subtree 3 is screened out from the dataset, then in subtree 3, the execution cost corresponding to each operator node in subtree 3 can be estimated in a bottom-up manner at the granularity of the operator node, and the execution cost estimated for the topmost operator node "Project (id, name)" in subtree 3 can be used as the execution cost corresponding to subtree 3 for subsequent estimation of the execution cost corresponding to subtree 2. Similarly for subtree 4, or rather, the same applies to subtrees that have no matching resource data screened out and contain multiple operator nodes.

[0133] Here, if a piece of resource data consumed by a historical subtree is screened out and is matching subtree 3, that is, this historical subtree and this subtree 3 can be regarded as the same, then the resource data consumed by this historical subtree in the dataset can be directly used as the reference resource data of subtree 3.

[0134] In this application, since the reference resource data of the subtree can be obtained at the granularity of the subtree, the execution cost corresponding to the subtree can also be estimated at the granularity of the subtree, thereby improving the estimation efficiency of the execution cost corresponding to the candidate plan tree.

[0135] The processing device can obtain the execution cost corresponding to each candidate plan tree of the operation statement according to the principle described above. A candidate plan tree can have a corresponding execution cost. As the name implies, the execution cost corresponding to a candidate plan tree is the cost required to execute the candidate plan tree. The smaller the execution cost, the less overhead (such as computer resource overhead or time overhead, etc.) required to execute the operation statement. Therefore, the processing device can select the candidate plan tree with the smallest corresponding execution cost from multiple candidate plan trees of the operation statement as the target plan tree of the operation statement. This target plan tree is the physical plan tree ultimately used to execute the operation statement, and this target plan tree can be the optimal plan tree of the operation statement. The processing device can use this target plan tree to execute the operation statement. Executing this target plan tree is ultimately the execution of the operation statement. For example, by executing this target plan tree, corresponding operations (such as query, insert, delete, modify, etc.) on the data in the data table can be realized for the operation statement.

[0136] The resource data consumed by each historical subtree in the dataset can respectively have a mapping relationship with the standard representation information of each historical subtree. The standard representation information of a historical subtree is the information obtained after performing standard representation processing on the historical subtree. Among them, for the specific process of obtaining the standard representation information of the historical subtree, reference can be made to the relevant description of obtaining the standard representation information of N subtrees in the corresponding embodiment described below. Figure 6 Corresponding to the relevant description of obtaining the standard representation information of N subtrees in the embodiment.

[0137] Any subtree of the above target plan tree can be called a second subtree. After executing the target plan tree, if the dataset does not contain the standard representation information of this second subtree at this time (this standard representation information actually refers to the standard representation information of the historical subtree similar to the second subtree that has been historically executed), then this second subtree can be used as a new historical subtree, and the resource data actually consumed by this second subtree during the actual execution process can be newly added and added to the above dataset, as described below.

[0138] Therefore, after executing the target plan tree, if the dataset does not contain the standard representation information of the second subtree, it indicates that the dataset does not yet contain the resource data actually consumed by the historical subtree similar to this second subtree that has been historically executed. Then, after executing the second subtree, the processing device can obtain the resource data actually consumed by this second subtree during the execution process. The processing device can construct a mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree, and can newly add and add the constructed mapping relationship to the dataset, so that the dataset can contain the resource data consumed by this second subtree during the actual execution process.

[0139] If the dataset contains the standard representation information of the second subtree (which is actually the standard representation information of a historical subtree similar to the second subtree that has been historically executed), indicating that the dataset already contains the resource data actually consumed by the historical subtree similar to the second subtree that has been historically executed, then after executing the second subtree, the processing device does not need to repeatedly add the resource data actually consumed by the second subtree to the dataset.

[0140] In one implementation, the method for determining whether the dataset contains the standard representation information of the second subtree may include:

[0141] The dataset may contain the M standard representation information of the above M historical subtrees. The processing device may obtain the information similarity between the standard representation information of the second subtree and each of the M standard representation information. Among them, the principle of obtaining the information similarity between the standard representation information of the second subtree and each of the M standard representation information is the same as that in the following Figure 6 corresponding embodiment for obtaining the information similarity between the standard representation information of the first subtree and the M standard representation information. Specifically, reference may be made to the relevant description in the following Figure 6 corresponding embodiment. Among them, the second subtree and the first subtree may be the same or different.

[0142] The processing device may obtain the standard representation information with the largest information similarity between the second subtree and each of the above M standard representation information, and may refer to the standard representation information with the largest information similarity between the second subtree as the reference representation information. That is, the reference representation information is the one with the largest information similarity between the second subtree and each of the M standard representation information.

[0143] If the information similarity between the standard representation information of the second subtree and the reference representation information is greater than or equal to the set similarity threshold, it can be considered that the standard representation information of the second subtree is the same as the reference representation information, that is, it can be considered that the second subtree is the same as the historical subtree to which the reference representation information belongs. At this time, it can be considered that the dataset contains the standard representation information of the second subtree. The similarity threshold may be the set minimum information similarity for evaluating that the standard representation information is the same, and the specific value of the similarity threshold may be determined according to the actual application scenario.

[0144] If the information similarity between the standard representation information of the second subtree and the reference representation information is less than the above set similarity threshold, it can be considered that the standard representation information of the second subtree is different from the reference representation information, that is, it can be considered that the second subtree is different from the historical subtree to which the reference representation information belongs. At this time, it can be considered that the dataset does not contain the standard representation information of the second subtree.

[0145] Exemplarily, in one implementation, the process by which the processing device obtains the resource data actually consumed during the execution of the second subtree may include: after executing the second subtree, the processing device may obtain the input / output data (i.e., IO data) of the last operator node in the second subtree, and may use the input / output data of the last operator node as the input / output data of the second subtree. The input / output data may include the amount of input data and the amount of output data of the last operator node, or may further include the number of logical interactions of the operator node (such as the number of times of logically performing data interactions). The last operator node may be the bottommost operator node in the second subtree. In one implementation, the input / output data of the subtree may include the input / output data of the subtree with respect to the disk and the input / output data of the subtree with respect to the network (Net). Among them, the input / output data of the subtree with respect to the disk may represent the number of logical interactions of the subtree on the disk, and the input / output data of the subtree with respect to the network may represent the number of logical interactions of the subtree on the network and the data transfer amount (such as may include the amount of input data and the amount of output data).

[0146] The processing device may also obtain the memory data consumed by each operator node in the second subtree during the running process. The memory data consumed by an operator node may be the memory data occupied by the operator node during the running process. The memory data may be the memory size occupied by the operator node during the running process. For example, the unit of the memory data may be GB (gigabyte), etc. The processing device may use the memory data consumed the most by the operator nodes in the second subtree as the memory data consumed by the second subtree. Among them, the memory data consumed the most by the operator nodes in the second subtree may be the memory data consumed by the operator node that consumes the most memory data in the second subtree, that is, the memory data is the peak value of the memory data consumed by each operator node in the second subtree.

[0147] In addition, the processing device may obtain the processor data (which may be referred to as CPU data) consumed by each operator node in the second subtree. The processor data consumed by an operator node may be the duration occupied by the operator node on the CPU (processor) during the running process. The processing device may use the processor data consumed the most by the operator nodes in the second subtree as the processor data consumed by the second subtree. Among them, the processor data consumed the most by the operator nodes in the second subtree may be the processor data consumed by the operator node that consumes the most (i.e., the longest) processor data in the second subtree. The processor data may be the peak value of the processor data consumed by each operator node in the second subtree. [[ID=​10]]

[0148] The processing device may use the input / output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree obtained above as the resource data consumed by the second subtree. That is, the resource data of the second subtree may include the input / output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree.

[0149] Furthermore, in this application, the operator nodes included in the operation statement can also be diagnosed. For example, the operator nodes can be diagnosed based on the input data volume, input record count, output data volume, and output record count of the operator nodes to determine whether there are unreasonable settings or abnormalities in the operator nodes. For example, if the difference between the input data volume and the output data volume of the operator node is too large (e.g., greater than a certain data volume threshold), it can be considered that there is an unreasonable place in the operator node, and diagnostic information for the operator node can be generated. The diagnostic information can also include the direction and suggestions for optimizing the operator node. Please refer to Figure 5 , Figure 5 which is a schematic diagram of the interface of a diagnostic interface for operator nodes provided by an embodiment of this application. As Figure 5 shown, the operator name "SortMergeJoin", input data volume "57245 bytes (Byte)", input record count "30995894 rows", output data volume "57245 bytes (Byte)", output record count "5 rows", and diagnostic information of the operator node can be included in the diagnostic interface. Among them, the diagnostic information can include "Big table preposition: In this Join stage (table join stage), the input of the right table is larger, and it is suitable to be preposed as the Base table. Input of the left table: 5 rows, input of the right table: 30995889 rows." and "MapJoin diagnosis: The left table is suitable for using MapJoin (an optimization technique).".

[0150] Among them, the specific logic for diagnosing operator nodes (which can be called diagnostic logic or detection logic) can be flexibly set according to actual application requirements.

[0151] In this application, through the similarity matching mechanism between subtrees, matching resource data for subtrees is screened in the dataset, which can provide a decision for root cause analysis during cost prediction of the physical plan tree, thereby achieving accurate and fast cost estimation of the physical plan tree. In addition, in this application, by using the physical plan tree generated from the logical plan tree for subtree matching, the information connection between the logical plan tree and the physical plan tree can be realized. Matching the subtrees of the physical plan tree can be understood as realizing the matching of the logical plan tree.

[0152] The present application can obtain a candidate plan tree for an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; and it can obtain a data set, which includes resource data consumed by M historical subtrees during their respective historical executions. The historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer; moreover, it can perform a first matching process on the resource data for the N subtrees in the data set. The first matching process is used to screen for matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees; thus, it can determine the reference resource data for the N subtrees based on the first matching result obtained from the first matching process; and use the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree. It can be seen from this that the method proposed in the present application can obtain the reference resource data of each subtree in the candidate plan tree at the granularity of the subtree, and moreover, the reference resource data can be obtained by matching from the resource data actually consumed by the historical subtrees during the actual historical execution process. Therefore, it also ensures the accuracy and reliability of the reference resource data obtained for each subtree. Thus, through the accurate and reliable reference resource data of each subtree, the execution cost corresponding to the candidate plan tree can also be accurately estimated.

[0153] Please refer to Figure 6 , Figure 6 FIG. is a schematic flowchart of a first matching process for resource data of N subtrees in a data set provided by an embodiment of the present application. In the embodiment of the present application, the process of performing a first matching process on the resource data of N subtrees in the data set to obtain a first matching result is specifically described. As Figure 6 shown, the process may include:

[0154] Step S201: Perform a standard representation process on each of the N subtrees respectively to generate standard representation information for each subtree.

[0155] Specifically, the processing device may perform a standard representation process (also referred to as a normalization process) on each of the above-mentioned N subtrees of the candidate plan tree to generate standard representation information for each subtree. A subtree may have one piece of standard representation information.

[0156] Any one of the N subtrees may be referred to as a first subtree. Hereinafter, taking the first subtree as an example to perform a standard representation process to generate the standard representation information of the first subtree for specific illustration. It should be noted that the principles of performing a standard representation process on each of the N subtrees to generate the respective standard representation information of each subtree are independent and the same.

[0157] The processing device can perform data parsing processing on the first subtree from K data domains to obtain the representation data of the first subtree under these K data domains. K is a positive integer, and these K data domains are preset domains for standardizing the representation of subtrees. Among them, if no corresponding representation information is parsed out for the first subtree under a certain data domain among these K data domains, the representation information of the first subtree under this data domain can be represented as null.

[0158] The processing device can perform representation processing on the representation data of the first subtree under these K data domains according to the set standard format to generate the standard representation information of the first subtree. The standard representation information of the first subtree can include the data obtained by performing representation processing on the representation data of the first subtree under these K data domains according to this standard format. The standard representation information of the first subtree can be understood as being obtained by representing the first subtree according to a unified standard.

[0159] The above-mentioned K data domains can include at least one of the following: the operator domain corresponding to the operator called by the subtree, the object domain corresponding to the data object in the table data called by the subtree, and the condition domain corresponding to the condition included in the operator called by the subtree. That is, the standardization representation processing of the subtree can include abstraction on these K data domains.

[0160] Among them, the operator called by the subtree can be the operator included in the subtree, such as the TableScan operator (table scan operator), Filter operator (filter operator), Project operator (projection operator), etc. All the operators in the operator domain can belong to the elements in this operator domain.

[0161] The table data called by the subtree in the data table can be the data called by the subtree in the data table. This table data can be the entire data table or one or more partitions in the data table. This table data can contain one or more data objects (i.e., Object). A data object can be an entity in the data table. For example, a data object can be an employee, a commodity, or a customer, etc. A data object usually corresponds to a row of data in the data table, and the object attributes of this data object can be recorded in this row of data. All the data objects in the object domain can belong to the elements in this object domain.

[0162] The condition included in the operator called by the subtree can be the condition set within this operator. For example, for the Filter operator: Filter (gender ='male' and age > 18), the condition included in this Filter operator is "gender ='male' and age > 18", and this condition means gender is male and age is greater than 18. All the conditions in the condition domain can belong to the elements in this condition domain.

[0163] Therefore, the representation data of the first subtree in the operator domain can include each operator called in the first subtree. Each of these operators can be referred to as each element of the representation data of the first subtree in the operator domain, and each of these elements belongs to the elements in this operator domain. The representation data of the first subtree in the object domain can include each data object in the table data called by the first subtree. Similarly, each of these data objects can be referred to as each element of the representation data of the first subtree in the object domain, and each of these elements belongs to the elements in this object domain. The representation information of the first subtree in the condition domain can include each condition included in the operator called in the first subtree. Each of these conditions can also be referred to as each element of the representation data of the first subtree in the condition domain, and each of these elements belongs to the elements in this condition domain.

[0164] The above standard format can include the arrangement order and writing format (such as unified capitalization, text of the same font type, etc.) respectively and uniformly set for each element in the above data domain (here it can refer to all elements in each data domain). That is, there can be a preset and unified arrangement order and unified writing format among the elements in the same data domain.

[0165] Therefore, the above representation processing of the representation data of the first subtree in K data domains according to the set standard format can mean representing the representation data of the first subtree in each data domain according to the unified arrangement order and unified writing format of the elements in each data domain, and there can also be a fixed representation order among the representation data in each data domain. For example, according to the order of the operator domain, object domain, and condition domain, the elements of the first subtree in each data domain can be represented in turn according to their respective unified arrangement order and writing format, and the standard representation information of the first subtree can be generated and obtained.

[0166] The processing device can obtain the standard representation information of each of the above N subtrees according to the principle of obtaining the standard representation information of the first subtree. Among them, the above standardization representation processing can also be called normalization processing, which is used to represent each subtree in a unified format. The standard representation information obtained by performing standardization representation processing on the subtree can be called the signature of the subtree, which is used to identify the subtree.

[0167] Moreover, in one embodiment, since the resource data in the dataset may be collected by multiple SQL computing engines, and the same operator may have different forms in different SQL computing engines, before performing the standardized representation processing on the subtree, the present application may first perform the standardized representation processing (i.e., normalization processing) on each operator node in the subtree. For example, the physical execution operators in different SQL computing engines (i.e., the operators in the physical plan tree) can be uniformly normalized to relational algebra operators (which can be called standard format operators). That is, when normalizing, the "relational algebra" theory can be used as the standard. For example, operator nodes such as BroadCastJoin (broadcast join operator), SortMergeJoin (sort merge join), and HashJoin (hash join operator) can be uniformly normalized to Join (join operator), and the Scan operators (a type of scan operator) of various SQL computing engines can be uniformly classified into projection operators, etc.

[0168] In other words, the above N subtrees of the present application can be obtained by uniformly representing the operator nodes in the N initial subtrees into which the candidate plan tree is divided as relational algebra operators. The present application may first uniformly represent the operator nodes in the subtrees divided from the physical plan tree as relational algebra operators, and then perform the standardized representation processing to generate the standard representation information of the subtree. This can ensure the accuracy of the standardized representation processing of the subtree, obtain more accurate standard representation information of the subtree, and thus ensure the accuracy of matching the resource data in the dataset.

[0169] In the present application, by representing each subtree as its respective standard representation information, the problem that the same subtree is misjudged as different subtrees due to different representation methods can be eliminated, enabling the processing device to more accurately evaluate the similarity between subtrees subsequently.

[0170] Please refer to Figure 7 , Figure 7 which is a schematic diagram of a framework for collecting resource data consumed by historical subtrees provided by an embodiment of the present application. As Figure 7As shown, in this application, a collector (which can also be referred to as a data collector and can belong to an SDK, software package) can be used to collect the resource data consumed by historical subtrees. Here, the resource database is the above-mentioned data set and can include the resource data collected for historical subtrees. In this application, the collector can, in a streaming manner, through a real-time operation entry, parse and obtain the resource data actually consumed by each subtree in the physical plan tree currently (with very little latency and can be regarded as real-time) executed by the SQL computing engine (there can be one or more SQL computing engines, and multiple SQL computing engines can be distributed). The executed physical plan tree can be used as the above-mentioned historical plan tree, and the subtrees of the historical plan tree are historical subtrees. The collector can store the mapping relationship between the resource data actually consumed by each subtree in the collected physical plan tree and the standard representation information of each subtree in the resource database.

[0171] In addition, the collector can also, in a batch manner, through an offline operation entry, parse and obtain the resource data actually consumed by each subtree in the physical plan tree executed by the SQL computing engine previously (such as a long time ago from the current time, such as before a specified time interval). The executed physical plan tree can also be used as the above-mentioned historical plan tree, and the subtrees of the historical plan tree are historical subtrees. The collector can store the mapping relationship between the resource data actually consumed by each subtree in the collected physical plan tree and the standard representation information of each subtree in the resource database.

[0172] In one implementation, the collector can obtain Metrics data (data for measuring or analyzing data) during the running of SQL statements in a real-time or offline manner. The Metrics data can include the resource data consumed by each operator node in the physical plan tree of the SQL statement during execution. The collector can, through the resource data consumed by each operator node during execution, obtain the resource data consumed by each subtree of the physical plan tree. By way of example, the SQL computing engine can include Spark (a fast and general big data processing engine), Presto (a distributed SQL query engine), Starrocks (a distributed SQL database for online analytical processing), etc.

[0173] Through the above process, the present application can parallelly and synchronously collect the resource data of each subtree of the physical plan tree executed by each SQL computing engine through two collection methods: real-time collection and offline collection. Among them, real-time collection can be used as the main collection method, and offline collection can be used as the auxiliary collection method. The offline collection method can solve the problem that a large amount of resource data consumed by historical subtrees cannot be collected in time during real-time collection. By parallelly adopting the two collection methods of real-time collection and offline collection, the present application can efficiently and comprehensively collect the resource data actually consumed by each subtree of the physically executed plan tree in the past, which is used for estimating the corresponding execution cost of other physical plan trees in the future.

[0174] Step S202: Based on the standard representation information of N subtrees, perform a first matching process on the resource data of the N subtrees in the dataset.

[0175] Specifically, the processing device can perform a first matching process on the resource data of the N subtrees in the dataset through the standard representation information obtained for the N subtrees, so as to obtain a first matching result, as described below.

[0176] There may be a mapping relationship between the resource data consumed by each historical subtree in the above dataset and the standard representation information of each historical subtree itself. That is, the mapping relationship between the above M historical subtrees and their respective standard representation information can be stored in the dataset. The dataset may not directly contain historical subtrees, but can refer to historical subtrees through the standard representation information of historical subtrees. The dataset may contain the standard representation information of each historical subtree.

[0177] Among them, since the standard representation information of a subtree is obtained by standardizing the subtree without modifying the actual operation logic contained in the subtree, the standard representation information of the subtree can also well refer to and represent the subtree. Therefore, the similarity between the standard representation information of the above N subtrees and the standard representation information of the M historical subtrees can be used to reflect (i.e., embody) the similarity between the N subtrees and the M historical subtrees. In short, the similarity between the standard representation information of subtrees can be used to reflect the similarity between subtrees.

[0178] The above first matching process may include a matching process of the resource data of each subtree in the above N subtrees in the dataset, and the principles of the resource data matching process for each subtree are independent and the same. Therefore, any one of the N subtrees can be the first subtree. The following will take the process of matching the resource data of the first subtree in the dataset as an example for specific description.

[0179] The dataset may include the standard representation information of the above-mentioned M historical subtrees, and one historical subtree has one standard representation information. The processing device can obtain the similarity (which can be called information similarity) between the standard representation information of the first subtree and each of the M pieces of standard representation information. There can be an information similarity between the standard representation information of the first subtree and one piece of the M pieces of standard representation information.

[0180] In one implementation, the manner in which the processing device obtains the information similarity between the standard representation information of the first subtree and each of the M pieces of standard representation information may include: The processing device can use the Jaccard algorithm (an algorithm for evaluating the similarity between two sets) to obtain the information similarity between the standard representation information of the first subtree and the M pieces of standard representation information. In this process, a piece of standard representation information can be represented as a corresponding set, and a character in the standard representation information can be used as an element in the corresponding set, that is, the set corresponding to a piece of standard representation information is composed of each character in the standard representation information. Therefore, the processing device can represent the standard representation information of the first subtree and the M pieces of standard representation information as corresponding sets respectively. Thus, the processing device can calculate the set similarity between the set corresponding to the standard representation information of the first subtree and the sets corresponding to the M pieces of standard representation information through the Jaccard algorithm, and use this set similarity as the information similarity between the standard representation information of the first subtree and each of the M pieces of standard representation information.

[0181] In another implementation, the manner in which the processing device obtains the information similarity between the standard representation information of the first subtree and each of the M pieces of standard representation information may further include: The processing device can convert the standard representation information of the first subtree and the M pieces of standard representation information into corresponding feature vectors respectively. A piece of standard representation information can be converted into a corresponding feature vector, and this feature vector can be generated by performing feature embedding processing on the standard representation information, and this feature vector can be used to characterize the features of the standard representation information. Thus, the processing device can calculate the vector similarity (which can be cosine similarity) between the feature vector corresponding to the standard representation information of the first subtree and the feature vectors corresponding to the M pieces of standard representation information, and use this vector similarity as the information similarity between the standard representation information of the first subtree and each of the M pieces of standard representation information.

[0182] Exemplarily, the manner in which the processing device converts the standard representation information into the corresponding feature vector may include: The processing device may obtain a feature embedding model. Since the standard representation information may be text information, this feature embedding model may be a text feature embedding model, and this text feature embedding model may be a trained model that can be used to generate the feature vector of the text. Therefore, the processing device may call this text feature embedding model to perform feature embedding processing (which may also be referred to as feature extraction processing) on the standard representation information, and thus generate the feature vector corresponding to the standard representation information.

[0183] The processing device may obtain the standard representation information with the largest information similarity between the above-mentioned M standard representation information and the standard representation information of the first subtree, and call the standard representation information with the largest information similarity between the above-mentioned M standard representation information and the standard representation information of the first subtree the to-be-determined representation information. That is, the to-be-determined representation information is the one with the largest information similarity between the M standard representation information and the standard representation information of the first subtree.

[0184] If the information similarity between the standard representation information of the first subtree and the to-be-determined representation information is greater than or equal to the set similarity threshold, it can be considered that the standard representation information of the first subtree is the same as the to-be-determined representation information, that is, it can be considered that the first subtree is the same as the historical subtree to which the to-be-determined representation information belongs. At this time, the processing device may use the resource data in the dataset that has a mapping relationship with the to-be-determined representation information as the resource data filtered out and matching the first subtree. The first matching result may include the resource data filtered out and matching the first subtree. The similarity threshold may be the set minimum information similarity used to evaluate whether the standard representation information is the same, and the specific value of the similarity threshold may be determined according to the actual application scenario.

[0185] If the information similarity between the standard representation information of the first subtree and the to-be-determined representation information is less than the above-mentioned set similarity threshold, it can be considered that the standard representation information of the first subtree is different from the to-be-determined representation information, that is, it can be considered that the first subtree is different from the historical subtree to which the to-be-determined representation information belongs. At this time, the processing device may determine that no resource data matching the first subtree is filtered out in the dataset. The first matching result may include that no resource data matching the first subtree is filtered out in the dataset.

[0186] The processing device may perform resource data matching processing on the above-mentioned N subtrees respectively according to the same principle as the above resource data matching processing for the first subtree, so as to obtain the matching result for each subtree. The matching result of a subtree may be the resource data filtered out and matching the subtree, or the resource data that is not filtered out and matching the subtree. The first matching result may include the respective matching results of each of the N subtrees.

[0187] Please refer to Figure 8 , Figure 8 which is a schematic diagram of a framework for executing operation statements provided by an embodiment of the present application. As Figure 8 shown, the data collector can collect the sub-tree operation data, which is the resource data actually consumed by the historical sub-tree during the historical execution process and can be included in the above dataset. The data collector can also obtain the metadata and statistical information of the data table. The metadata of the data table can define each partition of the data table, and the statistical information of the data table can include the data volume of the data table and the data volume of each partition in the data table.

[0188] In the present application, when a data table has not been run yet, the operator associated with the partition in the data table (which can be called a derived operator) can also be derived through the metadata and statistical information of the data table. For example, the processing device can obtain one or more operator types that can operate on the branches of the data table (such as the operator type of scanning, the operator type of projection, etc.). Thus, by combining the one or more operator types with the partition, the derived operator associated with the partition can be generated. For example, the derived operator can be a scanning operator or a projection operator for the associated partition. The processing device can also obtain the resource data consumed by the derived operator. Since the partition associated with the derived operator has not been run yet, the resource data consumed by the derived operator at this time can include the input and output data of the derived operator. The input data volume in the input and output data can be the data volume of the data table, and the output data volume in the input and output data can be the data volume of the partition associated with the derived operator. Thus, the present application can generate the standard representation information of the derived operator and add the mapping relationship between the standard representation information of the derived operator and the resource data consumed by the derived operator to the above dataset. Similar to the resource data consumed by the historical sub-tree, it can also be used to screen the resource data that matches the subsequent sub-tree.

[0189] Moreover, if the derived operator is actually executed subsequently, the resource data consumed by the derived operator during the actual execution process can also be supplemented to the data table. Since the input and output data of the derived operator are already included in the data table, the resource data supplemented here can include the memory data and processor data actually consumed during the execution of the derived operator.

[0190] Please refer to again Figure 8The SQL computing engine in it may include an SQL engine connector (for connecting to the computing engine), a parser (for parsing SQL statements), an optimizer (for optimizing SQL statements, such as optimizing Join (a table join) into Mapjoin (an optimization technique)), an executor (for executing SQL statements), and an algorithm service connector. The algorithm service connector may be newly added in this application and is used to connect to an algorithm service (which may be the algorithm provided in this application for screening resource data matching a subtree, and the principle may be as described in the above embodiments of this application) to screen resource data matching the subtree of the physical plan tree of the SQL statement currently to be executed from the data set through the algorithm service. The processing device may input an SQL statement (such as the above operation statement) into the SQL computing engine and may process the input SQL statement step by step through the parser, optimizer, and executor in the SQL computing engine to obtain a final processing result (such as the processing result may include data queried from a data table). The processing device may store the processing result (corresponding to Figure 8 "data storage" in

[0191] Please refer to Figure 9 again, Figure 9 which is a schematic diagram of a scenario for resource data matching provided by an embodiment of this application. As Figure 9 shown, the data collector can read the resource data actually consumed during the execution of the executed historical plan tree from the SQL computing engine. Through this resource data, the resource data consumed by each operator node in the historical plan tree can be obtained. Thus, the resource data consumed by each historical subtree of the historical plan tree can be obtained through the resource data consumed by each operator node. The data collector can map the resource data consumed by each historical subtree to the data set through the standard representation information of each historical subtree.

[0192] Subsequently, the data collector can obtain a new subtree signature (i.e., the standard representation information of the subtree. The new subtree signature may be, for example, the standard representation information of the above N subtrees). The data collector can perform subtree matching and recall (such as the above first matching process) in the data set through the new subtree signature using a similarity algorithm (such as a similarity algorithm between standard representation information) to screen resource data matching the N subtrees for estimating the execution cost corresponding to the candidate plan tree.

[0193] The method provided by this application converts the subtrees of the physical plan tree of SQL statements into standard representation information for representation, establishes the correlation between SQL statements, and pulls through the subtree data (such as the resource data consumed by subtrees) between different SQL statements. Through this subtree data, the cost optimization and estimation of SQL in the CBO model are supported, the similarity between subtrees is accurately evaluated, and thus the resource data matching the subtrees can be accurately screened out from the dataset. Moreover, by adopting the above method of this application, in the CBO optimization of the internal SuperSQL (a SQL computing platform), the optimization of the Mapjoin operator is supported. In the test verification of Presto (an open-source distributed SQL query engine) and TPC-DS (a decision support system benchmark), 99 SQL statements in the standard test set (such as the subtrees of the physical plan trees of these 99 SQL statements have all screened out matching resource data) have achieved excellent test results.

[0194] Please refer to Figure 10 , Figure 10 FIG. is a schematic flowchart of a process for determining the reference resource data of N subtrees through a first matching result provided by an embodiment of this application. Since the principle of obtaining the reference resource data of each subtree in the N subtrees through the first matching result is the same, therefore, the following still takes the process of obtaining the reference resource data of the first subtree as an example for specific description. As Figure 10 shown, this process may include:

[0195] Step S301, if the first matching result includes the resource data screened out that matches the first subtree, then use the screened resource data that matches the first subtree as the reference resource data of the first subtree.

[0196] Specifically, if the first matching result includes the resource data screened out that matches the first subtree, the processing device can directly use the screened resource data that matches the first subtree as the reference resource data of the first subtree.

[0197] Step S302, if the first matching result includes no resource data screened out that matches the first subtree, then determine the reference resource data of the first subtree based on the number of operator nodes included in the first subtree.

[0198] Specifically, the above N subtrees can all be composed of one or more operator nodes of the candidate plan tree. Therefore, if the first matching result includes no resource data screened out that matches the first subtree, the processing device can determine the reference resource data of the first subtree through the number of operator nodes included in the first subtree, as described below.

[0199] When the first subtree contains an operator node of the candidate plan tree (i.e., when the first subtree contains an operator node), the processing device can obtain the resource data consumed by the table data called by the first subtree, and can use the resource data consumed by the table data called by the first subtree as the reference resource data of the first subtree. Among them, the table data called by the first subtree can be the entire data table or at least one partition in the data table, and the resource data consumed by the data table or each partition in the data table can be set in advance for the data table and each partition in the data table respectively. For example, the resource data consumed by the data table / partition in the data table can be obtained by analyzing and statistics through data attributes such as the data scale, data distribution, and field type of the data contained in the data table / partition in the data table. The specific business logic of the analysis and statistics can be configured by developers according to the actual application scenario.

[0200] And when the first subtree contains multiple operator nodes of the candidate plan tree (i.e., when the first subtree contains multiple operator nodes), the processing device can perform a second matching process of resource data on the first subtree in the dataset, and the matching result obtained from the second matching process can be called the second matching result. The processing device can determine the reference resource data of each operator node in the first subtree through the second matching result obtained from the second matching process. In this case, the reference resource data of the first subtree can include the reference resource data of each operator node in the first subtree. The second matching process is used to screen the matching resource data for each operator node in the first subtree in the dataset through the similarity between each operator node in the first subtree and M historical subtrees.

[0201] Among them, the principle of the second matching process is actually the same as that of the above-mentioned first matching process, that is, the principle of screening the resource data matching each operator node in the first subtree in the dataset through the similarity between each operator node in the first subtree and the M historical subtrees is the same as the principle of screening the resource data matching the N subtrees in the dataset through the similarity between the above-mentioned N subtrees and the M historical subtrees. The only difference is that in the process of the second matching process compared with the first matching process, the N subtrees are replaced by each operator node in the first subtree. And each operator node in the first subtree can also be regarded as a corresponding subtree respectively. However, here a subtree can only contain one operator node, that is, an operator node in the first subtree can be regarded as a subtree that only contains this operator node. Therefore, in the same principle as screening the resource data matching the N subtrees in the dataset through the similarity between the N subtrees and the M historical subtrees, through the similarity between multiple subtrees of the first subtree (that is, multiple operator nodes included in the first subtree) and the M historical subtrees, screen the resource data matching these multiple subtrees in the dataset to obtain the second matching result.

[0202] The second matching result can also include the matching results of each operator node in the first subtree. The matching result of an operator node can be the resource data screened out that matches the operator node, or the resource data that has not been screened out to match the operator node.

[0203] Similarly, any operator node in the first subtree can be called a target operator node. Since the principle of determining the reference resource data of each operator node in the first subtree is the same, therefore, the following still takes the process of determining the reference resource data of the target operator node as an example for specific description, as described below.

[0204] If the above second matching result includes the resource data screened out that matches the target operator node, then the resource data screened out that matches the target operator node can be directly used as the reference resource data of the target operator node. In this case, it can be considered that the historical subtree to which the resource data screened out that matches the target operator node belongs is the same as the target operator node.

[0205] ​

[0206] Among them, when the first subtree contains multiple operator nodes of the candidate plan tree and the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, the cost estimation model can also follow the original principle. In this first subtree, through the execution cost corresponding to the previous operator node of the current operator node and the reference resource data of the current operator node, estimate the execution cost corresponding to the current operator node, so as to use the execution cost estimated for the topmost operator node in the first subtree as the execution cost of the first subtree, for subsequent estimation of the execution cost corresponding to the subtree (such as the next subtree of the first subtree).

[0207] Please refer to Figure 11 , Figure 11 FIG. is a schematic structural diagram of a subtree structure of a first subtree provided by an embodiment of the present application. Assume that the first subtree includes the operator node 1 "Project (id, name)" (a projection operator), the operator node 2 "Filter (gender=male and age=18)" (a filtering operator), and the operator node 3 "TableScan (student)" (a scanning operator) here. Since no matching resource data is filtered out for the first subtree, the execution cost can be estimated at the granularity of the operator node within the first subtree. For example, the processing device can pre-estimate the execution cost corresponding to the operator node 3 through the reference resource data of the operator node 3, and can pre-estimate the execution cost corresponding to the operator node 2 through the execution cost corresponding to the operator node 3 and the reference resource data of the operator node 2, and can pre-estimate the execution cost corresponding to the operator node 1 through the execution cost corresponding to the operator node 2 and the reference resource data of the operator node 1. Thus, the execution cost corresponding to the operator node 1 can be used as the execution cost estimated for the first subtree.

[0208] Through the above method of the present application, the resource data of the subtree can be accurately collected and obtained at the granularity of the subtree (including the granularity of the operator node, which belongs to a small granularity), rather than obtaining the resource data at the granularity of the data table or the partition of the data table (which belongs to a large granularity). Thus, the execution cost corresponding to the subtree can be accurately estimated at the granularity of the subtree, and finally the execution cost corresponding to the entire candidate plan tree can be estimated, improving the accuracy of estimating the execution cost corresponding to the candidate plan tree. And by obtaining the reference resource data of the subtree through the actually consumed resource data collected by the historical subtree, the convenience and reliability of obtaining the reference resource data of the subtree are also ensured, greatly improving the performance of executing the SQL statement.

[0209] Please refer to Figure 12 , Figure 12This is a schematic structural diagram of a data processing device provided by an embodiment of the present application. As Figure 12 shown, the data processing device 120 may include: a first acquisition module 1201, a second acquisition module 1202, a matching module 1203, a determination module 1204, and an estimation module 1205.

[0210] The first acquisition module 1201 is configured to acquire a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement;

[0211] The second acquisition module 1202 is configured to acquire a data set, where the data set includes resource data consumed by each of M historical subtrees during historical execution, and the historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer;

[0212] The matching module 1203 is configured to perform a first matching process on the resource data of the N subtrees in the data set, and the first matching process is used to screen out matching resource data for the N subtrees by the similarity between the N subtrees and the M historical subtrees;

[0213] The determination module 1204 is configured to determine the reference resource data of the N subtrees based on the first matching result obtained from the first matching process;

[0214] The estimation module 1205 is configured to use the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree.

[0215] In one implementation, the manner in which the matching module 1203 performs the first matching process on the resource data of the N subtrees in the data set includes:

[0216] Performing a standardization representation process on each of the N subtrees respectively to generate standard representation information for each subtree;

[0217] Based on the standard representation information of the N subtrees, performing the first matching process on the resource data of the N subtrees in the data set;

[0218] Among them, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree respectively; the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees is used to reflect the similarity between the N subtrees and the M historical subtrees.

[0219] In one implementation, any one of the N subtrees is a first subtree, and the data set includes M standard representation information of M historical subtrees;

[0220] The matching module 1203 performs the first matching process of resource data for N sub-trees in the data set based on the standard representation information of the N sub-trees, including:

[0221] Obtain the information similarity between the standard representation information of the first sub-tree and M pieces of standard representation information respectively;

[0222] Determine the standard representation information with the largest information similarity between the M pieces of standard representation information and the standard representation information of the first sub-tree as the to-be-determined representation information;

[0223] If the information similarity between the standard representation information of the first sub-tree and the to-be-determined representation information is greater than or equal to the set similarity threshold, then use the resource data in the data set that has a mapping relationship with the to-be-determined representation information as the resource data filtered out and matched with the first sub-tree;

[0224] Among them, the first matching result includes the resource data filtered out and matched with the first sub-tree.

[0225] In one implementation, the matching module 1203 is further configured to:

[0226] If the information similarity between the standard representation information of the first sub-tree and the to-be-determined representation information is less than the similarity threshold, then determine that no resource data matching the first sub-tree is filtered out in the data set;

[0227] Among them, the first matching result includes that no resource data matching the first sub-tree is filtered out in the data set.

[0228] In one implementation, all N sub-trees are composed of one or more operator nodes of the candidate plan tree; the manner in which the determination module 1204 determines the reference resource data of the N sub-trees based on the first matching result includes:

[0229] If the first matching result includes the resource data filtered out and matched with the first sub-tree, then use the filtered resource data matched with the first sub-tree as the reference resource data of the first sub-tree;

[0230] If the first matching result includes that no resource data matching the first sub-tree is filtered out, then determine the reference resource data of the first sub-tree based on the number of operator nodes included in the first sub-tree.

[0231] In one implementation, the manner in which the determination module 1204 determines the reference resource data of the first sub-tree based on the number of operator nodes included in the first sub-tree includes:

[0232] When the first sub-tree includes one operator node of the candidate plan tree, use the resource data consumed by the table data called by the first sub-tree as the reference resource data of the first sub-tree;

[0233] When the first subtree contains multiple operator nodes of the candidate plan tree, perform a second matching process on the resource data for the first subtree in the dataset, and based on the second matching result obtained from the second matching process, determine the reference resource data for each operator node in the first subtree; wherein, the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, and the second matching process is used to screen out the matching resource data for each operator node in the first subtree through the similarity between each operator node in the first subtree and M historical subtrees.

[0234] In one implementation, any operator node in the first subtree is a target operator node; the manner in which the determination module 1204 determines the reference resource data for each operator node in the first subtree based on the second matching result obtained from the second matching process includes:

[0235] If the second matching result includes the screened resource data that matches the target operator node, then use the screened resource data that matches the target operator node as the reference resource data of the target operator node;

[0236] If the second matching result includes no screened resource data that matches the target operator node, then use the resource data consumed by the table data called by the target operator node as the reference resource data of the target operator node.

[0237] In one implementation, any one of the N subtrees is the first subtree; the manner in which the matching module 1203 performs a normalization representation process on each of the N subtrees to generate the standard representation information for each subtree includes:

[0238] Perform data parsing processing on the first subtree from K data domains to obtain the representation data of the first subtree under the K data domains; K is a positive integer;

[0239] Perform a representation process on the representation data of the first subtree under the K data domains according to a set standard format to generate the standard representation information of the first subtree;

[0240] Among them, the K data domains include at least one of the following: the operator domain corresponding to the operator called by the subtree, the object domain corresponding to the data object in the table data called by the subtree, and the condition domain corresponding to the condition included in the operator called by the subtree.

[0241] In one implementation, the process by which the first acquisition module 1201 acquires the N subtrees of the candidate plan tree includes:

[0242] Identify the branch nodes in the candidate plan tree, and divide each branch under the branch nodes into the first type of subtrees of the candidate plan tree;

[0243] For each operator node in the candidate plan tree except the first type of subtree, construct the second type of subtree of the candidate plan tree, and one operator node in the candidate plan tree except the first type of subtree is used to construct one second type of subtree;

[0244] Take the first type of subtree and the second type of subtree as the N subtrees obtained by partitioning the candidate plan tree.

[0245] In one implementation, there are multiple candidate plan trees for the operation statement, and the data processing device further includes an execution module 1206, and the execution module 1206 is configured to:

[0246] Select the candidate plan tree with the minimum corresponding execution cost from the multiple candidate plan trees as the target plan tree of the operation statement;

[0247] Execute the operation statement using the target plan tree.

[0248] In one implementation, there is a mapping relationship between the resource data consumed by each historical subtree in the dataset and the standard representation information of each historical subtree respectively, and any subtree of the target plan tree is the second subtree;

[0249] After executing the target plan tree, the execution module 1206 is further configured to:

[0250] If the standard representation information of the second subtree is not included in the dataset, then after executing the second subtree, obtain the resource data consumed by the second subtree during the execution process;

[0251] Construct a mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree, and add the constructed mapping relationship to the dataset.

[0252] In one implementation, the manner in which the execution module 1206 obtains the resource data consumed by the second subtree includes:

[0253] Obtain the input and output data of the last operator node in the second subtree, and use the input and output data of the last operator node as the input and output data of the second subtree;

[0254] Obtain the memory data consumed by each operator node in the second subtree, and use the maximum memory data consumed by the operator nodes in the second subtree as the memory data consumed by the second subtree;

[0255] Obtain the processor data consumed by each operator node in the second subtree, and use the maximum processor data consumed by the operator nodes in the second subtree as the processor data consumed by the second subtree;

[0256] Among them, the resource data consumed by the second subtree includes the input and output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree.

[0257] In one implementation, the process by which the first acquisition module 1201 acquires the candidate plan tree of the operation statement includes:

[0258] Perform statement parsing processing on the operation statement to generate the logical plan tree of the operation statement;

[0259] Perform conversion processing on the logical plan tree to generate multiple candidate plan trees of the operation statement.

[0260] In one implementation, the manner in which the estimation module 1205 estimates the execution cost corresponding to the candidate plan tree by using the reference resource data of N subtrees includes:

[0261] Input the reference resource data of N subtrees into the cost estimation model;

[0262] Call the cost estimation model to estimate the execution cost corresponding to the candidate plan tree based on the reference resource data of N subtrees.

[0263] According to an embodiment of the present application, Figure 3 the steps involved in the data processing method shown can be Figure 12 executed by each module in the data processing device 120 shown. For example, Figure 3 the step S101 shown in Figure 12 can be executed by the first acquisition module 1201 in Figure 3 the step S102 shown in Figure 12 can be executed by the second acquisition module 1202 in Figure 3 the step S103 shown in Figure 12 can be executed by the matching module 1203 in Figure 3 the step S104 shown in Figure 12 can be executed by the determination module 1204 in Figure 3 the step S105 shown in Figure 12 can be executed by the estimation module 1205 in

[0264] The present application can obtain a candidate plan tree for an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; and it can obtain a data set, which includes resource data consumed by M historical subtrees during their respective historical executions. The historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer; and, it can perform a first matching process on the N subtrees in the data set, and the first matching process is used to screen out matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees; thus, it can determine the reference resource data of the N subtrees based on the first matching result obtained from the first matching process; and use the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree. It can be seen that the device proposed in the present application can obtain the reference resource data of each subtree in the candidate plan tree at the granularity of the subtree, and moreover, the reference resource data can be obtained by matching from the resource data actually consumed by the historical subtrees during the actual historical execution process. Therefore, it also ensures the accuracy and reliability of the reference resource data obtained for each subtree. Thus, through the accurate and reliable reference resource data of each subtree, the execution cost corresponding to the candidate plan tree can also be accurately estimated.

[0265] According to an embodiment of the present application, Figure 12 Each module in the data processing device 120 shown can be separately or all combined into one or several units to form, or a certain one (or some) of the units can be further split into multiple smaller sub-units in terms of function, and the same operations can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In actual applications, the function of one module can also be realized by multiple units, or the functions of multiple modules can be realized by one unit. In other embodiments of the present application, the data processing device 120 can also include other units. In actual applications, these functions can also be assisted by other units and can be realized through the cooperation of multiple units.

[0266] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially realized by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to realize one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of that module or unit.

[0267] According to an embodiment of the present application, a computer program capable of executing the steps involved in the corresponding methods shown in the embodiments of the present application can be run on a general-purpose computer device (which may include processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM)) to construct a data processing device 120 as shown in Figure 12 . The above computer program can be recorded on a computer-readable recording medium, and can be loaded into the above computer device through the computer-readable recording medium and run therein.

[0268] Please refer to Figure 13 , Figure 13 which is a schematic structural diagram of a computer device provided by an embodiment of the present application. As shown in Figure 13 , the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, in some embodiments, the computer device 1000 may further include: a user interface 1003 and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display) and a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. Optionally, the memory 1005 may also be at least one storage device located far from the aforementioned processor 1001. As shown in Figure 13 , the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0269] In the computer device 1000 shown in Figure 13 , the network interface 1004 can provide network communication functions; the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call the device control application program stored in the memory 1005 to implement:

[0270] Obtain a candidate plan tree for an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement;

[0271] Obtain a data set, where the data set includes the resource data consumed by each of M historical subtrees during the historical execution process. The historical subtrees are subtrees of a historical plan tree that has been historically executed, and M is a positive integer;

[0272] Perform a first matching process on resource data for N subtrees in the dataset. The first matching process is used to screen out matching resource data for the N subtrees based on the similarity between the N subtrees and M historical subtrees;

[0273] Based on the first matching result obtained from the first matching process, determine the reference resource data for the N subtrees;

[0274] Adopt the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree.

[0275] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the descriptions of the above data processing methods in the embodiments of the present application, and can also execute the descriptions of the above data processing device 120 in the corresponding embodiments described above, which will not be elaborated here. In addition, the beneficial effects of adopting the same method will not be elaborated either. Figure 12 In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium. When the processor executes the computer program, it can execute the descriptions of the data processing methods in the embodiments of the present application. Therefore, it will not be elaborated here. In addition, the beneficial effects of adopting the same method will not be elaborated either. For the technical details not disclosed in the embodiments of the computer storage medium involved in the present application, please refer to the descriptions of the method embodiments of the present application.

[0276] As an example, the above computer program can be deployed to be executed on a single computer device, or deployed to be executed on multiple computer devices located at one location. Or, it can be executed on multiple computer devices distributed at multiple locations and interconnected through a communication network. The multiple computer devices distributed at multiple locations and interconnected through a communication network can form a blockchain network.

[0277] The above computer-readable storage medium can be an internal storage unit of the above computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0278] ​

[0279] The present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the descriptions of the above data processing methods in the embodiments of the present application. Therefore, details will not be repeated here. In addition, the beneficial effects of using the same method will not be described in detail either. For the technical details not disclosed in the embodiments of the computer-readable storage medium involved in the present application, please refer to the descriptions of the method embodiments of the present application.

[0280] The terms "first", "second", etc. in the description, claims and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or modules, but may optionally further include steps or modules not listed, or may optionally further include other step units inherent to these processes, methods, devices, products or equipment.

[0281] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0282] The above-disclosed are only the preferred embodiments of the present application, and of course, the scope of the rights of the present application cannot be limited thereby. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that, The method includes: Obtaining a candidate plan tree of an operation statement and N sub-trees of the candidate plan tree, where the candidate plan tree is composed of the N sub-trees, any one of the sub-trees includes one or more operator nodes in the candidate plan tree, there are no overlapping operator nodes among the N sub-trees, N is a positive integer, and the candidate plan tree is used to execute the operation statement; Obtaining a data set, where the data set includes resource data consumed by each of M historical sub-trees during historical execution. The resource data consumed by any one of the historical sub-trees includes input and output data, memory data consumed, and processor data consumed by any one of the historical sub-trees. The historical sub-tree is a sub-tree of a historical plan tree that has been historically executed, and M is a positive integer; Performing a first matching process on the resource data of the N sub-trees in the data set, where the first matching process is used to screen out matching resource data for the N sub-trees based on the similarity between the N sub-trees and the M historical sub-trees; Determining reference resource data of the N sub-trees based on the first matching result obtained from the first matching process; Estimating the execution cost corresponding to the candidate plan tree by using the reference resource data of the N sub-trees.

2. The method according to claim 1, characterized in that, The performing a first matching process on the resource data of the N sub-trees in the data set includes: Performing a standard representation process on each of the N sub-trees respectively to generate standard representation information of each sub-tree; Based on the standard representation information of the N sub-trees, performing the first matching process on the resource data of the N sub-trees in the data set; Wherein, the resource data consumed by each historical sub-tree in the data set has a mapping relationship with the standard representation information of each historical sub-tree respectively; the similarity between the standard representation information of the N sub-trees and the standard representation information of the M historical sub-trees is used to reflect the similarity between the N sub-trees and the M historical sub-trees.

3. The method according to claim 2, characterized in that, Any one of the N sub-trees is a first sub-tree, and the data set includes M standard representation information of the M historical sub-trees; The performing a first matching process on the resource data of the N sub-trees in the data set based on the standard representation information of the N sub-trees includes: Obtaining the information similarity between the standard representation information of the first sub-tree and each of the M standard representation information; Determining the standard representation information with the largest information similarity between the first sub-tree and the M standard representation information as the to-be-determined representation information; If the information similarity between the standard representation information of the first sub-tree and the to-be-determined representation information is greater than or equal to a set similarity threshold, then using the resource data in the data set that has a mapping relationship with the to-be-determined representation information as the screened resource data that matches the first sub-tree; Wherein, the first matching result includes the screened resource data that matches the first sub-tree.

4. The method according to claim 3, wherein The method further includes: If the information similarity between the standard representation information of the first subtree and the to-be-determined representation information is less than the similarity threshold, it is determined that no resource data matching the first subtree is screened out in the dataset; Among them, the first matching result includes that no resource data matching the first subtree is screened out in the dataset.

5. The method according to claim 4, characterized in that, Determining the reference resource data of the N subtrees based on the first matching result obtained from the first matching process includes: If the first matching result includes the screened resource data matching the first subtree, the screened resource data matching the first subtree is used as the reference resource data of the first subtree; If the first matching result includes that no resource data matching the first subtree is screened out, the reference resource data of the first subtree is determined based on the number of operator nodes included in the first subtree.

6. The method according to claim 5, wherein Determining the reference resource data of the first subtree based on the number of operator nodes included in the first subtree includes: When the first subtree includes one operator node of the candidate plan tree, the resource data consumed by the table data called by the first subtree is used as the reference resource data of the first subtree; When the first subtree includes multiple operator nodes of the candidate plan tree, a second matching process for resource data of the first subtree is performed in the dataset, and based on the second matching result obtained from the second matching process, the reference resource data of each operator node in the first subtree is determined; among them, the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, and the second matching process is used to screen out the matching resource data for each operator node in the first subtree through the similarity between each operator node in the first subtree and the M historical subtrees.

7. The method according to claim 6, characterized in that, Any operator node in the first subtree is a target operator node; determining the reference resource data of each operator node in the first subtree based on the second matching result obtained from the second matching process includes: If the second matching result includes the screened resource data matching the target operator node, the screened resource data matching the target operator node is used as the reference resource data of the target operator node; If the second matching result includes that no resource data matching the target operator node is screened out, the resource data consumed by the table data called by the target operator node is used as the reference resource data of the target operator node.

8. The method according to claim 2, wherein Any one of the N subtrees is the first subtree; performing a standardization representation process on each of the N subtrees respectively to generate the standard representation information of each subtree includes: Performing data parsing processing on the first subtree from K data domains to obtain the representation data of the first subtree under the K data domains; K is a positive integer; Performing a representation process on the representation data of the first subtree under the K data domains according to a set standard format to generate the standard representation information of the first subtree; Among them, the K data fields include at least one of the following: an operator field corresponding to an operator called by a subtree, an object field corresponding to a data object in table data called by a subtree, and a condition field corresponding to a condition included in an operator called by a subtree.

9. The method according to claim 1, wherein The process of obtaining N subtrees of the candidate plan tree includes: Identifying branch nodes in the candidate plan tree and dividing each branch under the branch nodes into first-type subtrees of the candidate plan tree; Constructing second-type subtrees of the candidate plan tree respectively through each operator node in the candidate plan tree except the first-type subtrees, and one operator node in the candidate plan tree except the first-type subtrees is used to construct one second-type subtree; Regarding the first-type subtrees and the second-type subtrees as the N subtrees obtained by dividing the candidate plan tree.

10. The method according to claim 1, wherein There are multiple candidate plan trees for the operation statement, and the method further includes: Selecting the candidate plan tree with the minimum corresponding execution cost from the multiple candidate plan trees as the target plan tree of the operation statement; Executing the operation statement using the target plan tree.

11. The method according to claim 10, wherein The resource data consumed by each historical subtree in the dataset has a mapping relationship with the respective standard representation information of each historical subtree, and any subtree of the target plan tree is a second subtree; After executing the target plan tree, the method further includes: If the standard representation information of the second subtree is not included in the dataset, after executing the second subtree, obtaining the resource data consumed by the second subtree during the execution process; Constructing a mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree, and adding the constructed mapping relationship to the dataset.

12. The method according to claim 11, wherein The obtaining of the resource data consumed by the second subtree during the execution process includes: Obtaining the input and output data of the last operator node in the second subtree, and using the input and output data of the last operator node as the input and output data of the second subtree; Obtaining the memory data consumed by each operator node in the second subtree, and using the maximum memory data consumed by the operator nodes in the second subtree as the memory data consumed by the second subtree; Obtaining the processor data consumed by each operator node in the second subtree, and using the maximum processor data consumed by the operator nodes in the second subtree as the processor data consumed by the second subtree; Among them, the resource data consumed by the second subtree includes the input and output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree.

13. The method according to claim 1, wherein The process of obtaining the candidate plan tree of the operation statement includes: Performing statement parsing processing on the operation statement to generate a logical plan tree of the operation statement; Performing conversion processing on the logical plan tree to generate multiple candidate plan trees of the operation statement.

14. The method according to claim 1, characterized in that, The estimating of the execution cost corresponding to the candidate plan tree by using the reference resource data of the N subtrees includes: Inputting the reference resource data of the N subtrees into a cost estimation model; Call the cost estimation model to estimate the execution cost corresponding to the candidate plan tree based on the reference resource data of the N subtrees.

15. A data processing device, characterized in that, The device includes: A first acquisition module, configured to acquire a candidate plan tree of an operation statement and N subtrees of the candidate plan tree. The candidate plan tree is composed of the N subtrees. Any one of the subtrees includes one or more operator nodes in the candidate plan tree. There are no overlapping operator nodes among the N subtrees. N is a positive integer. The candidate plan tree is used to execute the operation statement; A second acquisition module, configured to acquire a data set. The data set includes resource data consumed by each of the M historical subtrees during historical execution. The resource data consumed by any one of the historical subtrees includes input and output data of any one of the historical subtrees, memory data consumed, and processor data consumed. The historical subtrees are subtrees of historical plan trees that have been historically executed. M is a positive integer; A matching module, configured to perform a first matching process on the resource data of the N subtrees in the data set. The first matching process is used to screen the resource data matching the N subtrees through the similarity between the N subtrees and the M historical subtrees; A determination module, configured to determine the reference resource data of the N subtrees based on the first matching result obtained from the first matching process; An estimation module, configured to estimate the execution cost corresponding to the candidate plan tree by using the reference resource data of the N subtrees.

16. A computer program product, characterized in that, It includes a computer program, which when executed by a processor, implements the steps of the method according to any one of claims 1-14.

17. A computer device, characterized in that, It includes a memory and a processor. The memory stores a computer program, which when executed by the processor, causes the processor to execute the steps of the method according to any one of claims 1-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor to execute the steps of the method according to any one of claims 1-14.

Images