Data processing method and device, product and equipment

Abstract

The invention discloses a data processing method and device, a product and equipment, and the method comprises the steps: obtaining a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, N being a positive integer, and the candidate plan tree being used for executing the operation statement; a data set is obtained, the data set comprises resource data consumed by M historical sub-trees in the historical execution process, the historical sub-trees are sub-trees of the historical plan tree which has been historically executed, and M is a positive integer; performing first matching processing of resource data on the N sub-trees in the data set, wherein the first matching processing is used for screening matched resource data for the N sub-trees through the similarity between the N sub-trees and the M historical sub-trees; determining reference resource data of the N sub-trees based on a first matching result obtained by the first matching processing; and estimating the execution cost corresponding to the candidate plan tree by adopting the reference resource data of the N sub-trees. By adopting the method and the device, the accuracy of the estimated execution cost of the candidate plan tree can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a data processing method, device, product and equipment. Background Art

[0002] SQL statements (structured statements) are statements used to query and operate data in a database. In actual applications, SQL statements are converted into multiple candidate plan trees, and then the candidate plan tree used to execute the SQL statement is determined from the multiple candidate plan trees according to the execution costs corresponding to each of the multiple candidate plan trees. It can be seen that the execution cost corresponding to the candidate plan tree of the SQL statement plays an extremely important role in the execution process of the SQL statement. Therefore, how to more accurately obtain the execution cost corresponding to the candidate plan tree of the SQL statement is a hot issue. Summary of the invention

[0003] The present application provides a data processing method, apparatus, product and device, which can improve the accuracy of the estimated execution cost of a candidate plan tree.

[0004] On the one hand, the present application provides a data processing method, the method comprising: Obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; Obtain a data set, which includes resource data consumed by M historical subtrees during historical execution, where the historical subtree is a subtree of the historical plan tree that has been historically executed, and M is a positive integer; Performing a first matching process of resource data on the N subtrees in the data set, the first matching process being used to screen matching resource data for the N subtrees based on similarities between the N subtrees and the M historical subtrees; Determine reference resource data of N subtrees based on a first matching result obtained by the first matching process; Using the reference resource data of N subtrees, the execution cost corresponding to the candidate plan tree is estimated.

[0005] On one hand, the present application provides a data processing device, the device comprising: A first acquisition module is used to acquire a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; The second acquisition module is used to acquire a data set, wherein the data set includes resource data consumed by each of M historical subtrees during the historical execution process, where the historical subtree is a subtree of the historical plan tree that has been historically executed, and M is a positive integer; A matching module, used for performing a first matching process of resource data on the N subtrees in the data set, wherein the first matching process is used for selecting matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees; A determination module, configured to determine reference resource data of N subtrees based on a first matching result obtained by the first matching process; The estimation module is used to estimate the execution cost corresponding to the candidate plan tree using the reference resource data of N subtrees.

[0006] In one implementation, the matching module performs a first matching process of resource data on N subtrees in a data set, including: Performing standardized representation processing on each of the N subtrees to generate standard representation information for each subtree; Based on the standard representation information of the N subtrees, performing a first matching process of resource data on the N subtrees in the data set; Among them, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree; the similarity between the standard representation information of N subtrees and the standard representation information of M historical subtrees is used to reflect the similarity between N subtrees and M historical subtrees.

[0007] In one embodiment, any one of the N subtrees is a first subtree, and the data set includes M standard representation information of the M historical subtrees; The matching module performs a first matching process on the resource data of the N subtrees in the data set based on the standard representation information of the N subtrees, including: Obtaining information similarities between the standard representation information of the first subtree and the M standard representation information respectively; The standard representation information having the greatest information similarity with the standard representation information of the first subtree among the M standard representation information is determined as the pending representation information; If the information similarity between the standard representation information of the first subtree and the pending representation information is greater than or equal to the set similarity threshold, the resource data in the data set that has a mapping relationship with the pending representation information is used as the selected resource data that matches the first subtree; The first matching result includes the filtered resource data that matches the first subtree.

[0008] In one embodiment, the matching module is further configured to: If the information similarity between the standard representation information of the first subtree and the pending representation information is less than the similarity threshold, it is determined that resource data matching the first subtree is not screened out in the data set; The first matching result includes resource data that is not matched with the first subtree in the data set.

[0009] In one implementation, each of the N subtrees is composed of one or more operator nodes of the candidate plan tree; the determination module determines the reference resource data of the N subtrees based on the first matching result obtained by the first matching process, including: If the first matching result includes the filtered resource data that matches the first subtree, the filtered resource data that matches the first subtree is used as reference resource data for the first subtree; If the first matching result includes resource data that is not screened out and matches the first subtree, the reference resource data of the first subtree is determined based on the number of operator nodes included in the first subtree.

[0010] In one implementation, the determination module determines the reference resource data of the first subtree based on the number of operator nodes included in the first subtree, including: When the first subtree includes an operator node of the candidate plan tree, resource data consumed by the table data called by the first subtree is used as reference resource data of the first subtree; When the first subtree contains multiple operator nodes of the candidate plan tree, a second matching process of resource data is performed on the first subtree in the data set, and based on a second matching result obtained by the second matching process, the reference resource data of each operator node in the first subtree is determined; wherein the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, and the second matching process is used to screen matching resource data for each operator node in the first subtree through the similarity between each operator node in the first subtree and the M historical subtrees.

[0011] In one implementation, any operator node in the first subtree is a target operator node; and the determination module determines the reference resource data of each operator node in the first subtree based on the second matching result obtained by the second matching process, including: If the second matching result includes the filtered resource data that matches the target operator node, the filtered resource data that matches the target operator node is used as the reference resource data of the target operator node; If the second matching result includes resource data that is not screened out and matches the target operator node, the resource data consumed by the table data called by the target operator node is used as reference resource data of the target operator node.

[0012] In one implementation, any one of the N subtrees is a first subtree; and the matching module performs standardized representation processing on each of the N subtrees to generate standard representation information of each subtree, including: Perform data parsing processing on the first subtree from the K data domains to obtain the representation data of the first subtree under the K data domains; K is a positive integer; Perform representation processing on the representation data of the first subtree under the K data domains according to a set standard format to generate standard representation information of the first subtree; The K data domains include at least one of the following: an operator domain corresponding to the operator called by the subtree, an object domain corresponding to the data object in the table data called by the subtree, and a condition domain corresponding to the condition contained in the operator called by the subtree.

[0013] In one implementation, the process of the first acquisition module acquiring N subtrees of the candidate plan tree includes: Identify a branch node in the candidate plan tree, and divide each branch under the branch node into a first type subtree of the candidate plan tree; Through each operator node except the first type subtree in the candidate plan tree, a second type subtree of the candidate plan tree is constructed respectively, and an operator node except the first type subtree in the candidate plan tree is used to construct a second type subtree; The first type of subtree and the second type of subtree are used as N subtrees obtained by partitioning the candidate plan tree.

[0014] In one embodiment, there are multiple candidate plan trees for the operation statement, and the data processing device further includes an execution module, which is used to: Select a candidate plan tree with the minimum execution cost from multiple candidate plan trees as the target plan tree of the operation statement; Execute operation statements using the target plan tree.

[0015] In one embodiment, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree, and any subtree of the target plan tree is the second subtree; After executing the target plan tree, the execution module is also used to: If the data set does not contain standard representation information of the second subtree, after executing the second subtree, obtaining resource data consumed by the second subtree during execution; A mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree is constructed, and the constructed mapping relationship is added to the data set.

[0016] In one implementation, the execution module obtains resource data consumed by the second subtree during execution, including: Obtain the input and output data of the last operator node in the second subtree, and use the input and output data of the last operator node as the input and output data of the second subtree; Obtain the memory data consumed by each operator node in the second subtree, and use the memory data consumed most by the operator nodes in the second subtree as the memory data consumed by the second subtree; Obtain the processor data consumed by each operator node in the second subtree, and use the processor data consumed most by the operator nodes in the second subtree as the processor data consumed by the second subtree; The resource data consumed by the second subtree includes input and output data of the second subtree, memory data consumed by the second subtree, and processor data consumed by the second subtree.

[0017] In one implementation, the process of the first acquisition module acquiring the candidate plan tree of the operation statement includes: Parse the operation statements and generate a logical plan tree for the operation statements; The logical plan tree is converted to generate multiple candidate plan trees for the operation statements.

[0018] In one implementation, the estimation module uses the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree, including: Input the reference resource data of N subtrees into the cost estimation model; The call cost estimation model estimates the execution cost corresponding to the candidate plan tree based on the reference resource data of N subtrees.

[0019] In one aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes a method in one aspect of the present application.

[0020] In one aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the method in the above aspect.

[0021] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional modes such as the above-mentioned one aspect.

[0022] The present application can obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; and can obtain a data set, which includes resource data consumed by M historical subtrees during historical execution, where the historical subtree is a subtree of a historical plan tree that has been historically executed, and M is a positive integer; and, can perform a first matching process on the resource data of the N subtrees in the data set, and the first matching process is used to filter matching resource data for the N subtrees through the similarity between the N subtrees and the M historical subtrees; thereby, the reference resource data of the N subtrees can be determined based on the first matching result obtained by the first matching process; and the reference resource data of the N subtrees can be used to estimate the execution cost corresponding to the candidate plan tree. It can be seen that the method proposed in the present application can obtain the reference resource data of each subtree in the candidate plan tree with the subtree as the granularity, and the reference resource data can be matched from the resource data actually consumed by the historical subtree in the actual historical execution process. Therefore, the accuracy and reliability of the reference resource data obtained for each subtree are guaranteed. Therefore, through the accurate and reliable reference resource data of each subtree, the execution cost corresponding to the candidate plan tree can also be accurately estimated. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 It is a structural diagram of a network architecture for data processing provided by an embodiment of the present application; Figure 2 It is a schematic diagram of a scenario for obtaining the execution cost corresponding to the candidate plan tree provided by an embodiment of the present application; Figure 3 It is a flowchart of a data processing method provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of a candidate plan tree provided in an embodiment of the present application; Figure 5 is a schematic diagram of a diagnostic interface for an operator node provided in an embodiment of the present application; Figure 6 It is a flow chart of performing a first matching process on resource data of N subtrees in a data set provided by an embodiment of the present application; Figure 7 It is a schematic diagram of a framework for collecting resource data consumed by a historical subtree provided in an embodiment of the present application; Figure 8 It is a schematic diagram of a framework for executing an operation statement provided in an embodiment of the present application; Fig. 9 This is a schematic diagram of a scenario for performing resource data matching provided in an embodiment of the present application; Fig.10 It is a schematic diagram of a process of determining reference resource data of N subtrees through a first matching result provided by an embodiment of the present application; Fig.11 is a structural schematic diagram of a subtree structure of a first subtree provided in an embodiment of the present application; Fig.12 is a structural schematic diagram of a data processing device provided in an embodiment of the present application; Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] All data collected by this application (such as operation statements, candidate plan trees, subtrees of candidate plan trees, historical subtrees, and resource data consumed by historical subtrees and other related data) are collected with the consent and authorization of the owner of the data (such as users, institutions or enterprises), and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of the relevant regions.

[0027] SQL: Structured Query Language, is a standard programming language for managing and operating relational databases. SQL statements are instructions for performing specific tasks or queries, including creating, reading, updating, and deleting data. These operations are usually referred to as CRUD operations (basic operations in database management).

[0028] Relational algebra: A mathematical theory used to describe and manipulate data in a database. This theory abstracts a series of operators for data operations, such as selection, projection, union, intersection, difference, Cartesian product, natural join, division, etc. The combination of these operators can form complex data processing operations and provide a theoretical basis for the design of the SQL language.

[0029] Abstract syntax tree: AST for short. The lexical analyzer can decompose the SQL source code into lexical units (tokens), and then the syntax analyzer organizes these lexical units into a tree structure according to the grammatical rules. This tree structure can be called an abstract syntax tree.

[0030] Logical execution plan: a plan generated during the SQL query optimization process, used to describe the logical operation sequence and execution mode of the SQL query without considering the specific physical implementation details. This part translates the SQL logic into an execution sequence composed of operators in relational algebra theory through AST. Among them, the logical execution plan can also be a tree structure, and the logical execution plan can be called a logical plan tree.

[0031] Physical execution plan: a plan generated during the SQL query optimization process, which is used to describe the specific physical operation sequence and execution method of the SQL query, taking into account details such as the underlying storage structure, index, and execution algorithm. The physical execution plan is the code logic of the actual SQL operation. The physical execution plan can also be a tree structure, which can be called a physical plan tree.

[0032] Execution operator subtree (can be referred to as subtree for short): Both the logical execution plan and the physical execution plan can be expressed as a directed acyclic graph. The execution operator subtree is the local logic of the directed acyclic graph and can fully express the local fragment of the data processing logic.

[0033] CBO model: A cost-based optimization model (also called a cost model) is an optimization method that considers multiple factors and selects the optimal execution plan (such as the optimal physical execution plan) through cost calculation. The CBO model can be used to estimate the execution cost corresponding to the physical plan tree.

[0034] See also Figure 1 , Figure 1 Schematic diagram of a data processing network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a server 200 and a terminal device cluster, and the terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 1, terminal device 2, terminal device 3, ..., terminal device n; Figure 1 As shown, terminal device 1, terminal device 2, terminal device 3, ..., terminal device n can all be connected to the server 200 through a network, so that each terminal device can exchange data with the server 200 through the network connection.

[0035] like Figure 1The server 200 shown can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (content distribution network), and big data and artificial intelligence platforms. The terminal device can be: a smart phone, a tablet computer, a laptop, a desktop computer, a smart TV, a car terminal, a smart home, and other smart terminals. The following takes the communication between the terminal device 1 and the server 200 as an example to describe the specific embodiments of the present application.

[0036] Here, the present application takes the processed SQL statement as a statement for data query as an example for relevant explanation, but in actual application scenarios, the processed SQL statement can not only be a statement for data query, but also a statement for performing other operations on data (such as deletion, modification, addition, etc.). Among them, the user can trigger the data query operation in the terminal device 1, and the terminal device 1 can generate a data query request and send the data query request to the server 200. The server 200 can generate a data query statement (an SQL statement, which can be used as an operation statement described in the following process) according to the data query request, and the data query statement is used to query the data requested by the user.

[0037] The server 200 can first generate multiple candidate plan trees (physical plan trees) for the data query statement, and can estimate the execution cost corresponding to each candidate plan tree. Thus, the data query statement can be finally executed through the candidate plan tree with the smallest estimated corresponding execution cost to query and obtain the data requested by the user. The server 200 can return the query and obtained data to the terminal device 1, so that the terminal device 1 can display the obtained data on the terminal interface for the user to view.

[0038] The process of server 200 estimating the execution cost corresponding to each candidate plan tree can be seen in Figure 2 , Figure 2 Schematic diagram of a scenario for obtaining the execution cost corresponding to a candidate plan tree provided by an embodiment of the present application. Figure 2 As shown above Figure 1The server 200 in the data set can obtain a data set, which can include the resource data actually consumed by M historical subtrees during the historical execution process, where M is a positive integer, and M is assumed to be equal to 5 here. The M historical subtrees belong to the subtrees of the historical plan tree that has been actually executed in the history. In addition, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree. The standard representation information of the historical subtree is the representation information obtained after the historical subtree is processed by the standardized representation. The principle of the standardized representation processing can be referred to as follows Figure 6 Corresponding to the relevant description in the embodiment.

[0039] As shown here, the M historical subtrees may include historical subtree L1, historical subtree L2, historical subtree L3, historical subtree L4 and historical subtree L5, the standard representation information of historical subtree L1 is standard representation information B1, the standard representation information of historical subtree L2 is standard representation information B2, the standard representation information of historical subtree L3 is standard representation information B3, the standard representation information of historical subtree L4 is standard representation information B4, and the standard representation information of historical subtree L5 is standard representation information B5. Therefore, the data set may include the mapping relationship between the standard representation information B1 and the resource data Z1 actually consumed by the historical subtree L1, the mapping relationship between the standard representation information B2 and the resource data Z2 actually consumed by the historical subtree L2, the mapping relationship between the standard representation information B3 and the resource data Z3 actually consumed by the historical subtree L3, the mapping relationship between the standard representation information B4 and the resource data Z4 actually consumed by the historical subtree L4, and the mapping relationship between the standard representation information B5 and the resource data Z5 actually consumed by the historical subtree L5.

[0040] The server 200 may also perform subtree division on the candidate plan tree to obtain N subtrees of the candidate plan tree, where N is a positive integer, and it is assumed here that N is equal to 4. The N subtrees may include subtree H1, subtree H2, subtree H3, and subtree H4. The server 200 may also perform standardized representation processing on the N subtrees to obtain standard representation information of each of the N subtrees, including standard representation information S1 of subtree H1, standard representation information S2 of subtree H2, standard representation information S3 of subtree H3, and standard representation information S4 of subtree H4.

[0041] The server 200 can use the standard representation information of the N subtrees to perform a first matching process on the resource data in the data set. The first matching process is used to screen the matching resource data for the N subtrees through the similarity between the N subtrees and the M historical subtrees to obtain a first matching result. Here, the first matching result may include reference resource data C1 matched for subtree H1, reference resource data C2 matched for subtree H2, reference resource data C3 matched for subtree H3, and reference resource data C4 matched for subtree H4. The similarity between the N subtrees and the M historical subtrees can be reflected by the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees. Therefore, it can be understood that the server 200 can screen the resource data matching the N subtrees in the data set through the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees to obtain a first matching result. The specific process of obtaining the first matching result through the first matching process can also be referred to the detailed description of the following embodiments.

[0042] The server 200 can determine the reference resource data of each of the N subtrees through the first matching result obtained above, and thus, the execution cost corresponding to the candidate plan tree can be estimated through the reference resource data of each of the N subtrees. The specific process can also refer to the relevant descriptions in the following embodiments.

[0043] By adopting the method provided in the present application, the reference resource data of the subtree can be obtained at the fine granularity of the subtree, and the reference resource data can be obtained by matching the resource data actually consumed by the historical subtree in the data set during the actual execution process. Therefore, the reference resource data determined by the subtree of the candidate plan tree in this way can be used to accurately estimate the execution cost corresponding to the candidate plan tree and reduce the overhead caused by executing SQL statements.

[0044] See also Figure 3 , Figure 3 This is a flowchart of a data processing method provided by an embodiment of the present application. The execution subject in the embodiment of the present application may be a data processing device (referred to as a processing device), which may be a computer device or a computer device cluster composed of multiple computer devices. The computer device may be a server, a terminal device, or other devices, and the present application does not limit this. Figure 3 As shown, the method may include: Step S101, obtaining a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement.

[0045] Specifically, the operation statement may be an SQL statement, which may be used to query or operate (such as add, delete, modify, etc.) the data in the data table. The processing device may obtain the operation statement, which may be any SQL statement that needs to be executed currently, and the specific SQL statement may be determined according to the actual application scenario.

[0046] The processing device may also obtain a candidate plan tree of the operation statement, and the candidate plan tree may be a physical plan tree of the operation statement, and there may be multiple candidate plan trees. The process of the processing device obtaining the candidate plan tree of the operation statement may include: the processing device may perform syntax parsing on the operation statement to generate an abstract syntax tree of the operation statement, such as the processing device may use a configured lexical analyzer and syntax analyzer to perform syntax parsing on the operation statement to generate an abstract syntax tree of the operation statement.

[0047] The processing device can perform relevant processing of logic optimization on the abstract syntax tree of the operation statement to generate a logical plan tree of the operation statement. For example, the processing device can optimize and convert the nodes in the abstract syntax tree into nodes in the logical plan tree according to the configured processing logic, thereby obtaining the logical plan tree of the operation statement. Thus, the processing device can convert the logical plan tree of the operation statement to obtain multiple physical plan trees of the operation statement, and the multiple physical plan trees can all be used as candidate plan trees for the operation statement in this application. For example, the processing device can obtain a configured query optimizer (Query Optimizer), which can have a series of optimization rules and rewriters (Rewriter), through which the physical plan tree can be generated and obtained according to the structure and attributes of the logical plan tree, as well as the characteristics of the underlying storage system and the execution engine.

[0048] Since the principles of obtaining the execution cost corresponding to each candidate plan tree are independent and the same, the following process is specifically described by taking the process of obtaining the execution cost corresponding to a candidate plan tree as an example, that is, the number of candidate plan trees is not emphasized in the following process, and they are collectively referred to as candidate plan trees, as described below.

[0049] The processing device can also obtain N subtrees of the candidate plan tree of the operation statement, where N is a positive integer. The specific value of N can be determined according to the actual application scenario, and the number of subtrees divided into different candidate plan trees can be the same or different. That is, the candidate plan tree can be divided into the N subtrees. Among them, the candidate plan tree can contain at least one operator node (can be simply referred to as a node), and the operator node can be the basic component unit of the candidate plan tree. An operator node can represent a specific execution operation on a data table. Based on this, a subtree of the candidate plan tree can be composed of at least one operator node in the candidate plan tree, that is, a subtree of the candidate plan tree can contain one or more operator nodes in the candidate plan tree.

[0050] In an exemplary embodiment, the process of a processing device obtaining N subtrees of a candidate plan tree may include: the processing device may identify branch nodes in the candidate plan tree (also operator nodes of the candidate plan tree), there may be one or more branch nodes, or in special circumstances, no branch node may be identified (indicating that no branch node exists in the candidate plan tree), and each branch node may be connected to at least 2 branches.

[0051] Therefore, the processing device can divide each branch under the branch node into a subtree of the candidate plan tree, and the subtree can be called a first-type subtree. A branch under a branch node can be divided into a first-type subtree, so the number of first-type subtrees can be equal to the number of all branches under each branch node.

[0052] Furthermore, the processing device can also construct a subtree of the candidate plan tree through each operator node in the candidate plan tree except the first type of subtree, and the subtree can be called a second type of subtree. An operator node in the candidate plan tree except the first type of subtree can be used to construct a second type of subtree, that is, the number of second type subtrees can be equal to the number of operator nodes in the candidate plan tree except the first type of subtree.

[0053] The processing device may use the first type subtree and the second type subtree obtained above as N subtrees obtained by dividing the candidate plan tree, and the N subtrees may include each of the first type subtrees and each of the second type subtrees obtained by the above division.

[0054] Among them, the above-mentioned method of dividing the candidate plan tree into subtrees is only described by way of example. In actual application requirements, the various subtrees of the candidate plan tree can be divided by any other feasible method, so as to obtain the reference resource data of the candidate plan tree (which may include the reference resource data of each subtree of the candidate plan tree) by using the granularity of the subtree. The specific process can be found in the subsequent content description.

[0055] Step S102, obtaining a data set, wherein the data set includes resource data consumed by M historical subtrees during historical execution, wherein the historical subtree is a subtree of the historical plan tree that has been historically executed, and M is a positive integer.

[0056] Specifically, the processing device can obtain a data set, which may include resource data actually consumed by M historical subtrees during the historical execution process. M is a positive integer. The specific value of M can be determined according to the actual application scenario. Here, M can be the number of resource data contained in the current data set when obtaining the execution cost corresponding to the above-mentioned candidate plan tree. A historical subtree can have one resource data actually consumed.

[0057] The historical subtree may be a subtree of a historical plan tree that has actually been executed historically. The historical plan tree may be a physical plan tree that is actually executed for a historically executed SQL statement. The division method of the historical subtree of the historical plan tree may be the same as the division method of the subtree of the above-mentioned candidate plan tree.

[0058] It can be seen from the data set of the present application that when executing the physical plan tree of each SQL statement, the present application can collect the resource data actually consumed by each subtree in the physical plan tree during the execution process according to the granularity of the subtree, and store it in the data set, so as to accurately and reliably obtain the reference resource data of the subsequent physical plan tree to be executed through the data set, as described below.

[0059] Step S103 , performing a first matching process of resource data on the N subtrees in the data set, wherein the first matching process is used to screen matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees.

[0060] Specifically, the processing device can perform resource data matching processing on the above-mentioned N subtrees in the data set. This matching processing can be called the first matching processing. The first matching processing can be used to filter matching resource data for the N subtrees from the data set through the similarity between the N subtrees and the N historical subtrees.

[0061] During the first matching process, the more similar the resource data of the historical subtree in the data set is to the subtree of the candidate plan tree, the more it matches the subtree of the candidate plan tree. The first matching result can be obtained through the first matching process. The first matching result can include resource data that matches the subtree in the N subtrees selected from the data set, and can also include resource data that does not match the subtree in the N subtrees selected from the data set. The specific implementation process of the first matching process can be referred to below. Figure 6 Corresponding to the description in the embodiment.

[0062] Step S104: determining reference resource data of N subtrees based on the first matching result obtained by the first matching process.

[0063] Specifically, the processing device can determine the reference resource data of each subtree in the N subtrees through the first matching result obtained by the above-mentioned first matching processing. A subtree can have a reference resource data, and the reference resource data of a subtree is the resource data required to be consumed by the referenced subtree during subsequent actual execution. In other words, the reference resource data of a subtree can be the resource data that is pre-evaluated for the subtree and may be consumed by the subtree during actual execution.

[0064] The specific process of determining the reference resource data of the N subtrees of the candidate plan tree through the first matching result obtained by the first matching process can be referred to as follows: Fig.10 Detailed description in the corresponding embodiment.

[0065] Step S105, using the reference resource data of the N subtrees, to estimate the execution cost corresponding to the candidate plan tree.

[0066] Specifically, the processing device may use the reference resource data of the N subtrees obtained above to estimate the execution costs corresponding to the candidate plan trees to which the N subtrees belong.

[0067] For example, the process of estimating the execution cost corresponding to the candidate plan tree to which the N subtrees belong through the reference resource data of the N subtrees may include: the processing device may obtain a cost estimation model, which may be referred to as a cost model (which may be a CBO model), and the cost estimation model may be a trained model that can be used to estimate the cost of the physical plan tree. The processing device may input the reference resource data of the N subtrees into the cost estimation model to call the cost estimation model to estimate the execution cost corresponding to the candidate plan tree through the reference resource data of the N subtrees.

[0068] In one embodiment, the cost estimation model can estimate the execution cost corresponding to the current operator node through the execution cost corresponding to the previous operator node of the current operator node and the reference resource data of the current operator node, and use this principle to estimate the execution cost corresponding to each operator node in the candidate plan tree from bottom to top, so that the corresponding execution cost estimated for the topmost operator node (i.e., the root node) in the candidate plan tree can be used as the estimated execution cost of the entire candidate plan tree.

[0069] In this application, since the reference resource data of a subtree can be obtained at the granularity of a subtree, the cost estimation model can also change the original method of estimating the execution cost corresponding to a subnode at the granularity of an operator node to obtaining the execution cost corresponding to a subtree at the granularity of a subtree. This principle can also be to estimate the execution cost corresponding to the current subtree through the execution cost corresponding to the previous subtree of the current subtree and the reference resource data of the current subtree. The subtree estimated at the end in this way will also be the root node of the candidate plan tree, so the estimated execution cost of the root node can also be used as the estimated execution cost of the entire candidate plan tree.

[0070] See also Figure 4 , Figure 4 Schematic diagram of a candidate plan tree provided in an embodiment of the present application. Figure 4 As shown, the candidate plan tree may include the operator node "Project(id, name, subject, score)" (a projection operator, where id, name, subject, and score can be called data objects or fields called in a data table), the operator node "SortMergeJoin(a.id=b.id)" (a join operator that can be used for table join operations), the operator node "Project(id, name)" (a projection operator, where id and name represent called fields or data objects), and the operator node "Filter(gender=male and age>18)" (a filtering operator, where "gender=male and age>18" is the filtering condition, which means filtering out females older than 18), operator node "TableScan(student)" (a data scanning operator, which can be referred to as a scanning operator. Here, student is the called data object, meaning student), operator node "Project(id, subject, score)" (a projection operator. id, subject, and score represent the called fields or data objects), operator node "Filter(subject=english)" (a filtering operator), and operator node "TableScan(score)" (a data scanning operator).

[0071] The operator node "Project (id, name, subject, score)" is the root node of the candidate plan tree. The operator node "SortMergeJoin (a.id = b.id)" is a branch node of the candidate plan tree, which is connected to two branches, including branch 1 (left branch) and branch 2 (right branch). Both branch 1 and branch 2 contain 3 operator nodes.

[0072] For example, the candidate plan tree can be divided into subtree 1 composed of the operator node "Project (id, name, subject, score)", subtree 2 composed of the operator node "SortMergeJoin (a.id=b.id)", subtree 3 composed of branch 1, and subtree 4 composed of branch 2. Among them, if there are more branch nodes connected to branch 1 or branch 2, branch 1 or branch 2 can be further divided into finer subtrees through the connected branch nodes. The division principle can be the same, and both can be to divide the branches under the branch nodes into subtrees. Since the execution cost of the subtree can be evaluated from the bottom to the top, subtree 3 and subtree 4 can both belong to the previous subtree of subtree 2, and subtree 2 can be the previous subtree of subtree 1. Similarly, if according to the operator granularity, the operator node "TableScan (student)" is the previous operator node of the operator node "Filter (gender=male and age>18)", and the operator node "Filter (gender=maleand age>18)" is the previous operator node of the operator node "Project(id, name)", the operator node "Project(id, name)" and the operator node "Project(id, subject, score)" are both the previous operator node of the operator node "SortMergeJoin(a.id=b.id)", the operator node "SortMergeJoin(a.id=b.id)" is the previous operator node of the operator node "Project(id, name, subject, score)", the operator node "TableScan(score)" is the previous operator node of the operator node "Filter(subject=english)", and the operator node "Filter(subject=english)" is the previous operator node of the operator node "Project(id, subject, score)".

[0073] Here, since there are no more subtrees connected to subtree 3 and subtree 4, neither subtree 3 nor subtree 4 has a previous subtree. The cost estimation model can be called to estimate the execution cost corresponding to subtree 3 through the reference resource data of subtree 3, and the cost estimation model can be called to estimate the execution cost corresponding to subtree 4 through the reference resource data of subtree 4. In addition, the execution cost corresponding to subtree 2 can be estimated by the execution cost corresponding to subtree 3, the execution cost corresponding to subtree 4, and the reference resource data of subtree 2. The execution cost corresponding to subtree 1 can also be estimated by the execution cost corresponding to subtree 2 and the reference resource data of subtree 1. Finally, the execution cost corresponding to subtree 1 (i.e., the root node) can be used as the execution cost estimated for the candidate plan tree.

[0074] Among them, if in the following Figure 6 and Fig.10 In the corresponding embodiment, if the resource data matching subtree 3 is not screened out from the data set, the execution cost corresponding to each operator node in subtree 3 can be estimated in a bottom-up manner according to the granularity of the operator node in subtree 3, and the estimated execution cost of the top operator node "Project (id, name)" in subtree 3 can be used as the execution cost corresponding to subtree 3, which is used for the subsequent estimation of the execution cost corresponding to subtree 2. And the same is true for subtree 4, or in other words, the same is true for subtrees that do not screen out matching resource data and contain multiple operator nodes.

[0075] Here, if the resource data consumed by a historical subtree is screened out to match subtree 3, that is, the historical subtree and subtree 3 can be regarded as the same, then the resource data consumed by the historical subtree in the data set can be directly taken as the reference resource data of subtree 3.

[0076] In the present application, since the reference resource data of a subtree can be obtained at the granularity of a subtree, the execution cost corresponding to the subtree can also be estimated at the granularity of the subtree, thereby improving the efficiency of estimating the execution cost corresponding to the candidate plan tree.

[0077] The processing device can obtain the execution cost corresponding to each candidate plan tree of the operation statement according to the principle described above. A candidate plan tree can have a corresponding execution cost. As the name implies, the execution cost corresponding to a candidate plan tree is the cost required to execute the candidate plan tree. The smaller the execution cost, the less the cost (such as computer resource cost, or time cost, etc.) required to execute the operation statement. Therefore, the processing device can select the candidate plan tree with the smallest execution cost from multiple candidate plan trees of the operation statement as the target plan tree of the operation statement. The target plan tree is the physical plan tree ultimately used to execute the operation statement. The target plan tree can be the optimal plan tree of the operation statement. The processing device can use the target plan tree to execute the operation statement. Executing the target plan tree is the final execution of the operation statement. For example, by executing the target plan tree, the corresponding operation of the operation statement on the data in the data table (such as query, add, delete, modify, etc.) can be realized.

[0078] The resource data consumed by each historical subtree in the data set can be mapped to the standard representation information of each historical subtree. The standard representation information of a historical subtree is the information obtained after the standard representation processing of the historical subtree. The specific process of obtaining the standard representation information of the historical subtree can be referred to as follows Figure 6 The corresponding embodiment provides a description of obtaining standard representation information of N subtrees.

[0079] Any subtree of the above-mentioned target plan tree can be called the second subtree. After executing the target plan tree, if the data set does not contain the standard representation information of the second subtree at this time (the standard representation information actually refers to the standard representation information of the historical subtree that has been historically executed and is similar to the second subtree), then the second subtree can be regarded as a new historical subtree, and the resource data actually consumed by the second subtree during the actual execution process can be newly added and added to the above-mentioned data set, as described below.

[0080] Therefore, after executing the target plan tree, if the data set does not contain the standard representation information of the second subtree, indicating that the data set does not yet contain the resource data actually consumed by the historical subtree similar to the second subtree that has been executed historically, then after executing the second subtree, the processing device can obtain the resource data actually consumed by the second subtree during the execution process. The processing device can construct a mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree, and can add and add the constructed mapping relationship to the data set, so that the data set can contain the resource data consumed by the second subtree during the actual execution process.

[0081] If the data set contains standard representation information of the second subtree (the standard representation information is actually the standard representation information of a historical subtree that has been historically executed and is similar to the second subtree), it indicates that the data set already contains the resource data actually consumed by the historical subtree that has been historically executed and is similar to the second subtree. Then, after executing the second subtree, the processing device does not need to repeatedly add the resource data actually consumed by the second subtree to the data set.

[0082] In one implementation, a method of determining whether the data set contains standard representation information of the second subtree may include: The data set may include M standard representation information of the M historical subtrees. The processing device may obtain the information similarity between the standard representation information of the second subtree and the M standard representation information, wherein the principle of obtaining the information similarity between the standard representation information of the second subtree and the M standard representation information is the same as the following Figure 6 The principle of obtaining the information similarity between the standard representation information of the first subtree and the M standard representation information in the corresponding embodiment is the same, and the details can be referred to as follows Figure 6 The second subtree may be the same as or different from the first subtree.

[0083] The processing device may obtain the standard representation information having the greatest information similarity with the standard representation information of the second subtree among the M standard representation information, and may refer to the standard representation information having the greatest information similarity with the standard representation information of the second subtree as reference representation information. That is, the reference representation information is the standard representation information having the greatest information similarity with the standard representation information of the second subtree among the M standard representation information.

[0084] If the information similarity between the standard representation information of the second subtree and the reference representation information is greater than or equal to the set similarity threshold, the standard representation information of the second subtree can be regarded as the same as the reference representation information, that is, the second subtree can be regarded as the same as the historical subtree to which the reference representation information belongs, and at this time, it can be considered that the data set contains the standard representation information of the second subtree. The similarity threshold can be the minimum information similarity set for evaluating whether the standard representation information is the same, and the specific value of the similarity threshold can be determined according to the actual application scenario.

[0085] If the information similarity between the standard representation information of the second subtree and the reference representation information is less than the above-mentioned similarity threshold, it can be regarded that the standard representation information of the second subtree is different from the reference representation information, that is, it can be regarded that the second subtree is different from the historical subtree to which the reference representation information belongs. At this time, it can be considered that the data set does not contain the standard representation information of the second subtree.

[0086] For example, in one embodiment, the process of the processing device obtaining the resource data actually consumed by the second subtree during execution may include: after executing the second subtree, the processing device may obtain the input and output data (i.e., IO data) of the last operator node in the second subtree, and may use the input and output data of the last operator node as the input and output data of the second subtree, and the input and output data may include the input data volume and output data volume of the last operator node, or may also include the number of logical interactions of the operator node (such as the number of logical data interactions). The last operator node may be the bottom operator node in the second subtree. In one embodiment, the input and output data of the subtree may include the input and output data of the subtree for the disk, and the input and output data of the subtree for the network (Net), wherein the input and output data of the subtree for the disk may represent the number of logical interactions of the subtree on the disk, and the input and output data of the subtree for the network may represent the number of logical interactions of the subtree on the network and the amount of data transmission (such as the amount of input data and the amount of output data).

[0087] The processing device may also obtain the memory data consumed by each operator node in the second subtree during operation. The memory data consumed by an operator node may be the memory data occupied by the operator node during operation. The memory data may be the memory size occupied by the operator node during operation. For example, the unit of the memory data may be GB (gigabyte), etc. The processing device may use the memory data consumed most by the operator nodes in the second subtree as the memory data consumed by the second subtree. The memory data consumed most by the operator nodes in the second subtree may be the memory data consumed by the operator node with the most memory data consumed in the second subtree, that is, the memory data is the peak value of the memory data consumed by each operator node in the second subtree.

[0088] Furthermore, the processing device may obtain the processor data (which may be referred to as CPU data) consumed by each operator node in the second subtree. The processor data consumed by an operator node may be the length of time that the operator node occupies the CPU (processor) during operation. The processing device may use the processor data consumed most by the operator nodes in the second subtree as the processor data consumed by the second subtree, wherein the processor data consumed most by the operator nodes in the second subtree may be the processor data consumed by the operator node with the most (i.e., the longest) processor data consumed in the second subtree, and the processor data may be the peak value of the processor data consumed by each operator node in the second subtree.

[0089] The processing device may regard the input and output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree as the resource data consumed by the second subtree, that is, the resource data of the second subtree may include the input and output data of the second subtree, the memory data consumed by the second subtree, and the processor data consumed by the second subtree.

[0090] Furthermore, the present application can also diagnose the operator nodes contained in the operation statements, such as the input data volume, input record number, output data volume, and output record number of the operator nodes to determine whether the operator nodes have unreasonable settings or abnormalities. For example, if the difference between the input data volume and the output data volume of the operator node is too large (such as greater than a certain data volume threshold), it can be considered that the operator node is unreasonable, and diagnostic information for the operator node can be generated. The diagnostic information can also include directions and suggestions for optimizing the operator node. Please refer to Figure 5 , Figure 5 is a schematic diagram of a diagnostic interface for an operator node provided in an embodiment of the present application. Figure 5 As shown, the diagnostic interface may include the operator name "SortMergeJoin", input data volume "57245 bytes", input record number "30995894 rows", output data volume "57245 bytes", output record number "5 rows", and diagnostic information of the operator node. The diagnostic information may include the large table pre-position "In this Join stage (table connection stage), the right table input is large and is suitable for pre-position as a Base table (base table), the left table input: 5 rows, the right table input: 30995889 rows." and the MapJoin diagnosis "The left table is suitable for MapJoin (an optimization technology)".

[0091] The specific logic for diagnosing the operator node (which may be referred to as diagnostic logic or detection logic) may be flexibly set according to actual application requirements.

[0092] In this application, by using a similar matching mechanism between subtrees, matching resource data is screened for subtrees in the data set, which can provide decisions for the root cause analysis when predicting the cost of the physical plan tree, thereby achieving accurate and rapid cost estimation of the physical plan tree. In addition, in this application, by matching the subtrees through the physical plan tree produced by the logical plan tree, the information between the logical plan tree and the physical plan tree can be pulled together, and the matching of the subtrees of the physical plan tree can be understood as achieving the matching of the logical plan tree.

[0093] The present application can obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; and can obtain a data set, which includes resource data consumed by M historical subtrees during historical execution, where the historical subtree is a subtree of a historical plan tree that has been historically executed, and M is a positive integer; and, can perform a first matching process on the resource data of the N subtrees in the data set, and the first matching process is used to filter matching resource data for the N subtrees through the similarity between the N subtrees and the M historical subtrees; thereby, the reference resource data of the N subtrees can be determined based on the first matching result obtained by the first matching process; and the reference resource data of the N subtrees can be used to estimate the execution cost corresponding to the candidate plan tree. It can be seen that the method proposed in the present application can obtain the reference resource data of each subtree in the candidate plan tree with the subtree as the granularity, and the reference resource data can be matched from the resource data actually consumed by the historical subtree in the actual historical execution process. Therefore, the accuracy and reliability of the reference resource data obtained for each subtree are also guaranteed. Therefore, through the accurate and reliable reference resource data of each subtree, the execution cost corresponding to the candidate plan tree can also be accurately estimated.

[0094] See also Figure 6 , Figure 6 1 is a flow chart of performing a first matching process on resource data of N subtrees in a data set provided by an embodiment of the present application. The embodiment of the present application specifically describes the process of performing a first matching process on resource data of N subtrees in a data set to obtain a first matching result. Figure 6 As shown, the process may include: Step S201 , performing standardized representation processing on each of the N subtrees to generate standardized representation information for each subtree.

[0095] Specifically, the processing device may perform standardized representation processing (also referred to as normalization processing) on ​​each of the N subtrees of the candidate plan tree to generate standard representation information for each subtree. One subtree may have one standard representation information.

[0096] Any one of the N subtrees can be called the first subtree, and the following is a specific description of performing standardized representation processing on the first subtree to generate standard representation information of the first subtree as an example. It should be noted that the principles of performing standardized representation processing on each of the N subtrees to generate standard representation information for each subtree are independent and the same.

[0097] The processing device may perform data parsing processing on the first subtree from the K data domains to obtain representation data of the first subtree under the K data domains, where K is a positive integer and the K data domains are preset domains for performing standardized representation of the subtrees. If the first subtree does not parse out corresponding representation information under a data domain in the K data domains, the representation information of the first subtree under the data domain may be represented as null.

[0098] The processing device may perform representation processing on the representation data of the first subtree under the K data domains according to a set standard format to generate standard representation information of the first subtree. The standard representation information of the first subtree may include data obtained by performing representation processing on the representation data of the first subtree under the K data domains according to the standard format, and the standard representation information of the first subtree may be understood as the representation of the first subtree according to a unified standard.

[0099] The K data domains may include at least one of the following: an operator domain corresponding to an operator called by a subtree, an object domain corresponding to a data object in table data called by a subtree, and a condition domain corresponding to a condition contained in an operator called by a subtree. That is, the standardized representation processing of a subtree may include an abstraction on the K data domains.

[0100] The operator called by the subtree can be the operator contained in the subtree, such as TableScan operator, Filter operator, Project operator, etc. All operators in the operator domain can belong to the elements in the operator domain.

[0101] The table data called by the subtree in the data table can be the data called by the subtree in the data table. The table data can be the entire data table or one or more partitions in the data table. The table data can contain one or more data objects (i.e., Objects). A data object can be an entity in the data table. For example, a data object can be an employee, a product, or a customer. A data object can usually correspond to a row of data in the data table, and the row of data can record the object attributes of the data object. All data objects under the object domain can belong to the elements under the object domain.

[0102] The conditions contained in the operator called by the subtree can be the conditions set in the operator. For example, for the Filter operator: Filter (gender='male'and age>18), the condition contained in the Filter operator is "gender='male'and age>18", which means that the gender is female and the age is greater than 18. All conditions under the condition domain can belong to each element under the condition domain.

[0103] Therefore, the representation data of the first subtree under the operator domain can include each operator called in the first subtree, and each operator can be referred to as each element of the representation data of the first subtree under the operator domain, and each element belongs to the element under the operator domain. The representation data of the first subtree under the object domain can include each data object in the table data called by the first subtree. Similarly, each data object can be referred to as each element of the representation data of the first subtree under the object domain, and each element belongs to the element under the object domain. The representation information of the first subtree under the condition domain can include each condition contained in the operator called in the first subtree, and each condition can also be referred to as each element of the representation data of the first subtree under the condition domain, and each element belongs to the element under the condition domain.

[0104] The standard format may include the arrangement order and writing format (such as unified uppercase and lowercase letters, unified font type, etc.) of the elements in each data field (here, it may refer to all elements in each data field) set by the unified standard. That is, the elements in the same data field may have a preset and unified arrangement order and a unified writing format.

[0105] Therefore, the above-mentioned representation processing of the representation data of the first subtree under K data domains according to the set standard format may refer to representing the representation data of the first subtree under each data domain according to the uniform arrangement order and uniform writing format of the elements under each data domain, and the representation data under each data domain may also have a fixed representation order. For example, the elements of the first subtree under each data domain may be represented in the order of the operator domain, the object domain, and the condition domain in turn according to the uniform arrangement order and writing format, so as to generate and obtain the standard representation information of the first subtree.

[0106] The processing device can obtain the standard representation information of each of the N subtrees according to the principle of obtaining the standard representation information of the first subtree. The above-mentioned standardized representation processing can also be called normalization processing, which is used to represent each subtree in a unified format. The standard representation information obtained by performing standardized representation processing on the subtree can be called the signature of the subtree, which is used to identify the subtree.

[0107] Furthermore, in one implementation, since the resource data in the data set may be collected through multiple SQL computing engines, and the same operator may be expressed differently in different SQL computing engines, before the application performs standardized representation processing on the subtree, each operator node in the subtree may also be standardized (i.e., normalized), such as the physical execution operators (i.e., operators in the physical plan tree) in different SQL computing engines may be uniformly normalized to relational algebra operators (which may be called standard format operators), that is, the "relational algebra" theory may be used as the standard for normalization. For example, operator nodes such as BroadCastJoin (broadcast join operator), SortMergeJoin (sort merge join), and HashJoin (hash join operator) may be uniformly normalized to Join (join operator), and Scan operators (a type of scan operator) of various SQL computing engines may be uniformly classified as projection operators, etc.

[0108] In other words, the N subtrees mentioned above in the present application can be obtained by uniformly representing the operator nodes in the N initial subtrees into which the candidate plan tree is divided as relational algebra operators. The present application can be to uniformly represent the operator nodes in the subtrees into which the physical plan tree is divided as relational algebra operators, and then perform standardized representation processing to generate standard representation information of the subtrees. In this way, the accuracy of the standardized representation processing of the subtrees can be guaranteed, and more accurate standard representation information of the subtrees can be obtained, thereby ensuring the accuracy of matching the resource data in the data set.

[0109] In the present application, by representing each subtree as its own standard representation information, the problem of mistaking the subtrees for different representations for the same subtree can be eliminated, so that the processing device can subsequently perform a more accurate assessment of the similarity between the subtrees.

[0110] See also Figure 7 , Figure 7 Schematic diagram of a framework for collecting resource data consumed by a historical subtree provided by an embodiment of the present application. Figure 7As shown, the present application can collect resource data consumed by historical subtrees through a collector (which can also be called a data collector, which can belong to an SDK, a software package). The resource database here is the above-mentioned data set, which can contain the resource data collected for the historical subtrees. In the present application, the collector can parse and obtain the resource data actually consumed by each subtree in the physical plan tree currently (with very small delay, which can be regarded as real-time) executed by the SQL computing engine (there can be one or more SQL computing engines, and multiple SQL computing engines can be distributed) in a streaming manner through a real-time running entrance. The executed physical plan tree can be used as the above-mentioned historical plan tree, and the subtree of the historical plan tree is the historical subtree. The collector can store the mapping relationship between the collected resource data actually consumed by each subtree in the physical plan tree and the standard representation information of each subtree in the resource database.

[0111] Furthermore, the collector can also parse and obtain the resource data actually consumed by each subtree in the physical plan tree executed by the SQL computing engine before (such as a long time ago, such as a specified time ago) in batches through an offline operation portal. The executed physical plan tree can also be used as the above-mentioned historical plan tree, and the subtree of the historical plan tree is the historical subtree. The collector can store the mapping relationship between the collected resource data actually consumed by each subtree in the physical plan tree and the standard representation information of each subtree in the resource database.

[0112] In one embodiment, the collector can obtain the metrics data (indicator data used to measure or analyze data) when the SQL statement is running in real time or offline. The metrics data may include the resource data consumed by each operator node in the physical plan tree of the SQL statement during execution. The collector can obtain the resource data consumed by each subtree of the physical plan tree through the resource data consumed by each operator node during execution. For example, the SQL computing engine may include Spark (a fast and general big data processing engine), Presto (a distributed SQL query engine), Starrocks (a distributed SQL database for online analytical processing), etc.

[0113] Through the above process, the present application can collect resource data of each subtree of the physical plan tree executed by each SQL computing engine in parallel and synchronously through the two collection methods of real-time collection and offline collection, wherein real-time collection can be used as the main collection method, and offline collection can be used as the auxiliary collection method. The offline collection method can solve the problem of not having enough time to collect a large amount of resource data consumed by historical subtrees during real-time collection. The present application can efficiently and fully collect the resource data actually consumed by each subtree of the physical plan tree executed historically by adopting the two collection methods of real-time collection and offline collection in parallel, which is used to estimate the corresponding execution cost of other subsequent physical plan trees.

[0114] Step S202: Based on the standard representation information of the N subtrees, a first matching process of resource data is performed on the N subtrees in the data set.

[0115] Specifically, the processing device may perform a first matching process on the resource data of the N subtrees in the data set based on the standard representation information obtained for the N subtrees, thereby obtaining a first matching result, as described below.

[0116] The resource data consumed by each historical subtree in the above-mentioned data set may respectively have a mapping relationship with the standard representation information of each historical subtree, that is, the data set may store the mapping relationship between the above-mentioned M historical subtrees and their respective standard representation information. The data set may not directly include the historical subtrees, but may refer to the historical subtrees through the standard representation information of the historical subtrees. The data set may include the standard representation information of each historical subtree.

[0117] Among them, since the standard representation information of the subtree is obtained by standardizing the representation of the subtree without modifying the actual operation logic contained in the subtree, the standard representation information of the subtree can also well refer to and represent the subtree. Therefore, the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees can be used to reflect (i.e., embody) the similarity between the N subtrees and the M historical subtrees. In short, the similarity between the standard representation information of the subtrees can be used to reflect the similarity between the subtrees.

[0118] The first matching process may include matching process of resource data of each of the N subtrees in the data set, and the principle of matching process of resource data of each subtree is independent and the same. Therefore, any one of the N subtrees may be the first subtree, and the following is a specific description of the process of matching process of resource data of the first subtree in the data set as an example.

[0119] The data set may include M standard representation information of the above-mentioned M historical subtrees, and one historical subtree has one standard representation information. The processing device may obtain the similarity (which may be referred to as information similarity) between the standard representation information of the first subtree and the M standard representation information respectively, and the standard representation information of the first subtree may have one information similarity with one standard representation information of the M standard representation information.

[0120] In one embodiment, the processing device may obtain the information similarity between the standard representation information of the first subtree and the M standard representation information respectively, which may include: the processing device may adopt the jaccard algorithm (an algorithm for evaluating the similarity between two sets) to obtain the information similarity between the standard representation information of the first subtree and the M standard representation information. In this process, a standard representation information may be represented as a corresponding set, and a character in the standard representation information may be used as an element in the corresponding set, that is, a set corresponding to a standard representation information is composed of each character in the standard representation information. Therefore, the processing device may represent the standard representation information of the first subtree and the M standard representation information as corresponding sets, so that the set similarity between the sets corresponding to the standard representation information of the first subtree and the sets corresponding to the M standard representation information may be calculated by the jaccard algorithm, and the set similarity may be used as the information similarity between the standard representation information of the first subtree and the M standard representation information respectively.

[0121] In another embodiment, the processing device obtains the information similarity between the standard representation information of the first subtree and the M standard representation information respectively, which may also include: the processing device may convert the standard representation information of the first subtree and the M standard representation information into corresponding feature vectors, one standard representation information may be converted into a corresponding feature vector, the feature vector may be generated by feature embedding processing of the standard representation information, and the feature vector may be used to characterize the features of the standard representation information. Thus, the processing device may calculate the vector similarity (which may be cosine similarity) between the feature vectors corresponding to the standard representation information of the first subtree and the feature vectors corresponding to the M standard representation information respectively, and use the vector similarity as the information similarity between the standard representation information of the first subtree and the M standard representation information respectively.

[0122] For example, the processing device converts the standard representation information into the corresponding feature vector in a manner that may include: the processing device may obtain a feature embedding model, the standard representation information may be text information, and therefore, the feature embedding model may be a text feature embedding model, and the text feature embedding model may be a trained model that can be used to generate a feature vector of text. Therefore, the processing device may call the text feature embedding model to perform feature embedding processing (also referred to as feature extraction processing) on ​​the standard representation information, and thus generate a feature vector corresponding to the standard representation information.

[0123] The processing device may obtain the standard representation information with the greatest information similarity to the standard representation information of the first subtree among the M standard representation information, and refer to the standard representation information with the greatest information similarity to the standard representation information of the first subtree as pending representation information. That is, the pending representation information is the standard representation information with the greatest information similarity to the standard representation information of the first subtree among the M standard representation information.

[0124] If the information similarity between the standard representation information of the first subtree and the pending representation information is greater than or equal to the set similarity threshold, the standard representation information of the first subtree can be regarded as the same as the pending representation information, that is, the first subtree can be regarded as the same as the historical subtree to which the pending representation information belongs. At this time, the processing device can use the resource data in the data set that has a mapping relationship with the pending representation information as the filtered resource data that matches the first subtree. The first matching result can include the filtered resource data that matches the first subtree. The similarity threshold can be the minimum information similarity set for evaluating whether the standard representation information is the same. The specific value of the similarity threshold can be determined according to the actual application scenario.

[0125] If the information similarity between the standard representation information of the first subtree and the pending representation information is less than the set similarity threshold, it can be regarded that the standard representation information of the first subtree is different from the pending representation information, that is, it can be regarded that the first subtree is different from the historical subtree to which the pending representation information belongs. At this time, the processing device can determine that resource data matching the first subtree has not been screened out in the data set. The first matching result can include resource data matching the first subtree that has not been screened out in the data set.

[0126] The processing device can perform resource data matching processing on the above-mentioned N subtrees respectively according to the same principle as the above-mentioned resource data matching processing on the first subtree, so as to obtain a matching result for each subtree. The matching result of a subtree can be the resource data that is screened out to match the subtree, or the resource data that is not screened out to match the subtree. The first matching result can include the matching results for each of the N subtrees.

[0127] See also Figure 8 , Figure 8 Schematic diagram of a framework for executing an operation statement provided in an embodiment of the present application. Figure 8 As shown, the data collector can collect subtree operation data, which is the resource data actually consumed by the historical subtree during the historical execution process, and can be included in the above data set. The data collector can also obtain metadata and statistical information of the data table. The metadata of the data table can define each partition of the data table, and the statistical information of the data table can include the data volume of the data table and the data volume of each partition in the data table.

[0128] In the present application, when a data table has not been run, the metadata and statistical information of the data table can also be used to derive the operator associated with the partition in the data table (which can be called a derived operator). For example, the processing device can obtain one or more operator types that can operate on the branches of the data table (such as scanning operator types, projection operator types, etc.), so that the one or more operator types can be combined with the partition to generate the derived operator associated with the partition. For example, the derived operator can be a scanning operator or a projection operator for the associated partition. The processing device can also obtain the resource data consumed by the derived operator. Since the partition associated with the derived operator has not been run, the resource data consumed by the derived operator can include the input and output data of the derived operator at this time. The input data volume in the input and output data can be the data volume of the data table, and the output data volume in the input and output data can be the data volume of the partition associated with the derived operator. Therefore, the present application can generate standard representation information of the derivative operator, and add the mapping relationship between the standard representation information of the derivative operator and the resource data consumed by the derivative operator to the above data set. Like the resource data consumed by the historical subtree, it can also be used to screen resource data that matches the subsequent subtrees.

[0129] Furthermore, if the derivative operator is actually executed later, the resource data consumed by the derivative operator during the actual execution process can be added to the data table. Since the data table already contains the input and output data of the derivative operator, the resource data added here can include the memory data and processor data actually consumed during the execution of the derivative operator.

[0130] See also Figure 8The SQL computing engine in the SQL computing engine may include a SQL engine connector (for connecting to a computing engine), a parser (for parsing SQL statements), an optimizer (for optimizing SQL statements, such as optimizing Join (a table connection) to Mapjoin (an optimization technology), an executor (for executing SQL statements), and an algorithm service connector. The algorithm service connector may be newly added in the present application, and is used to connect to an algorithm service (which may be an algorithm provided in the present application for screening resource data matching a subtree, and the principle may be as described above in the embodiment of the present application), so as to screen resource data matching a subtree of a physical plan tree of a SQL statement currently to be executed from a data set through the algorithm service. The processing device may input an SQL statement (such as the above-mentioned operation statement) into the SQL computing engine, and may gradually process the input SQL statement through the parser, optimizer, and executor in the SQL computing engine to obtain a final processing result (such as the processing result may include data queried from a data table), and the processing device may store the processing result (corresponding to Figure 8 "Data Storage" in the table) for subsequent display to users.

[0131] See also Fig. 9 , Fig. 9 Schematic diagram of a scenario for resource data matching provided by an embodiment of the present application. Fig. 9 As shown, the data collector can read the resource data actually consumed by the executed historical plan tree during the execution process from the SQL calculation engine. Through the resource data, the resource data consumed by each operator node in the historical plan tree can be obtained. Thus, the resource data consumed by each historical subtree of the historical plan tree can be obtained through the resource data consumed by each operator node. The data collector can map the resource data consumed by each historical subtree to the data set through the standard representation information of each historical subtree.

[0132] Subsequently, the data collector can obtain a new subtree signature (i.e., the standard representation information of the subtree, which may be the standard representation information of the N subtrees mentioned above). The data collector can use the new subtree signature through a similarity algorithm (such as a similarity algorithm between standard representation information) to match and recall subtrees in the data set (such as the first matching process mentioned above) to screen resource data that matches the N subtrees in order to estimate the execution cost corresponding to the candidate plan tree.

[0133] The method provided in this application converts the subtree of the physical plan tree of the SQL statement into standard representation information for representation, establishes the correlation between SQL statements, and pulls the subtree data (such as the resource data consumed by the subtree) between different SQL statements. The subtree data supports the cost optimization and estimation of SQL in the CBO model, and realizes the accurate evaluation of the similarity between subtrees, so that the resource data matching the subtree can be accurately screened out from the data set. In addition, the above method of this application supports the optimization of the Mapjoin operator in the CBO optimization of the internal SuperSQL (an SQL computing platform), and in the test verification of Presto (an open source distributed SQL query engine) and TPC-DS (a decision support system benchmark), 99 SQL statements covering the standard test set (such as the subtrees of the physical plan tree of the 99 SQL statements have screened out matching resource data), all of which have achieved excellent test results.

[0134] See also Fig.10 , Fig.10 1 is a flow chart of determining the reference resource data of N subtrees through the first matching result provided by an embodiment of the present application. Since the principle of obtaining the reference resource data of each subtree in the N subtrees through the first matching result is the same, the following is still a specific description of the process of obtaining the reference resource data of the first subtree as an example. Fig.10 As shown, the process may include: Step S301: If the first matching result includes the filtered resource data that matches the first subtree, the filtered resource data that matches the first subtree is used as reference resource data of the first subtree.

[0135] Specifically, if the first matching result includes the filtered resource data that matches the first subtree, the processing device may directly use the filtered resource data that matches the first subtree as the reference resource data of the first subtree.

[0136] Step S302: If the first matching result includes resource data that is not matched with the first subtree, then the reference resource data of the first subtree is determined based on the number of operator nodes included in the first subtree.

[0137] Specifically, the above-mentioned N subtrees can all be composed of one or more operator nodes of the candidate plan tree. Therefore, if the first matching result includes resource data that is not filtered out and matches the first subtree, the processing device can determine the reference resource data of the first subtree by the number of operator nodes contained in the first subtree, as described below.

[0138] When the first subtree contains an operator node of the candidate plan tree (i.e., the first subtree contains an operator node), the processing device can obtain the resource data consumed by the table data called by the first subtree, and can use the resource data consumed by the table data called by the first subtree as the reference resource data of the first subtree. The table data called by the first subtree can be the entire data table or at least one partition in the data table, and the resource data consumed by the data table or the resource data consumed by each partition in the data table can be obtained by setting the data table and each partition in the data table in advance. For example, the resource data consumed by the data table / partition in the data table can be obtained by analyzing and counting the data attributes such as the data scale, data distribution, and field type of the data contained in the data table / partition in the data table in advance. The business logic of the specific analysis and statistics can be configured by the developer according to the actual application scenario.

[0139] And when the first subtree contains multiple operator nodes of the candidate plan tree (that is, when the first subtree contains multiple operator nodes), the processing device can perform a second matching process on the resource data of the first subtree in the data set, and the matching result obtained by the second matching process can be referred to as the second matching result. The processing device can determine the reference resource data of each operator node in the first subtree through the second matching result obtained by the second matching process. In this case, the reference resource data of the first subtree can include the reference resource data of each operator node in the first subtree. The second matching process is used to screen matching resource data for each operator node in the first subtree in the data set through the similarity between each operator node in the first subtree and the M historical subtrees.

[0140] The principle of the second matching process is actually the same as the principle of the first matching process, that is, the principle of selecting matching resource data for each operator node in the first subtree in the data set by the similarity between each operator node in the first subtree and the M historical subtrees is the same as the principle of selecting matching resource data for the N subtrees in the data set by the similarity between the N subtrees and the M historical subtrees, except that the second matching process replaces the N subtrees with the operator nodes in the first subtree compared to the first matching process. Each operator node in the first subtree can also be regarded as a corresponding subtree, but here a subtree can only contain one operator node, that is, an operator node in the first subtree can be regarded as a subtree containing only the operator node. Therefore, according to the same principle as above, by using the similarity between the N subtrees and the M historical subtrees to filter matching resource data for the N subtrees in the data set, by using the similarity between the multiple subtrees of the first subtree (i.e., the multiple operator nodes contained in the first subtree) and the M historical subtrees, matching resource data can be filtered in the data set for the multiple subtrees to obtain a second matching result.

[0141] The second matching result may also include a matching result for each operator node in the first subtree. The matching result for an operator node may be resource data that is screened out to match the operator node, or resource data that is not screened out to match the operator node.

[0142] Similarly, any operator node in the first subtree can be called a target operator node. Since the principle of determining the reference resource data of each operator node in the first subtree is the same, the following is a specific explanation using the process of determining the reference resource data of the target operator node as an example, as described below.

[0143] If the second matching result includes the filtered resource data that matches the target operator node, the filtered resource data that matches the target operator node can be directly used as the reference resource data of the target operator node. In this case, it can be considered that the historical subtree to which the filtered resource data that matches the target operator node belongs is the same as the target operator node.

[0144] If the second matching result includes resource data that is not matched with the target operator node, the processing device may obtain the resource data consumed by the table data called by the target operator node, and may use the resource data consumed by the table data called by the target operator node as reference resource data of the target operator node. Similarly, the table data called by the target operator node may also be the entire data table or a partition in the data table, and the resource data consumed by the data table or the partition in the data table may also be pre-set.

[0145] Among them, when the first subtree contains multiple operator nodes of the candidate plan tree, and the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, the cost estimation model can also follow the original principle. In the first subtree, the execution cost corresponding to the current operator node is estimated by the execution cost corresponding to the previous operator node of the current operator node and the reference resource data of the current operator node, so that the estimated execution cost of the top operator node in the first subtree is used as the execution cost of the first subtree, so as to estimate the corresponding execution cost of subsequent subtrees (such as the next subtree of the first subtree).

[0146] See also Fig.11 , Fig.11 It is a structural diagram of a subtree structure of a first subtree provided in an embodiment of the present application. Assume that the first subtree contains operator node 1 "Project (id, name)" (a projection operator), operator node 2 "Filter (gender = male and age = 18)" (a filtering operator) and operator node 3 "TableScan (student)" (a scanning operator). Since no matching resource data is screened out for the first subtree, the execution cost can be estimated at the granularity of the operator node within the first subtree. For example, the processing device can estimate the execution cost corresponding to the child node 3 through the reference resource data of the operator node 3, and can estimate the execution cost corresponding to the child node 2 through the execution cost corresponding to the operator node 3 and the reference resource data of the operator node 2, and can estimate the execution cost corresponding to the child node 1 through the execution cost corresponding to the operator node 2 and the reference resource data of the operator node 1. Therefore, the execution cost corresponding to the operator node 1 can be used as the estimated execution cost of the first subtree.

[0147] Through the above method of the present application, the resource data of the subtree can be accurately collected and obtained at the granularity of the subtree (including the granularity of the operator node, which is a small granularity), rather than obtaining resource data according to the granularity of the data table or the partition of the data table (which is a large granularity), so that the execution cost corresponding to the subtree can be accurately estimated according to the granularity of the subtree, and finally the estimation of the corresponding execution cost of the entire candidate plan tree is realized, which improves the accuracy of the estimation of the corresponding execution cost of the candidate plan tree. And by obtaining the reference resource data of the subtree through the actual consumption resource data collected from the historical subtree, it also ensures the convenience and reliability of obtaining the reference resource data of the subtree, greatly improving the performance of executing SQL statements.

[0148] See also Fig.12 , Fig.12Schematic diagram of a data processing device provided in an embodiment of the present application. Fig.12 As shown, the data processing device 120 may include: a first acquisition module 1201 , a second acquisition module 1202 , a matching module 1203 , a determination module 1204 and an estimation module 1205 .

[0149] The first acquisition module 1201 is used to acquire a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; The second acquisition module 1202 is used to acquire a data set, wherein the data set includes resource data consumed by M historical subtrees during historical execution, where the historical subtree is a subtree of the historical plan tree that has been historically executed, and M is a positive integer; A matching module 1203 is used to perform a first matching process of resource data on the N subtrees in the data set, wherein the first matching process is used to select matching resource data for the N subtrees based on the similarity between the N subtrees and the M historical subtrees; A determination module 1204, configured to determine reference resource data of N subtrees based on a first matching result obtained by the first matching process; The estimation module 1205 is used to estimate the execution cost corresponding to the candidate plan tree using the reference resource data of the N subtrees.

[0150] In one implementation, the matching module 1203 performs a first matching process on resource data of N subtrees in the data set, including: Performing standardized representation processing on each of the N subtrees to generate standard representation information for each subtree; Based on the standard representation information of the N subtrees, performing a first matching process of resource data on the N subtrees in the data set; Among them, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree; the similarity between the standard representation information of N subtrees and the standard representation information of M historical subtrees is used to reflect the similarity between N subtrees and M historical subtrees.

[0151] In one embodiment, any one of the N subtrees is a first subtree, and the data set includes M standard representation information of the M historical subtrees; The matching module 1203 performs a first matching process on the resource data of the N subtrees in the data set based on the standard representation information of the N subtrees, including: Obtaining information similarities between the standard representation information of the first subtree and the M standard representation information respectively; The standard representation information having the greatest information similarity with the standard representation information of the first subtree among the M standard representation information is determined as the pending representation information; If the information similarity between the standard representation information of the first subtree and the pending representation information is greater than or equal to the set similarity threshold, the resource data in the data set that has a mapping relationship with the pending representation information is used as the selected resource data that matches the first subtree; The first matching result includes the filtered resource data that matches the first subtree.

[0152] In one implementation, the matching module 1203 is further configured to: If the information similarity between the standard representation information of the first subtree and the pending representation information is less than the similarity threshold, it is determined that resource data matching the first subtree is not screened out in the data set; The first matching result includes resource data that is not matched with the first subtree in the data set.

[0153] In one implementation, each of the N subtrees is composed of one or more operator nodes of the candidate plan tree; the determination module 1204 determines the reference resource data of the N subtrees based on the first matching result obtained by the first matching process, including: If the first matching result includes the filtered resource data that matches the first subtree, the filtered resource data that matches the first subtree is used as reference resource data for the first subtree; If the first matching result includes resource data that is not screened out and matches the first subtree, the reference resource data of the first subtree is determined based on the number of operator nodes included in the first subtree.

[0154] In one implementation, the determination module 1204 determines the reference resource data of the first subtree based on the number of operator nodes included in the first subtree, including: When the first subtree includes an operator node of the candidate plan tree, resource data consumed by the table data called by the first subtree is used as reference resource data of the first subtree; When the first subtree contains multiple operator nodes of the candidate plan tree, a second matching process of resource data is performed on the first subtree in the data set, and based on a second matching result obtained by the second matching process, the reference resource data of each operator node in the first subtree is determined; wherein the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, and the second matching process is used to screen matching resource data for each operator node in the first subtree through the similarity between each operator node in the first subtree and the M historical subtrees.

[0155] In one implementation, any operator node in the first subtree is a target operator node; the determination module 1204 determines the reference resource data of each operator node in the first subtree based on the second matching result obtained by the second matching process, including: If the second matching result includes the filtered resource data that matches the target operator node, the filtered resource data that matches the target operator node is used as the reference resource data of the target operator node; If the second matching result includes resource data that is not screened out and matches the target operator node, the resource data consumed by the table data called by the target operator node is used as reference resource data of the target operator node.

[0156] In one implementation, any one of the N subtrees is the first subtree; the matching module 1203 performs standardized representation processing on each of the N subtrees to generate standard representation information of each subtree, including: Perform data parsing processing on the first subtree from the K data domains to obtain the representation data of the first subtree under the K data domains; K is a positive integer; Perform representation processing on the representation data of the first subtree under the K data domains according to a set standard format to generate standard representation information of the first subtree; The K data domains include at least one of the following: an operator domain corresponding to the operator called by the subtree, an object domain corresponding to the data object in the table data called by the subtree, and a condition domain corresponding to the condition contained in the operator called by the subtree.

[0157] In one implementation, the process of the first acquisition module 1201 acquiring N subtrees of the candidate plan tree includes: Identify a branch node in the candidate plan tree, and divide each branch under the branch node into a first type subtree of the candidate plan tree; Through each operator node except the first type subtree in the candidate plan tree, a second type subtree of the candidate plan tree is constructed respectively, and an operator node except the first type subtree in the candidate plan tree is used to construct a second type subtree; The first type of subtree and the second type of subtree are used as N subtrees obtained by partitioning the candidate plan tree.

[0158] In one implementation, there are multiple candidate plan trees for the operation statement, and the data processing device further includes an execution module 1206, which is used to: Select a candidate plan tree with the minimum execution cost from multiple candidate plan trees as the target plan tree of the operation statement; Execute operation statements using the target plan tree.

[0159] In one embodiment, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree, and any subtree of the target plan tree is the second subtree; After executing the target plan tree, the execution module 1206 is further used to: If the data set does not contain standard representation information of the second subtree, after executing the second subtree, obtaining resource data consumed by the second subtree during execution; A mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree is constructed, and the constructed mapping relationship is added to the data set.

[0160] In one implementation, the execution module 1206 acquires resource data consumed by the second subtree during execution, including: Obtain the input and output data of the last operator node in the second subtree, and use the input and output data of the last operator node as the input and output data of the second subtree; Obtain the memory data consumed by each operator node in the second subtree, and use the memory data consumed most by the operator nodes in the second subtree as the memory data consumed by the second subtree; Obtain the processor data consumed by each operator node in the second subtree, and use the processor data consumed most by the operator nodes in the second subtree as the processor data consumed by the second subtree; The resource data consumed by the second subtree includes input and output data of the second subtree, memory data consumed by the second subtree, and processor data consumed by the second subtree.

[0161] In one implementation, the process of the first acquisition module 1201 acquiring a candidate plan tree of an operation statement includes: Parse the operation statements and generate a logical plan tree for the operation statements; The logical plan tree is converted to generate multiple candidate plan trees for the operation statements.

[0162] In one implementation, the estimation module 1205 uses the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree, including: Input the reference resource data of N subtrees into the cost estimation model; The call cost estimation model estimates the execution cost corresponding to the candidate plan tree based on the reference resource data of N subtrees.

[0163] According to one embodiment of the present application, Figure 3 The steps involved in the data processing method shown can be represented by Fig.12 The data processing device 120 shown in FIG. 1 is executed by each module in the data processing device 120. For example, Figure 3 The step S101 shown in FIG. 1 can be performed by Fig.12 The first acquisition module 1201 in is executed, Figure 3 The step S102 shown in FIG. 1 can be performed by Fig.12 The second acquisition module 1202 is executed; Figure 3 The step S103 shown in FIG. 1 can be performed by Fig.12 The matching module 1203 is used to execute, Figure 3 The step S104 shown in FIG. 1 can be performed by Fig.12 The determination module 1204 is executed, Figure 3 The step S105 shown in FIG. 1 can be performed by Fig.12 The estimation module 1205 in is used for execution.

[0164] The present application can obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; and can obtain a data set, which includes resource data consumed by M historical subtrees during historical execution, where the historical subtree is a subtree of a historical plan tree that has been historically executed, and M is a positive integer; and, can perform a first matching process on the resource data of the N subtrees in the data set, and the first matching process is used to filter matching resource data for the N subtrees through the similarity between the N subtrees and the M historical subtrees; thereby, the reference resource data of the N subtrees can be determined based on the first matching result obtained by the first matching process; and the reference resource data of the N subtrees can be used to estimate the execution cost corresponding to the candidate plan tree. It can be seen that the device proposed in the present application can obtain the reference resource data of each subtree in the candidate plan tree with the subtree as the granularity, and the reference resource data can be matched from the resource data actually consumed by the historical subtree in the actual historical execution process. Therefore, the accuracy and reliability of the reference resource data obtained for each subtree are also guaranteed. Therefore, through the accurate and reliable reference resource data of each subtree, the execution cost corresponding to the candidate plan tree can also be accurately estimated.

[0165] According to one embodiment of the present application, Fig.12 The various modules in the data processing device 120 shown can be separately or all combined into one or several units to constitute, or one (some) of the units can be further divided into multiple functionally smaller sub-units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of one module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the data processing device 120 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0166] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0167] According to one embodiment of the present application, a computer program capable of executing the steps involved in the corresponding methods shown in the embodiments of the present application can be run on a general-purpose computer device (the computer device may include processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM)) to construct the following. Fig.12 The data processing device 120 shown in FIG. The computer program may be recorded on a computer-readable recording medium, and may be loaded into the computer device through the computer-readable recording medium and run therein.

[0168] See also Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, in some embodiments, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Fig.13 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0169] exist Fig.13In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve: Obtain a candidate plan tree of the operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; Obtain a data set, which includes resource data consumed by M historical subtrees during historical execution, where the historical subtree is a subtree of the historical plan tree that has been historically executed, and M is a positive integer; Performing a first matching process of resource data on the N subtrees in the data set, the first matching process being used to screen matching resource data for the N subtrees based on similarities between the N subtrees and the M historical subtrees; Determine reference resource data of N subtrees based on a first matching result obtained by the first matching process; Using the reference resource data of N subtrees, the execution cost corresponding to the candidate plan tree is estimated.

[0170] It should be understood that the computer device 1000 described in the embodiments of the present application can execute the description of the above data processing method in each embodiment of the present application, and can also execute the above Fig.12 The description of the data processing device 120 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0171] In addition, it should be pointed out here that: the present application also provides a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium. When the processor executes the computer program, the description of the data processing method in each embodiment of the present application can be executed, so it will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0172] As an example, the above-mentioned computer program may be deployed on one computer device for execution, or deployed on multiple computer devices located at one location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0173] The computer-readable storage medium may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may include both an internal storage unit and an external storage device of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0174] The present application provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the description of the above-mentioned data processing method in each embodiment of the present application, and therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0175] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0176] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0177] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used to execute the operation statement; Acquire a data set, wherein the data set includes resource data consumed by M historical subtrees during historical execution, wherein the historical subtree is a subtree of a historical plan tree that has been historically executed, and M is a positive integer; Performing a first matching process of resource data on the N subtrees in the data set, wherein the first matching process is used to filter matching resource data for the N subtrees based on similarities between the N subtrees and the M historical subtrees; Determining reference resource data of the N subtrees based on a first matching result obtained by the first matching process; The reference resource data of the N subtrees are used to estimate the execution cost corresponding to the candidate plan tree.

2. The method according to claim 1, characterized in that The performing a first matching process of resource data on the N subtrees in the data set includes: Performing standardized representation processing on each of the N subtrees to generate standard representation information of each subtree; Based on the standard representation information of the N subtrees, performing the first matching processing of resource data on the N subtrees in the data set; Among them, the resource data consumed by each historical subtree in the data set has a mapping relationship with the standard representation information of each historical subtree respectively; the similarity between the standard representation information of the N subtrees and the standard representation information of the M historical subtrees is used to reflect the similarity between the N subtrees and the M historical subtrees.

3. The method according to claim 2, characterized in that Any one of the N subtrees is a first subtree, and the data set includes M standard representation information of the M historical subtrees; The performing the first matching processing of resource data on the N subtrees in the data set based on the standard representation information of the N subtrees includes: Obtaining information similarities between the standard representation information of the first subtree and the M standard representation information respectively; Determine the standard representation information having the greatest information similarity with the standard representation information of the first subtree among the M standard representation information as the pending representation information; If the information similarity between the standard representation information of the first subtree and the pending representation information is greater than or equal to a set similarity threshold, the resource data in the data set that has a mapping relationship with the pending representation information is used as the selected resource data that matches the first subtree; The first matching result includes the filtered resource data that matches the first subtree.

4. The method according to claim 3, characterized in that The method further comprises: If the information similarity between the standard representation information of the first subtree and the pending representation information is less than the similarity threshold, determining that resource data matching the first subtree is not screened out in the data set; The first matching result includes resource data that is not matched with the first subtree in the data set.

5. The method according to claim 4, characterized in that The N subtrees are all composed of one or more operator nodes of the candidate plan tree; and the first matching result obtained based on the first matching process, determining the reference resource data of the N subtrees, comprises: If the first matching result includes the filtered resource data that matches the first subtree, the filtered resource data that matches the first subtree is used as reference resource data of the first subtree; If the first matching result includes resource data that is not screened out and matches the first subtree, the reference resource data of the first subtree is determined based on the number of operator nodes included in the first subtree.

6. The method according to claim 5, characterized in that The determining, based on the number of operator nodes included in the first subtree, reference resource data of the first subtree includes: When the first subtree includes an operator node of the candidate plan tree, resource data consumed by the table data called by the first subtree is used as reference resource data of the first subtree; When the first subtree contains multiple operator nodes of the candidate plan tree, a second matching process of resource data is performed on the first subtree in the data set, and based on a second matching result obtained by the second matching process, the reference resource data of each operator node in the first subtree is determined; wherein the reference resource data of the first subtree includes the reference resource data of each operator node in the first subtree, and the second matching process is used to screen matching resource data for each operator node in the first subtree through the similarity between each operator node in the first subtree and the M historical subtrees.

7. The method according to claim 6, characterized in that Any operator node in the first subtree is a target operator node; and determining reference resource data of each operator node in the first subtree based on the second matching result obtained by the second matching process includes: If the second matching result includes the filtered resource data that matches the target operator node, the filtered resource data that matches the target operator node is used as reference resource data of the target operator node; If the second matching result includes resource data that is not screened out and matches the target operator node, the resource data consumed by the table data called by the target operator node is used as reference resource data of the target operator node.

8. The method according to claim 2, characterized in that Any one of the N subtrees is a first subtree; and performing standardized representation processing on each of the N subtrees to generate standard representation information of each subtree includes: Performing data parsing processing on the first subtree from K data domains to obtain representation data of the first subtree under the K data domains; K is a positive integer; Perform representation processing on the representation data of the first subtree under the K data domains according to a set standard format to generate standard representation information of the first subtree; The K data domains include at least one of the following: an operator domain corresponding to the operator called by the subtree, an object domain corresponding to the data object in the table data called by the subtree, and a condition domain corresponding to the condition contained in the operator called by the subtree.

9. The method according to claim 1, characterized in that The process of obtaining N subtrees of the candidate plan tree includes: Identify a branch node in the candidate plan tree, and divide each branch under the branch node into a first type subtree of the candidate plan tree; By using each operator node in the candidate plan tree except the first type of subtree, respectively construct a second type of subtree of the candidate plan tree, and an operator node in the candidate plan tree except the first type of subtree is used to construct a second type of subtree; The first type of subtree and the second type of subtree are used as the N subtrees obtained by dividing the candidate plan tree.

10. The method according to claim 1, characterized in that There are multiple candidate plan trees for the operation statement, and the method further includes: Selecting a candidate plan tree with the minimum execution cost from the plurality of candidate plan trees as the target plan tree of the operation statement; The target plan tree is used to execute the operation statement.

11. The method according to claim 10, characterized in that The resource data consumed by each historical subtree in the data set is respectively mapped to the standard representation information of each historical subtree, and any subtree of the target plan tree is a second subtree; After executing the target plan tree, the method further includes: If the data set does not include the standard representation information of the second subtree, after executing the second subtree, obtaining resource data consumed by the second subtree during the execution process; A mapping relationship between the resource data consumed by the second subtree and the standard representation information of the second subtree is constructed, and the constructed mapping relationship is added to the data set.

12. The method according to claim 11, characterized in that The obtaining of resource data consumed by the second subtree during execution includes: Obtaining input and output data of the last operator node in the second subtree, and using the input and output data of the last operator node as the input and output data of the second subtree; Obtain memory data consumed by each operator node in the second subtree, and use the memory data consumed most by the operator nodes in the second subtree as the memory data consumed by the second subtree; Obtaining processor data consumed by each operator node in the second subtree, and taking the processor data consumed most by the operator nodes in the second subtree as the processor data consumed by the second subtree; The resource data consumed by the second subtree includes input and output data of the second subtree, memory data consumed by the second subtree, and processor data consumed by the second subtree.

13. The method according to claim 1, characterized in that The process of obtaining the candidate plan tree of the operation statement includes: Performing statement parsing processing on the operation statement to generate a logic plan tree of the operation statement; The logical plan tree is converted to generate a plurality of candidate plan trees for the operation statement.

14. The method according to claim 1, wherein: The using the reference resource data of the N subtrees to estimate the execution cost corresponding to the candidate plan tree includes: Inputting the reference resource data of the N subtrees into a cost estimation model; The cost estimation model is called to estimate the execution cost corresponding to the candidate plan tree based on the reference resource data of the N subtrees.

15. A data processing device, characterized in that: The device comprises: A first acquisition module, used for acquiring a candidate plan tree of an operation statement and N subtrees of the candidate plan tree, where N is a positive integer, and the candidate plan tree is used for executing the operation statement; A second acquisition module is used to acquire a data set, wherein the data set includes resource data consumed by M historical subtrees during historical execution, wherein the historical subtree is a subtree of a historical plan tree that has been historically executed, and M is a positive integer; A matching module, configured to perform a first matching process of resource data on the N subtrees in the data set, wherein the first matching process is configured to screen matching resource data for the N subtrees based on similarities between the N subtrees and the M historical subtrees; A determination module, configured to determine the reference resource data of the N subtrees based on a first matching result obtained by the first matching process; The estimation module is used to estimate the execution cost corresponding to the candidate plan tree by using the reference resource data of the N subtrees.

16. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 14 when being executed by a processor.

17. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 14.

Images