Data processing method and related equipment

Abstract

The embodiment of the invention discloses a data processing method which can be applied to a distributed database. The method comprises the steps that at least one transaction is obtained, a first relation graph is constructed based on operation information of SQL statements in the transaction, and the first relation graph comprises a plurality of first nodes and first edges among the first nodes; each first node in the plurality of first nodes comprises a name of a table accessed by the SQL statement and index information (such as an index field or an index name and the like) of the table accessed by the SQL statement, and the first edge is used for describing an association relationship among the index information in the accessed table; and partitioning the accessed table according to the first relation graph. Compared with partitioning according to manual experience, the method for partitioning the table accessed by the SQL statements based on the running information of the SQL statements is more reasonable (for example, the SQL accessing the same data is distributed to the same node), and frequent transfer of the data among the nodes can be reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of databases, and in particular to a data processing method and related equipment. Background Art

[0002] A database is usually an organized and uniformly manageable collection of large amounts of data, and it usually needs to have four characteristics: atomicity, consistency, isolation, and durability, which can be referred to as ACID characteristics.

[0003] In a shared disk database architecture, all computing nodes can access all data. When a computing node receives a Structured Query Language (SQL) request, and the data required by this SQL is not on this node, this node will ask other nodes whether they have this data. If so, this data will be transferred from other nodes to this node. If other nodes do not have this data either, this computing node directly loads the page where the data is located from the storage. If SQL access is randomly distributed to multiple nodes, then data pages will be frequently transferred from one computing node to another computing node according to the needs of SQL calculations. The frequent transfer of pages between nodes increases the communication overhead between nodes, affecting the response delay of SQL statements and the overall throughput of the system.

[0004] Therefore, how to reduce the frequent transfer between nodes is a technical problem that needs to be solved urgently; Summary of the invention

[0005] The embodiment of the present application provides a data processing method and related equipment, which can identify the aggregation relationship between SQL statements accessing data, so as to more reasonably determine the partition strategy of data in the database according to the aggregation relationship.

[0006] The first aspect of the embodiment of the present application provides a data processing method, which can be applied to a distributed database (share disk), etc. The method can be executed by a data processing device, or by a component of the data processing device (such as a processor, a chip, or a chip system, etc.). The data processing device contains a database. In the method, at least one transaction is first obtained, and then a first relationship graph is constructed based on the running information of the SQL statement in the transaction. The first relationship graph includes a plurality of first nodes and a first edge between the plurality of first nodes. Each of the plurality of first nodes includes the name of the table accessed by the SQL statement and the index information of the table accessed by the SQL statement (such as an index field or an index name, etc.). The first edge is used to describe the association relationship between the index information in the accessed table; and then the accessed table is partitioned based on the first relationship graph to obtain a partition result, and the partition result is used to represent the association relationship between the partition index and the constant field in the SQL statement.

[0007] In the embodiment of the present application, a first relationship graph is constructed through the operation information of the SQL statement, and the accessed table is partitioned according to the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by the SQL statement based on the operation information of the SQL statement is more reasonable (for example, distributing the SQL accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by the SQL statement can be identified, so as to more reasonably determine the partition strategy of the data in the database based on the aggregation relationship.

[0008] Optionally, in a possible implementation of the first aspect, the above steps also include: obtaining database information of the database, the database information including primary keys and foreign keys of at least two tables; or the database information including index information of at least two tables; constructing a second relationship graph based on the database information, the second relationship graph including multiple second nodes, each of the multiple second nodes including the names of at least two tables and index information of at least two tables; constructing a first relationship graph based on the execution information of the SQL statement, including: updating the second relationship graph based on the execution information to obtain the first relationship graph.

[0009] In this possible implementation, a static relationship graph (i.e., the second relationship graph) can be first constructed through database information, and then the static relationship graph can be updated according to the aggregation relationship of the data accessed by the SQL statement to obtain a dynamic relationship graph (i.e., the first relationship graph), thereby ensuring that the tables in the database are in the first relationship graph as much as possible.

[0010] Optionally, in a possible implementation of the first aspect, the above-mentioned operation information includes at least one of the following: a field used for constant value comparison in an SQL statement, a join relationship between multiple tables accessed by an SQL statement, read and write types of multiple tables, index information used during operation, and the order of SQL statements in a transaction.

[0011] In this possible implementation, the aggregation relationship of SQL access data can be identified through the running information of the SQL statement, so that a more accurate partitioning result can be determined.

[0012] Optionally, in a possible implementation of the first aspect, when the above-mentioned operation information includes a join relationship, updating the second relationship graph based on the operation information includes: adding a first edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with an index field of a table in a database.

[0013] In this possible implementation, the edges in the static relationship graph are added through the join relationship in the SQL statement execution information, so that the static relationship graph can be updated according to the accessed vertices, so that the obtained dynamic relationship graph is more consistent with the access behavior.

[0014] Optionally, in a possible implementation of the first aspect, the above-mentioned operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; updating the second relationship graph based on the operation information includes: increasing the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0015] In this possible implementation, the weight of the edge in the static relationship graph is increased through the join relationship in the SQL statement execution information, and the access frequency can be determined according to the weight, which facilitates the subsequent determination of the partition result according to the edge.

[0016] Optionally, in a possible implementation manner of the first aspect, the above steps also include: determining index information used during the execution of the first SQL statement in each transaction as an alternative partition index; partitioning multiple tables based on the first relationship graph, including: determining a partition index based on the alternative partition index and the first relationship graph; and determining a partition result based on the partition index and a constant field in the first SQL statement.

[0017] In this possible implementation, the candidate partition index is first determined by the first SQL statement, the partition index is then determined according to the first relationship graph, and then the partition result is determined according to the partition index and the constant field in the first SQL statement.

[0018] Optionally, in a possible implementation of the first aspect, the above steps: determining the partition index based on the alternative partition index and the first relationship graph, include: determining the sum of edge weights of different paths corresponding to the nodes corresponding to at least one alternative partition index in the first relationship graph; determining the alternative partition index that meets a preset condition as the partition index, and the preset condition is related to the sum of edge weights.

[0019] In this possible implementation, the candidate partition index that meets the preset conditions in the first relationship graph is determined as the partition index through edge weights, so that the partition index better meets the access habits.

[0020] Optionally, in a possible implementation manner of the first aspect, the at least one transaction includes multiple transactions, and the running information includes a join relationship.

[0021] In this possible implementation, the join relationship can reflect the association relationship between the various tables, and the weight of the edge in the relationship graph can be updated according to the join relationship, thereby indirectly reflecting the importance of the node corresponding to the edge.

[0022] Optionally, in a possible implementation of the first aspect, the above-mentioned operating information includes a read-write type, and the read-write type is used to increase the read weight or write weight corresponding to multiple tables; the method also includes: determining whether the accessed table participates in the partition based on the read weight and the write weight; partitioning the accessed table based on the first relationship graph, including: when the accessed table participates in the partition, partitioning the accessed table based on the first relationship graph.

[0023] In this possible implementation, the read and write weights of multiple tables during the running of SQL statements can be used to determine which tables participate in the partitioning according to the read and write weights.

[0024] Optionally, in a possible implementation of the first aspect, the above step: before constructing the first relationship graph based on the running information of the SQL statement, the method also includes: obtaining a third relationship graph from other devices; constructing the first relationship graph based on the running information of the SQL statement, including: constructing the first relationship graph based on the running information and the third relationship graph.

[0025] In this possible implementation, the first relationship graph may be constructed through static relationship graphs at other devices to improve the scope of application of the solution.

[0026] Optionally, in a possible implementation manner of the first aspect, the above steps further include: sending a partition result to a client device, where the partition result is used to determine a device for executing an SQL statement.

[0027] In this possible implementation, the determined partitioning result may be sent to the client, so that the client can route the SQL statement according to the partitioning result, thereby improving processing efficiency.

[0028] The second aspect of the embodiment of the present application provides a data processing method, which can be applied to a distributed database (share disk), etc. The method can be executed by a client device, or by a component of the client device (such as a processor, a chip, or a chip system, etc.). In the method, a partition result is obtained, and the partition result is used to indicate the association relationship between the partition index and the constant field in the SQL statement; an SQL statement is obtained; and a route of the SQL statement is determined based on the partition result.

[0029] In the embodiment of the present application, since the partition result obtained is the association between the partition index and the constant field in the SQL statement, it is possible to first obtain the constant field in the SQL statement, then look up the table to see which range the value of the partition index falls into (the partition method will affect the determination of the range), and then access the database corresponding to the range.

[0030] Optionally, in a possible implementation manner of the second aspect, the above steps further include: caching partition results.

[0031] In this possible implementation, the partition results can be cached to facilitate the subsequent determination of the SQL statement routing and improve processing efficiency.

[0032] The third aspect of the embodiment of the present application provides a data processing device, which can be applied to a distributed database (share disk) and the like. The data processing device contains a database, and the data processing device includes: an acquisition unit, which is used to acquire at least one transaction, and the at least one transaction includes a structured query language SQL statement; a construction unit, which is used to construct a first relationship graph based on the running information of the SQL statement, and the first relationship graph includes a plurality of first nodes and a first edge between the plurality of first nodes, and each of the plurality of first nodes includes the name of a table accessed by the SQL statement and the index information of the table accessed by the SQL statement, and the first edge is used to describe the association relationship between the index information in the accessed table; a partitioning unit, which is used to partition the accessed table based on the first relationship graph to obtain a partitioning result, and the partitioning result is used to represent the association relationship between the partition index and the constant field in the SQL statement.

[0033] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is also used to obtain database information of the database, the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of at least two tables; the construction unit is also used to construct a second relationship graph based on the database information, the second relationship graph includes multiple second nodes, each of the multiple second nodes includes the names of at least two tables and index information of at least two tables; the construction unit is specifically used to update the second relationship graph based on the operation information to obtain the first relationship graph.

[0034] Optionally, in a possible implementation of the third aspect, the above-mentioned operation information includes at least one of the following: a field used for constant value comparison in an SQL statement, a join relationship between multiple tables accessed by an SQL statement, read and write types of multiple tables, index information used during operation, and the order of SQL statements in a transaction.

[0035] Optionally, in a possible implementation of the third aspect, when the above-mentioned operation information includes a join relationship, a construction unit is specifically used to add a first edge in the second relationship graph based on the join relationship to obtain a first relationship graph, and the join relationship has an association relationship with an index field of a table in a database.

[0036] Optionally, in a possible implementation of the third aspect, the above-mentioned operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; the construction unit is specifically used to increase the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0037] Optionally, in a possible implementation of the third aspect, the above-mentioned data processing device also includes: a first determination unit, used to determine that the index information used in the execution of the first SQL statement in each transaction is an alternative partition index; a partition unit, specifically used to determine the partition index based on the alternative partition index and the first relationship graph; the partition unit, specifically used to determine the partition result based on the partition index and a constant field in the first SQL statement.

[0038] Optionally, in a possible implementation of the third aspect, the above-mentioned partition unit is specifically used to determine the sum of edge weights of different paths corresponding to the nodes corresponding to at least one alternative partition index in the first relationship graph; the partition unit is specifically used to determine the alternative partition index that meets the preset conditions as the partition index, and the preset conditions are related to the sum of edge weights.

[0039] Optionally, in a possible implementation manner of the third aspect, the at least one transaction includes multiple transactions, and the running information includes a join relationship.

[0040] Optionally, in a possible implementation of the third aspect, the above-mentioned operating information includes read and write types, and the read and write types are used to increase the read weight or write weight corresponding to multiple tables; the data processing device also includes: a second determination unit, used to determine whether the accessed table participates in partitioning based on the read weight and the write weight; a partitioning unit, specifically used to partition the accessed table based on the first relationship graph when the accessed table participates in partitioning.

[0041] Optionally, in a possible implementation manner of the third aspect, the above-mentioned acquisition unit is also used to acquire a third relationship graph from other devices; and the construction unit is specifically used to construct the first relationship graph based on the operation information and the third relationship graph.

[0042] Optionally, in a possible implementation manner of the third aspect, the above-mentioned data processing device also includes: a sending unit, used to send a partition result to a client device, and the partition result is used to determine a device for executing an SQL statement.

[0043] The fourth aspect of the embodiment of the present application provides a client device, which can be applied to a distributed database (sharedisk), etc. The client device includes: an acquisition unit, which is used to acquire a partition result, and the partition result is used to represent the association relationship between the partition index and the constant field in the SQL statement; the acquisition unit is also used to acquire the SQL statement; and the determination unit is used to determine the route of the SQL statement based on the partition result.

[0044] Optionally, in a possible implementation manner of the fourth aspect, the client device further includes: a storage unit, configured to cache partitioning results.

[0045] A fifth aspect of an embodiment of the present application provides a data processing device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the data processing device implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0046] The sixth aspect of an embodiment of the present application provides a client device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the client device implements the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0047] A seventh aspect of an embodiment of the present application provides a communication system, including the data processing device of the fifth aspect above, and / or the client device of the sixth aspect.

[0048] An eighth aspect of an embodiment of the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction is run on a computer, the computer executes the method of the first aspect or any possible implementation of the first aspect, or the computer executes the method of the second aspect or any possible implementation of the second aspect.

[0049] A ninth aspect of an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, it enables the computer to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, or enables the computer to execute the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0050] The tenth aspect of an embodiment of the present application provides a chip system, which includes at least one processor for supporting a data processing device to implement the functions involved in the above-mentioned first aspect or any possible implementation of the first aspect, or for supporting a client device to implement the functions involved in the above-mentioned second aspect or any possible implementation of the second aspect.

[0051] In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the data processing device or client device. The chip system may be composed of a chip, or may include a chip and other discrete devices. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data for at least one processor.

[0052] Among them, the technical effects brought about by the third, fifth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0053] The technical effects brought about by the fourth, sixth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the second aspect or different possible implementation methods of the second aspect, and will not be repeated here.

[0054] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: the present application constructs a first relationship graph through the running information of the SQL statement, and partitions the accessed table according to the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by the SQL statement based on the running information of the SQL statement is more reasonable (for example, distributing the SQL accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by the SQL statement can be identified, so as to more reasonably determine the partitioning strategy of the data in the database based on the aggregation relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of the structure of the system architecture provided in the embodiment of the present application;

[0056] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0057] Figure 3 A relationship diagram corresponding to Example 1 provided in an embodiment of the present application;

[0058] Figure 4 A relationship diagram corresponding to Example 1 and Example 2 provided in an embodiment of the present application;

[0059] Figure 5 A relationship diagram corresponding to Example 1, Example 2, and Example 3 provided in the embodiments of the present application;

[0060] Figure 6 A relationship diagram corresponding to Example 1, Example 2, Example 3 and Example 4 provided in the embodiments of the present application;

[0061] Figure 7 A relationship diagram corresponding to Example 1, Example 2, Example 3, Example 4 and Example 5 provided in the embodiments of the present application;

[0062] Figure 8 Another flowchart of the data processing method provided in the embodiment of the present application;

[0063] Fig. 9 A structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0064] Fig.10 A schematic diagram of the structure of a client device provided in an embodiment of the present application;

[0065] Fig.11 Another structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0066] Fig.12 Another structural schematic diagram of a client device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The embodiment of the present application provides a data processing method and related equipment, which can identify the aggregation relationship between SQL statements accessing data, so as to more reasonably determine the partition strategy of data in the database according to the aggregation relationship.

[0068] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of the present application are first introduced below.

[0069] 1. Affairs

[0070] A transaction is a sequence of database operations defined by the user. These operations are either all or none, and are an indivisible unit of work. A transaction can be a SQL statement, a group of SQL statements, or an entire program.

[0071] In the process of writing or updating data, the database management system must have four characteristics to ensure that the transaction is correct and reliable: atomicity (also known as indivisibility), consistency (consistency), isolation (also known as independence), and durability.

[0072] Atomicity: All operations in a transaction are either completed or not completed, and will not end at some stage in the middle. If an error occurs during the execution of a transaction, it will be rolled back to the state before the transaction started, as if the transaction had never been executed. That is, transactions are indivisible and irreducible.

[0073] Consistency: The integrity of the database is not compromised before and after the transaction begins. This means that the written data must fully comply with all preset constraints, triggers, cascading rollbacks, etc.

[0074] Isolation: The ability of a database to allow multiple concurrent transactions to read, write, and modify its data at the same time. Isolation can prevent data inconsistency caused by cross-execution when multiple transactions are executed concurrently. Transaction isolation is divided into different levels, including read uncommitted, read committed, repeatable read, and serializable.

[0075] Persistence: After the transaction is completed, the changes to the data are permanent and will not be lost even if the system fails.

[0076] 2. Partition

[0077] Partitioning is an important function of database systems. Partitioning divides large data tables into parts, which are represented by partition tables. Partition tables can also include partition fragments. Reasonable partitioning can improve the performance of database systems, reduce the number of times the system reads and writes data, reduce the scale of data processing by database servers, and facilitate database management. Partition strategy is a clustering strategy for large data table partitions set by designers. The partition strategy specifies which partition fragment a data record in a data table is mapped to. The database system also provides management functions for data partitions to achieve partition management.

[0078] Horizontal Partitioning: Partition the rows of a table. Physically separated data in different groups are combined together. All columns in the table can be found in each partition, maintaining the attribute structure of the table.

[0079] Vertical partitioning: Divide certain specific columns into specific partitions to reduce the width of the table. Each partition stores the rows where the columns are located.

[0080] Partitioned table: A partitioned table is a data organization scheme that distributes table data into multiple storage objects (called data partitions or ranges) based on the values ​​in one or more table partitioning key columns in the table.

[0081] 3. Primary key and foreign key

[0082] The primary key (PK) generally refers to the primary keyword, which is one or more fields in a table. Its value is used to uniquely identify a record in the table. It cannot be repeated or empty, ensuring the atomicity of each column.

[0083] Foreign key (foreign key) generally refers to a foreign keyword, which refers to a column or columns in another table. The referenced columns should have a primary key constraint or a unique constraint. Foreign keys are used to establish and strengthen the connection between the data in two tables, and control the data stored in the foreign key table to maintain data consistency and integrity.

[0084] 4. Graph

[0085] A graph consists of a finite non-empty set of vertices and a set of edges between vertices, usually represented as: G(V,E), where G represents a graph, V is the set of vertices in the graph G, and E is the set of edges in the graph G. The data elements in the graph are called vertices. It is not allowed to have no vertices in the graph (finite non-emptiness). The relationship between the vertices in the graph is represented by edges (the edge set can be empty).

[0086] Undirected graphs: If the edge between any two vertices in the graph is an undirected edge (in short, it is an edge without direction), then the graph is called an undirected graph.

[0087] Directed graphs: If the edge between any two vertices in the graph is a directed edge (in short, an edge with a direction), then the graph is called a directed graph.

[0088] Path: In an undirected graph, if there is a set of edges starting from vertex Vi that can reach vertex Vj, then the vertex sequence from vertex Vi to vertex Vj is called the path from vertex Vi to vertex Vj.

[0089] Weight: Some edges or arcs of a graph have numbers associated with them. This number associated with an edge or arc of a graph is called a weight.

[0090] 5. Table connection (Join)

[0091] Join is used to combine rows from two or more tables. There are many join methods, such as inner join, left outer join, right outer join, and full join.

[0092] INNER JOIN: Returns rows if there is at least one match in the table.

[0093] LEFT JOIN: Returns all rows from the left table even if there is no match in the right table.

[0094] RIGHT JOIN: Returns all rows from the right table even if there is no match in the left table.

[0095] FULL JOIN: Returns rows if there is a match in one of the tables.

[0096] 6. TPC-C

[0097] TPC-C is a benchmark for measuring online transaction processing. TPC-C provides a specification to describe this benchmark. The TPC-C model simulates the sales model of a commodity wholesale company. This model covers the process of a wholesale company selling a series of commodities to customers, including order management, inventory management, account income and expenditure management, and other operations. These operations involve concepts such as warehouses, commodities, customers, and orders. Data tables and corresponding database operations are constructed around these concepts. TPC-C's wholesale operations conceptually include five operations: new order operations, payment operations, order status operations, shipping operations, and inventory operations. These operations together constitute a complete wholesale process. The new order operation is the core of measuring tpmC, which is the number of transactions that the database system can process per minute.

[0098] Currently, under the shared disk database architecture, all computing nodes can access all data. When a computing node receives an SQL request and the data required by this SQL is not on this node, this node will ask other nodes whether they have this data. If so, this data will be transferred from other nodes to this node. If other nodes do not have this data either, this computing node directly loads the page where the data is located from the storage. If SQL access is randomly distributed to multiple nodes, then data pages will be frequently transferred from one computing node to another computing node according to the needs of SQL calculations. The frequent transfer of pages between nodes increases the communication overhead between nodes, affecting the response delay of SQL statements and the overall throughput of the system.

[0099] Among these SQL statements, some SQL statements will modify the data, and the modified pages will become dirty pages. If the transactions of these SQL statements have been committed, but the corresponding dirty pages have not been persisted to the storage in time, if this node goes down, this page needs to be restored in the memory of other nodes through the WAL log before it can be accessed. Before this page is restored, if there are SQL statements from other nodes that need to access the data on this page, they can only wait. If the SQL statements that modify the data are randomly distributed, then each node caches random dirty pages. The failure of any computing node may cause the SQL statements of other nodes to wait for the recovery of these dirty pages, thus affecting the availability of the system.

[0100] To solve the above problems, we found that by reasonably classifying and scheduling SQL access and routing SQL distribution to access closely related data to one node, we can reduce the interaction between nodes and improve system performance, and avoid the impact of node failure on SQL access of other nodes, thereby improving system availability.

[0101] Based on the above ideas, an embodiment of the present application provides a data processing method, which constructs a first relationship graph through the operation information of SQL statements, and partitions the accessed table according to the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by SQL statements based on the operation information of SQL statements is more reasonable and can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by SQL statements can be identified, so as to more reasonably determine the partitioning strategy of the data in the database based on the aggregation relationship.

[0102] The following describes the system architecture provided by the embodiment of the present application. The system architecture can be applied to a distributed database (share disk), etc., and is not specifically limited here.

[0103] See attached Figure 1 , an embodiment of the present invention provides a system architecture. The system architecture includes: an application 101 , a driver 102 , a database instance 103 , a database instance 104 and a shared storage management 105 .

[0104] In the application 101, at least one business module is generally included, and each business module is connected to a database instance (e.g., database instance 103 and / or database instance 104) through a driver 102. Usually, each business module is connected to two database instances, such as database instance 103 and database instance 104, and then the business module distributes SQL statements to database instance 103 and database instance 104 for execution. In addition, in the case of multiple business modules, since the data correlation between different business modules is relatively small, the link between the business module and the database instance is fixed, and the SQL is fixedly distributed to a specific database instance, so as to avoid the frequent transfer of data pages between multiple nodes caused by random distribution of SQL, thereby improving the response delay of SQL statements and the overall throughput of the system.

[0105] Driver 102 includes client link / session management 1021, SQL distribution and reading results 1022, and client SQL routing 1023. Client link / session management 1021 is responsible for establishing a link between application 101 and the database instance. SQL distribution and reading results 1022 is responsible for receiving the SQL request of the application and distributing the SQL routing to a database instance. Waiting for the SQL execution result of the database instance and returning the execution result to application 101. Client SQL routing 1023 is responsible for routing the database instance accessed by SQL. In the embodiment of the present application, the client SQL routing 1023 receives the partition result from database instance 103 or database instance 104. The client can determine the database instance (or computing node) of the SQL routing based on the differentiation result. In addition, driver 102 can be Open Database Connectivity (ODBC) or Java Database Connectivity (JGBC), etc., which is not limited here.

[0106] The database instance 103 includes a server link / session management 1031, an SQL engine 1032, and an inter-node communication 1033. The SQL engine 1032 includes an SQL parsing / optimization / execution 10321 and a server SQL routing 10322. The server link / session management 1031 is responsible for establishing a link with the application 101 through the driver 102, and the SQL parsing / optimization / execution 10321 is responsible for parsing / optimizing / executing SQL, etc. The server SQL routing 10322 determines the SQL routing according to the SQL parsing / optimization / execution 10321's SQL operation information, and returns the SQL routing to the application 101 through the driver 102. The inter-node communication 1033 is responsible for communicating with other data instances (such as the database instance 104). Among them, the process of determining the SQL routing can be understood as the database server cache and the client cache building a lookup table. The input of the lookup table is the SQL statement string and the parameter condition. The lookup table maps the parameter condition to the partition field of the table according to the SQL statement string and the parameter condition, and determines the routing node according to the range of the partition field value.

[0107] The database instance 104 includes a server link / session management 1041, a SQL engine 1042, and an inter-node communication 1043. The SQL engine 1042 includes an SQL parsing / optimization / execution 10421 and a server SQL routing 10422. The server link / session management 1041 is responsible for establishing a link with the application 101 through the driver 102, and the SQL parsing / optimization / execution 10421 is responsible for parsing / optimizing / executing SQL, etc. The server SQL routing 10422 determines the SQL routing according to the SQL operation information of the SQL parsing / optimization / execution 10421, and returns the SQL routing to the application 101 through the driver 102. The inter-node communication 1043 is responsible for communicating with other data instances (such as the database instance 103).

[0108] The shared storage management 105 includes a Redo log file 1051 and a data file 1052 .

[0109] Database instance 103 and database instance 104 receive the SQL sent by driver 102 and parse / execute the SQL. If the data page accessed by the SQL is in this instance, it is processed directly in this instance. If it is not in this instance, a request is sent to the instance to transfer the data page to this node, and this node then accesses and calculates. If all database instances do not have the data page to be accessed, the data page is read from the data file to the memory of this node through shared storage management 105, and then accessed and calculated. If the data page is modified, the redo log of the page modification needs to be persisted to Redo log file 1051 through shared storage management before the transaction is committed. After the SQL is executed, the execution result is returned to driver 102. The database instance background task periodically performs checkpoints. For modified data pages, when the checkpoint is performed, these pages are persisted to data file 1052 through shared storage management 104. Redo logs record the modified value of a data block and are used to restore data updated by successful transactions that have not been written to data files.

[0110] In addition, the driver 102 can be deployed on the same computing server as the database instance (such as database instance 103 or database instance 104) in the application 101, or can be deployed separately on another computing server. The database instance runs on the computing server (the database instance 103 and the database instance 104 can be on the same computing server or on different computing servers), and the shared storage management 105 runs on the storage server. The database instances are interconnected through the network of switches, and the database instances to the shared storage management.

[0111] exist Figure 1 In the architecture, the data page buffer of the database instance will determine the node where the data page is currently located according to the SQL access needs, and transfer the data page from other nodes to this node. This embodiment analyzes the characteristics of SQL access data, reasonably schedules and distributes SQL, reduces the transfer of data pages between nodes, and improves SQL access performance. For example, in the database architecture of shared disk, SQL accessing the same data is distributed to the same node through SQL routing distribution.

[0112] In addition, the ways to execute SQL in the client application include: Parse-Bind-Execute (PBE), direct method, through stored procedures, indirect triggering through triggers, and anonymous blocks. SQL distribution routing divides these ways of executing SQL into three categories: the first category is that ODBC / JDBC can perceive parameter conditions, the second category is that the server perceives SQL parameter conditions through lexical and grammatical analysis, and the third category is that there are no parameter conditions at all. ODBC / JDBC can perceive parameter conditions mainly in the PBE method. The SQL statements executed in this way are routed by the client, and the client routes the SQL to the corresponding node for execution according to the bound parameter conditions. If the server can perceive the parameter conditions, the server will perform lexical and grammatical analysis and then route it. For SQL that has no parameter conditions at all and cannot be routed, it will be directly executed by this node without routing distribution.

[0113] Since transactions can only be run on one node in the Share Disk database, if a transaction includes multiple SQL statements, the node where the transaction runs is also determined by the node where the first SQL statement is distributed. Therefore, ODBC / JDBC only searches the routing table for the first SQL statement in the transaction and distributes it according to the transaction running status. For subsequent SQL statements in the transaction, the routing table is no longer searched, and they are directly distributed to the node where the previous SQL statement is executed. Note that it is the node that executes the SQL, not the node to which it is distributed, because the server may forward it to other nodes for execution.

[0114] Understandably, Figure 1 The system architecture shown is only an example. In actual applications, there may be more database instances, Redo log files, etc., which are not specifically limited here.

[0115] The data processing method provided in the embodiment of the present application is described below. The method can be executed by a data processing device, or by a component of the data processing device (such as a processor, a chip, or a chip system, etc.). The data processing device can be a server, or a client (also called a client device), or a system composed of a server and a client (such as the aforementioned Figure 1 shown).

[0116] The above-mentioned client may be a mobile phone, tablet computer (pad), portable game console, personal digital assistant (PDA), laptop computer, ultra mobile personal computer (UMPC), handheld computer, netbook, in-vehicle media player, wearable electronic device, virtual reality (VR) terminal device, augmented reality (AR) terminal device and other digital display products. The above-mentioned server may be a computing server, cloud server, etc. There is no limitation on the specific situation of the client and the server.

[0117] The method provided in the embodiment of the present application is mainly applicable to the distributed database (share disk) scenario.

[0118] See also Figure 2 , a flow chart of a data processing method provided in an embodiment of the present application, the method may include steps 201 to 205.

[0119] Step 201: The client sends a transaction to the server. Correspondingly, the server receives the transaction sent by the client. This step is optional.

[0120] Step 202: The server constructs a first relationship graph based on the running information of the SQL statement.

[0121] Step 203: The server partitions the accessed table based on the first relationship graph to obtain a partition result.

[0122] Step 204: The server sends the partition result to the client. Correspondingly, the client receives the partition result sent by the server. This step is optional.

[0123] Step 205: The client determines the SQL route based on the partition result. This step is optional.

[0124] Steps 201 to 205 are described in detail below:

[0125] Step 201: The client sends a transaction to the server. Correspondingly, the server receives the transaction sent by the client. This step is optional.

[0126] Optionally, the client first obtains at least one transaction, each transaction including at least one SQL statement, and then sends the at least one transaction to the server.

[0127] Of course, the server may also obtain transactions in other ways, such as obtaining transactions from a database, receiving transactions sent by other devices, determining transactions based on user operations, etc., which are not specifically limited here.

[0128] It can be understood that in the embodiment of the present application, the number of at least one transaction can be one or more, and the number of at least one SQL statement included in each transaction can also be one or more, which is not specifically limited here.

[0129] For example, the application in the TPC-C model and the number of transactions is 5, and the 5 transactions include: creating an order, order payment, order status query, delivery, and inventory check. The overall simulation simulates the operation of a wholesale system with a warehouse as the core.

[0130] For the convenience of description, the following first describes the example of the above TPC-C model.

[0131] The model used in the TPC-C test is a large commodity wholesale sales company, which has several commodity warehouses distributed in different regions. When the business expands, the company will add new warehouses. Each warehouse is responsible for supplying 10 sales points, each of which provides services to 3,000 customers. Among the orders submitted by each customer, there are an average of 10 products per order. About 1% of the products in all orders are not in stock in the warehouse to which they are directly affiliated and must be supplied by warehouses in other regions. At the same time, each warehouse must maintain inventory records of the 100,000 commodities sold by the company.

[0132] The TPC-C evaluation system database includes 9 tables, namely, warehouse table (Warehouse), district table (District), customer table (Customer), history table (History), order table (Order), new order table (New-Order), order line table (Order-Line), supply item table (Item), and stock table (Stock). Among them, the order line table can also be called order item or order entry.

[0133] Among them, the warehouse table (Warehouse) is used to record the warehouses where goods are stored. The district table (District) is used to record district information. Each warehouse provides wholesale for 10 districts. The customer table (Customer) has 3,000 users in each district. These users conduct transactions with the warehouses in the district (in special cases, they will go to other warehouses for transactions). The transaction table (Order) is used to record the user's commodity transactions. There may be 5 to 10 specific transaction items for commodities under one transaction. The new order table (New-Order) is used to record new orders. When the order is completed, the corresponding record will be deleted from the New-Order table.

[0134] Tables 1 to 9 below give detailed field examples of the above 9 tables:

[0135] Table 1-Warehouse

[0136]

[0137] Table 2-District

[0138]

[0139] Table 3 - Customer Table (Customer)

[0140]

[0141]

[0142] Table 4-History table

[0143]

[0144] Table 5-New Order Table (New-Order)

[0145]

[0146] Table 6-Order Table (Order)

[0147]

[0148] Table 7-Order-Line

[0149]

[0150] Table 8-Supply Commodity Item List (Item)

[0151]

[0152] Table 9-Stock Table (Stock)

[0153]

[0154]

[0155] It is understandable that the fields of the above 9 tables are just examples. In actual applications, there may be other forms (for example, the number of rows, columns, names, length limits, etc. of the table can be set according to actual needs), which are not specifically limited here.

[0156] The following describes the process of building the index relationship between the SQL statements and tables involved in each transaction:

[0157] Example 1, the SQL statements involved in Transaction 1 - Create Order Transaction (The New-Order Transaction) are as follows:

[0158]

[0159]

[0160]

[0161] Example 2, Transaction 2 - Order Payment Transaction (The Payment Transaction) involves the following SQL statements:

[0162]

[0163]

[0164]

[0165]

[0166] Example 3, the SQL statements involved in Transaction 3-Order Status Query Transaction (The Order-Status Transaction) are as follows:

[0167]

[0168]

[0169] Example 4, Transaction 4 - The Delivery Transaction involves the following SQL statements:

[0170]

[0171]

[0172] Example 5, the SQL statements involved in Transaction 5-Stock-Level Transaction are as follows:

[0173]

[0174]

[0175] It is understandable that the above is only an illustrative description using five transactions of TPC-C as an example. In actual applications, there may be scenarios such as TPC-H, TPC-T, and TPC-E, and the number of transactions and the specific structure of the SQL statements may vary, which are not limited here.

[0176] Step 202: The server constructs a first relationship graph based on the running information of the SQL statement.

[0177] After the server obtains the transaction, it constructs a first relationship graph based on the running information of at least one SQL statement in the transaction. The first relationship graph includes a plurality of first nodes and a first edge between the plurality of first nodes, each of the plurality of first nodes includes the name of a table accessed by the SQL statement and index information of the accessed table, and the first edge is used to describe the association relationship between the index information in the accessed table.

[0178] The operation information in the embodiment of the present application includes at least one of the following: fields used for constant value comparison in SQL statements, join relationships between multiple tables accessed by SQL statements, read and write types of multiple tables, index information used during operation, and the order of SQL statements in transactions.

[0179] In the embodiments of the present application, there are many situations for adding edge information. For example, edges may be constructed or edge weights may be increased based on join relationships, or edges may be constructed or edge weights may be increased between index fields of tables accessed by different SQLs, etc. The specifics are not limited here.

[0180] Exemplarily, continuing with Example 1 above, during the execution of the first SQL statement in Example 1, access to the warehouse table (Warehouse) will use the warehouse table_PK (W_ID), and access to the customer table (Customer) will use the customer table_PK (C_W_ID, C_D_ID, C_ID). After this SQL statement is executed, the warehouse table_PK (W_ID) and the customer table_PK (C_W_ID, C_D_ID, C_ID) are used as two first nodes, and a first edge is added between the two first nodes, and the weight of the first edge is 1. At the same time, the read weight of the warehouse table and the customer table is increased by 1. It should be noted that the join field c_w_id=w_id is useless here, because although w_id is the index vertex of WAREHOUSE, c_w_id is not.

[0181] In Example 1, during the execution of the second SQL statement, the district table PK (D_W_ID, D_ID) will be used to query the district table (District). This SQL statement is a single-table query, and there is no join relationship between tables. The query of the district table PK (D_W_ID, D_ID) is implicitly determined by the candidate partition index of the warehouse table PK (W_ID), so the district table PK (D_W_ID, D_ID) is used as the first node, and an edge is added between the warehouse table PK (W_ID) and the district table PK (D_W_ID, D_ID), and the weight of the edge is 1. At the same time, the read weight of the district table is increased by 1.

[0182] The process of executing other SQL statements in Example 1 to construct the first node and the first edge is similar to the above two SQL statements. After executing all SQL statements in Example 1, the first relationship graph obtained is as follows: Figure 3 In addition, after all SQL statements in Example 1 are executed, the read and write weights of each table are shown in Table 10;

[0183] Table 10 - Read and write weights of each table after running Example 1

[0184]

[0185]

[0186] As shown in Table 10, the read weight of the warehouse table is 1 and the write weight is 0. The read weight of the customer table is 1 and the write weight is 0. The read weight of the region table is 1 and the write weight is 1. The read weight of the order table is 0 and the write weight is 1. The read weight of the new order table is 0 and the write weight is 1. The read weight of the supply item table is 10 and the write weight is 0. The read weight of the inventory table is 10 and the write weight is 10. The read weight of the order line table is 0 and the write weight is 10.

[0187] For example, in the above example 2, after executing each SQL statement in example 2, Figure 3 The relationship diagram is updated to obtain Figure 4 relationship diagram. Figure 4 The relationship diagram with Figure 3 Compared with the relationship diagram, on the one hand, two first nodes (history table_FK2 (H_W_ID, H_D_ID) and history table_FK1 (H_C_W_ID, H_C_D_ID, H_C_ID)) and the edges between the two newly added nodes and the warehouse table_PK (W_ID) node are added. On the other hand, the weight of the edge is increased (such as Figure 4 black part in the middle).

[0188] In addition, after executing all SQL statements in Example 1 and Example 2, the read and write weights of each table are shown in Table 11;

[0189] Table 11 - Read and write weights of each table after running Example 1 and Example 2

[0190] Table name Read Weight Write Weight Warehouse table 2 1 Customer Table 4 1 Region Table 2 2 Order Form 0 1 New Order Form 0 1 Supply Item List 10 0 Inventory Table 10 10 Order Line Table 0 10 History Table 0 1

[0191] Among them, the newly added parts of Table 11 relative to Table 10 are the black parts in the above Table 11. After the SQL statements in Example 1 and Example 2 are executed, the weights of each table are as follows: The read weight of the warehouse table is 2, and the write weight is 1. The read weight of the customer table is 4, and the write weight is 1. The read weight of the region table is 2, and the write weight is 2. The read weight of the order table is 0, and the write weight is 1. The read weight of the new order table is 0, and the write weight is 1. The read weight of the supply item item table is 10, and the write weight is 0. The read weight of the inventory table is 10, and the write weight is 10. The read weight of the order line table is 0, and the write weight is 10. The read weight of the history table is 0, and the write weight is 1.

[0192] For example, after each SQL statement in Example 3 is executed, Figure 4 The relationship diagram is updated to obtain Figure 5 relationship diagram. Figure 5 The relationship diagram with Figure 4 Compared with the relationship diagram, on the one hand, a first node is added: Order Line Table_FK1(OL_W_ID,OL_D_ID,OL_O_ID). On the other hand, an edge is added between the Order Table_PK(O_W_ID,O_D_ID,O_ID) node and the Customer Table_PK(C_W_ID,C_D_ID,C_ID) node (such as Figure 5 black part in the middle).

[0193] In addition, after executing all SQL statements in Example 1, Example 2, and Example 3, the read and write weights of each table are shown in Table 12;

[0194] Table 12 - Read and write weights of each table after running Example 1, Example 2, and Example 3

[0195] Table name Read Weight Write Weight Warehouse table 2 1 Customer Table 6 1 Region Table 2 2 Order Form 1 1 New Order Form 0 1 Supply Item List 10 0 Inventory Table 10 10 Order Line Table 1 10 History Table 0 1

[0196] Among them, the newly added parts of Table 12 relative to Table 11 are the black parts in the above Table 12. After the SQL statements in Example 1, Example 2 and Example 3 are executed, the weights of each table are as follows: The read weight of the warehouse table is 2, and the write weight is 1. The read weight of the customer table is 6, and the write weight is 1. The read weight of the region table is 2, and the write weight is 2. The read weight of the order table is 1, and the write weight is 1. The read weight of the new order table is 0, and the write weight is 1. The read weight of the supply item item table is 10, and the write weight is 0. The read weight of the inventory table is 10, and the write weight is 10. The read weight of the order line table is 1, and the write weight is 10. The read weight of the history table is 0, and the write weight is 1.

[0197] For example, after each SQL statement in Example 4 is executed, Figure 5 The relationship diagram is updated to obtain Figure 6 relationship diagram. Figure 6 The relationship diagram with Figure 5 Compared with the relationship diagram, several first edges are added: the edge between the order table_PK(O_W_ID,O_D_ID,O_ID) node and the new order table_PK(NO_W_ID,NO_D_ID,NO_O_ID) node, the edge between the new order table_PK(NO_W_ID,NO_D_ID,NO_O_ID) node and the order line table_FK1(OL_W_ID,OL_D_ID,OL_O_ID) node (such as Figure 6 The bold part in the figure), and the edge weight between the customer table_PK(C_W_ID,C_D_ID,C_ID) node and the order table_PK(O_W_ID,O_D_ID,O_ID) node is +1.

[0198] In addition, after executing all SQL statements in Example 1, Example 2, Example 3, and Example 4, the read and write weights of each table are shown in Table 13;

[0199] Table 13 - Read and write weights of each table after running Example 1, Example 2, Example 3, and Example 4

[0200] Table name Read Weight Write Weight Warehouse table 2 1 Customer Table 6 2 Region Table 2 2 Order Form 2 2 New Order Form 1 2 Supply Item List 10 0 Inventory Table 10 10 Order Line Table 2 11 History Table 0 1

[0201] Among them, the newly added parts of Table 13 relative to Table 12 are the black parts in the above Table 13. After the SQL statements in Example 1, Example 2, Example 3 and Example 4 are executed, the weights of each table are as follows: The read weight of the warehouse table is 2, and the write weight is 1. The read weight of the customer table is 6, and the write weight is 2. The read weight of the region table is 2, and the write weight is 2. The read weight of the order table is 2, and the write weight is 2. The read weight of the new order table is 1, and the write weight is 2. The read weight of the supply item item table is 10, and the write weight is 0. The read weight of the inventory table is 10, and the write weight is 10. The read weight of the order line table is 2, and the write weight is 11. The read weight of the history table is 0, and the write weight is 1.

[0202] For example, after each SQL statement in Example 5 is executed, Figure 6 The relationship diagram is updated to obtain Figure 7 relationship diagram. Figure 7 The relationship diagram with Figure 6 Compared with the relationship diagram, several first edges are added: the edge between the inventory table_PK(S_W_ID,S_I_ID) node and the region table_PK(D_W_ID,D_ID) node, the edge between the region table_PK(D_W_ID,D_ID) node and the order line table_FK1(OL_W_ID,OL_D_ID,OL_O_ID) node (such as Figure 7 black part in the middle).

[0203] In addition, after executing all SQL statements in Example 1, Example 2, Example 3, Example 4, and Example 5, the read and write weights of each table are obtained as shown in Table 14;

[0204] Table 14 - Read and write weights of each table after running Example 1, Example 2, Example 3, Example 4, and Example 5

[0205] Table name Read Weight Write Weight Warehouse table 2 1 Customer Table 6 2 Region Table 3 2 Order Form 2 2 New Order Form 1 2 Supply Item List 10 0 Inventory Table 11 10 Order Line Table 3 11 History Table 0 1

[0206] Among them, the newly added parts of Table 14 relative to Table 13 are the black parts in the above Table 14. After the SQL statements in Example 1, Example 2, Example 3, Example 4 and Example 5 are executed, the weights of each table are as follows: The read weight of the warehouse table is 2, and the write weight is 1. The read weight of the customer table is 6, and the write weight is 2. The read weight of the region table is 3, and the write weight is 2. The read weight of the order table is 2, and the write weight is 2. The read weight of the new order table is 1, and the write weight is 2. The read weight of the supply item entry table is 10, and the write weight is 0. The read weight of the inventory table is 11, and the write weight is 10. The read weight of the order line table is 3, and the write weight is 11. The read weight of the history table is 0, and the write weight is 1.

[0207] Step 203: The server partitions the accessed table based on the first relationship graph to obtain a partition result.

[0208] After the server obtains the first relationship graph, it can partition the accessed table based on the first relationship graph to obtain a partition result, which is used to represent the association relationship between the partition index and the constant field in the SQL statement.

[0209] Specifically, the server determines that the index information used in the execution of the first SQL statement in each transaction is the candidate partition index, the server determines the partition index based on the candidate partition index and the first relationship graph, and the server determines the partition result based on the partition index and the constant field or parameter condition field in the first SQL statement.

[0210] Exemplarily, in transaction 1 shown in the aforementioned example 1, since the first SQL statement of transaction 1 first uses warehouse table_PK(W_ID) to execute the query during the query process, warehouse table_PK(W_ID) is an alternative partition index. In transaction 2 shown in the aforementioned example 2, since the first SQL statement of transaction 2 still uses warehouse table_PK(W_ID) for access during the query process, warehouse table_PK(W_ID) is an alternative partition index. In transaction 3 shown in the aforementioned example 3, since the first SQL statement of transaction 3 accesses customer table_PK(C_W_ID, C_D_ID, C_ID), customer table_PK(C_W_ID, C_D_ID, C_ID) is an alternative partition index. In transaction 4 shown in the aforementioned example 4, since the first SQL statement of transaction 4 uses new order table_PK(NO_W_ID, NO_D_ID, NO_O_ID) for access, new order table_PK(NO_W_ID, NO_D_ID, NO_O_ID) is an alternative partition index. In the transaction 5 shown in the above example 5, since the first SQL statement of transaction 5 uses the region table_PK(D_W_ID,D_ID), the region table_PK(D_W_ID,D_ID) is the candidate partition index. And the read weight and write weight of the accessed table recorded after the transaction is executed are shown in Table 14.

[0211] Among them, the above-mentioned process of determining the partition index based on the alternative partition index and the first relationship graph can specifically be: first determine the sum of edge weights of different paths corresponding to the nodes in the first relationship graph corresponding to at least one alternative partition index, and then determine the alternative partition index that meets the preset conditions as the partition index, and the preset conditions are related to the sum of edge weights (for example, select the alternative partition index with the highest weight as the partition index).

[0212] Exemplarily, the candidate partition indexes of the five transactions shown in the previous examples 1 to 5 are: warehouse table_PK (W_ID), warehouse table_PK (W_ID), customer table_PK (C_W_ID, C_D_ID, C_ID), new order table_PK (NO_W_ID, NO_D_ID, NO_O_ID), and region table_PK (D_W_ID, D_ID). Since the previous two are the same, there are only four candidate partition indexes in total. The sum of the weights of the graph edges of each link is determined based on the four candidate partition indexes. The partition index is determined based on the sum of the edge weights. For example, the candidate partition corresponding to the largest sum of edge weights is selected as the partition index. Of course, the candidate partition corresponding to the second largest sum of edge weights can also be selected as the partition index, and the specifics are not limited here.

[0213] Depend on Figure 7 It can be seen that the sum of the edge weights of the warehouse table _PK (W_ID) link is the largest, so the index information (W_ID) of the warehouse table _PK is used as the partition index.

[0214] In addition, in order to reduce unnecessary partitioning steps, it is possible to first determine whether the accessed table participates in the partitioning. The determination may be based on the value or ratio of the read weight and the write weight (for example, a table without a read weight, or a table with a write weight and a read weight greater than a preset threshold participates in the partitioning). And if the accessed table participates in the partitioning, then perform this step. For example, in the entire system of the aforementioned example, the read and write weights of the table are considered first. Since the read weight of the supply commodity entry table is 10 and the write weight is 0. There is only a read operation, so the supply commodity entry table is removed from the entire first relationship graph, that is, the supply commodity entry table does not participate in the partitioning.

[0215] Optionally, in the case of multiple transactions, the server determines the partition result based on the partition index, constant field, and join relationship. Alternatively, after determining the partition index of a table, the partition index of other tables can be determined based on the partition index and join relationship of the table. Since each table in a relational database can only have one horizontal partition, after the partition field of a table is determined, the partition of another table can be indirectly determined based on the join relationship between the tables.

[0216] If the partition fields of other transactions are in the same graph as the warehouse table_PK (W_ID), that is, the warehouse table_PK (W_ID) can be connected directly or indirectly, the partitions of these transactions are consistent with the warehouse table_PK (W_ID), and these transactions can directly use the alternative partition index. If they are not consistent, that is, they are indirectly connected through other indexes of a table (such as foreign keys, etc.), then this table is partitioned using other indexes. Of course, foreign keys, etc. can also appear in the first relationship graph and be linked to the primary key of the same table.

[0217] If the partition fields of other transactions and warehouse table _PK (W_ID) are not in the same graph, refer to the above process and analyze this graph separately.

[0218] It is understandable that the above partitioning result may also include a partitioning method, such as range partitioning, hash partitioning, etc. Wherein, when the partitioning method is range partitioning, the partitioning result also includes a partition value range corresponding to the range partitioning. When the partitioning method is hash range, the partitioning result also includes a value corresponding to the hash partitioning.

[0219] Alternatively, if the partitioning method is range partitioning, read the minimum and maximum values ​​of the W_ID field from the warehouse table, and the difference between the maximum and minimum values ​​divided by the number of cluster nodes (i.e., the number of servers) is the average range of each node. For example, the minimum value of the W_ID field is 1, the maximum value is 1000, and the number of cluster nodes is 3, then (1000-1) / 3=333. That is, the partition range of the first node is 1-333, the partition range of the second node is 334-667, and the partition range of the third node is 668-1000.

[0220] For transaction 1, the w_id field in the SQL statement is the conditional field, corresponding to the (W_ID) of the warehouse table. This SQL statement is routed according to :w_id in w_id=:w_id. If the :w_id value range is within 1-333, this SQL is sent to the first node; if the :w_id value range is within 334-667, it is sent to the second node; if the :w_id value range is within 668-1000, it is sent to the third node.

[0221] Optionally, after obtaining the partition result, the server may construct a routing table as shown in Table 15 below, the content of which is a transaction for creating an order, and W_ID in the first SQL statement is a routing field.

[0222] Table 15 - Routing Table

[0223] Partition value range Corresponding Node 1-333 Node 1 334-667 Node 2 668-1000 Node 3

[0224] Step 204: The server sends the partition result to the client. Correspondingly, the client receives the partition result sent by the server. This step is optional.

[0225] Optionally, after obtaining the partitioning result, the server may send the partitioning result or the routing table to the client.

[0226] Step 205: The client determines the SQL route based on the partition result. This step is optional.

[0227] Optionally, after the client obtains the partition result or routing table from the server, the SQL route can be determined according to the partition result or routing table.

[0228] For example, taking the creation of an order transaction as an example, the client routes according to the :w_id in the first SQL statement where w_id=:w_id. If the :w_id value is in the range of 1-333, the SQL is sent to the first node; if the :w_id value is in the range of 334-667, it is sent to the second node; if the :w_id value is in the range of 668-1000, it is sent to the third node.

[0229] It is understandable that the processing procedures for other transactions are similar and will not be repeated here.

[0230] In the embodiment of the present application, a first relationship graph is constructed through the operation information of the SQL statement, and the accessed table is partitioned according to the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by the SQL statement based on the operation information of the SQL statement is more reasonable (for example, distributing the SQL accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by the SQL statement can be identified, so as to more reasonably determine the partition strategy of the data in the database based on the aggregation relationship.

[0231] See also Figure 8 , another flow chart of a data processing method provided in an embodiment of the present application, the method may include steps 801 to 808.

[0232] Step 801: The server obtains database information.

[0233] Step 802: The server constructs a second relationship graph based on the database information.

[0234] Step 803: The client sends a transaction to the server. Correspondingly, the server receives the transaction sent by the client. This step is optional.

[0235] Step 804: Other servers send the third relationship graph to the server. Correspondingly, the server receives the third relationship graph sent by other servers. This step is optional.

[0236] Step 805: The server constructs a first relationship graph based on the running information of the SQL statement.

[0237] Step 806: The server partitions the accessed table based on the first relationship graph to obtain a partition result.

[0238] Step 807: The server sends the partition result to the client. Correspondingly, the client receives the partition result sent by the server. This step is optional.

[0239] Step 808: The client determines the SQL route based on the partition result. This step is optional.

[0240] Steps 801 to 808 are described in detail below:

[0241] Step 801: The server obtains database information.

[0242] The server obtains database information, where the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of at least two tables.

[0243] Optionally, the above database information is schema information, and the server SQL routing module can read the schema information of the database, and the schema information includes the primary key and foreign key information of the table.

[0244] Exemplarily, the database instance is started, the data dictionary management module is initialized, the server SQL routing module is initialized, and the data dictionary management module interface is called to read the schema information of the database.

[0245] Step 802: The server constructs a second relationship graph based on the database information.

[0246] After acquiring the database information, the server constructs a second relationship graph, the second relationship graph includes a plurality of second nodes, each of the plurality of second nodes includes the names of at least two tables and index information of at least two tables.

[0247] Optionally, after obtaining the schema information, the server constructs a second relationship graph between table fields according to the primary key and foreign key of the table.

[0248] In one possible implementation, primary and foreign keys are used to explicitly specify the relationship between tables when creating tables. In this case, the server reads the schema information to know the index association relationship between tables, and can directly build a second relationship graph between indexes of different tables.

[0249] In another possible implementation, if the relationship between tables is not explicitly specified, only indexes on the corresponding table fields are created. In this case, the server only knows the index vertex information when reading the schema information, and can only construct the vertices of the second relationship graph (i.e., the second nodes). The edges between the second nodes are added through the execution information of subsequent SQL statements.

[0250] Exemplarily, the server SQL routing module analyzes the primary and foreign key relationships of the table, and builds a relationship graph between the table fields based on the primary and foreign key relationships of the table. The vertices of the graph are the table name and the primary key field or the table name and the foreign key field. If there is a primary and foreign key constraint relationship between the fields of the two tables, an edge is built. The index information of the table is read, and the table name and all the fields of each index are added to the graph as a vertex.

[0251] Step 803: The client sends a transaction to the server. Correspondingly, the server receives the transaction sent by the client. This step is optional.

[0252] This step 803 is similar to the previous embodiment Figure 2 The corresponding step 201 is similar and will not be repeated here.

[0253] Step 804: Other servers send the third relationship graph to the server. Correspondingly, the server receives the third relationship graph sent by other servers. This step is optional.

[0254] The other servers send the third relationship graph to the server. Correspondingly, the server receives the third relationship graph sent by the other servers. The third relationship graph includes a plurality of third nodes.

[0255] Optionally, the third relationship graph further includes edges between a plurality of third nodes. The construction process of the third relationship graph may refer to the construction process of the second relationship graph, which will not be expanded here.

[0256] Step 805: The server constructs a first relationship graph based on the running information of the SQL statement.

[0257] After the server obtains the second relationship graph, the server may update the second relationship graph based on the operation information to obtain the first relationship graph.

[0258] Optionally, if the server has previously received a third relationship graph sent by other servers, the server constructs the first relationship graph based on the operation information and the third relationship graph. Of course, if the server has received a third relationship graph sent by other servers, the server can also update the second relationship graph based on the operation information and the third relationship graph to obtain the first relationship graph.

[0259] In one possible implementation, when the operation information includes a join relationship, the server specifically adds a first edge to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with an index field of a table in the database.

[0260] In another possible implementation, the operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; the server specifically increases the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0261] The above two situations can also be understood as the server SQL routing module checking whether the edge of the index vertex involved in the join relationship between the tables exists in the graph. If it does, the weight is increased by 1. If not, an edge is added between the index vertices, and the weight of the edge is initialized to 1.

[0262] This step can also be understood as the server determining the relationship between tables according to the SQL in the running process, and further improving the relationship diagram between the tables.

[0263] Part of the description of constructing the first relationship diagram in step 805 can refer to the above embodiment. Figure 2 The description of the corresponding step 202 is not repeated here.

[0264] Step 806: The server partitions the accessed table based on the first relationship graph to obtain a partition result.

[0265] The description of step 806 can refer to the above embodiment. Figure 2 The description of the corresponding step 203 is not repeated here.

[0266] The following steps are applied to Figure 1 The process in the shown architecture is described as follows:

[0267] Exemplarily, the server SQL routing module checks whether the SQL statement is the first SQL statement. If it is the first SQL statement, the index vertex matched by the field compared with the constant value of the SQL statement is marked as a partition candidate vertex. The server SQL routing module checks the number of SQL statements involved in the transaction. If there is only one SQL statement, it is not processed. If there are multiple SQL statements, an edge is added between the index vertex accessed by the first SQL statement and the index vertex accessed by the second SQL statement according to the order of the SQL statements, and the weight of the edge is initialized to 1. If the edge already exists, the weight is increased by 1. Add an edge between the index vertex accessed by the second SQL statement and the index vertex accessed by the third SQL statement, and the weight of the edge is initialized to 1. If the edge already exists, the weight is increased by 1. And so on, until the last SQL statement. Record the mapping relationship between SQL statement binding parameters, stored procedure parameters, and index fields.

[0268] The server SQL routing module selects a node as an analysis node for the association weight graph (for example, selects the node with the smallest node number) in an analysis cycle (for example, one minute), and notifies other nodes to send the association weight graph of this node to the analysis node. The server SQL routing module receives the relationship graph between tables constructed by other nodes through inter-node communication interaction, and merges it with the relationship graph constructed locally. The read and write weights of the table are added with the statistical values ​​of other nodes respectively. If the vertex does not exist in this graph, it is added to the relationship graph. If the edge does not exist, an edge between the two vertices is constructed, and the weight of the received edge is copied. If the edge exists, the weight of the edge counted by other nodes is added to the edge of this graph. The vertices marked as partition candidate vertices remain unchanged and are still marked as partition candidate vertices.

[0269] The server SQL routing module checks whether most of the schema tables are already in the graph. If not, it returns directly and waits for the next round of collection.

[0270] Check the read and write weights of the table. For read-only tables (write weight is 0) or tables with a majority read weight (for example, read weight exceeds 90%), do not partition them and delete them from the graph.

[0271] Traverse each graph, take each candidate partition vertex in the graph as a candidate partition strategy, count the weights of all edges determined by each partition strategy, and select the partition strategy with the largest weight as the final partition strategy for this graph.

[0272] According to the candidate vertices selected by the partitioning strategy, read the maximum value of the key of the field, divide the key partition range and the corresponding nodes according to the maximum value and the number of nodes running the database instance. Then, according to the association between tables, determine the partition fields, partition field ranges and corresponding nodes of other tables.

[0273] The server SQL routing module determines the SQL statement, SQL statement binding parameters, stored procedure parameters and index field mapping, table partition index field, partition mode, partition range and corresponding nodes. If a SQL statement includes multiple tables, the table close to the partition strategy vertex is selected as the basis for routing distribution judgment.

[0274] Step 807: The server sends the partition result to the client. Correspondingly, the client receives the partition result sent by the server. This step is optional.

[0275] Step 808: The client determines the SQL route based on the partition result. This step is optional.

[0276] The description of steps 807 and 808 can refer to the above embodiment. Figure 2The corresponding descriptions in step 204 and step 305 are not repeated here.

[0277] The following steps are applied to Figure 1 The process in the shown architecture is described as follows:

[0278] The client routes according to the partition strategy of the cache, and the server routes according to the partition strategy of the cache. The application executes the SQL statement and calls the ODBC / JDBC driver to send the SQL statement to the database instance. Distribute SQL and read the query results. The client SQL routing determines the routing node and sends it to the corresponding node through the communication connection. The server instance receives the SQL statement through the server connection and calls the SQL engine for execution. The SQL engine parses and optimizes the SQL.

[0279] If the node executes, the SQL engine executes the SQL and accesses the data page through the data page buffer. The execution result is returned through the server connection. The distribution and reading result module returns the SQL execution result to the application.

[0280] If the execution is not on this node, the SQL engine forwards it to the corresponding node through inter-node communication. The SQL engine receives the forwarded SQL through inter-node communication. It parses, optimizes, and executes the SQL. It returns the result to the original instance. The SQL engine receives the execution results of other nodes. It returns the execution results through the server connection. The distribution and reading result module returns the SQL execution results to the application.

[0281] It is understandable that if SQL is routed by the server, the execution result of SQL can be returned directly to the ODBC / JDBC driver by the database instance that executes SQL, without being returned to the original database instance. The original database instance returns the result to the ODBC / JDBC driver.

[0282] In the embodiment of the present application, the link relationship between the tables is determined by Schema, SQL statement access and business load, the link relationship is analyzed to determine the partition strategy, and the client and server forward and route the SQL according to the partition strategy. On the one hand, the database Schema information is read, SQL access is tracked in units of transactions, the link relationship and weight between the tables, the read and write weight of the table, the alternative partition field of the table, the SQL parameter value and the mapping relationship between the index partition field of the table are determined. On the other hand, each link relationship graph is traversed, and the best partition strategy is selected according to the read and write weight of the table, the alternative index partition field of the table, and the relationship weight between the index fields of the table, and the SQL statement, the SQL statement binding parameters, the mapping relationship between the stored procedure parameters and the table index field, the index partition field of the table, the partition mode, the partition range and the corresponding node are determined. On the other hand, if the SQL statement can determine the node of the database instance according to the binding parameters, it is routed by the client, otherwise it is routed by the server, the server parses the SQL, and routes according to the parsed table field constant value and the index partition information of the table.

[0283] The data processing method in the embodiment of the present application is described above. The data processing device in the embodiment of the present application is described below. Fig. 9 , an embodiment of the data processing device in the embodiment of the present application includes:

[0284] An acquisition unit 901 is used to acquire at least one transaction, where the at least one transaction includes a structured query language SQL statement;

[0285] A construction unit 902 is used to construct a first relationship graph based on the running information of the SQL statement, where the first relationship graph includes a plurality of first nodes and first edges between the plurality of first nodes, each of the plurality of first nodes includes a name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, and the first edge is used to describe an association relationship between the index information in the accessed table;

[0286] The partitioning unit 903 is used to partition the accessed table based on the first relationship graph to obtain a partitioning result, where the partitioning result is used to represent the association relationship between the partition index and the constant field in the SQL statement.

[0287] Optionally, the acquisition unit 901 is also used to obtain database information of the database, the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of at least two tables; the construction unit 902 is also used to construct a second relationship graph based on the database information, the second relationship graph includes multiple second nodes, each of the multiple second nodes includes the names of at least two tables and index information of at least two tables; the construction unit 902 is specifically used to update the second relationship graph based on the operation information to obtain the first relationship graph.

[0288] Optionally, the running information includes at least one of the following: a field used for constant value comparison in an SQL statement, a join relationship between multiple tables accessed by an SQL statement, read and write types of multiple tables, index information used during the running process, and the order of SQL statements in a transaction.

[0289] Optionally, when the operation information includes a join relationship, the construction unit 902 is specifically used to add a first edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with an index field of a table in the database.

[0290] Optionally, the operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; the construction unit 902 is specifically used to increase the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0291] Optionally, the data processing device also includes: a first determination unit 904, used to determine that the index information used in the execution of the first SQL statement in each transaction is an alternative partition index; a partition unit 903, specifically used to determine the partition index based on the alternative partition index and the first relationship graph; the partition unit 903, specifically used to determine the partition result based on the partition index and a constant field in the first SQL statement.

[0292] Optionally, the partition unit 903 is specifically used to determine the sum of edge weights of different paths corresponding to the nodes in the first relationship graph corresponding to at least one alternative partition index; the partition unit 903 is specifically used to determine that the alternative partition index that meets the preset conditions is the partition index, and the preset conditions are related to the sum of edge weights.

[0293] Optionally, at least one transaction includes multiple transactions, and the running information includes a join relationship.

[0294] Optionally, the operation information includes a read-write type, which is used to increase the read weight or write weight corresponding to multiple tables; the data processing device also includes: a second determination unit 905, used to determine whether the accessed table participates in the partition based on the read weight and the write weight; a partitioning unit 903, specifically used to partition the accessed table based on the first relationship graph when the accessed table participates in the partition.

[0295] Optionally, the acquisition unit 901 is further used to acquire a third relationship graph from other devices; the construction unit 902 is specifically used to construct the first relationship graph based on the operation information and the third relationship graph.

[0296] Optionally, the data processing device further includes: a sending unit 906, configured to send a partition result to a client device, where the partition result is used to determine a device for executing an SQL statement.

[0297] In this embodiment, the operations performed by each unit in the data processing device are the same as those described above. Figures 1 to 8 The steps performed by the server in the illustrated embodiment are similar and will not be repeated here.

[0298] In this embodiment, the construction unit 902 constructs a first relationship graph through the operation information of the SQL statement, and the partitioning unit 903 partitions the accessed table according to the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by the SQL statement based on the operation information of the SQL statement is more reasonable (for example, distributing the SQL statements accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by the SQL statement can be identified, so as to more reasonably determine the partition strategy of the data in the database based on the aggregation relationship.

[0299] See also Fig.10 , an embodiment of the client device in the embodiment of the present application includes:

[0300] An acquisition unit 1001 is used to acquire a partition result, where the partition result is used to indicate an association relationship between a partition index and a constant field in an SQL statement;

[0301] The acquisition unit 1001 is also used to acquire SQL statements;

[0302] The determining unit 1002 is used to determine the routing of the SQL statement based on the partitioning result.

[0303] Optionally, the client device further includes: a storage unit 1003, used for caching partition results.

[0304] In this embodiment, the operations performed by each unit in the client device are the same as those described above. Figures 1 to 8 The steps performed by the client in the illustrated embodiment are similar and will not be repeated here.

[0305] In this embodiment, since the partition result obtained by the acquisition unit 1001 is the association between the partition index and the constant field in the SQL statement, it is possible to first obtain the constant field in the SQL statement, then look up the table to see which range the value of the partition index falls into (the partition method will affect the determination of the range), and then access the database corresponding to the range.

[0306] See also Fig.11, a schematic diagram of the structure of another data processing device provided by the present application. The data processing device may include a processor 1101, a memory 1102 and a communication port 1103. The processor 1101, the memory 1102 and the communication port 1103 are interconnected via a line. The memory 1102 stores program instructions and data.

[0307] The memory 1102 stores the aforementioned Figures 1 to 8 In the corresponding implementation manner shown, the steps executed by the data processing device correspond to program instructions and data.

[0308] Processor 1101, used to execute the above Figures 1 to 8 The steps shown in any of the illustrated embodiments are performed by a data processing device.

[0309] The communication port 1103 can be used to receive and send data, and to perform the aforementioned Figures 1 to 8 The steps related to obtaining, sending and receiving in any of the embodiments shown.

[0310] In one implementation, the data processing device may include Fig.11 More or fewer components are merely illustrative in this application and are not limiting.

[0311] See also Fig.12 , a schematic diagram of the structure of another client device provided by the present application. The client device may include a processor 1201, a memory 1202 and a communication port 1203. The processor 1201, the memory 1202 and the communication port 1203 are interconnected via a line. The memory 1202 stores program instructions and data.

[0312] The memory 1202 stores the aforementioned Figures 1 to 8 In the corresponding implementation manner shown, the steps executed by the client correspond to program instructions and data.

[0313] Processor 1201, used to execute the above Figures 1 to 8 The steps performed by the client in any of the embodiments shown in the illustrated embodiments.

[0314] The communication port 1203 can be used to receive and send data, and to perform the aforementioned Figures 1 to 8 The steps related to obtaining, sending and receiving in any of the embodiments shown.

[0315] In one implementation, the client device may include a Fig.12 More or fewer components are merely illustrative in this application and are not limiting.

[0316] An embodiment of the present application further provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the possible implementation manner of the data processing device in the aforementioned embodiment.

[0317] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation mode of the above-mentioned data processing device.

[0318] The embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device to implement the functions involved in the possible implementation of the above-mentioned data processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data for the at least one processor. In one possible design, the chip system may also include a memory, which is used to store the necessary program instructions and data for the terminal device. The chip system may be composed of chips, or may include chips and other discrete devices.

[0319] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0320] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0321] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0322] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data processing method, characterized in that: The method is applied to a data processing device, the data processing device includes a database, and the method comprises: Obtaining at least one transaction, wherein the at least one transaction includes a structured query language SQL statement; Building a first relationship graph based on the running information of the SQL statement, the first relationship graph includes a plurality of first nodes and first edges between the plurality of first nodes, each of the plurality of first nodes includes a name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, and the first edge is used to describe an association relationship between the index information in the accessed table; The accessed table is partitioned based on the first relationship graph to obtain a partition result, where the partition result is used to represent an association relationship between a partition index and a constant field in the SQL statement.

2. The method according to claim 1, characterized in that The method further comprises: Acquire database information of the database, wherein the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of the at least two tables; constructing a second relationship graph based on the database information, wherein the second relationship graph includes a plurality of second nodes, and each second node in the plurality of second nodes includes the name of the at least two tables and index information of the at least two tables; The constructing a first relationship graph based on the running information of the SQL statement includes: The second relationship graph is updated based on the operation information to obtain the first relationship graph.

3. The method according to claim 1 or 2, characterized in that: The operation information includes at least one of the following: a field used for constant value comparison in the SQL statement, a join relationship between multiple tables accessed by the SQL statement, read and write types of the multiple tables, index information used during operation, and the order of the SQL statements in the transaction.

4. The method according to claim 3, characterized in that When the operation information includes the join relationship, updating the second relationship graph based on the operation information includes: The first edge is added to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

5. The method according to claim 3, characterized in that: The operation information includes the join relationship, and the second relationship graph also includes second edges between the plurality of second nodes; The updating the second relationship diagram based on the operation information includes: The weight of the second edge is increased in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

6. The method according to any one of claims 3 to 5, characterized in that The method further comprises: Determine the index information used during the execution of the first SQL statement in each transaction as the candidate partition index; Partitioning the multiple tables based on the first relationship graph includes: Determine the partition index based on the candidate partition index and the first relationship graph; The partition result is determined based on the partition index and a constant field in the first SQL statement.

7. The method according to claim 6, characterized in that The determining of the partition index based on the candidate partition index and the first relationship graph includes: Determine the sum of edge weights of different paths corresponding to the node corresponding to the at least one candidate partition index in the first relationship graph; A candidate partition index satisfying a preset condition is determined as the partition index, where the preset condition is related to the edge weight sum.

8. The method according to claim 6 or 7, characterized in that: The at least one transaction includes multiple transactions, and the running information includes the join relationship.

9. The method according to any one of claims 6 to 8, characterized in that The operation information includes the read / write type, and the read / write type is used to increase the read weight or write weight corresponding to the multiple tables; the method further includes: Determine, based on the read weight and the write weight, that the accessed table participates in the partition; Partitioning the accessed table based on the first relationship graph includes: In the case where the accessed table participates in partitioning, the accessed table is partitioned based on the first relationship graph.

10. The method according to any one of claims 1 to 9, characterized in that Before constructing the first relationship graph based on the execution information of the SQL statement, the method further includes: Obtaining a third relationship graph from another device; The step of constructing a first relationship graph based on the running information of the SQL statement includes: The first relationship graph is constructed based on the operation information and the third relationship graph.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The partition result is sent to the client device, and the partition result is used to determine the device that executes the SQL statement.

12. A data processing method, characterized in that: The method is applied to a client device, and the method comprises: Obtaining a partition result, where the partition result is used to indicate an association relationship between a partition index and a constant field in an SQL statement; Obtain the SQL statement; A route for the SQL statement is determined based on the partitioning result.

13. The method according to claim 12, characterized in that The method further comprises: The partition results are cached.

14. A data processing device, characterized in that: The data processing device contains a database, and the data processing device includes: An acquisition unit, configured to acquire at least one transaction, wherein the at least one transaction includes a structured query language SQL statement; a construction unit, configured to construct a first relationship graph based on the running information of the SQL statement, wherein the first relationship graph includes a plurality of first nodes and first edges between the plurality of first nodes, each of the plurality of first nodes includes a name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, and the first edge is used to describe an association relationship between the index information in the accessed table; A partitioning unit is used to partition the accessed table based on the first relationship graph to obtain a partitioning result, wherein the partitioning result is used to represent an association relationship between a partition index and a constant field in the SQL statement.

15. The data processing device according to claim 14, characterized in that The acquisition unit is further used to acquire database information of the database, wherein the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of the at least two tables; The construction unit is further used to construct a second relationship graph based on the database information, wherein the second relationship graph includes a plurality of second nodes, and each second node in the plurality of second nodes includes the name of the at least two tables and index information of the at least two tables; The construction unit is specifically configured to update the second relationship diagram based on the operation information to obtain the first relationship diagram.

16. The data processing device according to claim 14 or 15, characterized in that The operation information includes at least one of the following: a field used for constant value comparison in the SQL statement, a join relationship between multiple tables accessed by the SQL statement, read and write types of the multiple tables, index information used during operation, and the order of the SQL statements in the transaction.

17. The data processing device according to claim 16, characterized in that When the operation information includes the join relationship, the construction unit is specifically used to add the first edge to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

18. The data processing device according to claim 16, characterized in that The operation information includes the join relationship, and the second relationship graph also includes second edges between the plurality of second nodes; The construction unit is specifically used to increase the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

19. The data processing device according to any one of claims 16 to 18, characterized in that: The data processing device further comprises: A first determination unit is used to determine that index information used in the execution of the first SQL statement in each transaction is a candidate partition index; The partition unit is specifically configured to determine the partition index based on the candidate partition index and the first relationship graph; The partitioning unit is specifically configured to determine the partitioning result based on the partitioning index and a constant field in the first SQL statement.

20. The data processing device according to claim 19, characterized in that The partition unit is specifically used to determine the sum of edge weights of different paths corresponding to the node corresponding to the at least one candidate partition index in the first relationship graph; The partition unit is specifically used to determine a candidate partition index that meets a preset condition as the partition index, and the preset condition is related to the edge weight sum.

21. The data processing device according to claim 19 or 20, characterized in that The at least one transaction includes multiple transactions, and the running information includes the join relationship.

22. The data processing device according to any one of claims 19 to 21, characterized in that: The operation information includes the read / write type, and the read / write type is used to increase the read weight or write weight corresponding to the multiple tables; The data processing device further comprises: A second determining unit, configured to determine whether the accessed table participates in a partition based on the read weight and the write weight; The partitioning unit is specifically configured to partition the accessed table based on the first relationship graph when the accessed table participates in partitioning.

23. The data processing device according to any one of claims 14 to 22, characterized in that: The acquisition unit is further used to acquire a third relationship graph from other devices; The construction unit is specifically configured to construct the first relationship diagram based on the operation information and the third relationship diagram.

24. The data processing device according to any one of claims 14 to 23, characterized in that: The data processing device further comprises: The sending unit is used to send the partition result to the client device, and the partition result is used to determine the device that executes the SQL statement.

25. A client device, characterized in that: The client device comprises: An acquisition unit, used for acquiring a partition result, wherein the partition result is used for indicating an association relationship between a partition index and a constant field in an SQL statement; The acquisition unit is further used to acquire the SQL statement; A determination unit is used to determine a route of the SQL statement based on the partition result.

26. The client device according to claim 25, characterized in that The client device also includes: A storage unit is used to cache the partition result.

27. A data processing device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the data processing device executes the method according to any one of claims 1 to 11.

28. A client device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the client device executes the method according to claim 12 or 13.

29. A communication system, characterized in that: Comprising the data processing device as claimed in claim 27 and the client device as claimed in claim 28.

30. A computer storage medium, characterized in that The method comprises computer instructions, and when the computer instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 13.

31. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 13.

Images