Data processing method and related device

Abstract

The embodiments of the present application disclose a data processing method, which can be applied to a distributed database. The method comprises: first acquiring at least one transaction; constructing a first relationship graph on the basis of running information of an SQL statement in the transaction, wherein the first relationship graph comprises a plurality of first nodes and a first edge between a plurality of first nodes, each of the plurality of first nodes comprises a name of a table accessed by the SQL statement and index information (for example, an index field or an index name) of the table accessed by the SQL statement, and the first edge is used for describing an association relationship between the index information in the accessed table; and then, partitioning the accessed table on the basis of the first relationship graph. Compared with partitioning based on human experience, the manner of partitioning, on the basis of the running information of the SQL statement, the table accessed by the SQL statement is more rational (for example, SQLs accessing the same data are distributed to the same node), thereby reducing frequent transfer of data between nodes.

Description

A data processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 30, 2023, with application number 202311113815.6 and invention name “A data processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of databases, and in particular to a data processing method and related equipment. Background Art

[0003] A database is usually an organized and uniformly managed collection of large amounts of data, and it usually needs to have four characteristics: atomicity, consistency, isolation, and durability, which can be referred to as the ACID characteristics.

[0004] In a shared disk database architecture, all compute nodes can access all data. When a compute node receives a Structured Query Language (SQL) request, and the data required by this SQL is not on this node, the node will ask other nodes whether they have this data. If so, the data will be transferred from other nodes to this node. If other nodes do not have this data either, the compute node directly loads the page containing the data from the storage. If SQL access is randomly distributed to multiple nodes, then data pages will be frequently transferred from one compute node to another according to the needs of SQL calculations. The frequent transfer of pages between nodes increases the communication overhead between nodes, affecting the response latency of SQL statements and the overall throughput of the system.

[0005] Therefore, how to reduce the frequent transfers between nodes is a technical problem that needs to be solved urgently;

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a data processing method and related devices that can identify the aggregation relationship between data accessed by SQL statements, thereby more reasonably determining the partitioning strategy of data in the database based on the aggregation relationship.

[0008] In a first aspect, an embodiment of the present application provides a data processing method that can be applied to a distributed database (share disk), etc. The method can be executed by a data processing device, or by a component of the data processing device (such as a processor, a chip, or a chip system, etc.). The data processing device contains a database. In this method, at least one transaction is first obtained, and then a first relationship graph is constructed based on the execution information of the SQL statement in the transaction. The first relationship graph includes a plurality of first nodes and a first edge between the plurality of first nodes. Each of the plurality of first nodes includes the name of the table accessed by the SQL statement and the index information of the table accessed by the SQL statement (such as an index field or an index name, etc.). The first edge is used to describe the association relationship between the index information in the accessed table; then, the accessed table is partitioned based on the first relationship graph to obtain a partition result, and the partition result is used to represent the association relationship between the partition index and the constant field in the SQL statement.

[0009] In an embodiment of the present application, a first relationship graph is constructed using the SQL statement execution information, and the accessed table is partitioned based on the first relationship graph. Compared to partitioning based on manual experience, this method of partitioning the table accessed by SQL statements based on the SQL statement execution information is more reasonable (for example, distributing SQL statements accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by SQL statements can be identified, so that a more reasonable partitioning strategy for the data in the database can be determined based on the aggregation relationship.

[0010] Optionally, in a possible implementation of the first aspect, the above steps also include: obtaining database information of the database, the database information including primary keys and foreign keys of at least two tables; or the database information including index information of at least two tables; constructing a second relationship graph based on the database information, the second relationship graph including multiple second nodes, each of the multiple second nodes including the names of at least two tables and index information of at least two tables; constructing a first relationship graph based on the execution information of the SQL statement, including: updating the second relationship graph based on the execution information to obtain the first relationship graph.

[0011] In this possible implementation, a static relationship graph (i.e., the second relationship graph) can be first constructed using database information, and then the aggregation relationship of the data accessed according to the SQL statement can be updated to obtain a dynamic relationship graph (i.e., the first relationship graph), thereby ensuring that the tables in the database are as much as possible in the first relationship graph.

[0012] Optionally, in a possible implementation of the first aspect, the above-mentioned execution information includes at least one of the following: a field used for constant value comparison in the SQL statement, a join relationship between multiple tables accessed by the SQL statement, the read and write types of multiple tables, index information used during the execution process, and the order of the SQL statements in the transaction.

[0013] In this possible implementation, the aggregation relationship of SQL access data can be identified through the execution information of the SQL statement, so that a more accurate partitioning result can be determined.

[0014] Optionally, in a possible implementation of the first aspect, when the above-mentioned operation information includes a join relationship, updating the second relationship graph based on the operation information includes: adding a first edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0015] In this possible implementation, edges in the static relationship graph are added through the join relationship in the SQL statement execution information, so that the static relationship graph can be updated according to the accessed vertices, so that the obtained dynamic relationship graph is more consistent with the access behavior.

[0016] Optionally, in a possible implementation of the first aspect, the above-mentioned operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; updating the second relationship graph based on the operation information includes: increasing the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0017] In this possible implementation, the weight of the edge in the static relationship graph is increased by the join relationship in the SQL statement execution information, and the access frequency can be determined based on the weight, which facilitates the subsequent determination of the partition result based on the edge.

[0018] Optionally, in a possible implementation of the first aspect, the above steps also include: determining index information used during the execution of the first SQL statement in each transaction as an alternative partition index; partitioning multiple tables based on the first relationship graph, including: determining a partition index based on the alternative partition index and the first relationship graph; and determining a partition result based on the partition index and a constant field in the first SQL statement.

[0019] In this possible implementation, the candidate partition index is first determined by the first SQL statement, the partition index is then determined according to the first relationship graph, and the partition result is further determined according to the partition index and the constant field in the first SQL statement.

[0020] Optionally, in a possible implementation of the first aspect, the above steps of: determining the partition index based on the alternative partition index and the first relationship graph, include: determining the sum of edge weights of different paths corresponding to the nodes in the first relationship graph corresponding to at least one alternative partition index; determining the alternative partition index that meets the preset conditions as the partition index, and the preset conditions are related to the sum of edge weights.

[0021] In this possible implementation, the candidate partition index that meets the preset conditions in the first relationship graph is determined as the partition index through the edge weight, so that the partition index can better meet the access habits.

[0022] Optionally, in a possible implementation manner of the first aspect, the at least one transaction includes multiple transactions, and the execution information includes a join relationship.

[0023] In this possible implementation, the join relationship can reflect the association relationship between the various tables. The weight of the edge in the relationship graph can be updated based on the join relationship, thereby indirectly reflecting the importance of the node corresponding to the edge.

[0024] Optionally, in a possible implementation of the first aspect, the above-mentioned operating information includes a read-write type, and the read-write type is used to increase the read weight or write weight corresponding to multiple tables; the method also includes: determining whether the accessed table participates in the partition based on the read weight and the write weight; partitioning the accessed table based on the first relationship graph, including: when the accessed table participates in the partition, partitioning the accessed table based on the first relationship graph.

[0025] In this possible implementation, the read and write weights of multiple tables during the execution of SQL statements can be used to determine which tables participate in the partitioning according to the read and write weights.

[0026] Optionally, in a possible implementation of the first aspect, the above step: before constructing the first relationship graph based on the execution information of the SQL statement, the method also includes: obtaining a third relationship graph from other devices; constructing the first relationship graph based on the execution information of the SQL statement, including: constructing the first relationship graph based on the execution information and the third relationship graph.

[0027] In this possible implementation, the first relationship graph may be constructed through static relationship graphs at other devices, thereby improving the applicability of the solution.

[0028] Optionally, in a possible implementation manner of the first aspect, the above steps further include: sending a partitioning result to the client device, where the partitioning result is used to determine a device for executing the SQL statement.

[0029] In this possible implementation, the determined partitioning result may be sent to the client, so that the client can route the SQL statement according to the partitioning result, thereby improving processing efficiency.

[0030] A second aspect of an embodiment of the present application provides a data processing method that can be applied to a distributed database (shared disk), etc. The method can be executed by a client device or a component of the client device (e.g., a processor, chip, or chip system). In the method, a partition result is obtained, which is used to represent the association between a partition index and a constant field in an SQL statement; an SQL statement is obtained; and a routing of the SQL statement is determined based on the partition result.

[0031] In the embodiment of the present application, since the partition result obtained is the association between the partition index and the constant field in the SQL statement, it is possible to first obtain the constant field in the SQL statement, then look up the table to see which range the value of the partition index falls into (the partitioning method will affect the determination of the range), and then access the database corresponding to the range.

[0032] Optionally, in a possible implementation manner of the second aspect, the above steps further include: caching partition results.

[0033] In this possible implementation, partition results can be cached to facilitate subsequent determination of SQL statement routing and improve processing efficiency.

[0034] A third aspect of an embodiment of the present application provides a data processing device that can be applied to a distributed database (share disk), etc. The data processing device includes a database and comprises: an acquisition unit for acquiring at least one transaction, wherein the at least one transaction includes a structured query language (SQL) statement; a construction unit for constructing a first relationship graph based on the execution information of the SQL statement, wherein the first relationship graph includes a plurality of first nodes and first edges between the plurality of first nodes, wherein each of the plurality of first nodes includes the name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, and the first edge is used to describe the association between the index information in the accessed table; and a partitioning unit for partitioning the accessed table based on the first relationship graph to obtain a partitioning result, wherein the partitioning result is used to represent the association between the partition index and the constant field in the SQL statement.

[0035] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is also used to obtain database information of the database, the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of at least two tables; the construction unit is also used to construct a second relationship graph based on the database information, the second relationship graph includes multiple second nodes, and each second node in the multiple second nodes includes the names of at least two tables and index information of at least two tables; the construction unit is specifically used to update the second relationship graph based on the operation information to obtain the first relationship graph.

[0036] Optionally, in a possible implementation of the third aspect, the above-mentioned execution information includes at least one of the following: a field used for constant value comparison in the SQL statement, a join relationship between multiple tables accessed by the SQL statement, the read and write types of multiple tables, index information used during the execution process, and the order of the SQL statements in the transaction.

[0037] Optionally, in a possible implementation of the third aspect, when the above-mentioned operation information includes a join relationship, a construction unit is constructed, specifically for adding a first edge in the second relationship graph based on the join relationship to obtain a first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0038] Optionally, in a possible implementation of the third aspect, the above-mentioned operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; the construction unit is specifically used to increase the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0039] Optionally, in a possible implementation of the third aspect, the above-mentioned data processing device further includes: a first determination unit, used to determine that index information used during the execution of the first SQL statement in each transaction is an alternative partition index; a partitioning unit, specifically used to determine the partition index based on the alternative partition index and the first relationship graph; and a partitioning unit, specifically used to determine the partition result based on the partition index and a constant field in the first SQL statement.

[0040] Optionally, in a possible implementation of the third aspect, the above-mentioned partition unit is specifically used to determine the sum of edge weights of different paths corresponding to the node corresponding to at least one alternative partition index in the first relationship graph; the partition unit is specifically used to determine the alternative partition index that meets the preset conditions as the partition index, and the preset conditions are related to the sum of edge weights.

[0041] Optionally, in a possible implementation manner of the third aspect, the at least one transaction includes multiple transactions, and the execution information includes a join relationship.

[0042] Optionally, in a possible implementation of the third aspect, the above-mentioned operating information includes read and write types, which are used to increase the read weight or write weight corresponding to multiple tables; the data processing device also includes: a second determination unit, used to determine whether the accessed table participates in partitioning based on the read weight and the write weight; a partitioning unit, specifically used to partition the accessed table based on the first relationship graph when the accessed table participates in partitioning.

[0043] Optionally, in a possible implementation of the third aspect, the above-mentioned acquisition unit is further used to obtain a third relationship diagram from other devices; and the construction unit is specifically used to construct the first relationship diagram based on the operation information and the third relationship diagram.

[0044] Optionally, in a possible implementation of the third aspect, the above-mentioned data processing device further includes: a sending unit, configured to send a partitioning result to a client device, where the partitioning result is used to determine a device for executing an SQL statement.

[0045] A fourth aspect of an embodiment of the present application provides a client device that can be applied to a distributed database (share disk), etc. The client device includes: an acquisition unit for acquiring a partition result, where the partition result is used to represent an association between a partition index and a constant field in an SQL statement; the acquisition unit is also used to acquire an SQL statement; and a determination unit is used to determine a route for the SQL statement based on the partition result.

[0046] Optionally, in a possible implementation manner of the fourth aspect, the client device further includes: a storage unit, configured to cache partitioning results.

[0047] A fifth aspect of an embodiment of the present application provides a data processing device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, and when the program or instructions are executed by the processor, the data processing device implements the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0048] The sixth aspect of an embodiment of the present application provides a client device, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the client device implements the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0049] A seventh aspect of an embodiment of the present application provides a communication system, including the data processing device of the fifth aspect above, and / or the client device of the sixth aspect.

[0050] An eighth aspect of an embodiment of the present application provides a computer-readable medium having a computer program or instruction stored thereon. When the computer program or instruction runs on a computer, the computer executes the method of the aforementioned first aspect or any possible implementation of the first aspect, or the computer executes the method of the aforementioned second aspect or any possible implementation of the second aspect.

[0051] A ninth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the aforementioned first aspect or any possible implementation of the first aspect, or enables the computer to execute the method of the aforementioned second aspect or any possible implementation of the second aspect.

[0052] The tenth aspect of an embodiment of the present application provides a chip system, which includes at least one processor for supporting a data processing device to implement the functions involved in the above-mentioned first aspect or any possible implementation of the first aspect, or for supporting a client device to implement the functions involved in the above-mentioned second aspect or any possible implementation of the second aspect.

[0053] In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the data processing device or client device. The chip system may be composed of a chip or may include a chip and other discrete components. Optionally, the chip system may also include an interface circuit that provides program instructions and / or data to at least one processor.

[0054] Among them, the technical effects brought about by the third, fifth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0055] The technical effects brought about by the fourth, sixth, seventh, eighth, ninth, and tenth aspects or any possible implementation methods thereof can be referred to the technical effects brought about by the second aspect or different possible implementation methods of the second aspect, and will not be repeated here.

[0056] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: the present application constructs a first relationship graph through the running information of the SQL statement, and partitions the accessed table according to the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by the SQL statement based on the running information of the SQL statement is more reasonable (for example, distributing SQLs accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by the SQL statement can be identified, so that the partitioning strategy of the data in the database can be more reasonably determined based on the aggregation relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] FIG1 is a schematic diagram of the structure of the system architecture provided in an embodiment of the present application;

[0058] FIG2 is a flow chart of a data processing method provided in an embodiment of the present application;

[0059] FIG3 is a relationship diagram corresponding to Example 1 provided in an embodiment of the present application;

[0060] FIG4 is a relationship diagram corresponding to Example 1 and Example 2 provided in an embodiment of the present application;

[0061] FIG5 is a relationship diagram corresponding to Example 1, Example 2, and Example 3 provided in an embodiment of the present application;

[0062] FIG6 is a relationship diagram corresponding to Example 1, Example 2, Example 3 and Example 4 provided in an embodiment of the present application;

[0063] FIG7 is a relationship diagram corresponding to Example 1, Example 2, Example 3, Example 4, and Example 5 provided in an embodiment of the present application;

[0064] FIG8 is another flow chart of the data processing method provided in an embodiment of the present application;

[0065] FIG9 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0066] FIG10 is a schematic diagram of the structure of a client device provided in an embodiment of the present application;

[0067] FIG11 is another structural diagram of a data processing device provided in an embodiment of the present application;

[0068] FIG12 is another schematic diagram of the structure of the client device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The embodiments of the present application provide a data processing method and related devices that can identify the aggregation relationship between data accessed by SQL statements, thereby more reasonably determining the partitioning strategy of data in the database based on the aggregation relationship.

[0070] To facilitate understanding, the following first introduces the relevant terms and concepts mainly involved in the embodiments of this application.

[0071] 1. Affairs

[0072] A transaction is a user-defined sequence of database operations that are either all or nothing, and is an indivisible unit of work. A transaction can be a single SQL statement, a group of SQL statements, or an entire program.

[0073] In the process of writing or updating data, a database management system must have four characteristics to ensure that transactions are correct and reliable: atomicity (or indivisibility), consistency (consistency), isolation (also known as independence), and durability.

[0074] Atomicity: All operations within a transaction are either fully completed or incomplete, with no interruptions occurring at any point in the process. If an error occurs during transaction execution, the transaction is rolled back to the state before the transaction began, as if the transaction had never been executed. This means that transactions are indivisible and irreducible.

[0075] Consistency: The integrity of the database is not compromised before and after the transaction begins. This means that the written data must fully comply with all pre-set constraints, triggers, cascading rollbacks, etc.

[0076] Isolation: A database's ability to allow multiple concurrent transactions to read, write, and modify its data simultaneously prevents data inconsistencies caused by cross-execution of multiple transactions. Transaction isolation is divided into different levels, including read uncommitted, read committed, repeatable read, and serializable.

[0077] Persistence: After the transaction is completed, the changes to the data are permanent and will not be lost even if the system fails.

[0078] 2. Partition

[0079] Partitioning is a key feature of database systems. It divides and conquers large data tables, represented by partitioned tables. Partitioned tables can also include partition segments. Appropriate partitioning can improve database system performance, reduce the number of data reads and writes required by the system, reduce the amount of data processed by the database server, and facilitate database management. A partitioning strategy is a clustering strategy set by the designer for partitioning large data tables. It specifies which partition segment a data record in a table is mapped to. Database systems also provide data partition management capabilities to facilitate partition management.

[0080] Horizontal Partitioning: Partition the rows of a table so that physically separated data in different groups are combined together. All columns in the table can be found in each partition, maintaining the attribute structure of the table.

[0081] Vertical partitioning: Divide certain specific columns into specific partitions to reduce the width of the table. Each partition stores the rows where the columns are located.

[0082] Partitioned table: A partitioned table is a data organization scheme that distributes table data into multiple storage objects (called data partitions or ranges) based on the values ​​in one or more table partitioning key columns in the table.

[0083] 3. Primary key and foreign key

[0084] A primary key (PK) is a key that is one or more fields in a table whose value uniquely identifies a record in the table. It cannot be repeated or empty, ensuring atomicity of each column.

[0085] A foreign key (FK) is a foreign keyword that references one or more columns in another table. The referenced columns should have a primary key constraint or a unique constraint. Foreign keys are used to establish and strengthen the connection between two tables and control the data stored in the foreign key table, maintaining data consistency and integrity.

[0086] 4. Graph

[0087] A graph consists of a finite, non-empty set of vertices and a set of edges between them, typically represented as G(V,E), where G represents a graph, V is the set of vertices in G, and E is the set of edges in G. The data elements in a graph are called vertices. A graph cannot be empty of vertices (finite non-emptiness). The relationships between vertices in a graph are represented by edges (the set of edges can be empty).

[0088] Undirected graphs: If the edge between any two vertices in the graph is an undirected edge (in short, it is an edge without direction), then the graph is called an undirected graph.

[0089] Directed graphs: If the edge between any two vertices in the graph is a directed edge (in short, it is an edge with a direction), then the graph is called a directed graph.

[0090] Path: In an undirected graph, if there is a set of edges starting from vertex Vi that can reach vertex Vj, then the vertex sequence from vertex Vi to vertex Vj is called the path from vertex Vi to vertex Vj.

[0091] Weight: Some edges or arcs of a graph have numbers associated with them. This number associated with an edge or arc of a graph is called a weight.

[0092] 5. Table Join

[0093] A join is used to combine rows from two or more tables. There are various join methods, such as inner join, left outer join, right outer join, and full join.

[0094] INNER JOIN: Returns rows if there is at least one match in the table.

[0095] LEFT JOIN: Returns all rows from the left table even if there is no match in the right table.

[0096] RIGHT JOIN: Returns all rows from the right table even if there is no match in the left table.

[0097] FULL JOIN: Returns rows if there is a match in either table.

[0098] 6. TPC-C

[0099] TPC-C is a benchmark for measuring online transaction processing. TPC-C provides a specification to describe this benchmark. The TPC-C model simulates the sales model of a commodity wholesale company. This model covers the entire process of selling a series of products to customers, including operations such as order management, inventory management, and account balance management. These operations involve concepts such as warehouse, product, customer, and order. Data tables and corresponding database operations are constructed around these concepts. TPC-C wholesale operations conceptually include five operations: new order operation, payment operation, order status operation, shipment operation, and inventory operation. Together, these operations constitute the complete wholesale process. The new order operation is the core of the TPMC measurement, which measures the number of transactions per minute (tpm).

[0100] Currently, under the shared disk database architecture, all compute nodes can access all data. When a compute node receives a SQL request that requires data not on this node, it will query other nodes for the required data. If so, the data will be transferred from other nodes to this node. If other nodes also do not have the data, the compute node will directly load the page containing the data from storage. If SQL access is randomly distributed to multiple nodes, data pages will be frequently transferred from one compute node to another as needed for SQL calculations. This frequent transfer of pages between nodes increases the communication overhead between nodes, affecting the response latency of SQL statements and the overall throughput of the system.

[0101] Among these SQL statements, some will modify data, and the modified pages become dirty pages. If the transactions for these SQL statements have been committed, but the corresponding dirty pages have not been persisted to storage in a timely manner, if this node goes down, this page must be restored in the memory of other nodes through the WAL log before it can be accessed. Before this page is restored, if SQL statements from other nodes need to access the data on this page, they will have no choice but to wait. If the SQL statements that modify data are randomly distributed, then each node caches random dirty pages. The failure of any computing node may cause SQL statements from other nodes to wait for these dirty pages to be restored, thus affecting system availability.

[0102] To solve the above problems, we found that by properly classifying and scheduling SQL access and routing SQL access to closely related data to one node, we can reduce the interaction between nodes and improve system performance. It can also avoid the impact of node failure on SQL access on other nodes, thereby improving system availability.

[0103] Based on the above ideas, an embodiment of the present application provides a data processing method that constructs a first relationship graph based on the execution information of SQL statements and partitions the accessed table based on the first relationship graph. Compared with partitioning based on manual experience, this method of partitioning the table accessed by SQL statements based on the execution information of SQL statements is more reasonable and can reduce the frequent transfer of data between nodes. In other words, it is possible to identify the aggregation relationship between the data accessed by SQL statements, thereby more reasonably determining the partitioning strategy of the data in the database based on this aggregation relationship.

[0104] The following describes the system architecture provided by the embodiments of the present application. The system architecture can be applied to a distributed database (shared disk), etc., and is not specifically limited here.

[0105] 1 , an embodiment of the present invention provides a system architecture, which includes: an application 101 , a driver 102 , a database instance 103 , a database instance 104 , and a shared storage management 105 .

[0106] Application 101 generally includes at least one business module, and each business module is connected to a database instance (e.g., database instance 103 and / or database instance 104) via driver 102. Typically, each business module is connected to two database instances, such as database instance 103 and database instance 104, and the business module then distributes SQL statements to database instance 103 and database instance 104 for execution. In addition, in the case of multiple business modules, since the data correlation between different business modules is relatively small, the link between the business module and the database instance is fixed, and the SQL is fixedly distributed to a specific database instance, avoiding the frequent transfer of data pages between multiple nodes caused by random distribution of SQL, thereby improving the response delay of SQL statements and the overall throughput of the system.

[0107] Driver 102 includes client link / session management 1021, SQL distribution and reading results 1022, and client SQL routing 1023. Client link / session management 1021 is responsible for establishing a link between application 101 and the database instance. SQL distribution and reading results 1022 is responsible for receiving the SQL request of the application and distributing the SQL routing to a database instance. Waiting for the SQL execution result of the database instance and returning the execution result to application 101. Client SQL routing 1023 is responsible for routing the database instance accessed by SQL. In the embodiment of the present application, the client SQL routing 1023 receives the partitioning result from database instance 103 or database instance 104. The client can determine the database instance (or computing node) of the SQL routing based on the differentiation result. In addition, driver 102 can be Open Database Connectivity (ODBC) or Java Database Connectivity (JGBC), etc., which are not limited here.

[0108] Database instance 103 includes server link / session management 1031, SQL engine 1032, and inter-node communication 1033. SQL engine 1032 includes SQL parsing / optimization / execution 10321 and server SQL routing 10322. Server link / session management 1031 is responsible for establishing a link with application 101 through driver 102, while SQL parsing / optimization / execution 10321 is responsible for parsing, optimizing, and executing SQL statements. Server SQL routing 10322 determines an SQL route based on SQL execution information from SQL parsing / optimization / execution 10321 and returns the SQL route to application 101 through driver 102. Inter-node communication 1033 is responsible for communicating with other data instances (e.g., database instance 104). The process of determining an SQL route can be understood as the database server cache and client cache building a lookup table. The lookup table inputs are the SQL statement string and parameter conditions. Based on the SQL statement string and parameter conditions, the lookup table maps the parameter conditions to the table's partition fields and determines the routing node based on the partition field value range.

[0109] Database instance 104 includes server connection / session management 1041, SQL engine 1042, and inter-node communication 1043. SQL engine 1042 includes SQL parsing / optimization / execution 10421 and server SQL routing 10422. Server connection / session management 1041 is responsible for establishing a connection with application 101 through driver 102, while SQL parsing / optimization / execution 10421 is responsible for parsing, optimizing, and executing SQL. Server SQL routing 10422 determines an SQL route based on SQL execution information generated by SQL parsing / optimization / execution 10421 and returns the SQL route to application 101 through driver 102. Inter-node communication 1043 is responsible for communicating with other data instances (e.g., database instance 103).

[0110] The shared storage management 105 includes a redo log file 1051 and a data file 1052 .

[0111] Database instance 103 and database instance 104 receive the SQL sent by driver 102 and parse / execute the SQL. If the data page accessed by the SQL is in this instance, it is processed directly in this instance. If it is not in this instance, a request is sent to the instance to transfer the data page to this node, and this node then accesses and calculates it. If all database instances do not have the data page to be accessed, the data page is read from the data file to the memory of this node through shared storage management 105, and then accessed and calculated. If the data page is modified, the redo log of the page modification needs to be persisted to the Redo log file 1051 through shared storage management before the transaction is committed. After the SQL is executed, the execution result is returned to the driver 102. The database instance background task periodically performs checkpoints. For modified data pages, when the checkpoint is performed, these pages are persisted to the data file 1052 through shared storage management 104. The Redo log records the modified value of a data block and is used to restore data updated by a successful transaction that has not been written to the data file.

[0112] Furthermore, within application 101, driver 102 can be deployed on the same compute server as a database instance (e.g., database instance 103 or database instance 104), or it can be deployed separately on a separate compute server. Database instances run on compute servers (database instance 103 and database instance 104 can be on the same or different compute servers), and shared storage management 105 runs on a storage server. Database instances are interconnected, and so are shared storage management. Network connections between database instances and between database instances and the shared storage management are established via switches.

[0113] In the architecture shown in Figure 1, the database instance's data page buffer determines the node where the data page is currently located based on SQL access needs and transfers the data page from other nodes to the current node. This embodiment analyzes the characteristics of SQL access data and rationally schedules and distributes SQL, reducing the transfer of data pages between nodes and improving SQL access performance. For example, in a shared disk database architecture, SQL routing distribution can distribute SQL queries accessing the same data to the same node.

[0114] In addition, client applications can execute SQL statements in various ways: Parse-Bind-Execute (PBE), directly, through stored procedures, indirectly through triggers, and through anonymous blocks. SQL dispatch routing categorizes these SQL execution methods into three categories: those where ODBC / JDBC can detect parameter conditions, those where the server detects parameter conditions through lexical and syntactic parsing, and those where there are no parameter conditions at all. ODBC / JDBC primarily detects parameter conditions using the PBE method. SQL statements executed using this method are routed by the client, which routes the SQL statement to the appropriate node for execution based on the bound parameter conditions. If the server detects parameter conditions, it performs lexical and syntactic analysis before routing the SQL statement. SQL statements without parameter conditions, which cannot be routed, are directly executed by the node itself without routing.

[0115] Because transactions in a Share Disk database can only run on a single node, if a transaction includes multiple SQL statements, the node where the transaction runs is determined by the node where the first SQL statement was routed. Therefore, ODBC / JDBC only searches the routing table for the first SQL statement in the transaction, based on the transaction's execution status. Subsequent SQL statements in the transaction are not searched in the routing table and are directly distributed to the node where the previous SQL statement was executed. Note that this refers to the node where the SQL statement is executed, not the node to which it is distributed, as the server may forward the statement to another node for execution.

[0116] It is understandable that the system architecture shown in FIG1 is only an example. In actual applications, there may be more database instances, Redo log files, etc., which are not specifically limited here.

[0117] The data processing method provided in the embodiments of the present application is described below. The method can be executed by a data processing device, or by a component of the data processing device (such as a processor, chip, or chip system, etc.). The data processing device can be a server, a client (also referred to as a client device), or a system consisting of a server and a client (as shown in Figure 1 above).

[0118] The aforementioned clients may include mobile phones, tablet computers (pads), portable game consoles, personal digital assistants (PDAs), laptop computers, ultra mobile personal computers (UMPCs), handheld computers, netbooks, in-vehicle media playback devices, wearable electronic devices, virtual reality (VR) terminals, augmented reality (AR) terminals, and other digital display products. The aforementioned servers may include computing servers, cloud servers, and the like. The specific circumstances of the clients and servers are not limited.

[0119] The method provided in the embodiment of the present application is mainly applicable to the distributed database (share disk) scenario.

[0120] Please refer to FIG. 2 , which is a flowchart of a data processing method provided in an embodiment of the present application. The method may include steps 201 to 205 .

[0121] Step 201: The client sends a transaction to the server. In response, the server receives the transaction sent by the client. This step is optional.

[0122] Step 202: The server constructs a first relationship graph based on the execution information of the SQL statement.

[0123] Step 203: The server partitions the accessed table based on the first relationship graph to obtain a partitioning result.

[0124] Step 204: The server sends the partitioning result to the client. Correspondingly, the client receives the partitioning result sent by the server. This step is optional.

[0125] Step 205: The client determines the SQL route based on the partitioning result. This step is optional.

[0126] Steps 201 to 205 are described in detail below:

[0127] Step 201: The client sends a transaction to the server. In response, the server receives the transaction sent by the client. This step is optional.

[0128] Optionally, the client first obtains at least one transaction, each transaction including at least one SQL statement, and then sends the at least one transaction to the server.

[0129] Of course, the server can also obtain transactions in other ways, such as obtaining transactions from a database, receiving transactions sent by other devices, determining transactions based on user operations, etc., which are not limited here.

[0130] It is understandable that in the embodiment of the present application, the number of at least one transaction may be one or more, and the number of at least one SQL statement included in each transaction may also be one or more, which is not specifically limited here.

[0131] For example, let's use the TPC-C model with five transactions: order creation, order payment, order status query, shipment, and inventory check. This simulates the operation of a warehouse-centric wholesale system.

[0132] For ease of description, the following first describes the example of the TPC-C model mentioned above.

[0133] The TPC-C test model is a large wholesale company with several geographically distributed warehouses. As the business expands, the company adds new warehouses. Each warehouse supplies 10 sales outlets, each of which serves 3,000 customers. Each customer orders an average of 10 items. Approximately 1% of all orders are out of stock at the warehouse directly responsible and must be supplied by warehouses in other regions. Furthermore, each warehouse maintains inventory records for the company's 100,000 products.

[0134] The TPC-C benchmark system database includes nine tables: Warehouse, District, Customer, History, Order, New-Order, Order-Line, Item, and Stock. The Order-Line table can also be called an Order Item or Order Entry.

[0135] The Warehouse table records warehouses storing goods. The District table records district information, with each warehouse providing wholesale services for 10 districts. The Customer table contains 3,000 users per district, who transact with the warehouse in that district (or, in special cases, with other warehouses). The Order table records user transactions, with each transaction potentially involving 5 to 10 specific items. The New-Order table records new orders, which are deleted from the New-Order table when they are completed.

[0136] Tables 1 to 9 below give examples of the detailed fields of the above nine tables:

[0137] Table 1-Warehouse

[0138] Table 2-District

[0139] Table 3 - Customer Table (Customer)

[0140] Table 4-History Table

[0141] Table 5-New-Order

[0142] Table 6-Order Table (Order)

[0143] Table 7-Order Line Table (Order-Line)

[0144] Table 8-Supply Commodity Item List (Item)

[0145] Table 9-Stock

[0146] It is understandable that the fields of the above 9 tables are just examples. In actual applications, there may be other forms (for example, the number of rows, columns, names, length limits, etc. of the table can be set according to actual needs), which are not specifically limited here.

[0147] The following describes the process of building index relationships between SQL statements and tables involved in each transaction:

[0148] In Example 1, the SQL statements involved in Transaction 1 - Create Order Transaction (The New-Order Transaction) are as follows:

[0149] Example 2, Transaction 2 - Order Payment Transaction (The Payment Transaction) involves the following SQL statements:

[0150] Example 3, Transaction 3 - Order Status Query Transaction (The Order-Status Transaction) involves the following SQL statements:

[0151] Example 4, Transaction 4 - The Delivery Transaction involves the following SQL statements:

[0152] Example 5, Transaction 5 - Stock-Level Transaction involves the following SQL statements:

[0153] It is understandable that the above description uses five TPC-C transactions as an example. In actual applications, scenarios such as TPC-H, TPC-T, and TPC-E may also be used, and the number of transactions and the specific structure of the SQL statements involved may vary. These are not limited here.

[0154] Step 202: The server constructs a first relationship graph based on the execution information of the SQL statement.

[0155] After obtaining the transaction, the server constructs a first relationship graph based on the execution information of at least one SQL statement in the transaction. The first relationship graph includes first edges between multiple first nodes. Each of the multiple first nodes includes the name of a table accessed by the SQL statement and index information of the accessed table. The first edges are used to describe the association between the index information in the accessed tables.

[0156] The operation information in the embodiment of the present application includes at least one of the following: the field used for constant value comparison in the SQL statement, the join relationship between multiple tables accessed by the SQL statement, the read and write types of multiple tables, the index information used during the operation process, and the order of the SQL statements in the transaction.

[0157] In the embodiments of the present application, there are many situations in which edge information is added. For example, edges can be constructed or edge weights can be increased based on join relationships, or edges can be constructed or edge weights can be increased between index fields of tables accessed by different SQLs, etc. The specific situations are not limited here.

[0158] For example, continuing with Example 1 above, during the execution of the first SQL statement in Example 1, access to the warehouse table (Warehouse) will use the warehouse table_PK (W_ID), and access to the customer table (Customer) will use the customer table_PK (C_W_ID, C_D_ID, C_ID). After this SQL statement is executed, the warehouse table_PK (W_ID) and the customer table_PK (C_W_ID, C_D_ID, C_ID) are used as two first nodes, and a first edge is added between the two first nodes, and the weight of the first edge is 1. At the same time, the read weight of the warehouse table and the customer table is increased by 1. It should be noted that the join field c_w_id = w_id is useless here, because although w_id is the index vertex of WAREHOUSE, c_w_id is not.

[0159] In Example 1, the second SQL statement uses the district_PK(D_W_ID, D_ID) to query the district table (District). This SQL statement is a single-table query, without any table joins. The district_PK(D_W_ID, D_ID) query implicitly uses the warehouse_PK(W_ID) candidate partition index. Therefore, the district_PK(D_W_ID, D_ID) is used as the first node, and an edge with a weight of 1 is added between the warehouse_PK(W_ID) and district_PK(D_W_ID, D_ID). The read weight of the district table is also increased by 1.

[0160] The process of executing the other SQL statements in Example 1 to construct the first node and the first edge is similar to the above two SQL statements. The first relationship graph obtained after executing all SQL statements in Example 1 is shown in Figure 3. In addition, the read and write weights of each table obtained after executing all SQL statements in Example 1 are shown in Table 10.

[0161] Table 10 - Read and write weights of each table after running Example 1

[0162] As shown in Table 10, the warehouse table has a read weight of 1 and a write weight of 0. The customer table has a read weight of 1 and a write weight of 0. The region table has a read weight of 1 and a write weight of 1. The order table has a read weight of 0 and a write weight of 1. The new order table has a read weight of 0 and a write weight of 1. The supply item entry table has a read weight of 10 and a write weight of 0. The inventory table has a read weight of 10 and a write weight of 10. The order line table has a read weight of 0 and a write weight of 10.

[0163] For example, continuing with Example 2 above, the relationship diagram in Figure 3 can be updated after executing the SQL statements in Example 2 to obtain the relationship diagram in Figure 4. Compared to the relationship diagram in Figure 3, the relationship diagram in Figure 4 adds two first nodes (history table_FK2(H_W_ID, H_D_ID) and history table_FK1(H_C_W_ID, H_C_D_ID, H_C_ID)) and edges between these two newly added nodes and the warehouse table_PK(W_ID) node. Furthermore, the edge weights are increased (as shown in the bold portion in Figure 4).

[0164] In addition, after executing all SQL statements in Example 1 and Example 2, the read and write weights of each table are shown in Table 11;

[0165] Table 11 - Read and write weights of each table after running Example 1 and Example 2

[0166] The new sections in Table 11 relative to Table 10 are the black sections in Table 11 above. After executing the SQL statements in Examples 1 and 2, the weights of the tables are as follows: The warehouse table has a read weight of 2 and a write weight of 1. The customer table has a read weight of 4 and a write weight of 1. The region table has a read weight of 2 and a write weight of 2. The order table has a read weight of 0 and a write weight of 1. The new order table has a read weight of 0 and a write weight of 1. The supply item item table has a read weight of 10 and a write weight of 0. The inventory table has a read weight of 10 and a write weight of 10. The order line table has a read weight of 0 and a write weight of 10. The history table has a read weight of 0 and a write weight of 1.

[0167] For example, after executing each SQL statement in Example 3, the relationship diagram in Figure 4 can be updated to produce the relationship diagram in Figure 5 . Compared to the relationship diagram in Figure 4 , the relationship diagram in Figure 5 adds a first node: OrderLineTable_FK1(OL_W_ID, OL_D_ID, OL_O_ID). Furthermore, an edge is added between the OrderTable_PK(O_W_ID, O_D_ID, O_ID) node and the CustomerTable_PK(C_W_ID, C_D_ID, C_ID) node (as highlighted in bold in Figure 5 ).

[0168] In addition, after executing all SQL statements in Example 1, Example 2, and Example 3, the read and write weights of each table are shown in Table 12.

[0169] Table 12 - Read and write weights of each table after running Example 1, Example 2, and Example 3

[0170] The new sections in Table 12 relative to Table 11 are the black sections in Table 12 above. After executing the SQL statements in Examples 1, 2, and 3, the weights of each table are as follows: The warehouse table has a read weight of 2 and a write weight of 1. The customer table has a read weight of 6 and a write weight of 1. The region table has a read weight of 2 and a write weight of 2. The order table has a read weight of 1 and a write weight of 1. The new order table has a read weight of 0 and a write weight of 1. The supply item item table has a read weight of 10 and a write weight of 0. The inventory table has a read weight of 10 and a write weight of 10. The order line table has a read weight of 1 and a write weight of 10. The history table has a read weight of 0 and a write weight of 1.

[0171] For example, after executing each SQL statement in Example 4, the relationship diagram in FIG5 can be updated to obtain the relationship diagram in FIG6. Compared with the relationship diagram in FIG5, the relationship diagram in FIG6 adds several first edges: the edge between the OrderTable_PK(O_W_ID, O_D_ID, O_ID) node and the NewOrderTable_PK(NO_W_ID, NO_D_ID, NO_O_ID) node, the edge between the NewOrderTable_PK(NO_W_ID, NO_D_ID, NO_O_ID) node and the OrderLineTable_FK1(OL_W_ID, OL_D_ID, OL_O_ID) node (as shown in bold in FIG6), and the edge between the CustomerTable_PK(C_W_ID, C_D_ID, C_ID) node and the OrderTable_PK(O_W_ID, O_D_ID, O_ID) node with a weight of +1.

[0172] In addition, after executing all SQL statements in Example 1, Example 2, Example 3, and Example 4, the read and write weights of each table are shown in Table 13.

[0173] Table 13 - Read and write weights of each table after running Example 1, Example 2, Example 3, and Example 4

[0174] The new sections in Table 13 relative to Table 12 are marked in bold in Table 13 above. After executing the SQL statements in Examples 1, 2, 3, and 4, the weights of each table are as follows: The warehouse table has a read weight of 2 and a write weight of 1. The customer table has a read weight of 6 and a write weight of 2. The region table has a read weight of 2 and a write weight of 2. The order table has a read weight of 2 and a write weight of 2. The new order table has a read weight of 1 and a write weight of 2. The supply item item table has a read weight of 10 and a write weight of 0. The inventory table has a read weight of 10 and a write weight of 10. The order line table has a read weight of 2 and a write weight of 11. The history table has a read weight of 0 and a write weight of 1.

[0175] For example, after executing each SQL statement in Example 5, the relationship diagram in FIG6 can be updated to obtain the relationship diagram in FIG7 . Compared with the relationship diagram in FIG6 , the relationship diagram in FIG7 adds several first edges: an edge between the InventoryTable_PK(S_W_ID, S_I_ID) node and the RegionTable_PK(D_W_ID, D_ID) node, and an edge between the RegionTable_PK(D_W_ID, D_ID) node and the OrderLineTable_FK1(OL_W_ID, OL_D_ID, OL_O_ID) node (as shown in bold in FIG7 ).

[0176] In addition, after executing all SQL statements in Example 1, Example 2, Example 3, Example 4, and Example 5, the read and write weights of each table are shown in Table 14.

[0177] Table 14 - Read and write weights of each table after running Example 1, Example 2, Example 3, Example 4, and Example 5

[0178] The new sections in Table 14 relative to Table 13 are marked in bold in Table 14 above. After executing the SQL statements in Examples 1, 2, 3, 4, and 5, the weights of each table are as follows: The warehouse table has a read weight of 2 and a write weight of 1. The customer table has a read weight of 6 and a write weight of 2. The region table has a read weight of 3 and a write weight of 2. The order table has a read weight of 2 and a write weight of 2. The new order table has a read weight of 1 and a write weight of 2. The supply item entry table has a read weight of 10 and a write weight of 0. The inventory table has a read weight of 11 and a write weight of 10. The order line table has a read weight of 3 and a write weight of 11. The history table has a read weight of 0 and a write weight of 1.

[0179] Step 203: The server partitions the accessed table based on the first relationship graph to obtain a partitioning result.

[0180] After obtaining the first relationship graph, the server can partition the accessed table based on the first relationship graph to obtain a partition result, which is used to represent the association relationship between the partition index and the constant field in the SQL statement.

[0181] Specifically, the server determines that the index information used during the execution of the first SQL statement in each transaction is the alternative partition index, the server determines the partition index based on the alternative partition index and the first relationship graph, and the server determines the partition result based on the partition index and the constant field or parameter condition field in the first SQL statement.

[0182] For example, in transaction 1 shown in Example 1 above, since the first SQL statement of transaction 1 first uses warehouse_PK(W_ID) to execute the query during the query process, warehouse_PK(W_ID) is an alternative partition index. In transaction 2 shown in Example 2 above, since the first SQL statement of transaction 2 also uses warehouse_PK(W_ID) for access during the query process, warehouse_PK(W_ID) is an alternative partition index. In transaction 3 shown in Example 3 above, since the first SQL statement of transaction 3 accesses customer_PK(C_W_ID, C_D_ID, C_ID), customer_PK(C_W_ID, C_D_ID, C_ID) is an alternative partition index. In transaction 4 shown in Example 4 above, since the first SQL statement of transaction 4 uses neworder_PK(NO_W_ID, NO_D_ID, NO_O_ID) for access, neworder_PK(NO_W_ID, NO_D_ID, NO_O_ID) is an alternative partition index. In transaction 5 shown in Example 5 above, because the first SQL statement in transaction 5 uses region_table_PK(D_W_ID, D_ID), region_table_PK(D_W_ID, D_ID) is a candidate partition index. The read and write weights of the accessed tables recorded after the transaction is executed are shown in Table 14.

[0183] Among them, the above-mentioned process of determining the partition index based on the alternative partition index and the first relationship graph can be specifically: first determine the sum of the edge weights of the nodes corresponding to different paths in the first relationship graph corresponding to at least one alternative partition index, and then determine the alternative partition index that meets the preset conditions as the partition index, and the preset conditions are related to the sum of the edge weights (for example, select the alternative partition index with the highest weight as the partition index).

[0184] For example, the alternative partition indexes of the five transactions shown in Examples 1 to 5 are: Warehouse Table_PK (W_ID), Warehouse Table_PK (W_ID), Customer Table_PK (C_W_ID, C_D_ID, C_ID), New Order Table_PK (NO_W_ID, NO_D_ID, NO_O_ID), and Region Table_PK (D_W_ID, D_ID). Since the previous two are the same, there are only four alternative partition indexes in total. The sum of the weights of the graph edges of each link is determined based on the four alternative partition indexes. The partition index is determined based on the sum of the edge weights. For example, the alternative partition corresponding to the largest sum of edge weights is selected as the partition index. Of course, the alternative partition corresponding to the second largest sum of edge weights can also be selected as the partition index, and the specific details are not limited here.

[0185] As can be seen from Figure 7, the sum of the edge weights of the warehouse table _PK (W_ID) link is the largest, so the index information (W_ID) of the warehouse table _PK is used as the partition index.

[0186] In addition, to reduce unnecessary partitioning steps, it is possible to first determine whether the accessed table participates in the partitioning. The judgment can be based on the value or ratio of the read weight to the write weight (for example, a table without a read weight, or a table with a write weight and a read weight greater than a preset threshold, participates in the partitioning). This step is then performed only if the accessed table participates in the partitioning. For example, in the entire system of the aforementioned example, the read and write weights of the tables are first considered. Since the supply commodity entry table has a read weight of 10 and a write weight of 0, and there are only read operations, the supply commodity entry table is removed from the entire first relationship graph, that is, the supply commodity entry table does not participate in the partitioning.

[0187] Optionally, when multiple transactions occur, the server determines the partitioning result based on the partition index, constant fields, and join relationships. Alternatively, after determining the partition index for one table, the partition indexes of other tables can be determined based on that table's partition index and join relationships. Since each table in a relational database can only have one horizontal partition, once the partition field for one table is determined, the partitions of other tables can be indirectly determined based on the join relationships between the tables.

[0188] If the partition fields of other transactions are in the same graph as the warehouse_PK(W_ID), meaning they are connected directly or indirectly to the warehouse_PK(W_ID), then the partitions of these transactions are consistent with the warehouse_PK(W_ID) and these transactions can directly use the alternative partition index. If they are inconsistent, meaning they are indirectly connected via another index of the same table (such as a foreign key), then the table can be partitioned using the other index. Of course, foreign keys can also appear in the first relationship graph and link to the primary key of the same table.

[0189] If the partition fields of other transactions and the warehouse table _PK (W_ID) are not in the same graph, refer to the above process and analyze this graph separately.

[0190] It is understood that the partitioning result may also include the partitioning mode, such as range partitioning, hash partitioning, etc. If the partitioning mode is range partitioning, the partitioning result also includes the partition value range corresponding to the range partitioning. If the partitioning mode is hash range partitioning, the partitioning result also includes the value corresponding to the hash partitioning.

[0191] Alternatively, if the partitioning method is range partitioning, read the minimum and maximum values ​​of the W_ID field from the warehouse table. The difference between the maximum and minimum values ​​divided by the number of cluster nodes (i.e., the number of servers) is the average range for each node. For example, if the minimum value of the W_ID field is 1, the maximum value is 1000, and the number of cluster nodes is 3, then (1000-1) / 3 = 333. This means that the partition range for the first node is 1-333, the partition range for the second node is 334-667, and the partition range for the third node is 668-1000.

[0192] For transaction 1, the w_id field in the SQL statement is the conditional field, corresponding to the (W_ID) of the warehouse table. This SQL statement is routed based on the :w_id in w_id=:w_id. If the :w_id value range is within 1-333, the SQL statement is sent to the first node; if the :w_id value range is within 334-667, it is sent to the second node; if the :w_id value range is within 668-1000, it is sent to the third node.

[0193] Optionally, after obtaining the partitioning result, the server can construct a routing table as shown in Table 15 below. The content of the routing table is the create order transaction. The W_ID in the first SQL statement is the routing field.

[0194] Table 15 - Routing Table

[0195] Step 204: The server sends the partitioning result to the client. Correspondingly, the client receives the partitioning result sent by the server. This step is optional.

[0196] Optionally, after obtaining the partitioning result, the server may send the partitioning result or routing table to the client.

[0197] Step 205: The client determines the SQL route based on the partitioning result. This step is optional.

[0198] Optionally, after the client obtains the partition result or routing table from the server, it can determine the SQL routing according to the partition result or routing table.

[0199] For example, taking the creation of an order transaction as an example, the client routes based on the :w_id in the first SQL statement where w_id=:w_id. If the :w_id value is in the range of 1-333, the SQL is sent to the first node; if the :w_id value is in the range of 334-667, it is sent to the second node; if the :w_id value is in the range of 668-1000, it is sent to the third node.

[0200] It is understandable that the processing procedures for other transactions are similar and will not be repeated here.

[0201] In an embodiment of the present application, a first relationship graph is constructed using the SQL statement execution information, and the accessed table is partitioned based on the first relationship graph. Compared to partitioning based on manual experience, this method of partitioning the table accessed by SQL statements based on the SQL statement execution information is more reasonable (for example, distributing SQL statements accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, the aggregation relationship between the data accessed by SQL statements can be identified, so that a more reasonable partitioning strategy for the data in the database can be determined based on the aggregation relationship.

[0202] Please refer to FIG. 8 , which is another flowchart of a data processing method provided in an embodiment of the present application. The method may include steps 801 to 808 .

[0203] Step 801: The server obtains database information.

[0204] Step 802: The server constructs a second relationship graph based on the database information.

[0205] Step 803: The client sends a transaction to the server. Correspondingly, the server receives the transaction sent by the client. This step is optional.

[0206] In step 804, the other servers send the third relationship graph to the server. Correspondingly, the server receives the third relationship graph sent by the other servers. This step is optional.

[0207] Step 805: The server constructs a first relationship graph based on the execution information of the SQL statement.

[0208] Step 806: The server partitions the accessed table based on the first relationship graph to obtain a partition result.

[0209] In step 807, the server sends the partitioning result to the client. Correspondingly, the client receives the partitioning result sent by the server. This step is optional.

[0210] Step 808: The client determines the SQL routing based on the partitioning result. This step is optional.

[0211] Steps 801 to 808 are described in detail below:

[0212] Step 801: The server obtains database information.

[0213] The server obtains database information, where the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of at least two tables.

[0214] Optionally, the above database information is schema information, and the server SQL routing module can read the schema information of the database, and the schema information includes the primary key and foreign key information of the table.

[0215] For example, a database instance is started, a data dictionary management module is initialized, a server SQL routing module is initialized, and an interface of the data dictionary management module is called to read the schema information of the database.

[0216] Step 802: The server constructs a second relationship graph based on the database information.

[0217] After acquiring the database information, the server constructs a second relationship graph, where the second relationship graph includes a plurality of second nodes, and each of the plurality of second nodes includes the names of at least two tables and index information of at least two tables.

[0218] Optionally, after obtaining the schema information, the server constructs a second relationship graph between table fields based on the primary key and foreign key of the table.

[0219] In one possible implementation, primary and foreign keys are used to explicitly specify the relationships between tables when creating them. In this case, the server reads the schema information to understand the index relationships between tables and can directly construct a second relationship graph between indexes in different tables.

[0220] In another possible implementation, if the relationships between tables are not explicitly specified and only indexes are created on the corresponding table fields, the server only knows the index vertex information when reading the schema information and can only construct the vertices of the second relationship graph (i.e., the second nodes). Edges between the second nodes are added based on the execution information of subsequent SQL statements.

[0221] Exemplarily, the server SQL routing module analyzes the primary and foreign key relationships of the tables and constructs a relationship graph between the table fields based on the primary and foreign key relationships. The vertices of the graph are the table name and the primary key field, or the table name and the foreign key field. If there is a primary or foreign key constraint relationship between the fields of two tables, an edge is constructed. The table index information is read, and the table name and all fields of each index are added as a vertex to the graph.

[0222] Step 803: The client sends a transaction to the server. Correspondingly, the server receives the transaction sent by the client. This step is optional.

[0223] This step 803 is similar to the corresponding step 201 of FIG. 2 in the aforementioned embodiment, and will not be described again here.

[0224] In step 804, the other servers send the third relationship graph to the server. Correspondingly, the server receives the third relationship graph sent by the other servers. This step is optional.

[0225] The other servers send the third relationship graph to the server. Correspondingly, the server receives the third relationship graph sent by the other servers. The third relationship graph includes multiple third nodes.

[0226] Optionally, the third relationship graph further includes edges between a plurality of third nodes. The construction process of the third relationship graph can refer to the construction process of the second relationship graph, which will not be expanded here.

[0227] Step 805: The server constructs a first relationship graph based on the execution information of the SQL statement.

[0228] After the server obtains the second relationship graph, the server may update the second relationship graph based on the operation information to obtain the first relationship graph.

[0229] Alternatively, if the server has previously received a third relationship graph sent by another server, the server constructs the first relationship graph based on the operation information and the third relationship graph. Of course, if the server has received a third relationship graph sent by another server, the server may also update the second relationship graph based on the operation information and the third relationship graph to obtain the first relationship graph.

[0230] In one possible implementation, when the operation information includes a join relationship, the server specifically adds a first edge to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0231] In another possible implementation, the operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; the server specifically increases the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0232] The above two situations can also be understood as the server SQL routing module checking whether the edge of the index vertex involved in the join relationship between the tables exists in the graph. If it does, the weight is increased by 1. If not, an edge is added between the index vertices, and the weight of the edge is initialized to 1.

[0233] This step can also be understood as the server determining the relationship between tables based on the SQL in the running process and further improving the relationship diagram between the tables.

[0234] Part of the description of constructing the first relationship diagram in this step 805 can refer to the description of step 202 corresponding to Figure 2 of the aforementioned embodiment, and will not be repeated here.

[0235] Step 806: The server partitions the accessed table based on the first relationship graph to obtain a partition result.

[0236] The description of this step 806 can refer to the description of step 203 corresponding to FIG. 2 of the aforementioned embodiment, and will not be repeated here.

[0237] The following describes the process of applying this step in the architecture shown in Figure 1:

[0238] Exemplarily, the server SQL routing module checks whether the SQL statement is the first SQL statement. If it is the first SQL statement, the index vertex matched by the field compared with the constant value of the SQL statement is marked as a partition candidate vertex. The server SQL routing module checks the number of SQL statements involved in the transaction. If there is only one SQL statement, no processing is performed. If there are multiple SQL statements, an edge is added between the index vertex accessed by the first SQL statement and the index vertex accessed by the second SQL statement according to the order of the SQL statements, and the weight of the edge is initialized to 1. If the edge already exists, the weight is increased by 1. Add an edge between the index vertex accessed by the second SQL statement and the index vertex accessed by the third SQL statement, and the weight of the edge is initialized to 1. If the edge already exists, the weight is increased by 1. And so on, until the last SQL statement. Record the mapping relationship between SQL statement binding parameters, stored procedure parameters and index fields.

[0239] The server SQL routing module selects a node as the association weight graph analysis node (for example, selects the node with the smallest node number) in an analysis cycle (for example, one minute), and notifies other nodes to send the association weight graph of this node to the analysis node. The server SQL routing module receives the relationship graph between tables constructed by other nodes through inter-node communication interaction, and merges it with the relationship graph constructed locally. The read and write weights of the table are added with the statistical values ​​of other nodes respectively. If the vertex does not exist in this graph, it is added to the relationship graph. If the edge does not exist, an edge between the two vertices is constructed, and the weight of the received edge is copied. If the edge exists, the weight of the edge counted by other nodes is added to the edge of this graph. The vertex marked as a partition candidate vertex remains unchanged and is still marked as a partition candidate vertex.

[0240] The server SQL routing module checks whether most of the schema tables are already in the graph. If not, it returns directly and waits for the next round of collection.

[0241] Check the read and write weights of the table. For read-only tables (write weight is 0) or tables with a majority read weight (for example, read weight exceeds 90%), do not partition them and delete them from the graph.

[0242] Traverse each graph, take each partition candidate vertex in the graph as a partition candidate strategy, count the weights of all edges determined by each partition strategy, and select the partition strategy with the largest weight as the final partition strategy for this graph.

[0243] Based on the candidate vertices selected by the partitioning strategy, the maximum value of the field key is read. The key partition range and corresponding nodes are divided according to the maximum value and the number of nodes running the database instance. Then, based on the associations between tables, the partition fields, partition field ranges, and corresponding nodes of other tables are determined.

[0244] The server's SQL routing module determines the SQL statement, the mapping between SQL statement binding parameters, stored procedure parameters, index fields, table partition index fields, partitioning methods, partition ranges, and corresponding nodes. If a SQL statement includes multiple tables, the table closest to the partitioning strategy vertex is selected as the basis for routing distribution.

[0245] In step 807, the server sends the partitioning result to the client. Correspondingly, the client receives the partitioning result sent by the server. This step is optional.

[0246] Step 808: The client determines the SQL routing based on the partitioning result. This step is optional.

[0247] The description of steps 807 and 808 can refer to the description of steps 204 and 305 corresponding to FIG. 2 of the aforementioned embodiment, and will not be repeated here.

[0248] The following describes the process of applying this step in the architecture shown in Figure 1:

[0249] The client performs routing based on the cache partitioning strategy, and the server performs routing based on the cache partitioning strategy. The application executes SQL statements, calling the ODBC / JDBC driver to send the SQL statements to the database instance. The client SQL routing determines the routing node for distributing SQL statements and reading query results, and sends the query to the corresponding node via the communication connection. The server instance receives the SQL statement through the server connection and calls the SQL engine for execution. The SQL engine parses and optimizes the SQL statement.

[0250] If executed on this node, the SQL engine executes the SQL, accessing the data page through the data page buffer. The execution results are returned through the server connection. The distribution and result reading module returns the SQL execution results to the application.

[0251] If the execution is not on the current node, the SQL engine forwards the SQL to the corresponding node through inter-node communication. The SQL engine receives the forwarded SQL through inter-node communication, parses, optimizes, and executes it, and returns the result to the original instance. The SQL engine receives the execution results from other nodes and returns the results through the server connection. The distribution and result reading module returns the SQL execution results to the application.

[0252] It is understandable that if SQL is routed by the server, the SQL execution result can be returned directly to the ODBC / JDBC driver by the database instance that executes the SQL, without being returned to the original database instance. The original database instance returns it to the ODBC / JDBC driver.

[0253] In an embodiment of the present application, the link relationship between tables is determined by Schema, SQL statement access and business load, the link relationship is analyzed to determine the partitioning strategy, and the client and server forward and route the SQL according to the partitioning strategy. On the one hand, the database Schema information is read, SQL access is tracked in units of transactions, the link relationship and weight between tables, the read and write weight of the table, the alternative partitioning field of the table, the mapping relationship between the SQL parameter value and the index partitioning field of the table are determined. On the other hand, each link relationship graph is traversed, and the best partitioning strategy is selected according to the read and write weight of the table, the alternative index partitioning field of the table, and the relationship weight between the index fields of the table, and the SQL statement, the SQL statement binding parameters, the mapping relationship between the stored procedure parameters and the table index field, the index partitioning field of the table, the partitioning method, the partitioning range and the corresponding node are determined. On the other hand, if the SQL statement can determine the node of the database instance according to the binding parameters, it is routed by the client, otherwise it is routed by the server, the server parses the SQL, and routes according to the parsed table field constant value and the index partitioning information of the table.

[0254] The data processing method in the embodiment of the present application is described above. The data processing device in the embodiment of the present application is described below. Please refer to Figure 9. An embodiment of the data processing device in the embodiment of the present application includes:

[0255] An acquiring unit 901 is configured to acquire at least one transaction, where the at least one transaction includes a structured query language SQL statement.

[0256] A construction unit 902 is configured to construct a first relationship graph based on the SQL statement execution information, the first relationship graph including a plurality of first nodes and first edges between the plurality of first nodes, each of the plurality of first nodes including a name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, the first edges being used to describe associations between the index information in the accessed tables;

[0257] The partitioning unit 903 is configured to partition the accessed table based on the first relationship graph to obtain a partitioning result, where the partitioning result is used to represent an association relationship between a partition index and a constant field in an SQL statement.

[0258] Optionally, the acquisition unit 901 is also used to obtain database information of the database, the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of at least two tables; the construction unit 902 is also used to construct a second relationship graph based on the database information, the second relationship graph includes multiple second nodes, and each second node in the multiple second nodes includes the names of at least two tables and index information of at least two tables; the construction unit 902 is specifically used to update the second relationship graph based on the operation information to obtain the first relationship graph.

[0259] Optionally, the execution information includes at least one of the following: fields used for constant value comparison in SQL statements, join relationships between multiple tables accessed by SQL statements, read and write types of multiple tables, index information used during execution, and the order of SQL statements in transactions.

[0260] Optionally, when the operation information includes a join relationship, the construction unit 902 is specifically configured to add a first edge to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with an index field of a table in the database.

[0261] Optionally, the operation information includes a join relationship, and the second relationship graph also includes a second edge between multiple second nodes; the construction unit 902 is specifically used to increase the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

[0262] Optionally, the data processing device further includes: a first determination unit 904, used to determine that the index information used during the execution of the first SQL statement in each transaction is an alternative partition index; a partitioning unit 903, specifically used to determine the partition index based on the alternative partition index and the first relationship graph; and the partitioning unit 903, specifically used to determine the partition result based on the partition index and a constant field in the first SQL statement.

[0263] Optionally, the partition unit 903 is specifically used to determine the sum of edge weights of different paths corresponding to the nodes in the first relationship graph corresponding to at least one alternative partition index; the partition unit 903 is specifically used to determine that the alternative partition index that meets the preset conditions is the partition index, and the preset conditions are related to the sum of edge weights.

[0264] Optionally, at least one transaction includes multiple transactions, and the execution information includes a join relationship.

[0265] Optionally, the operation information includes read and write types, which are used to increase the read weight or write weight corresponding to multiple tables; the data processing device also includes: a second determination unit 905, used to determine whether the accessed table participates in partitioning based on the read weight and the write weight; a partitioning unit 903, specifically used to partition the accessed table based on the first relationship graph when the accessed table participates in partitioning.

[0266] Optionally, the acquiring unit 901 is further configured to acquire a third relationship graph from other devices; and the constructing unit 902 is specifically configured to construct the first relationship graph based on the operation information and the third relationship graph.

[0267] Optionally, the data processing device further includes: a sending unit 906, configured to send a partitioning result to a client device, where the partitioning result is used to determine a device for executing an SQL statement.

[0268] In this embodiment, the operations performed by each unit in the data processing device are similar to the steps performed by the server in the embodiments shown in Figures 1 to 8 above, and will not be repeated here.

[0269] In this embodiment, construction unit 902 constructs a first relationship graph based on SQL statement execution information, and partitioning unit 903 partitions the accessed table based on the first relationship graph. Compared to partitioning based on manual experience, this method of partitioning the table accessed by SQL statements based on SQL statement execution information is more reasonable (for example, distributing SQL statements accessing the same data to the same node), which can reduce the frequent transfer of data between nodes. In other words, it can identify the aggregation relationship between the data accessed by SQL statements, thereby more reasonably determining the partitioning strategy for the data in the database based on this aggregation relationship.

[0270] Please refer to FIG10 , an embodiment of a client device in an embodiment of the present application includes:

[0271] An acquisition unit 1001 is used to acquire a partition result, where the partition result is used to represent an association relationship between a partition index and a constant field in an SQL statement;

[0272] The acquisition unit 1001 is also used to acquire SQL statements;

[0273] The determining unit 1002 is configured to determine a route of the SQL statement based on the partitioning result.

[0274] Optionally, the client device further includes: a storage unit 1003, configured to cache partitioning results.

[0275] In this embodiment, the operations performed by each unit in the client device are similar to the steps performed by the client in the embodiments shown in Figures 1 to 8 above, and will not be repeated here.

[0276] In this embodiment, since the partition result obtained by the acquisition unit 1001 is the association relationship between the partition index and the constant field in the SQL statement, it is possible to first obtain the constant field in the SQL statement, then look up the table to see which range the value of the partition index falls into (the partitioning method will affect the determination of the range), and then access the database corresponding to the range.

[0277] Referring to FIG. 11 , a schematic diagram of the structure of another data processing device provided herein is shown. The data processing device may include a processor 1101, a memory 1102, and a communication port 1103. The processor 1101, the memory 1102, and the communication port 1103 are interconnected via a circuit. The memory 1102 stores program instructions and data.

[0278] The memory 1102 stores program instructions and data corresponding to the steps executed by the data processing device in the corresponding implementation modes shown in FIG. 1 to FIG. 8 .

[0279] The processor 1101 is configured to execute the steps performed by the data processing device in any of the embodiments shown in FIG. 1 to FIG. 8 .

[0280] The communication port 1103 can be used to receive and send data, and to execute the steps related to acquisition, sending, and receiving in any of the embodiments shown in FIG. 1 to FIG. 8 .

[0281] In one implementation, the data processing device may include more or fewer components than those shown in FIG. 11 . This application is merely an illustrative description and does not limit this.

[0282] Referring to Figure 12, a schematic diagram of the structure of another client device provided herein is provided. The client device may include a processor 1201, a memory 1202, and a communication port 1203. The processor 1201, the memory 1202, and the communication port 1203 are interconnected via a line. The memory 1202 stores program instructions and data.

[0283] The memory 1202 stores program instructions and data corresponding to the steps executed by the client in the corresponding implementation modes shown in FIG. 1 to FIG. 8 .

[0284] The processor 1201 is configured to execute the steps performed by the client in any of the embodiments shown in FIG. 1 to FIG. 8 .

[0285] The communication port 1203 can be used to receive and send data, and to execute the steps related to acquisition, sending, and receiving in any of the embodiments shown in FIG. 1 to FIG. 8 .

[0286] In one implementation, the client device may include more or fewer components relative to those in FIG. 12 , which is merely an example and not limiting.

[0287] An embodiment of the present application further provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the possible implementation of the data processing device in the aforementioned embodiment.

[0288] An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers. When the computer program product is executed by the processor, the processor executes the method of the possible implementation method of the above-mentioned data processing device.

[0289] An embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device to implement the functions involved in the possible implementation of the above-mentioned data processing device. Optionally, the chip system also includes an interface circuit, which provides program instructions and / or data to the at least one processor. In one possible design, the chip system may also include a memory for storing the necessary program instructions and data for the terminal device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

[0290] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0291] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0292] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0293] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data processing method, characterized in that: The method is applied to a data processing device, the data processing device includes a database, and the method comprises: Obtaining at least one transaction, wherein the at least one transaction includes a structured query language SQL statement; Building a first relationship graph based on the running information of the SQL statement, the first relationship graph includes a plurality of first nodes and first edges between the plurality of first nodes, each of the plurality of first nodes includes a name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, and the first edge is used to describe an association relationship between the index information in the accessed table; The accessed table is partitioned based on the first relationship graph to obtain a partition result, where the partition result is used to represent an association relationship between a partition index and a constant field in the SQL statement.

2. The method according to claim 1, characterized in that The method further comprises: Acquire database information of the database, wherein the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of the at least two tables; constructing a second relationship graph based on the database information, wherein the second relationship graph includes a plurality of second nodes, and each second node in the plurality of second nodes includes the name of the at least two tables and index information of the at least two tables; The constructing a first relationship graph based on the running information of the SQL statement includes: The second relationship graph is updated based on the operation information to obtain the first relationship graph.

3. The method according to claim 1 or 2, characterized in that: The operation information includes at least one of the following: a field used for constant value comparison in the SQL statement, a join relationship between multiple tables accessed by the SQL statement, read and write types of the multiple tables, index information used during operation, and the order of the SQL statements in the transaction.

4. The method according to claim 3, characterized in that When the operation information includes the join relationship, updating the second relationship graph based on the operation information includes: The first edge is added to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

5. The method according to claim 3, characterized in that: The operation information includes the join relationship, and the second relationship graph also includes second edges between the plurality of second nodes; The updating the second relationship diagram based on the operation information includes: The weight of the second edge is increased in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

6. The method according to any one of claims 3 to 5, characterized in that The method further comprises: Determine the index information used during the execution of the first SQL statement in each transaction as the candidate partition index; Partitioning the multiple tables based on the first relationship graph includes: Determine the partition index based on the candidate partition index and the first relationship graph; The partition result is determined based on the partition index and a constant field in the first SQL statement.

7. The method according to claim 6, characterized in that The determining of the partition index based on the candidate partition index and the first relationship graph includes: Determine the sum of edge weights of different paths corresponding to the node corresponding to the at least one candidate partition index in the first relationship graph; A candidate partition index satisfying a preset condition is determined as the partition index, where the preset condition is related to the edge weight sum.

8. The method according to claim 6 or 7, characterized in that: The at least one transaction includes multiple transactions, and the running information includes the join relationship.

9. The method according to any one of claims 6 to 8, characterized in that The operation information includes the read / write type, and the read / write type is used to increase the read weight or write weight corresponding to the multiple tables; the method further includes: Determine, based on the read weight and the write weight, that the accessed table participates in the partition; Partitioning the accessed table based on the first relationship graph includes: In the case where the accessed table participates in partitioning, the accessed table is partitioned based on the first relationship graph.

10. The method according to any one of claims 1 to 9, characterized in that Before constructing the first relationship graph based on the execution information of the SQL statement, the method further includes: Obtaining a third relationship graph from another device; The constructing a first relationship graph based on the running information of the SQL statement includes: The first relationship graph is constructed based on the operation information and the third relationship graph.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: The partition result is sent to the client device, and the partition result is used to determine the device that executes the SQL statement.

12. A data processing method, characterized in that: The method is applied to a client device, and the method comprises: Obtaining a partition result, where the partition result is used to indicate an association relationship between a partition index and a constant field in an SQL statement; Obtain the SQL statement; A route for the SQL statement is determined based on the partitioning result.

13. The method according to claim 12, characterized in that The method further comprises: The partition results are cached.

14. A data processing device, characterized in that: The data processing device contains a database, and the data processing device includes: An acquisition unit, configured to acquire at least one transaction, wherein the at least one transaction includes a structured query language SQL statement; a construction unit, configured to construct a first relationship graph based on the running information of the SQL statement, wherein the first relationship graph includes a plurality of first nodes and first edges between the plurality of first nodes, each of the plurality of first nodes includes a name of a table accessed by the SQL statement and index information of the table accessed by the SQL statement, and the first edge is used to describe an association relationship between the index information in the accessed table; A partitioning unit is used to partition the accessed table based on the first relationship graph to obtain a partitioning result, wherein the partitioning result is used to represent an association relationship between a partition index and a constant field in the SQL statement.

15. The data processing device according to claim 14, characterized in that The acquisition unit is further used to acquire database information of the database, wherein the database information includes primary keys and foreign keys of at least two tables; or the database information includes index information of the at least two tables; The construction unit is further used to construct a second relationship graph based on the database information, wherein the second relationship graph includes a plurality of second nodes, and each second node in the plurality of second nodes includes the name of the at least two tables and index information of the at least two tables; The construction unit is specifically configured to update the second relationship diagram based on the operation information to obtain the first relationship diagram.

16. The data processing device according to claim 14 or 15, characterized in that The operation information includes at least one of the following: a field used for constant value comparison in the SQL statement, a join relationship between multiple tables accessed by the SQL statement, read and write types of the multiple tables, index information used during operation, and the order of the SQL statements in the transaction.

17. The data processing device according to claim 16, characterized in that When the operation information includes the join relationship, the construction unit is specifically used to add the first edge to the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

18. The data processing device according to claim 16, characterized in that The operation information includes the join relationship, and the second relationship graph also includes second edges between the plurality of second nodes; The construction unit is specifically used to increase the weight of the second edge in the second relationship graph based on the join relationship to obtain the first relationship graph, and the join relationship has an association relationship with the index field of the table in the database.

19. The data processing device according to any one of claims 16 to 18, characterized in that: The data processing device further comprises: A first determination unit is used to determine that index information used in the execution of the first SQL statement in each transaction is a candidate partition index; The partition unit is specifically configured to determine the partition index based on the candidate partition index and the first relationship graph; The partitioning unit is specifically configured to determine the partitioning result based on the partitioning index and a constant field in the first SQL statement.

20. The data processing device according to claim 19, characterized in that The partition unit is specifically used to determine the sum of edge weights of different paths corresponding to the node corresponding to the at least one candidate partition index in the first relationship graph; The partition unit is specifically used to determine a candidate partition index that meets a preset condition as the partition index, and the preset condition is related to the edge weight sum.

21. The data processing device according to claim 19 or 20, characterized in that The at least one transaction includes multiple transactions, and the running information includes the join relationship.

22. The data processing device according to any one of claims 19 to 21, characterized in that: The operation information includes the read / write type, and the read / write type is used to increase the read weight or write weight corresponding to the multiple tables; The data processing device further comprises: A second determining unit, configured to determine whether the accessed table participates in a partition based on the read weight and the write weight; The partitioning unit is specifically configured to partition the accessed table based on the first relationship graph when the accessed table participates in partitioning.

23. The data processing device according to any one of claims 14 to 22, characterized in that: The acquisition unit is further used to acquire a third relationship graph from other devices; The construction unit is specifically configured to construct the first relationship diagram based on the operation information and the third relationship diagram.

24. The data processing device according to any one of claims 14 to 23, characterized in that: The data processing device further comprises: The sending unit is used to send the partition result to the client device, and the partition result is used to determine the device that executes the SQL statement.

25. A client device, characterized in that: The client device comprises: An acquisition unit, used for acquiring a partition result, wherein the partition result is used for indicating an association relationship between a partition index and a constant field in an SQL statement; The acquisition unit is further used to acquire the SQL statement; A determination unit is used to determine a route of the SQL statement based on the partition result.

26. The client device according to claim 25, characterized in that The client device also includes: A storage unit is used to cache the partition result.

27. A data processing device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the data processing device executes the method according to any one of claims 1 to 11.

28. A client device, characterized in that include: A processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the client device executes the method according to claim 12 or 13.

29. A communication system, characterized in that: Comprising the data processing device as claimed in claim 27 and the client device as claimed in claim 28.

30. A computer storage medium, characterized in that The method comprises computer instructions, and when the computer instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 13.

31. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 13.

Images