Data processing method, device, electronic device and readable storage medium

By storing data rows and indexes uniformly in the target storage node and processing operations within the node, the delay problem caused by data rows and index distribution in a distributed database is solved, which simplifies consistency management and improves operation efficiency.

CN117743337BActive Publication Date: 2024-12-06PINGCAP XINGCHEN (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311766378.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-20
Publication Date
2024-12-06
Estimated Expiration
2043-12-20

AI Technical Summary

Technical Problem

In a distributed database, data rows and indexes are distributed on different storage nodes, resulting in multiple network interactions, resulting in large delays, and solving the consistency problem of data rows and indexes on different storage nodes is complex.

Method used

The index of each data row of the data table and the data table is stored in the target storage node at the same time. After receiving the request from the calculation node, the index column to be operated is determined and the operation is performed within the storage node to avoid cross-network operations and realize data rows and index uniform processing.

Benefits of technology

It significantly reduces latency, simplifies the consistency management of data rows and indexes, and improves operational efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117743337B_ABST
    Figure CN117743337B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a data processing method, device, electronic device and readable storage medium, which relate to the field of database technology. The method includes: receiving a first request sent by a computing node, the first request including a row identifier of a first data row in a first data table and a first operation; determining a first index column affected by the first operation from each index column, and performing a first operation on the first data row in the first data table and the index of the first data row in each first index column. The embodiments of the present application implement the execution of a first operation on a data row and an index of a data row within a storage node, and the execution efficiency is much higher than that when the data rows and indexes are distributed on different storage nodes, and there is no need to execute across the network, which can effectively reduce latency. In addition, compared to maintaining the consistency of data rows and indexes on different storage nodes, it is easier to maintain the consistency of the two within the same storage node.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of database technology, and more specifically, to a data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] A distributed database is usually composed of multiple nodes, including computing nodes and storage nodes. Computing nodes are responsible for processing SQL requests (requests for at least one of the operations of adding, deleting, modifying, and querying), while storage nodes are responsible for storing data rows or indexes of data rows.

[0003] In a distributed database, computing nodes can connect to various storage nodes through the network to operate on the data in the storage nodes. When computing nodes read data from storage nodes, the index and data are distributed on different storage nodes. It is necessary to read the index first, and then read the data based on the index. This requires multiple network interactions and causes a large delay. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can solve at least one technical problem existing in the background technology. The technical solution is as follows:

[0005] According to a first aspect of an embodiment of the present application, a data processing method is provided, which is applied to a target storage node of a distributed database node cluster, wherein the target storage node stores a first data table and an index of each data row in each index column in the first data table; the method comprises:

[0006] Receive a first request sent by a computing node, where the first request includes a row identifier of a first data row in a first data table and a first operation; the first request is generated by the computing node based on a database operation statement sent by a client;

[0007] A first index column affected by the first operation is determined from each index column, and the first operation is performed on the first data row in the first data table and the index of the first data row in each first index column.

[0008] According to a second aspect of an embodiment of the present application, another data processing method is provided, characterized in that it is applied to a computing node of a distributed database node cluster, and the method includes:

[0009] Receive a database operation statement sent by a client, and generate a first request based on the database operation statement, where the first request includes a row identifier of a first data row in a first data table and a first operation;

[0010] Determine a target storage node where the first data table is located, send a first request to the target storage, so that the target storage node determines a first index column affected by the first operation from each index column, and performs a first operation on a first data row in the first data table and an index of the first data row in each first index column;

[0011] The target storage node stores the first data table and the index of each data row in the first data table in each index column.

[0012] According to a third aspect of an embodiment of the present application, another data processing method is provided, which is applied to a management node of a distributed database node cluster, wherein the distributed database node cluster includes multiple storage nodes; each storage node stores at least one data table and an index of each data row in each index column in each data table;

[0013] Methods include:

[0014] Periodically receive heartbeat messages sent by each storage node; the heartbeat messages include the total storage space and remaining storage space of the corresponding storage node;

[0015] In each current cycle, for each storage node, the remaining space ratio of the storage node is determined according to the total storage space and the remaining storage space of the storage node, and a first storage node and a second storage node are determined from each storage node, and if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, the second data table in the first storage node and the index of each data row in the second data table are migrated to the second storage node;

[0016] The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node.

[0017] According to a fourth aspect of an embodiment of the present application, a data processing device is provided, which is applied to a target storage node of a distributed database node cluster, wherein the target storage node stores a first data table and an index of each data row in each index column in the first data table; the device comprises:

[0018] A first request receiving module is used to receive a first request sent by a computing node, where the first request includes a row identifier of a first data row in a first data table and a first operation; the first request is generated by the computing node based on a database operation statement sent by a client;

[0019] The first operation execution module is used to determine the first index column affected by the first operation from each index column, and perform the first operation on the first data row in the first data table and the index of the first data row in each first index column.

[0020] According to a fifth aspect of an embodiment of the present application, a data processing device is provided, which is applied to a computing node of a distributed database node cluster, and the device includes:

[0021] An operation statement receiving module, used for receiving a database operation statement sent by a client, and generating a first request based on the database operation statement, wherein the first request includes a row identifier of a first data row in a first data table and a first operation;

[0022] A first request sending module is used to determine a target storage node where the first data table is located, and send a first request to the target storage, so that the target storage node determines a first index column affected by the first operation from each index column, and performs a first operation on a first data row in the first data table and an index of the first data row in each first index column;

[0023] The target storage node stores the first data table and the index of each data row in the first data table in each index column.

[0024] According to a sixth aspect of an embodiment of the present application, there is provided a data processing device, which is applied to a management node of a distributed database node cluster, wherein the distributed database node cluster includes a plurality of storage nodes; each storage node stores at least one data table and an index of each data row in each index column in each data table;

[0025] The device includes:

[0026] A heartbeat message receiving module is used to periodically receive heartbeat messages sent by each storage node; the heartbeat message includes the total storage space and remaining storage space of the corresponding storage node;

[0027] A migration module is used to determine, for each storage node, the remaining space ratio of the storage node according to the total storage space and the remaining storage space of the storage node, determine a first storage node and a second storage node from each storage node, and if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, migrate the second data table in the first storage node and the index of each data row in the second data table to the second storage node;

[0028] The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node.

[0029] According to the seventh aspect of the embodiments of the present application, an electronic device is provided, which includes a memory, a processor and a computer program stored in the memory, and when the processor executes the program, the steps of the method provided in any one of the first aspect, the second aspect and the third aspect are implemented.

[0030] According to an eighth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in any one of the first aspect, the second aspect and the third aspect are implemented.

[0031] According to the ninth aspect of the embodiments of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device executes the steps of implementing the method provided in any one of the first aspect, the second aspect, and the third aspect.

[0032] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0033] In the embodiment of the present application, the target storage node stores the data table and the index of each data row in the data table in each index column at the same time. After receiving the first operation sent by the computing node, the first index column affected by the first operation is determined, and the storage node jointly performs the first operation on the first data row in the first data table and the column value of the first data row in each first index column, that is, the first operation is performed on the data row and the index of the data row within a storage node. The execution efficiency of performing the first operation on the data row and the index of the data row at a storage node is much higher than that when the data row and the index are distributed on different storage nodes. There is no need to perform across the network, which can effectively reduce the latency. In addition, it is much easier to solve the consistency problem of data rows and indexes within the same storage node than to solve the problem on different storage nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0035] Figure 1 A schematic diagram of the system architecture of a distributed database system for implementing a data processing method provided in an embodiment of the present application;

[0036] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of performing an insert operation in a distributed database node cluster provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of a process for performing data query in a distributed database node cluster provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of a process for creating an index for a third target column in a first data table provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of partitioning a storage node provided in an embodiment of the present application;

[0041] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;

[0042] Figure 8 A flowchart of another data processing method provided in an embodiment of the present application;

[0043] Fig. 9 A flowchart of a data redistribution method provided in an embodiment of the present application;

[0044] Fig.10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0045] Fig.11 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0046] Fig.12 A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;

[0047] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0049] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we call an element "connected" or "coupled" to another element, the one element can be directly connected or coupled to the other element, or it can refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here may include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0051] First, several terms involved in this application are introduced and explained:

[0052] Distributed databases store data in multiple servers and share and access data through network connections. Each server may have a complete or partial copy of the database management system (DBMS) and its own local database. Many servers located in different locations are interconnected through the network to form a complete, global, logically centralized, and physically distributed large database. Distributed database systems can provide higher availability, higher performance, and better scalability.

[0053] A distributed database node cluster is a cluster composed of nodes of a distributed database. It stores the data in the original centralized database in multiple storage nodes connected by a network to obtain larger storage capacity and higher concurrent access. For a distributed database based on a computing and storage separation architecture, the distributed database node cluster includes three types of nodes: computing nodes, storage nodes, and management nodes.

[0054] The computing node is responsible for processing the corresponding client's request and returning the result, providing computing resources, and being able to perform a series of data calculations.

[0055] Storage nodes provide storage resources and can store some data tables as well as indexes of these data tables.

[0056] The management node can independently expand and manage computing resources and storage resources based on business needs and the load of the node cluster.

[0057] In the financial field, a transaction may involve dozens or hundreds of SQL statements, and the requirements for transaction latency are particularly strict. In order to reduce latency, the where condition field of each SQL statement generally uses an index to improve execution efficiency and reduce latency.

[0058] In a relational database, there are mainly two types of data: data rows and indexes corresponding to data rows. Assuming that the index of the where condition field of the SQL statement is the primary key index of the data row, since the primary key index is an index based on the primary key, and the primary key is the data belonging to the data row itself, the data row to be operated can be directly located according to the primary key index. In this case, the query efficiency of the SQL statement is very high.

[0059] However, assuming that the index of the where condition field in the SQL statement is not a primary key index but a common index, then it is necessary to locate the primary key of the data row to be operated based on the common index, and then locate the data row to be operated based on the primary key. In the scenario of a distributed database, a data table with a large amount of data often needs to be distributed in different storage nodes, and the data row and its corresponding index will definitely not be on the same storage node. This means that the computing node of the distributed data needs to interact with the storage node at least twice, including interaction with the storage node storing the index and interaction with the storage node storing the data row. Interaction with different storage nodes requires crossing the network, which will bring a large delay.

[0060] The data processing method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0061] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0062] Figure 1 A schematic diagram of the system architecture of a distributed database system for implementing a data processing method provided in an embodiment of the present application. It can be understood that the data processing method provided in an embodiment of the present application can be applied to but not limited to the following Figure 1 In the application scenario shown.

[0063] like Figure 1 As shown, it includes a client 110 and a distributed database node cluster 120 , wherein the distributed database node cluster 120 includes a computing node 121 , multiple storage nodes 122 and a management node 123 .

[0064] The client 110 refers to a program that corresponds to the server and provides local services to the user. The user can operate the data rows in the data table through the client, triggering the client to generate a database statement (SQL statement). The database statement can be a database operation statement (insert statement, update statement, and delete statement, etc.) or a database query statement (select statement). The number of the client 110 can be one or more.

[0065] When the database statement is a database operation statement, after obtaining the database operation statement, the client 110 may send the database operation statement to the computing node 121 .

[0066] After receiving the database operation statement sent by the client 110, the computing node 121 generates a first request based on the database operation statement sent by the client. The first request includes a table identifier of a first data table, a row identifier of a first data row in the first data table, and a first operation.

[0067] The computing node 121 may search for routing information from the management node 123 . The routing information records the table identifier of the data table in each storage node. The computing node 121 may search for routing information to determine a target storage node 122 from the storage node 122 .

[0068] The computing node 121 sends the first request to the target storage node 122;

[0069] The target storage node 122 stores the first data table and the index of each data row in each index column in the first data table;

[0070] After receiving the first request sent by the computing node, the target storage node 122 determines the first index column affected by the first operation from each index column, and performs the first operation on the first data row in the first data table and the index of the first data row in each first index column.

[0071] When the database statement is a database query statement (select statement), after obtaining the database query statement, the client 110 may send the database query statement to the computing node 121 .

[0072] The computing node 121 may generate an execution plan based on the database query statement, where the execution plan includes a query condition for querying the second data row in the first data table; and send the execution plan to the target storage node 122 .

[0073] The target storage node 122 may receive the execution plan sent by the computing node 121 , and execute the execution plan to query the second data row in the first data table.

[0074] In addition to storing routing information, the management node 123 of the embodiment of the present application can also redistribute data. Specifically, the management node 123 can periodically receive heartbeat messages sent by each storage node 122; the heartbeat message includes the total storage space and remaining storage space of the corresponding storage node;

[0075] The management node 123 determines the remaining space ratio of each storage node according to the total storage space and the remaining storage space of the storage node in each current cycle, determines a first storage node and a second storage node from each storage node, and if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, migrates the second data table in the first storage node and the index of each data row in the second data table to the second storage node;

[0076] The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node.

[0077] The connections between the client and the computing node, between the computing node and the storage node, between the storage node and the storage node, and between the computing node and the management node are established through the network, and the network may include but is not limited to: wired network, wireless network. Among them, the wired network may include a local area network, a metropolitan area network, and a wide area network, etc., and the wireless network may include Bluetooth, Wi-Fi, and other networks that realize wireless communication. The specific network can also be determined based on the actual application scenario and is not limited here.

[0078] The present application provides a data processing method, which is applied to a target storage node in a distributed database node cluster. The target storage node stores a first data table and an index of each data row in each index column in the first data table, such as Figure 2 As shown, the method includes:

[0079] Step S201, receiving a first request sent by a computing node, the first request including a row identifier of a first data row in a first data table and a first operation; the first request is generated by the computing node based on a database operation statement sent by a client.

[0080] The distributed database node cluster in the embodiment of the present application includes computing nodes, storage nodes and management nodes. In this embodiment, a data processing method is executed by a target storage node in the distributed database node cluster.

[0081] The computing node of the embodiment of the present application can receive a database operation statement sent by a client. The database operation statement is a SQL operation statement, which can be any one of an insert statement, an update statement, and a delete statement.

[0082] The database operation statement is used to perform a preset operation (insert, modify and delete) on the first data row in the first data table. The database operation statement includes a table identifier of the first data table and a row identifier of the first data row.

[0083] After obtaining the database operation statement, the computing node in the embodiment of the present application converts the database operation statement into a first request, and the storage node can process the first request, which is used to instruct to perform a first operation on the first data row in the first data table, or to perform a first operation on the first data row and the index of the first data row. That is, if the first data table does not have an index, the first request is used to operate on the data row; if the first data table has an index, the first request is used to perform a first operation on the first data row and the index of the first data row. The subsequent embodiments are developed on the basis that the first data table contains an index.

[0084] The distributed database node cluster of the embodiment of the present application includes multiple storage nodes. Assuming that there is an index in a data table, the storage node stores the data table and the index of each data row in the data table in each index column, that is, the generation and management of the index of each data row in the data table is maintained by the storage node storing the data table, and is no longer generated and managed by the computing node. Assuming that there is no index in a data table, the storage node only needs to store each data row in the data table and does not maintain the index. A storage node can store multiple data tables and the index of each data row in the data table in each index column.

[0085] An index column refers to a column on which an index is created. The index establishes an association between the column value of the index column and the row identifier of the data row.

[0086] After receiving the database operation statement sent by the client, the computing node in the embodiment of the present application needs to obtain routing information from the management node. The routing information records the table identifier of the data table in each storage node. The computing node can read the routing information, determine at least one target storage node that stores the first data table from multiple storage nodes, and send the first request to the at least one target storage node respectively.

[0087] In addition, it is worth noting that if the target storage node includes a first data table, then the first request may include the row identifier of the first data row in the first data packet and the first operation; if the target storage node includes other data tables in addition to the first data table, then the first request also includes the table identifier of the first data table.

[0088] A storage node may include multiple data tables. To facilitate searching, each table will be assigned a table identifier (also called table ID), represented by TableID. The table identifier is an integer that is unique in the entire distributed database node cluster.

[0089] Row ID (also called row ID) is also an integer, which is unique in the data table. For row ID, if a data table has an integer primary key, the value of the primary key will be used as the row ID of this data row.

[0090] Step S202 : determining a first index column affected by a first operation from among the index columns, and performing a first operation on a first data row in a first data table and an index of the first data row in each first index column.

[0091] After receiving the first request, the storage node in the embodiment of the present application determines the first index column affected by the first operation from each index column. Specifically, if the first operation is an insert operation or a delete operation, that is, inserting or deleting the first data row, then each index column in the data table is the first index column; if the first operation is a modification operation, it is necessary to determine whether the first target column to be modified belongs to the index column. If the first target column to be modified belongs to the index column, the first target column is determined to be the first index column.

[0092] After determining that the first index column exists, the storage node in the embodiment of the present application performs the first operation on the first data row in the first data table and the column value of the first data row in each first index column, that is, the first operation is performed on the data row and the index of the data row within a storage node. The execution efficiency of performing the first operation on the data row and the index of the data row in a storage node is much higher than that when the data row and the index are distributed on different storage nodes, and there is no need to perform across the network, which can effectively reduce the latency. In addition, solving the consistency problem of data rows and indexes within the same storage node is much easier than solving the problem on different storage nodes.

[0093] In addition, assuming that there is no first index column affected by the first operation in each index column, the target storage node directly performs the first operation on the first data row of the first data table, and does not perform operations on each index.

[0094] In addition, in the case where the data table does not contain an index, the storage node only needs to operate on the data rows of the data table without operating on the index. The subsequent embodiments are mainly carried out for data tables with indexes.

[0095] In an embodiment of the present application, a possible implementation method is provided, in which the target storage node stores each data row in the first data table in the form of a data row key-value pair, wherein the key of each data row key-value pair includes a row identifier of the corresponding data row, and the value is a column value of each column in the corresponding data row;

[0096] The target storage node stores the index of each data row in each index column in the form of an index key-value pair. The key of each index key-value pair includes the column identifier of the corresponding index column and the column value of the corresponding data row in the corresponding index column, and the value includes the row identifier of the corresponding data row.

[0097] In a relational database, there are mainly two types of data: each data row in the data table and each index of each data row.

[0098] In the embodiment of the present application, the target storage node stores each data row in the first data table in the form of a data key-value pair. The key of each data row key-value pair includes the table identifier of the data table and the row identifier of the corresponding data row, and the value is the column value of each column in the corresponding data row. In a certain scenario, each data row can be encoded as a (Key, Value) key-value pair according to the following rules:

[0099] Key:tablePrefix{TableID}_recordPrefixSep{RowID},

[0100] Value:[col1,col2,col3,col4],

[0101] Among them, tablePrefix and recordPrefixSep are both specific string constants used to distinguish other data in the Key space, TableID represents the table ID, RowID represents the row ID, coli is the column value of the data row in the i-th column, i=1,2,3...

[0102] In the embodiment of the present application, the target storage node stores the index of each data row in each index column in the form of an index key-value pair. The key of each index key-value pair includes the table identifier of the data table, the column identifier of the corresponding index column, and the column value of the corresponding data row in the corresponding index column, and the value includes the row identifier of the corresponding data row. In one scenario, assuming that the index is a primary key index, each index key-value pair can be encoded as a (Key, Value) key-value pair according to the following rules:

[0103] Key:tablePrefix{tableID}_indexPrefixSep{indexID}_indexedColumnsValue,

[0104] Value:RowID,

[0105] Among them, tablePrefix and recordPrefixSep are specific string constants used to distinguish other data in the Key space, TableID represents the table ID, indexID represents the column ID, RowID represents the row ID, and indexedColumnsValue represents the column value of the index column in the primary key index.

[0106] In another scenario, if the index is a common secondary index that does not need to meet the uniqueness constraint, an index key-value pair may correspond to multiple data rows, and the corresponding RowID needs to be queried based on the index key-value pair range. Therefore, it is encoded into a (Key, Value) key-value pair according to the following rules:

[0107] Key:tablePrefix{TableID}_indexPrefixSep{IndexID}_indexedColumnsValue_{RowID1,RowID2......},

[0108] Value:null,

[0109] The key of the above index key-value pair includes the table identifier, the column identifier of the index column, the column value of the index column, and the row identifier of each data row that satisfies the column value of the index column, while the value of the index key-value pair is empty. For ease of understanding, the index key-value pair can be represented as follows:

[0110] Key:tablePrefix{TableID}_indexPrefixSep{IndexID}_indexedColumnsValue,

[0111] Value:RowID1,RowID2......,

[0112] In this form, the key of the index key-value pair includes the column identifier of the corresponding index column and the column value of the corresponding data row in the corresponding index column, and the value includes the row identifier of the corresponding data row.

[0113] The embodiment of the present application stores the data row key-value pairs corresponding to the data rows and the index key-value pairs corresponding to the various indexes of the data rows in the target storage node at the same time, so that the value of the index key-value pair can be quickly located according to the key of the index key value in the same target storage node, that is, the row identifier of the required data row can be quickly located, and then the data row can be quickly located, which can effectively reduce the latency of data operations.

[0114] A possible implementation method is provided in an embodiment of the present application. If the first operation is an insert operation, the first request also includes a data row key-value pair of the first data row;

[0115] Determining a first index column affected by the first operation from each index column, and performing the first operation on the first data row in the first data table and the index of the first data row in each first index column, including:

[0116] Using each index column of the first data table as the first index column;

[0117] Parse the data row key-value pair of the first data row to obtain the column value of each first index column in the first data row, and for each first index column, generate the key of the index key-value pair of the first index column in the first data row according to the column value of the first index column in the first data row, and generate the value of the index key-value pair of the first index column according to the row identifier of the first data row;

[0118] Insert the data row key-value pair of the first data row and the index key-value pairs of each first index column into the target storage node.

[0119] Since the computing node can process the database operation statements sent by the client, the database operation statements include insert statements, modify update statements and delete statements, and the storage node can process kv operations, it is necessary to convert the database operation statements into the first request. The first request belongs to the kv request, and the kv request includes the kv operation, that is, the first operation is the kv operation.

[0120] When the database operation statement in the embodiment of the present application is an insert statement, the first operation in the first request is an insert put operation. In addition, the first request also includes a data row key-value pair of the first data row. In this case, the first request is used to perform an insert put operation on the data row key-value pair of the first data row. In addition, if there is at least one index column in the first data table, then in this case, the at least one index column is the first index column, and an index needs to be created for the first data row in each first index column, that is, an index key-value pair of the first data row in each first index column is generated.

[0121] like Figure 3As shown, it exemplarily shows a schematic diagram of performing an insert operation in a distributed database node cluster, involving interactions between a client, a computing node in the distributed database node cluster, and a storage node, assuming that the storage node includes storage node 1 and storage node 2.

[0122] The client sends an insert statement to the computing node, where the insert statement includes a row identifier of a first data row to be inserted, a table identifier of a first data table to which the first data row belongs, and column values ​​of each column of the first data row.

[0123] The computing node obtains routing information from the management node. The routing information records the table identifiers of the data tables stored in the storage node. The computing node selects a target storage node from multiple storage nodes (i.e., storage node 1 and storage node 2) based on the routing information. Assuming that the first data table is stored in storage node 1, then storage node 1 is the target storage node. The computing node also converts the insert statement into a put operation and sends a kv request to storage node 1. The kv request includes a put operation and a data row key-value pair (key, value) of the first data row.

[0124] After receiving the kv request, storage node 1 parses the data row key-value pair (key, vlaue) in the kv request to determine the table identifier of the first data table to which the first data row belongs, the row identifier of the first data row, and the column values ​​of each column in the first data row; the storage node can also obtain the data dictionary information of the first data table according to the table identifier of the first data table, and the data dictionary information records the table information in the first data table, including the table identifier, table name, column names, column identifiers, data types of each column, etc. of the data table, and also records the index columns in the first data table (i.e., the columns in those columns). In the first data table, if there is an index column, then storage node 1 can determine the various index columns of the first data table according to the data dictionary information of the first data table. Each index column is a first index column affected by the insert operation. Therefore, it is necessary to establish an index for each first index column. Specifically, for each first index column, the key of the index key-value pair of the first index column in the first data row can be generated according to the column value of the first index column in the first data row, and the value of the index key-value pair of the first index column can be generated according to the row identifier of the first data row, thereby obtaining the index key-value pair of the first index column.

[0125] After obtaining each index key-value pair, storage node 1 stores the data row key-value pair of the first data row and each index key-value pair on disk, so that the data row and the index of the data row are stored on disk together.

[0126] In addition, after the operation is completed, the storage node 1 will also return the execution result to the computing node. If the insertion is successful, the first execution result will be returned to the computing node, and the first execution result indicates that the insertion is successful. If the insertion fails, the second execution result will be returned to the computing node, and the second execution result indicates that the insertion fails.

[0127] After receiving the execution result, computing node 1 forwards the execution result to the client.

[0128] A possible implementation method is provided in an embodiment of the present application. If the first operation is a delete operation, the first request also includes a key of a data row key-value pair of the first data row;

[0129] Determining a first index column affected by the first operation from each index column, and performing the first operation on the first data row in the first data table and the index of the first data row in each first index column, including:

[0130] Determine locally the data row key-value pair of the first data row according to the key of the data row key-value pair of the first data row;

[0131] Using each index column of the first data table as the first index column;

[0132] Determine, from the data row key-value pairs of the first data row, the column values ​​of the first data row in each first index column;

[0133] For each first index column, locally determine an index key-value pair of the first data row in the first index column according to the column value of the first data row in the first index column;

[0134] The data row key-value pair of the first data row and the index key-value pairs of the first data row in each first index column are deleted locally.

[0135] When the database operation statement is a delete operation, and the first operation is a delete operation in a kv operation, each index column in the first data table is a first index column affected by the first operation.

[0136] When the first operation is a delete operation, the first request includes the key of the data row key-value pair of the first data row. Based on the key of the data row key-value pair of the first data row, the data row key-value pair of the first data row can be determined locally on the target storage node, the first data row key-value pair can be parsed to determine the column values ​​of the first data row in each first index column, and for each first index column, the index key-value pair of the first data row in the first index column can be determined locally on the target storage node based on the column values ​​of the first data row in each first index column, and the data row key-value pair of the first data row and the index key-value pairs of the first data row in each first index column can be deleted locally.

[0137] In an embodiment of the present application, a possible implementation method is provided. If the first operation is a modification operation, the first request further includes a key of a data row key-value pair of a first data row, a column identifier of each first target column to be modified in the first data row, and a modified column value of the first target column;

[0138] Determining a first index column affected by the first operation from each index column, and performing the first operation on the first data row in the first data table and the index of the first data row in each first index column, including:

[0139] According to the key of the data row key-value pair of the first data row, locally determine the data row key-value pair of the first data row, update the data row key-value pair of the first data row to a new data row key-value pair, and the column value of each first target column in the value of the new data row key-value pair is the corresponding modified column value; record the column value of each first target column before modification;

[0140] For each first target column, if it is determined that the first target column belongs to an index column, the first target column is used as a first index column;

[0141] For each first index column, an index key-value pair of the first data row in the first index column is locally determined according to the column value of the first data row in the first index column before modification; the index key-value pair of the first data row in the first index column is updated to a new index key-value pair, the column value of the first index column in the key of the new index key-value pair is the modified column value; and the value of the first target column before modification is deleted.

[0142] If it is determined that the first target column does not belong to the index column, the value of the first target column before modification is deleted;

[0143] When the first operation is a modification operation, the first request further includes a key of a data row key-value pair of the first data row, a column identifier of each first target column to be modified in the first data row, and a modified column value of the first target column;

[0144] The target storage node can determine the data row key-value pair of the first data row locally according to the key of the data row key-value pair of the first data row, and update the data row key-value pair of the first data row to a new data row key-value pair, and the column value of each first target column in the value of the new data row key-value pair is the corresponding modified column value. At this time, it is also necessary to record the column value of each first target column before the modification, because the index key-value pair of each index column of the first data row stored in the target storage node is generated based on the column value of the index column before the modification, and therefore, it is necessary to record the column value of each first target column before the modification.

[0145] It is worth noting that each first target column to be modified is not necessarily the first index column affected by the first operation. If the first target column belongs to the index column, it is determined that the first target column belongs to the first index column affected by the first operation; if the first target column does not belong to the index column, there is no index for the first target column, and there is no need to operate the index.

[0146] For each first index column, the key of the index key-value pair of the first data row in the first index column can be determined based on the column value of the first data row before the modification of the first index column, and the index key-value pair of the first data row in the first index column can be determined locally based on the key of the index key-value pair of the first index column; the index key-value pair of the first data row in the first index column is updated to a new index key-value pair, and the column value of the first index column in the key of the new index key-value pair is the modified column value; and the value of the first target column before the modification is deleted.

[0147] A possible implementation method is provided in an embodiment of the present application, and the method further includes:

[0148] receiving an execution plan sent by the computing node, the execution plan including a query condition for querying a second data row in the first data table;

[0149] If it is determined that the query condition includes the row identifier of the second data row, a data row key-value pair of the second data row is determined locally according to the row identifier of the second data row, and a value of the data row key-value pair of the second data row is read as the second data row;

[0150] If it is determined that the query condition does not include the row identifier of the second data row, then determine the various index columns of the first data table; if it is determined that the query condition includes at least one second target column, and the second target column is an index column, then locally determine the index key-value pair of the second data row in the second target column according to the column value of the second target column; determine the row identifier of the second data row according to the index key-value pair of the second data row in the second target column, locally determine the data row key-value pair of the second data row according to the row identifier of the second data row, and read the value of the data row key-value pair of the second data row as the second data row;

[0151] The query result is returned to the computing node, where the query result includes the second data row.

[0152] like Figure 4 As shown, it exemplarily shows a flow chart of performing data query in a distributed database node cluster, including:

[0153] S1: The client sends a query select statement for the second data row in the first data table to a computing node of the distributed database node cluster;

[0154] S2: The computing node can receive the query statement, perform lexical and grammatical analysis on the select statement, and generate an execution plan. The execution plan refers to the performance of the database query statement when it is executed in the database. The execution plan can estimate in advance how long the database needs to run, the data tables and indexes involved in the query. The execution plan includes a query condition (where condition) for querying the second data row in the first data table, a table identifier of the first data table, and a row identifier of the second data row.

[0155] S3: The computing node determines whether the query condition includes a row identifier; if so, executes step S4; if not, executes step S5;

[0156] S4: If it is determined that the query condition includes the row identifier of the second data row, determine the target storage node where the second data row is located according to the distribution information of the row identifier, and directly send the execution plan to the target storage node where the second data row is located; execute S5;

[0157] S5: If the computing node determines that the query condition does not include the row identifier of the second data row, the computing node determines at least one target storage node storing the second data table according to the table identifier of the second data table, and sends an execution plan to the at least one target storage node storing the second data table; and executes S6;

[0158] S6: The target storage node locally determines a data row key-value pair of the second data row according to the row identifier of the second data row, and reads the value of the data row key-value pair of the second data row as the second data row; execute S10;

[0159] S7: If the target storage node determines that the query condition does not include the row identifier of the second data row, it determines the index columns of the first data table and judges whether the query condition includes the index column; if so, execute step S8; if not, execute step S9;

[0160] S8: If it is determined that the query condition includes at least one second target column, and the second target column is an index column, then locally determine the index key value pair of the second data row in the second target column according to the column value of the second target column; determine the row identifier of the second data row according to the index key value pair of the second data row in the second target column; and execute S6;

[0161] S9: If the target storage node determines that the query condition does not include an index column, it scans each data row in the first data table until a second data row is scanned;

[0162] S10: After obtaining the second data row, the target storage node returns a query result to the computing node, where the query result includes the second data row.

[0163] A possible implementation method is provided in an embodiment of the present application, and the method further includes:

[0164] receiving a second request sent by the computing node; the second request includes a column identifier of a third target column in the first data table; the second request is used to create an index for the third target column;

[0165] Determine a row identifier of each data row in the first data table and a column value of each data row in a third target column;

[0166] According to the row identifier of each data row in the first data table, the column identifier of the third target column and the column value of each data row in the third target column, an index key-value pair of each data row in the third target column is generated and stored.

[0167] like Figure 5 As shown, it exemplarily shows a flow chart of creating an index for the third target column in the first data table, including: S51: the client sends a create index statement to the computing node, and the create index statement is used to indicate to create an index for the third target column in the first data table; S52: after receiving the create index statement, the computing node performs lexical analysis and syntactic analysis on the create index statement to parse the create index statement and generate a syntax tree; S53: the computing node determines all target storage nodes storing the first data table according to the table identifier of the first data table in the syntax tree; S54: the computing node generates a second request and sends the second request to all target storage nodes, and the second request is used to create an index for the third target column in the first data table.

[0168] Since the target storage node includes multiple partitions (regions), each partition is used to store the data row key-value pairs of each data row in the first data table or the index key-value pairs of each data row, S55: after receiving the second request, the target storage node can create index key-value pairs in the third target column for each data row of the first data table in units of partitions. Specifically, the target partition storing the data row key-value pairs of the first data table can be determined, and each data row key-value pair in the target partition can be read to determine the column value of each data row key-value pair in the third target column. The target storage node generates and stores the index key-value pairs of each data row in the third target column based on the row identifier of each data row in the first data table, the column identifier of the third target column, and the column value of each data row in the third target column.

[0169] In an embodiment of the present application, a possible implementation method is provided, where the target storage node includes multiple partitions, each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix;

[0170] Performing a first operation on a first data row in a first data table and an index of the first data row in each first index column includes:

[0171] Matching the key of the data row key value of the first data row with the key of at least one index key-value pair of each partition, and determining a first target partition from the multiple partitions; the key of the data row key-value pair of the first data row and the key of at least one data row key-value pair in the first target partition have the same prefix;

[0172] Performing a first operation on the data row key-value pair corresponding to the first data row in the first target partition;

[0173] For each first index column, a key of an index key-value pair of a first data row in the first index column is matched with a key of at least one index key-value pair of each partition, and a second target partition is determined from the multiple partitions; the key of the index key-value pair of the first data row in the first index column and the key of at least one index key-value pair in the second target partition have the same prefix;

[0174] A first operation is performed on the index key-value pair corresponding to the first index column of the first data row in the second target partition.

[0175] For distributed databases based on the computing and storage separation architecture, the storage layer uses key-value pairs for distributed storage. Data can be regarded as a huge ordered KV Map. For each storage node, the storage node can be partitioned according to the key of the key-value pair. A storage node includes multiple partitions. The keys of the key-value pairs in each partition are continuous. Each partition can be described by a left-closed and right-open interval [StartKey, EndKey). The amount of data stored in each Region is maintained at about 96 MiB by default (can be modified through configuration). StartKey represents the first key-value pair of the partition, and EndKey represents the last key-value pair of the partition.

[0176] like Figure 6As shown, it exemplarily shows a schematic diagram of partitioning a storage node, the storage node includes 2 partitions, namely partition 1 and partition 2, each partition is used to store key-value pairs (key, value), which can be data row key-value pairs or index key-value pairs. Partition 1 includes (key1, value1), (key2, value2), (key3, value3) and (key4, value4), then key1 is the StartKey of partition 1, and key4 is the EndKey of partition 1; partition 2 includes (key5, value5), (key6, value6), (key7, value7) and (key8, value8), then key5 is the StartKey of partition 1, and key8 is the EndKey of partition 1.

[0177] The above-mentioned embodiments have explained the method of encoding data row key-value pairs and index key-value pairs. For data row key-value pairs, the data row key-value pairs include the table identifier of the data table to which the data row belongs. If each data row in a data table has the same table identifier, then each data row in the data table has the same Key prefix. Continuing with the above-mentioned embodiments, the key of the data row key-value pair is "tablePrefix{TableID}_recordPrefixSep{RowID}", RowID is variable, and "tablePrefix{TableID}_recordPrefixSep" is immutable. Therefore, the prefix of the data row key-value pair can be "tablePrefix{TableID}_recordPrefixSep".

[0178] For the index key-value pair, the key of the index key-value pair includes the table ID of the data table, the column ID of the corresponding index column, and the column value of the corresponding data row in the corresponding index column. The key-value pair of the data row is tablePrefix{tableID}_indexPrefixSep{indexID}_indexedColumnsValue. "indexedColumnsValue" is mutable, while "tablePrefix{tableID}_indexPrefixSep{indexID}" is immutable. Therefore, the prefix of the data row key-value pair can be "tablePrefix{tableID}_indexPrefixSep{indexID}".

[0179] In this way, data row key-value pairs or index key-value pairs with the same prefix are arranged together in the Key space of the storage node. After encoding in the above way, all data row key-value pairs of a table will be arranged in the Key space in the order of RowID, and index key-value pairs will also be arranged in the Key space in the order of specific values.

[0180] The embodiment of the present application performs a first operation on the first data row in the first data table and the index of the first data row in each first index column. In fact, the first target partition where the index key-value pair of the first data row is located performs the first operation on the data row key-value pair of the first data row, and the first operation is performed on each index key-value pair in the second partition where each index key-value pair is located.

[0181] Specifically, the key of the data row key value of the first data row can be matched with the key of at least one index key-value pair of each partition to determine the first target partition from multiple partitions; the key of the data row key-value pair of the first data row and the key of at least one data row key-value pair in the first target partition have the same prefix; and the first operation is performed on the data row key-value pair corresponding to the first data row in the first target partition.

[0182] For each first index column, the key of the index key-value pair of the first data row in the first index column is matched with the key of at least one index key-value pair of each partition, and a second target partition is determined from the multiple partitions; the key of the index key-value pair of the first data row in the first index column and the key of at least one index key-value pair in the second target partition have the same prefix; and a first operation is performed on the index key-value pair corresponding to the first data row in the first index column in the second target partition.

[0183] Another data processing method is provided in the embodiment of the present application, which is applied to a computing node of a distributed database node cluster, such as Figure 7 As shown, the method includes:

[0184] Step S701: receiving a database operation statement sent by a client, and generating a first request based on the database operation statement, wherein the first request includes a row identifier of a first data row in a first data table and a first operation.

[0185] Step S702, determining a target storage node where the first data table is located, sending a first request to the target storage, so that the target storage node determines a first index column affected by a first operation from each index column, and performs a first operation on a first data row in the first data table and an index of the first data row in each first index column;

[0186] The target storage node stores the first data table and the index of each data row in the first data table in each index column.

[0187] The computing node of the embodiment of the present application can receive a database operation statement sent by a client. The database operation statement is a SQL operation statement. The database operation statement can be any one of an insert statement, an update statement, and a delete statement.

[0188] The database operation statement is used to perform a preset operation (insert, modify and delete) on the first data row in the first data table. The database operation statement includes a table identifier of the first data table and a row identifier of the first data row.

[0189] After obtaining the database operation statement, the computing node in the embodiment of the present application converts the database operation statement into a first request, and the storage node can process the first request, which is used to indicate a first operation to be performed on a first data row in a first data table.

[0190] After receiving the database operation data sent by the client, the computing node in the embodiment of the present application needs to obtain routing information from the management node. The routing information records the table identifier of the data table in each storage node. The computing node can read the routing information, determine at least one target storage node that stores the first data table from multiple storage nodes, and send the first request to the at least one target storage node respectively.

[0191] In addition, it is worth noting that if the target storage node includes a first data table, then the first request may include the row identifier of the first data row in the first data packet and the first operation; if the target storage node includes other data tables in addition to the first data table, then the first request also includes the table identifier of the first data table.

[0192] After receiving the first request, the storage node in the embodiment of the present application determines the first index column affected by the first operation from each index column. Specifically, if the first operation is an insert operation or a delete operation, that is, inserting or deleting the first data row, then each index column in the data table is a first index column; if the first operation is a modification operation, it is necessary to determine whether the target column to be modified belongs to the index column. If the target column to be modified belongs to the index column, the target column is the first index column.

[0193] After determining that the first index column exists, the storage node in the embodiment of the present application performs the first operation on the first data row in the first data table and the column values ​​of the first data row in each first index column, that is, the first operation is performed on the data row and the index of the data row within a storage node. The execution efficiency of performing the first operation on the data row and the index of the data row in a storage node is much higher than that of data rows and indexes distributed on different storage nodes, which can effectively reduce latency. In addition, solving the consistency problem of data rows and indexes within the same storage node is much easier than solving the problem on different storage nodes.

[0194] In an embodiment of the present application, a possible implementation method is provided, which receives a database query statement sent by a client, generates an execution plan based on the database query statement, and the execution plan includes a query condition for querying a second data row in a first data table;

[0195] Send the execution plan to the target storage node.

[0196] When the client initiates a query select statement for the second data row in the first data table, the computing node of the distributed database node cluster can receive the query statement, parse the Select statement for lexical and grammatical analysis, and generate an execution plan. The execution plan refers to the performance of the database query statement when it is executed in the database. The execution plan can estimate in advance how long the database needs to run, the data tables and indexes involved in the query. The execution plan includes the query condition (where condition) for querying the second data row in the first data table, the table identifier of the first data table, and the row identifier of the second data row.

[0197] The computing node determines whether the query condition includes a row identifier; if it is determined that the query condition includes the row identifier of the second data row, the target storage node where the second data row is located is determined based on the distribution information of the row identifier, and the execution plan is directly sent to the target storage node; if the computing node determines that the query condition does not include the row identifier of the second data row, the computing node determines at least one target storage node that stores the second data table based on the table identifier of the second data table, and sends the execution plan to the at least one target storage node.

[0198] The target storage node may return a query result after execution is completed, and the query result includes the second data row.

[0199] See the detailed process Figure 4 The embodiments of the present application will not be described in detail here.

[0200] In an embodiment of the present application, a possible implementation method is provided, and determining a target storage node where a first data table is located includes:

[0201] Obtaining routing information of each node from the management node, where the table identifier of the data table contained in each storage node is recorded in the routing information;

[0202] A target storage node where the first data table is located is determined from the routing information.

[0203] The above-mentioned embodiment has already explained that the table identifier of the data table contained in each storage node is recorded in the specification routing information, and the computing node can search the routing information to determine the target storage node where the first data table is located, which will not be repeated here.

[0204] Another data processing method is provided in the embodiment of the present application, such as Figure 8 As shown, a management node is applied to a distributed database node cluster, and the distributed database node cluster includes multiple storage nodes; each storage node stores at least one data table and the index of each data row in each data table in each index column;

[0205] The method includes:

[0206] Step S801, periodically receiving a heartbeat message sent by each storage node; the heartbeat message includes the total storage space and the remaining storage space of the corresponding storage node;

[0207] Step S802, in each current cycle, for each storage node, determine the remaining space ratio of the storage node according to the total storage space and the remaining storage space of the storage node, determine a first storage node and a second storage node from each storage node, and if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, migrate the second data table in the first storage node and the index of each data row in the second data table to the second storage node;

[0208] The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node.

[0209] The embodiments of the present application also support redistribution of data stored in storage nodes. Specifically, data redistribution can be performed by a management node, and the management node can periodically receive heartbeat messages sent by each storage node; the heartbeat message includes the total storage space and remaining storage space of the corresponding storage node; in addition, the heartbeat message can also include the number of regions, the size of each region, and the occupied storage space, etc.

[0210] In each cycle, the management node can determine whether the data in each storage node is evenly distributed. Specifically, the remaining space ratio of the storage node can be determined based on the total storage space and the remaining storage space of the storage node, that is, the remaining space ratio = remaining storage space / total storage space. The remaining space ratio can be used to determine whether the data is evenly distributed.

[0211] Normally, the total storage space of each storage node is similar. The distributed database node cluster pursues concurrency and hopes that the data in each storage node can be evenly distributed so that each storage node can be accessed as relatively evenly as possible. After determining the remaining space ratio of each storage node, a first storage node and a second storage node can be determined from each storage node. The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node. If it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, it means that the data distribution in each storage node is uneven and the data needs to be redistributed, that is, the data in the first storage node with a smaller remaining storage space is migrated to the second storage node with sufficient remaining storage space, so that after redistribution, the data can be relatively evenly distributed on each storage node; if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is not greater than the preset threshold, it means that the data distribution of each storage node is relatively uniform. In this case, there is no need to redistribute the data.

[0212] When it is determined that the data distribution in each data storage node is uneven, the embodiment of the present application can migrate the second data table in the first storage node and the index of each data row in the second data table to the second storage node for data redistribution. On the one hand, the data can be relatively evenly distributed on each storage node. On the other hand, it can ensure that the indexes of the data rows in the data table are migrated together with the migration of the data table, so that the data table and the index of the data table are located on the same storage node. When the data table and the index in the data table are subsequently operated, it is only necessary to operate the data table and the index of the data table on one storage node, without crossing multiple storage nodes or across the network, which can effectively reduce latency.

[0213] In an embodiment of the present application, a possible implementation method is provided, where each storage node includes multiple partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix;

[0214] Migrating the second data table in the first storage node and the index of each data row in the second data table to the second storage node includes:

[0215] Determine a first partition to be migrated from the first storage node, where the first partition is a partition that records the data row key-value pairs of each data row in the second data table; add a second partition in the second storage node; and migrate the data row key-value pairs in the first partition to the second partition;

[0216] A third partition is added to the second storage node to instruct the second storage node to generate index key-value pairs in each index column for each data row of the second data table in the third partition according to each index column of the second data table and the column value of each index column in the second partition.

[0217] The embodiment of the present application migrates data tables in units of partitions to achieve data redistribution. Specifically, the first partition to be migrated can be determined from the first storage node, and the first partition is a partition that records the data row key-value pairs of each data row of each second data table. A second partition can be added in the second storage node. Specifically, a first instruction can be sent to the second storage node, and the first instruction is used to instruct the second storage node to create a second partition for itself; after the second partition is created, the data row key-value pairs in the first partition can be migrated to the second partition.

[0218] In addition, the indexes of each data row in the second data table in the first storage node in each index column can also be migrated. However, if the data row and the index of the data row are migrated together, it will bring higher complexity to maintain the consistency of the two. After the second data table is migrated, or in the process of migrating the second data table, the embodiment of the present application can instruct the second storage node to create a third partition in itself according to the various index columns of the second data table and the column values ​​of each index column in the second partition, and generate index key-value pairs in each index column for each data row of the second data table in the third partition, so that the second storage node includes both the data rows in the second data table and the indexes of each second data row in the second table in each index column, so that the two can maintain consistency and reduce the complexity of migrating indexes.

[0219] In an embodiment of the present application, a possible implementation method is provided to migrate the key-value pairs of data rows in the first partition to the second partition, including:

[0220] Determine a first replica synchronization group to which the first partition belongs; the first replica synchronization group contains at least three partitions, and at least the three partitions belong to different storage nodes; the data row key values ​​contained in each partition in the first replica synchronization group are the same; there is a primary partition in at least three partitions in the first replica synchronization group;

[0221] Add the second partition to the first replica synchronization group, determine the primary partition in the first replica synchronization group, synchronize the data row key-value pairs of each data row in the second data table in the primary partition in the first replica synchronization group to the second partition, and delete the first partition.

[0222] Specifically, when migrating a partition, it is necessary to determine the replica synchronization group (Raft group) where the partition is located. The replica synchronization group includes at least three partitions, and at least three partitions belong to different storage nodes. The replica synchronization group includes the primary partition (leader) and the secondary partition (follower). Following the replica synchronization (raft) algorithm, in the replica synchronization group, the writing and deletion of data are all in the primary partition leader, and then the primary partition leader synchronizes its own writing and deletion to the secondary partition follower. The number of partitions in the replica synchronization group is usually an odd number. For a write operation or delete operation, when more than half of the partitions in the replica synchronization group (usually the number of partitions / 2+1) have completed the write operation or delete operation, then the write operation or delete operation executed on the replica synchronization group can be considered successful.

[0223] When migrating each data row key-value pair in the first partition, the embodiment of the present application needs to determine the first replica synchronization cluster where the first partition is located, add the second partition to the first replica synchronization group, determine the primary partition of the first replica synchronization cluster, synchronize the data row key-value pairs of each data row in the second data table in the primary partition in the first replica synchronization group to the second partition, and delete the first partition.

[0224] Specifically, the third storage node where the primary partition of the first replica synchronization cluster is located can be determined, the node identifier of the third storage node and the partition identifier of the primary partition of the first replica synchronization cluster can be determined, and a synchronization instruction can be sent to the second storage node. The synchronization instruction includes the node identifier of the third storage node and the partition identifier of the primary partition of the first replica synchronization cluster. The synchronization instruction is used to instruct the second storage node to synchronize the data row key-value pairs in the primary partition of the first replica synchronization cluster contained in the third storage node.

[0225] In an embodiment of the present application, a possible implementation method is provided to delete the first partition, including:

[0226] If it is determined that the first partition is not the primary partition in the first replica synchronization group, deleting the first partition;

[0227] If it is determined that the first partition is the primary partition in the first replica synchronization group, a new primary partition is selected from the first replica synchronization group, and the new primary partition is not the first partition; and the first partition is deleted.

[0228] When deleting the first partition, the embodiment of the present application needs to determine whether the first partition is the primary partition in the first replica synchronization group. If it is determined that the first partition is not the primary partition in the first replica synchronization group, it means that the first partition does not affect the high availability of the first replica synchronization group, and the first partition can be directly deleted. The management node can send a first deletion instruction to the first storage node, and the first deletion instruction is used to delete the first partition; if it is determined that the first partition is the primary partition in the first replica synchronization group, in order to avoid that the deletion of the first partition affects the high availability of the first replica synchronization group, an election is initiated through the raft algorithm, and a new primary partition is selected from the first replica synchronization group. The new primary partition is not the first partition, and the first partition is deleted. That is, after selecting a new primary partition from the first replica synchronization group, the management node sends a first deletion instruction to the first storage node.

[0229] In addition, after the corresponding index (or index key-value pair) is created for each data row in the second data table in the second storage node, it is also necessary to delete the index (or index key-value pair) in the second data table in the first storage node. The fourth partition where the index key-value pairs of each data row in the second data table are located can be determined, and the fourth partition is deleted. Just like deleting the first partition, it is necessary to determine the second replica synchronization group where the fourth partition is located. The second replica synchronization group includes multiple partitions. The storage nodes where the multiple partitions of the second replica synchronization group and the multiple partitions of the first replica synchronization group are located are the same. The third partition is added to the second replica synchronization group. If it is determined that the fourth partition is not the primary partition of the second replica synchronization group, the fourth partition is directly deleted. If it is determined that the fourth partition is the primary partition of the second replica synchronization group, an election is initiated to select a new primary partition from the second replica synchronization group. The new primary partition is not the fourth partition, and the fourth partition is deleted.

[0230] like Fig. 9 As shown, it exemplarily shows a flow chart of a data redistribution method provided by an embodiment of the present application, involving a management node and multiple storage nodes of a distributed database node cluster, the multiple storage nodes are storage node 1 and storage node 2, wherein storage node 1 is a first storage node, the storage node includes region 1 (first partition) to be migrated, storage node 2 is a second storage node, the management node initiates a region scheduling task for storage node 2, instructs storage node 2 to add a new region 2 (second partition), the management node adds region 2 to the raft group (first replica synchronization group) where region 1 is located, determines the leader in the raft group, instructs storage node 2 to synchronize the data row key value in the leader for region 2, and creates an index for each data row in region 2, and an index can be created in region 3 (third partition).

[0231] In addition, the management node will also initiate a region scheduling task for storage node 1, instructing storage node 1 to delete region 1 from the raft group. If region 1 is the leader of the raft group, it will be downgraded to a follower; if region 1 is not the leader of the raft group, it will directly delete region 1. When deleting the data row key-value pairs of Region 1, storage node 1 needs to delete the corresponding index or index key-value pairs at the same time.

[0232] The present application embodiment provides a data processing device, such as Fig.10 As shown, a target storage node applied to a distributed database node cluster stores a first data table and an index of each data row in each index column in the first data table. The data processing device 1000 may include:

[0233] A first request receiving module 1001 is used to receive a first request sent by a computing node, where the first request includes a row identifier of a first data row in a first data table and a first operation; the first request is generated by the computing node based on a database operation statement sent by a client;

[0234] The first operation execution module 1002 is used to determine a first index column affected by a first operation from various index columns, and perform a first operation on a first data row in a first data table and an index of the first data row in each first index column.

[0235] In an embodiment of the present application, a possible implementation method is provided, in which the target storage node stores each data row in the first data table in the form of a data row key-value pair, wherein the key of each data row key-value pair includes a row identifier of the corresponding data row, and the value is a column value of each column in the corresponding data row;

[0236] The target storage node stores the index of each data row in each index column in the form of index key-value pairs. The key of each index key-value pair includes the column identifier of the corresponding index column and the column value of the corresponding data row in the corresponding index column, and the value is the row identifier of the corresponding data row.

[0237] A possible implementation method is provided in an embodiment of the present application. If the first operation is an insert operation, the first request also includes a data row key-value pair of the first data row;

[0238] The first operation execution module includes:

[0239] An insertion submodule is used to use each index column of the first data table as the first index column; parse the data row key-value pairs of the first data row to obtain the column values ​​of the first data row in each first index column, and for each first index column, generate the key of the index key-value pair of the first index column in the first data row according to the column value of the first data row in the first index column, and generate the value of the index key-value pair of the first index column according to the row identifier of the first data row; insert the data row key-value pair of the first data row and the index key-value pairs of each first index column in the target storage node.

[0240] In an embodiment of the present application, a possible implementation method is provided. If the first operation is a modification operation, the first request further includes a key of a data row key-value pair of a first data row, a column identifier of each first target column to be modified in the first data row, and a modified column value of the first target column;

[0241] The first operation execution module includes:

[0242] A modification submodule is used to locally determine the data row key-value pair of the first data row according to the key of the data row key-value pair of the first data row, update the data row key-value pair of the first data row to a new data row key-value pair, and the column value of each first target column in the value of the new data row key-value pair is the corresponding modified column value; record the column value of each first target column before modification; for each first target column, if it is determined that the first target column belongs to an index column, use the first target column as the first index column; for each first index column, locally determine the index key-value pair of the first data row in the first index column according to the column value of the first data row in the first index column before modification; update the index key-value pair of the first data row in the first index column to a new index key-value pair, and the column value of the first index column in the key of the new index key-value pair is the modified column value; delete the value of the first target column before modification.

[0243] A possible implementation method is provided in an embodiment of the present application. If the first operation is a delete operation, the first request also includes a key of a data row key-value pair of the first data row;

[0244] The first operation execution module includes:

[0245] A deletion submodule is used to locally determine the data row key-value pair of the first data row according to the key of the data row key-value pair of the first data row; use each index column of the first data table as the first index column; determine the column value of the first data row in each first index column from the data row key-value pair of the first data row; for each first index column, locally determine the index key-value pair of the first data row in the first index column according to the column value of the first data row in the first index column; locally delete the data row key-value pair of the first data row and the index key-value pairs of the first data row in each first index column.

[0246] A possible implementation method is provided in an embodiment of the present application. The data processing device 1000 further includes:

[0247] An execution plan receiving module, used to receive an execution plan sent by a computing node, wherein the execution plan includes a query condition for querying a second data row in the first data table;

[0248] A query module, used for, if it is determined that the query condition includes a row identifier of the second data row, locally determining a data row key-value pair of the second data row according to the row identifier of the second data row, and reading the value of the data row key-value pair of the second data row as the second data row; if it is determined that the query condition does not include the row identifier of the second data row, determining each index column of the first data table, and if it is determined that the query condition includes at least one second target column, and the second target column is an index column, locally determining the index key-value pair of the second data row in the second target column according to the column value of the second target column; determining the row identifier of the second data row according to the index key-value pair of the second data row in the second target column, locally determining the data row key-value pair of the second data row according to the row identifier of the second data row, and reading the value of the data row key-value pair of the second data row as the second data row;

[0249] The query result returning module is used to return the query result to the computing node, and the query result includes the second data row.

[0250] A possible implementation method is provided in an embodiment of the present application. The data processing device 1000 further includes:

[0251] A second request receiving module is used to receive a second request sent by the computing node after the node is completed; the second request includes a column identifier of a third target column in the first data table; the second request is used to create an index for the third target column;

[0252] A column value determination module, used to determine the row identifier of each data row in the first data table and the column value of each data row in the third target column;

[0253] The index key-value pair generation module is used to generate and store the index key-value pairs of each data row in the third target column according to the row identifier of each data row in the first data table, the column identifier of the third target column and the column value of each data row in the third target column.

[0254] The present application embodiment provides a data processing device, such as Fig.11 As shown, the data processing device 1100 is applied to a computing node of a distributed database node cluster and includes:

[0255] The operation statement receiving module 1101 is used to receive a database operation statement sent by a client, and generate a first request based on the database operation statement, wherein the first request includes a row identifier of a first data row in a first data table and a first operation;

[0256] A first request sending module 1102 is used to determine a target storage node where the first data table is located, and send a first request to the target storage, so that the target storage node determines a first index column affected by the first operation from each index column, and performs a first operation on the first data row in the first data table and the index of the first data row in each first index column;

[0257] The target storage node stores the first data table and the index of each data row in the first data table in each index column.

[0258] A possible implementation is provided in an embodiment of the present application, where the data processing device 1100 further includes:

[0259] A query statement receiving module, used to receive a database query statement sent by a client, and generate an execution plan based on the database query statement, wherein the execution plan includes a query condition for querying a second data row in the first data table;

[0260] The execution plan sending module is used to send the execution plan to the target storage node.

[0261] A possible implementation method is provided in an embodiment of the present application, where the first request sending module includes:

[0262] The routing information determination submodule is used to obtain the routing information of each node from the management node, and the routing information records the table identifier of the data table contained in each storage node;

[0263] The target storage node determination submodule is used to determine the target storage node where the first data table is located from the routing information.

[0264] The present application embodiment provides a data processing device, such as Fig.12 As shown, a management node is applied to a distributed database node cluster, and the distributed database node cluster includes multiple storage nodes; each storage node stores at least one data table and the index of each data row in each data table in each index column;

[0265] The data processing device 1200 includes:

[0266] The heartbeat message receiving module 1201 is used to periodically receive the heartbeat messages sent by each storage node; the heartbeat message includes the total storage space and the remaining storage space of the corresponding storage node;

[0267] Migration module 1202 is used to determine the remaining space ratio of the storage node according to the total storage space and the remaining storage space of each storage node in each current cycle, determine the first storage node and the second storage node from each storage node, and if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, migrate the second data table in the first storage node and the index of each data row in the second data table to the second storage node;

[0268] The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node.

[0269] In an embodiment of the present application, a possible implementation method is provided, where each storage node includes multiple partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix;

[0270] The migration module includes:

[0271] A migration submodule is used to determine a first partition to be migrated from the first storage node, where the first partition is a partition that records the data row key-value pairs of each data row in the second data table; add a second partition in the second storage node; and migrate the data row key-value pairs in the first partition to the second partition;

[0272] The index creation submodule is used to add a third partition to the second storage node, instructing the second storage node to generate index key-value pairs in each index column for each data row of the second data table in the third partition according to each index column of the second data table and the column value of each index column in the second partition.

[0273] A possible implementation method is provided in an embodiment of the present application, and the migration submodule includes:

[0274] The first replica synchronization group determination submodule is used to determine the first replica synchronization group to which the first partition belongs; the first replica synchronization group contains at least three partitions, and at least three partitions belong to different storage nodes; the data row key values ​​contained in each partition in the first replica synchronization group are the same; there is a primary partition in at least three partitions in the first replica synchronization group;

[0275] The synchronization submodule is used to add the second partition to the first replica synchronization group, determine the primary partition in the first replica synchronization group, synchronize the data row key-value pairs of each data row in the second data table in the primary partition in the first replica synchronization group to the second partition, and delete the first partition.

[0276] A possible implementation method is provided in an embodiment of the present application, and the synchronization submodule includes:

[0277] The deletion unit is used to delete the first partition if it is determined that the first partition is not the primary partition in the first replica synchronization group; if it is determined that the first partition is the primary partition in the first replica synchronization group, select a new primary partition from the first replica synchronization group, the new primary partition is not the first partition, and the first partition is deleted.

[0278] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0279] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0280] In an embodiment of the present application, an electronic device (computer device / equipment / system) is provided, including a memory, a processor and a computer program stored in the memory, and the processor executes the above-mentioned computer program to implement the steps of the data processing method, which can be implemented compared with the related art: the target storage node of the embodiment of the present application simultaneously stores the data table and the index of each data row of the data table in each index column, and after receiving the first operation sent by the computing node, determines the first index column affected by the first operation, and the storage node jointly performs the first operation on the first data row in the first data table and the column value of the first data row in each first index column, that is, the first operation is performed on the index of the data row and the data row within a storage node, and the execution efficiency of performing the first operation on the index of the data row and the data row at a storage node is much higher than that of the data row and the index distributed on different storage nodes, and there is no need to execute across the network, which can effectively reduce the delay. In addition, it is much easier to solve the consistency problem of data rows and indexes within the same storage node than to solve the problem on different storage nodes.

[0281] In an alternative embodiment, an electronic device is provided, such as Fig.13 As shown, Fig.13The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0282] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0283] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0284] Memory 4003 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disc, optical disk, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0285] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0286] Among them, the electronic equipment package may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0287] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented. Compared with the prior art, it can be implemented as follows: the target storage node of the embodiment of the present application simultaneously stores the data table and the index of each data row of the data table in each index column, and after receiving the first operation sent by the computing node, determines the first index column affected by the first operation, and the storage node jointly performs the first operation on the first data row in the first data table and the column value of the first data row in each first index column, that is, the first operation is performed on the index of the data row and the data row within a storage node, and the execution efficiency of performing the first operation on the index of the data row and the data row at a storage node is much higher than that of the data row and the index distributed on different storage nodes, and there is no need to execute across the network, which can effectively reduce the delay. In addition, it is much easier to solve the consistency problem of data rows and indexes within the same storage node than to solve the problem on different storage nodes.

[0288] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0289] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor. Compared with the prior art, it can be achieved that: the target storage node of the embodiment of the present application simultaneously stores the data table and the index of each data row of the data table in each index column, and after receiving the first operation sent by the computing node, determines the first index column affected by the first operation, and the storage node jointly performs the first operation on the first data row in the first data table and the column value of the first data row in each first index column, that is, the first operation is performed on the data row and the index of the data row within a storage node, and the execution efficiency of performing the first operation on the data row and the index of the data row at a storage node is much higher than that when the data row and the index are distributed on different storage nodes, and there is no need to execute across the network, which can effectively reduce the delay. In addition, it is much easier to solve the consistency problem of data rows and indexes within the same storage node than to solve the problem on different storage nodes.

[0290] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0291] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0292] The above are only optional implementation methods for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: A target storage node applied to a distributed database node cluster, the distributed database node cluster comprising a management node, a computing node and a plurality of storage nodes, the target storage node being any node among the plurality of storage nodes, the target storage node storing a first data table and an index of each data row in each index column in the first data table; The storage node includes a plurality of partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix, and the prefix includes a table identifier, which is used to indicate the data table corresponding to the corresponding partition; The management node stores routing information, and the routing information records the table identifier of the data table contained in each storage node; the method includes: Receive a first request sent by a computing node, the first request including a row identifier of a first data row in a first data table and a first operation; the first request is generated by the computing node based on a database operation statement sent by a client, the database operation statement including a table identifier of the first data table, and the target storage node is determined by the computing node by querying the management node for routing information according to the table identifier of the first data table; Determine a first index column affected by the first operation from each index column, and perform a first operation on the first data row in the first data table and the index of the first data row in each first index column; The performing a first operation on the first data row in the first data table and the index of the first data row in each first index column includes: Matching a prefix of a key of a data row key-value pair of the first data row with a prefix of a key of at least one data row key-value pair of each partition, and determining a first target partition from the multiple partitions; Performing a first operation on the data row key-value pair corresponding to the first data row in the first target partition; For each first index column, matching a prefix of a key of an index key-value pair of the first data row in the first index column with a prefix of a key of at least one index key-value pair of each partition, and determining a second target partition from the multiple partitions; A first operation is performed on the index key-value pair corresponding to the first index column of the first data row in the second target partition.

2. The method according to claim 1, characterized in that: The key of each data row key-value pair includes the row identifier of the corresponding data row, and the value is the column value of each column in the corresponding data row; The key of each index key-value pair includes the column identifier of the corresponding index column and the column value of the corresponding data row in the corresponding index column, and the value is the row identifier of the corresponding data row.

3. The method according to claim 2, characterized in that If the first operation is an insert operation, the first request further includes a data row key-value pair of the first data row; The determining, from each index column, a first index column affected by the first operation, and performing a first operation on the first data row in the first data table and the index of the first data row in each first index column includes: Using each index column of the first data table as the first index column; Parsing the data row key-value pair of the first data row to obtain column values ​​of the first data row in each first index column, generating, for each first index column, a key of the index key-value pair of the first index column in the first data row according to the column value of the first data row in the first index column, and generating a value of the index key-value pair of the first index column according to the row identifier of the first data row; Inserting the data row key-value pair of the first data row and the index key-value pairs of each first index column into the target storage node.

4. The method according to claim 2, characterized in that: If the first operation is a modification operation, the first request further includes a key of a data row key-value pair of the first data row, a column identifier of each first target column to be modified in the first data row, and a modified column value of the first target column; The determining, from each index column, a first index column affected by the first operation, and performing a first operation on the first data row in the first data table and the index of the first data row in each first index column includes: According to the key of the data row key-value pair of the first data row, locally determine the data row key-value pair of the first data row, update the data row key-value pair of the first data row to a new data row key-value pair, and the column value of each first target column in the value of the new data row key-value pair is the corresponding modified column value; record the column value of each first target column before modification; For each first target column, if it is determined that the first target column belongs to an index column, the first target column is used as a first index column; For each first index column, locally determine the index key-value pair of the first data row in the first index column according to the column value of the first data row in the first index column before modification; update the index key-value pair of the first data row in the first index column to a new index key-value pair, the column value of the first index column in the key of the new index key-value pair is the modified column value; delete the value of the first target column before modification.

5. The method according to claim 2, characterized in that: If the first operation is a delete operation, the first request further includes a key of a data row key-value pair of the first data row; The determining, from each index column, a first index column affected by the first operation, and performing a first operation on the first data row in the first data table and the index of the first data row in each first index column includes: Determine locally the data row key-value pair of the first data row according to the key of the data row key-value pair of the first data row; Using each index column of the first data table as the first index column; Determine the column value of each first index column of the first data row from the data row key-value pair of the first data row; For each first index column, locally determine an index key-value pair of the first data row in the first index column according to the column value of the first data row in the first index column; The data row key-value pair of the first data row and the index key-value pairs of the first data row in each first index column are deleted locally.

6. The method according to claim 2, characterized in that The method further comprises: Receiving an execution plan sent by the computing node, the execution plan including a query condition for querying a second data row in the first data table; If it is determined that the query condition includes the row identifier of the second data row, locally determining a data row key-value pair of the second data row according to the row identifier of the second data row, and reading the value of the data row key-value pair of the second data row as the second data row; If it is determined that the query condition does not include the row identifier of the second data row, then determine the various index columns of the first data table; if it is determined that the query condition includes at least one second target column, and the second target column is an index column, then locally determine the index key-value pair of the second data row in the second target column according to the column value of the second target column; determine the row identifier of the second data row according to the index key-value pair of the second data row in the second target column, locally determine the data row key-value pair of the second data row according to the row identifier of the second data row, and read the value of the data row key-value pair of the second data row as the second data row; Return a query result to the computing node, where the query result includes the second data row.

7. The method according to claim 1, characterized in that The method further comprises: receiving a second request sent by a computing node; the second request includes a column identifier of a third target column in the first data table; the second request is used to create an index for the third target column; Determine a row identifier of each data row in the first data table and a column value of each data row in the third target column; According to the row identifier of each data row in the first data table, the column identifier of the third target column and the column value of each data row in the third target column, an index key-value pair of each data row in the third target column is generated and stored.

8. A data processing method, characterized in that: A computing node applied to a distributed database node cluster, the distributed database node cluster includes a management node and multiple storage nodes, the management node stores routing information, the routing information records the table identifier of a data table contained in each storage node; each storage node includes multiple partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix, the prefix includes a table identifier, which is used to indicate the data table corresponding to the corresponding partition, the method includes: Receive a database operation statement sent by a client, and generate a first request based on the database operation statement, wherein the first request includes a row identifier of a first data row in a first data table and a first operation, and the database operation statement includes a table identifier of the first data table; According to the table identifier of the first data table, a routing information query is performed on the management node to determine the target storage node where the first data table is located, and the first request is sent to the target storage, so that the target storage node determines the first index column affected by the first operation from each index column, and performs a first operation on the first data row in the first data table and the index of the first data row in each first index column; wherein the target storage node performs a first operation on the first data row in the first data table and the index of the first data row in each first index column, including: matching the prefix of the data row key-value pair of the first data row with the prefix of the key of at least one data row key-value pair of each partition, and determining a first target partition from multiple partitions; performing a first operation on the data row key-value pair corresponding to the first data row in the first target partition; for each first index column, matching the prefix of the key of the index key-value pair of the first data row in the first index column with the prefix of the key of at least one index key-value pair of each partition, and determining a second target partition from multiple partitions; performing a first operation on the index key-value pair corresponding to the first data row in the first index column in the second target partition.

9. The method according to claim 8, characterized in that The method further comprises: Receive a database query statement sent by a client, and generate an execution plan based on the database query statement, wherein the execution plan includes a query condition for querying a second data row in a first data table; The execution plan is sent to the target storage node.

10. A data processing method, characterized in that: A management node applied to a distributed database node cluster, wherein the management node stores routing information, wherein the routing information records a table identifier of a data table contained in each storage node; the distributed database node cluster includes a computing node and a plurality of storage nodes; Each storage node stores at least one data table and the index of each data row in each data table in each index column, and each storage node includes multiple partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix, and the prefix includes a table identifier, which is used to indicate the data table corresponding to the corresponding partition; The method comprises: Periodically receiving heartbeat messages sent by each storage node; the heartbeat messages include the total storage space and remaining storage space of the corresponding storage node; In each current cycle, for each storage node, the remaining space ratio of the storage node is determined according to the total storage space and the remaining storage space of the storage node, and a first storage node and a second storage node are determined from each storage node. If it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, the second data table in the first storage node and the index of each data row in the second data table are migrated to the second storage node; The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node; Receive a routing information query request sent by a computing node, so that when the computing node receives a database operation statement including a table identifier of a first data table sent by a client, the computing node determines a target storage node according to the table identifier of the first data table in the database operation statement and the routing information stored by the management node, and performs the following operations: Generate a first request based on the database operation statement, the first request including a row identifier of a first data row in a first data table and a first operation; send the first request to the target storage, so that the target storage node determines a first index column affected by the first operation from each index column, and performs the first operation on the first data row in the first data table and the index of the first data row in each first index column; The target storage node performs a first operation on the first data row in the first data table and the index of the first data row in each first index column, including: Match the prefix of the data row key-value pair of the first data row with the prefix of the key of at least one data row key-value pair of each partition, and determine a first target partition from multiple partitions; perform a first operation on the data row key-value pair corresponding to the first data row in the first target partition; for each first index column, match the prefix of the key of the index key-value pair of the first data row in the first index column with the prefix of the key of at least one index key-value pair of each partition, and determine a second target partition from multiple partitions; perform a first operation on the index key-value pair corresponding to the first data row in the first index column in the second target partition.

11. The method according to claim 10, characterized in that The step of migrating the second data table in the first storage node and the index of each data row in the second data table to the second storage node includes: Determine a first partition to be migrated from the first storage node, where the first partition is a partition that records the data row key-value pairs of each data row in the second data table; add a second partition in the second storage node; and migrate the data row key-value pairs in the first partition to the second partition; A third partition is added to the second storage node to instruct the second storage node to generate index key-value pairs in each index column for each data row of the second data table in the third partition according to each index column of the second data table and the column value of each index column in the second partition.

12. The method according to claim 11, characterized in that The step of migrating the data row key-value pairs in the first partition to the second partition includes: Determine a first replica synchronization group to which the first partition belongs; the first replica synchronization group includes at least three partitions, and the at least three partitions belong to different storage nodes; the data row key values ​​included in each partition in the first replica synchronization group are the same; there is a primary partition among the at least three partitions in the first replica synchronization group; Add the second partition to the first replica synchronization group, determine the primary partition in the first replica synchronization group, synchronize the data row key-value pairs of each data row in the second data table in the primary partition in the first replica synchronization group to the second partition, and delete the first partition.

13. The method according to claim 12, characterized in that The deleting the first partition includes: If it is determined that the first partition is not a primary partition in the first replica synchronization group, deleting the first partition; If it is determined that the first partition is the primary partition in the first replica synchronization group, a new primary partition is selected from the first replica synchronization group, the new primary partition is not the first partition, and the first partition is deleted.

14. A data processing device, characterized in that: A target storage node applied to a distributed database node cluster, the distributed database node cluster comprising a management node, a computing node and a plurality of storage nodes, the target storage node being any node among the plurality of storage nodes, the target storage node storing a first data table and an index of each data row in each index column in the first data table; The target storage node includes a plurality of partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix, and the prefix includes a table identifier, which is used to indicate the data table corresponding to the corresponding partition; The management node stores routing information, and the routing information records the table identifier of the data table contained in each storage node; the device includes: A first request receiving module is used to receive a first request sent by a computing node, wherein the first request includes a row identifier of a first data row in a first data table and a first operation; the first request is generated by the computing node based on a database operation statement sent by a client, and the target storage node is determined by the computing node by querying the management node for routing information according to the table identifier of the first data table; The first operation execution module is configured to determine the first index column affected by the first operation from the various index columns, and perform the first operation on the first data row in the first data table and the index of the first data row in the various first index columns; the performing the first operation on the first data row in the first data table and the index of the first data row in the various first index columns includes: Matching the prefix of the data row key-value pair of the first data row with the prefix of the key of at least one data row key-value pair of each partition, and determining a first target partition from the multiple partitions; Performing a first operation on the data row key-value pair corresponding to the first data row in the first target partition; For each first index column, matching a prefix of a key of an index key-value pair of the first data row in the first index column with a prefix of a key of at least one index key-value pair of each partition, and determining a second target partition from the multiple partitions; A first operation is performed on the index key-value pair corresponding to the first index column of the first data row in the second target partition.

15. A data processing device, characterized in that: A computing node applied to a distributed database node cluster, the distributed database node cluster includes a management node and multiple storage nodes, the management node stores routing information, the routing information records the table identifier of the data table contained in each storage node; each storage node includes multiple partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix, the prefix includes a table identifier, which is used to indicate the data table corresponding to the corresponding partition, the device includes: An operation statement receiving module, configured to receive a database operation statement sent by a client, and generate a first request based on the database operation statement, wherein the first request includes a row identifier of a first data row in a first data table and a first operation, and the database operation statement includes a table identifier of the first data table; A first request sending module is used to query the management node for routing information according to the table identifier of the first data table, determine the target storage node where the first data table is located, and send the first request to the target storage, so that the target storage node determines the first index column affected by the first operation from each index column, and performs a first operation on the first data row in the first data table and the index of the first data row in each first index column, wherein the target storage node performs a first operation on the first data row in the first data table and the index of the first data row in each first index column, including: matching the prefix of the data row key-value pair of the first data row with the prefix of the key of at least one data row key-value pair of each partition, and determining a first target partition from multiple partitions; performing a first operation on the data row key-value pair corresponding to the first data row in the first target partition; for each first index column, matching the prefix of the key of the index key-value pair of the first data row in the first index column with the prefix of the key of at least one index key-value pair of each partition, and determining a second target partition from multiple partitions; performing a first operation on the index key-value pair corresponding to the first data row in the first index column in the second target partition.

16. A data processing device, characterized in that: A management node applied to a distributed database node cluster, wherein the management node stores routing information, wherein the routing information records a table identifier of a data table contained in each storage node; the distributed database node cluster includes a computing node and a plurality of storage nodes; Each storage node stores at least one data table and the index of each data row in each data table in each index column, and each storage node includes multiple partitions; each partition is used to store at least one data row key-value pair or index key-value pair; the key of at least one data row key-value pair or index key value of each partition has the same prefix, and the prefix includes a table identifier, which is used to indicate the data table corresponding to the corresponding partition; The device comprises: A heartbeat message receiving module, used for periodically receiving heartbeat messages sent by each storage node; the heartbeat message includes the total storage space and the remaining storage space of the corresponding storage node; A migration module, configured to determine, for each storage node, the remaining space ratio of the storage node according to the total storage space and the remaining storage space of the storage node, determine a first storage node and a second storage node from each storage node, and if it is determined that the difference between the remaining space ratios of the first storage node and the second storage node is greater than a preset threshold, migrate the second data table in the first storage node and the index of each data row in the second data table to the second storage node; The remaining space ratio of the first storage node is the minimum value among the remaining space ratios of each storage node; the remaining space ratio of the second storage node is the maximum value among the remaining space ratios of each storage node. The routing information query request sent by the computing node is received, so that when the computing node receives the database operation statement including the table identifier of the first data table sent by the client, the computing node determines the target storage node according to the table identifier of the first data table in the database operation statement and the routing information stored by the management node, and performs the following operations: Generate a first request based on the database operation statement, the first request including a row identifier of a first data row in a first data table and a first operation; send the first request to the target storage, so that the target storage node determines a first index column affected by the first operation from each index column, and performs the first operation on the first data row in the first data table and the index of the first data row in each first index column; The target storage node performs a first operation on the first data row in the first data table and the index of the first data row in each first index column, including: Match the prefix of the data row key-value pair of the first data row with the prefix of the key of at least one data row key-value pair of each partition, and determine a first target partition from multiple partitions; perform a first operation on the data row key-value pair corresponding to the first data row in the first target partition; for each first index column, match the prefix of the key of the index key-value pair of the first data row in the first index column with the prefix of the key of at least one index key-value pair of each partition, and determine a second target partition from multiple partitions; perform a first operation on the index key-value pair corresponding to the first data row in the first index column in the second target partition.

17. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of any one of claims 1-7, 8-9 and 10-13.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7, 8 to 9 and 11 to 13 are implemented.