A distributed database operation method, server and storage medium

Abstract

The present application relates to the field of database technology, and proposes an operating method, server and storage medium for a distributed database. The present application sets up a main server and at least one sub-server, and divides the data of the data node into the storage area of ​​each sub-server for storage. When a key-value pair needs to be added to the database, the main server will determine the data node where the key-value pair needs to be inserted, and then send the page number and key-value pair corresponding to the data node to the sub-server where the data of the data node is located. After receiving the page number and key-value pair, the sub-server will perform key-value pair merging and data node splitting operations on the data node, and return the obtained data node splitting results to the main server. Finally, the main server performs an index node splitting operation based on the data node splitting result, thereby completing the operation of adding the key-value pair to the database. By setting it in this way, the data transmission pressure brought to each server can be reduced.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to an operating method, server, and storage medium of a distributed database. Background Art

[0002] The Merkle B+ tree is an improved tree structure based on the B+ tree. It contains two different types of nodes: index nodes and data nodes. Index nodes store the minimum key and hash value of child nodes, while data nodes store data in the form of key-value pairs. The data stored in each node is stored on the server's disk in the form of a data page, that is, each node corresponds to a page number, which is the number of the data page where the data is stored. Currently, when adding key-value pairs to a database based on the Merkle B+ tree, it will bring high data transmission pressure to the server. For example, if N key-value pairs are inserted into the database at one time, these N key-value pairs may fall into N data nodes and may also fall into N-2N index nodes. This means that the read and write operations of 2N-3N data pages will cause high data transmission pressure. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a distributed database operation method, server, and storage medium, which can reduce the data transmission pressure on the server when operating the database.

[0004] A first aspect of an embodiment of the present application provides an operating method of a distributed database applied to a master server, including:

[0005] Obtaining a key-value pair to be added to a distributed database; wherein the distributed database is based on a Merkle B+ tree, the main server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of storage slices provided by all sub-servers is at least one, each storage slice stores a set number of data pages, and the data pages are used to store data of the data nodes of the Merkle B+ tree;

[0006] Determine, according to the key of the key-value pair, a target data node in the Merkle B+ tree into which the key-value pair is to be inserted;

[0007] Determining, from the at least one sub-server, a first sub-server where data of the target data node is located, based on a target page number corresponding to the target data node; wherein the target page number is a number of a target data page, the target data page stores the data of the target data node, and the first storage area of ​​the first sub-server stores the target data page;

[0008] Sending the key-value pair and the target page number to the first sub-server to instruct the first sub-server to perform a key-value pair merge and a data node splitting operation on the target data node according to the key-value pair and the target page number, obtaining a data node splitting result, and returning the data node splitting result to the main server;

[0009] After receiving the data node splitting result, the index node splitting operation of the Merkle B+ tree is performed according to the data node splitting result to complete the operation of adding the key-value pair to the distributed database.

[0010] In an embodiment of the present application, a main server and at least one sub-server are set up. Considering that the data nodes of the Merkle B+ tree store key-value pairs with a large amount of data, while the index nodes only store index information with a small amount of data, the data of the data nodes are divided into each sub-server for storage, wherein each sub-server is provided with at least zero storage areas, and each storage area stores a set number of data pages for storing the data of the corresponding data node. When a key-value pair needs to be added to the database, the main server will determine the data node where the key-value pair needs to be inserted, and then send the page number and key-value pair corresponding to the data node to the sub-server where the data of the data node is located. After receiving the page number and key-value pair, the sub-server will perform the key-value pair merge and data node splitting operations on the data node, and return the obtained data node splitting results to the main server. Finally, the main server performs the index node splitting operation based on the data node splitting results, thereby completing the operation of adding the key-value pair to the database. Through this setting, when operating the database, the data transmission pressure of the main server can be distributed to each sub-server, thereby reducing the data transmission pressure brought to each server.

[0011] A second aspect of an embodiment of the present application provides a method for operating a distributed database applied to a first sub-server, including:

[0012] Receive a target page number and a key-value pair to be added to a distributed database sent by a main server; wherein the distributed database is based on a Merkle B+ tree, the main server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of the storage slices provided by all the sub-servers is at least 1, each storage slice stores a set number of data pages, the data pages are used to store data of data nodes of the Merkle B+ tree, the first sub-server is the sub-server where the data of the target data node determined by the main server from the at least one sub-server is located, the target page number is the number of the target data page, the target data page stores the data of the target data node, the first storage slice of the first sub-server stores the target data page, the target data node is determined by the main server according to the key of the key-value pair, and is the data node in the Merkle B+ tree into which the key-value pair needs to be inserted;

[0013] Performing key-value pair merging and data node splitting operations on the target data node according to the key-value pair and the target page number to obtain a data node splitting result;

[0014] The data node splitting result is returned to the master server to instruct the master server to perform the index node splitting operation of the Merkel B+ tree according to the data node splitting result after receiving the data node splitting result, so as to complete the operation of adding the key-value pair to the distributed database.

[0015] A third aspect of an embodiment of the present application provides an operating device for a distributed database applied to a master server, including:

[0016] A key-value pair acquisition module, configured to acquire key-value pairs to be added to a distributed database; wherein the distributed database is based on a Merkle B+ tree, the main server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of storage slices provided by all sub-servers is at least one, each storage slice stores a set number of data pages, and the data pages are used to store data of the data nodes of the Merkle B+ tree;

[0017] A data node determination module is used to determine, according to the key of the key-value pair, a target data node in the Merkle B+ tree into which the key-value pair needs to be inserted;

[0018] a sub-server determining module, configured to determine, from the at least one sub-server, a first sub-server where the data of the target data node is located, based on a target page number corresponding to the target data node; wherein the target page number is the number of a target data page, the target data page stores the data of the target data node, and the first storage area of ​​the first sub-server stores the target data page;

[0019] a key-value pair sending module, configured to send the key-value pair and the target page number to the first sub-server, to instruct the first sub-server to perform a key-value pair merge and a data node splitting operation on the target data node according to the key-value pair and the target page number, to obtain a data node splitting result, and to return the data node splitting result to the main server;

[0020] The index node splitting module is used to perform the index node splitting operation of the Merkel B+ tree according to the data node splitting result after receiving the data node splitting result, so as to complete the operation of adding the key-value pair to the distributed database.

[0021] A fourth aspect of an embodiment of the present application provides an operating device for a distributed database applied to a first sub-server, including:

[0022] A key-value pair receiving module is configured to receive a target page number and a key-value pair to be added to a distributed database sent by a main server; wherein the distributed database is based on a Merkle B+ tree, the main server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of storage slices provided by all the sub-servers is at least 1, each storage slice stores a set number of data pages, the data pages are used to store data of data nodes of the Merkle B+ tree, the first sub-server is the sub-server where the data of the target data node determined by the main server from the at least one sub-server is located, the target page number is the number of the target data page, the target data page stores the data of the target data node, the first storage slice of the first sub-server stores the target data page, the target data node is the data node in the Merkle B+ tree determined by the main server according to the key of the key-value pair, and the key-value pair needs to be inserted;

[0023] A data node splitting module is used to perform key-value pair merging and data node splitting operations on the target data node according to the key-value pair and the target page number, to obtain a data node splitting result;

[0024] A split result returning module is used to return the data node splitting result to the main server to instruct the main server to perform the index node splitting operation of the Merkel B+ tree according to the data node splitting result after receiving the data node splitting result, so as to complete the operation of adding the key-value pair to the distributed database.

[0025] The fifth aspect of an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the operating method provided in the first aspect of the embodiment of the present application, or implements the operating method provided in the second aspect of the embodiment of the present application.

[0026] The sixth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the operating method provided in the first aspect of the embodiments of the present application, or implements the operating method provided in the second aspect of the embodiments of the present application.

[0027] The seventh aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a server, it enables the server to execute the operating method provided in the first aspect of the embodiments of the present application, or execute the operating method provided in the second aspect of the embodiments of the present application.

[0028] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of a Merkle B+ tree provided in an embodiment of the present application;

[0030] Figure 2 This is a schematic diagram of the structure of a server system provided in an embodiment of the present application;

[0031] Figure 3 This is a schematic diagram of setting storage areas for each sub-server provided in an embodiment of the present application;

[0032] Figure 4 This is a flow chart of a distributed database operation method provided by an embodiment of the present application;

[0033] Figure 5 This is a schematic diagram of an operation of determining a forwarded storage slice based on a global maximum page number provided by an embodiment of the present application;

[0034] Figure 6This is a schematic diagram of the operation of updating the global maximum page number and creating a new storage area provided by an embodiment of the present application;

[0035] Figure 7 This is a schematic diagram of the operation of adding a key-value pair to a distributed database provided by an embodiment of the present application;

[0036] Figure 8 This is a schematic diagram of an initial state Merkle B+ tree provided by an embodiment of the present application;

[0037] Figure 9 This embodiment of the present application provides Figure 8 Schematic diagram of the Merkle B+ tree performing key-value pair distribution and merging operations;

[0038] Figure 10 This embodiment of the present application provides Figure 9 Schematic diagram of the Merkle B+ tree performing a node splitting operation;

[0039] Figure 11 This is a schematic diagram of performing a data rollback operation on a database provided in an embodiment of the present application;

[0040] Figure 12 This is a structural diagram of an operating device for a distributed database applied to a master server provided in an embodiment of the present application;

[0041] Figure 13 1 is a schematic structural diagram of an operating device for a distributed database applied to a first sub-server provided in an embodiment of the present application;

[0042] Figure 14 This is a schematic diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details. In addition, in the description of the present application specification and the appended claims, the terms "first," "second," "third," etc. are only used to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0044] The Merkle B+ tree is based on a B+ tree and consists of data nodes (dataNode) at the bottom level and index nodes (indexNode) at each level above it. The highest level index node is called the root node, and the lowest level (i.e., the leaf level) index nodes are called leaf index nodes. Each node in a Merkle B+ tree (including data nodes and index nodes) has an ID. The child node list of an index node stores the minimum key and hash value of the child node. The child node list of a data node stores the key-value pairs of state data [key, value]. The hash value of each node can be calculated by the hash value of the sum of its child node lists.

[0045] like Figure 1 As shown in FIG, a schematic diagram of a Merkle B+ tree provided in an embodiment of the present application is provided. Figure 1 In the example, nodes n1, n2, and n3 are index nodes, and nodes n4, n5, and n6 are data nodes. Database data is stored in data nodes as key-value pairs. Each node (including data nodes and index nodes) is stored on the server's disk as a data page. To better adapt to disk read and write strategies, the size of each data page can be specified as 4K bytes (disk pages are typically 4K bytes). This allows the data for a particular node to be read from disk by calculating the offset of the page number (i.e., page ID) * page size (4K bytes). The data stored on disk includes index files (i.e., index node data) and data files (i.e., data node data).

[0046] by Figure 1 For example, when performing data query, assuming that a query with key c is required, starting from node n1 (root node), the minimum key (minimum key) of the two child nodes is compared with c. It can be found that c falls within the range of the first child node of n1, so the query reaches node n2. Repeat this process until it falls to the data node, and finally find the data node n5, and then query the position of c and the corresponding key value.

[0047] In terms of page number management, the server can set a maximum page ID and a list of free page numbers. When a database based on the Merkel B+ tree creates or updates a node (including index nodes and data nodes), it is equivalent to creating a new node. At this time, a page ID needs to be assigned to the new node. First, the server will go to the free page number list to check whether there are free page numbers (that is, page numbers that are not assigned to nodes). If there are, they will be taken out and assigned to the new node. Otherwise, the maximum page ID will be assigned to the new node and the maximum page ID will be incremented. There will naturally be page ID recycling when there is page ID allocation. When node A splits into node B and node C, the page ID of node A will be recycled and included in the free page number list. In addition, when a node is found to become an empty node, the page ID will also be recycled to the free page number list.

[0048] Currently, when performing data queries or adding key-value pairs in a Merkle B+ tree-based database, data read and write operations are usually performed on a large number of nodes, which can place a great deal of data transmission pressure on the server. In view of this, the embodiments of the present application provide a method for operating a distributed database that can reduce the data transmission pressure placed on the server when operating the database. For more specific technical implementation details of the embodiments of the present application, please refer to the various embodiments described below.

[0049] See also Figure 2 , is a structural diagram of a server system provided in an embodiment of the present application.

[0050] exist Figure 2 In the example, a main server and N (N≥1) sub-servers are included, and data can be exchanged between the servers. Considering that the data nodes of the Merkle B+ tree store key-value pairs with a large amount of data, while the index nodes only store index information with a small amount of data, the embodiment of the present application mainly fragments the data files of the data nodes, that is, the data of the data nodes are stored in the N sub-servers, and the main server only stores the data of the index nodes. Through such a setting, most of the data transmission pressure of the main server can be shared among the sub-servers when operating the database. On the one hand, the data transmission pressure of the main server can be reduced, and on the other hand, it will not cause excessive data transmission pressure on a single sub-server.

[0051] The above-mentioned sub-servers can also be called shard servers. At least zero storage slices can be set on each shard server. The total number of storage slices set by all shard servers is at least 1. Each storage slice stores a set number of data pages. This number is called the storage capacity of the storage slice. These data pages are used to store the data of the data nodes of the Merkle B+ tree. In addition, the data pages stored in each storage slice have their own corresponding page number range. Figure 3As shown, it can be stipulated that each storage area stores 100 data pages (i.e., storage capacity = 100, which can be set as needed). Each sub-server can have multiple storage areas. The main server can store various index files (such as the maximum page ID and free page number list of the index file, etc.), related sub-server information (such as the IP address and port number of the sub-server and the page number range of the stored data pages), a global maximum page number MaxDataID (the role of MaxDataID will be explained later) and a mapping information (the role of the mapping information will be explained later). Each sub-server needs to maintain the information of each storage area it has (mainly the page number range of the data pages stored in each storage area). Each storage area stores the data of the data node (including the maximum page ID and free page number list, etc.), the global maximum page number MaxDataID and a mapping information.

[0052] exist Figure 3 In the , each storage slice has been numbered from small to large according to the page number range corresponding to the data page stored in each, and each sub-server has completed the numbering. Each storage slice is distributed in order from small to large to each sub-server arranged in order from small to large for storage. For example, storage slice 1 (page number range: 0-99) is distributed on sub-server 1, storage slice 2 (page number range: 100-199) is distributed on sub-server 2, and storage slice 3 (page number range: 200-299) is distributed on sub-server 3; as for storage slice 4 (page number range: 300-399), since there are only 3 sub-servers currently, it is returned in order and inserted into sub-server 1, and so on. There are no data pages with duplicate page numbers in each storage slice. It should be noted that Figure 3 This is just an example provided in the embodiment of the present application. In actual operation, more (for example, more than 4) or fewer (for example, 1) sub-servers can be set up, and the number of storage slices and the storage capacity of the storage slices provided in each sub-server can be different. In addition, there is no restriction that the storage slices must be distributed to each sub-server in sequence according to the size of the corresponding page number range.

[0053] See also Figure 4 , shows a distributed database operation method provided by an embodiment of the present application, including:

[0054] 401. The primary server obtains the key-value pair to be added to the distributed database;

[0055] The embodiments of this application are based on Figure 2The server system architecture shown implements a distributed database based on a Merkle B+ tree. When adding a key-value pair to the distributed database, the master server first retrieves the key-value pair to be added. These key-value pairs can be one or more, supporting batch insertion of key-value pairs.

[0056] 402. The primary server determines, based on the key of the key-value pair, a target data node in the Merkle B+ tree into which the key-value pair is to be inserted.

[0057] After obtaining the key-value pair to be added, the master server will determine the data node (represented by the target data node) in the Merkle B+ tree where the key-value pair needs to be inserted based on the key value of the key-value pair. Since the master server stores the data of the index nodes of the Merkle B+ tree, it can find the target data node for the key-value pair based on the key value and the range of each index node.

[0058] Specifically, determining, based on the key of the key-value pair, the target data node in the Merkle B+ tree into which the key-value pair needs to be inserted may include:

[0059] (1) Distributing the key-value pairs in sequence from the root node of the Merkle B+ tree downwards according to the key and the minimum key of each index node in the Merkle B+ tree until the key-value pairs are distributed to the leaf index nodes of the Merkle B+ tree;

[0060] (2) Determine the target data node from the data nodes of the Merkle B+ tree connected to the leaf index node based on the key and the minimum key of the leaf index node.

[0061] In specific operations, the key value can be compared with the minimum key of each index node, starting from the root node of the Merkle B+ tree and distributing the key-value pair downwards until the key-value pair is distributed to the leaf index node (i.e., the index node of the last layer). Then, based on the key and the minimum key of the leaf index node, the target data node can be found from the various data nodes connected to the leaf index node. For example, in Figure 1 In the example, assuming that the key-value pair has been distributed to the leaf index node n2, the key value is b, and the data nodes connected to the leaf index node n2 are n4 and n5, it can be found that b falls within the range of the first child node of the leaf index node n2 (the minimum key is a, that is, the key value ≥ a), and then the data node n4 connected to the first child node of the leaf index node n2 can be determined as the target data node.

[0062] 403. The main server determines, based on the target page number corresponding to the target data node, a first sub-server where data of the target data node is located from the at least one sub-server.

[0063] After determining the target data node, the master server can obtain the page number corresponding to the target data node (represented by the target page number). The data of each data node is stored in a data page, and each data page has a corresponding number (i.e., a page number). Here, the data page storing the data file of the target data node is represented by the target data page, and the target page number is the number of the target data page. The storage area where the target data page is located is represented by the first storage area, and the sub-server where the first storage area is located is represented by the first sub-server.

[0064] The master server needs to find the first sub-server where the data of the target data node is located from each sub-server based on the target page number. In one implementation, the master server can store information about the storage slices provided by each sub-server (primarily including the page number range of the stored data pages). In this way, after obtaining the target page number, the master server can find the corresponding storage slice (i.e., the first storage slice) based on the page number range where the target page number is located, and then find the corresponding sub-server (i.e., the first sub-server).

[0065] In another implementation, for example Figure 3 The distribution of storage areas shown in the figure can use another method to index the sub-servers by page number. This indexing method includes:

[0066] (1) performing a modulo operation on the target page number and the target quantity to obtain a first value;

[0067] (2) performing a modulo operation on the first value and the number of each of the sub-servers to obtain a second value;

[0068] (3) Determine the sub-server numbered with the second value among the sub-servers as the first sub-server.

[0069] If the number of sub-servers remains unchanged and the storage capacity of each storage area is the same target number, the sub-server number can be calculated by (target page number % target number) % number of sub-servers, and then the sub-server with this number can be determined as the first sub-server, where % represents a modulo operation. For example, Figure 3Assuming the target page number is 150, the target quantity is 100, and the number of sub-servers is 3, the calculation is (150%100)%3=50%3=2. Therefore, sub-server 2 is determined to be the first sub-server, which is consistent with the actual situation. Using this calculation method, the main server does not need to store information about the storage area of ​​each sub-server, which can reduce the amount of data stored on the main server to a certain extent.

[0070] Furthermore, before performing a modulo operation on the first value and the number of each sub-server to obtain the second value, the method may further include:

[0071] (1) Obtaining preset mapping information, wherein the mapping information records the corresponding relationship between each preset page number range and the number of servers;

[0072] (2) determining a target page number range in which the target page number is located within each of the preset page number ranges;

[0073] (3) Find the number of servers corresponding to the target page number range from the mapping information as the number of each sub-server.

[0074] Since the distributed database of the embodiment of the present application supports the dynamic addition of the number of sub-servers, the main server also needs to obtain the current number of sub-servers before performing the modulo operation. The mapping information mentioned above will be used here, and the mapping information records the corresponding relationship between each preset page number range and the number of servers. According to the target page number range in each preset page number range, the number of servers corresponding to the target page number range can be found from the mapping information as the current number of sub-servers. For example, assuming that the mapping information records that the page number range 0-499 corresponds to the number of servers 3, the page number range 500-699 corresponds to the number of servers 4, the page number range 700-899 corresponds to the number of servers 5..., then if the target page number is 150, the corresponding number of sub-servers is 3, if the target page number is 550, the corresponding number of sub-servers is 4, and so on.

[0075] 404. The main server sends the key-value pair and the target page number to the first sub-server;

[0076] After determining the first sub-server, the main server will send the key-value pair and the target page number to the first sub-server, and then the first sub-server will perform key-value pair merging and data node splitting operations. Specifically, the steps of the main server sending the key-value pair and the target page number to the first sub-server may include:

[0077] (1) generating a gRPC request carrying the key-value pair and the target page number;

[0078] (2) Send the gRPC request to the first sub-server.

[0079] The master server can package the key-value pair and the target page number into a gRPC request and then send the gRPC request to the first child server. Using gRPC for data exchange between servers has the advantages of ease of use, high data transmission efficiency, and strong compatibility.

[0080] 405. The first sub-server performs a key-value pair merge and a data node split operation on the target data node according to the key-value pair and the target page number, and obtains a data node split result.

[0081] After receiving the key-value pair and target page number sent by the main server, the first sub-server will perform the corresponding key-value pair merging and data node splitting operations on the target data node to obtain the data node splitting result.

[0082] In one implementation of the embodiment of the present application, performing the key-value pair merging and data node splitting operations on the target data node according to the key-value pair and the target page number to obtain the data node splitting result may include:

[0083] (1) determining the first storage slice from all storage slices of the first sub-server according to the target page number and the storage slice information of the first sub-server; wherein the storage slice information records the page number range corresponding to each storage slice provided in the first sub-server;

[0084] (2) distributing the key-value pairs to the target data node stored in the first storage area, and performing a key-value pair merging operation on the target data node;

[0085] (3) If the amount of data on the target data node after the key-value pair merge operation exceeds a set threshold, the target data node is split into at least two new data nodes, so that the data of each new data node can be written into one data page;

[0086] (4) assigning a corresponding page number to each of the new data nodes, and writing the data of each of the new data nodes into the data page corresponding to the respective page number;

[0087] (5) Determine the data node splitting result based on the key value, hash value and corresponding page number of each new data node.

[0088] For the above step (1), the first sub-server can determine the first storage area from all storage areas of the first sub-server according to the target page number and the storage area information of the first sub-server. Figure 3In the example, if the target page number is 150, it can be found that it is within the page number range (100-199) corresponding to the storage slice 2, so it can be determined that the storage slice 2 is the first storage slice.

[0089] For the above step (2), the first sub-server will distribute the key-value pair to the target data node stored in the first storage area, and perform a key-value pair merge operation on the target data node, that is, merge the currently inserted key-value pair with the original key-value pair of the target data node.

[0090] Regarding step (3) above, if the first sub-server detects that the amount of data on the target data node after the key-value pair merge operation exceeds a set threshold (e.g., a 4K data page), it is necessary to perform a data node split operation. Here, the target data node will be split into at least two new data nodes according to the set split threshold, so that the data of each new data node can be written into a data page. Conversely, if the amount of data on the target data node after the key-value pair merge operation does not exceed the set threshold, the target data node does not need to be split.

[0091] For the above step (4), the first sub-server needs to assign a corresponding page number (i.e., the number of the data page storing the data node) to each new data node, and then write the data of each new data node into the data page corresponding to the respective page number.

[0092] Specifically, the first sub-server can record and store the minimum available page number and free page number list of each storage slice; wherein the minimum available page number is equal to the maximum page number currently allocated to the data node of the corresponding storage slice plus one, and the free page number list records the free page number of the corresponding storage slice that can currently be allocated to the data node. The minimum available page number here is the same concept as the maximum page ID described in the previous content of page number management, which represents a page number that is larger than all allocated page numbers and is unallocated, equivalent to the minimum value of the available page number. For example, assuming that the page numbers currently allocated to the node are 1-60, then 61 is a page number that is larger than all allocated page numbers and is unallocated, and 61 is also the minimum value among the available page numbers (61, 62, 63...). Each storage slice has a corresponding free page number list, which records the free page numbers that the storage slice can currently allocate to the data node. These free page numbers are generally page numbers that have been allocated to the node but have been recycled. Step (4) may include:

[0093] (4.1) Detecting whether the number of free page numbers recorded in the free page number list of the first storage area is greater than or equal to the number of each of the new data nodes;

[0094] (4.2) If the number of free page numbers recorded in the free page number list of the first storage area is greater than or equal to the number of the new data nodes, allocate one free page number to each of the new data nodes;

[0095] (4.3) If the number of free page numbers recorded in the free page number list of the first storage slice is less than the number of each of the new data nodes, each of the new data nodes is divided into a first data node set and a second data node set; wherein the number of data nodes included in the first data node set is equal to the number of free page numbers; one free page number is allocated to each data node included in the first data node set; for any data node included in the second data node set, whether the minimum available page number of the first storage slice has exceeded the upper limit of the page number range corresponding to the first storage slice is detected; if the minimum available page number of the first storage slice does not exceed the upper limit of the page number, the minimum available page number of the first storage slice is allocated to the arbitrary data node, and the minimum available page number of the first storage slice is incremented; if the minimum available page number of the first storage slice has exceeded the upper limit of the page number, a second storage slice is determined, and the data of the arbitrary data node is sent to the second sub-server where the second storage slice is located, so as to instruct the second sub-server to allocate the corresponding page number to the arbitrary data node, and return the page number allocated to the arbitrary data node to the first sub-server.

[0096] The first sub-server will first detect whether the number of free page numbers recorded in the free page number list of the first storage area is greater than or equal to the number of each new data node. If so, it means that the free page numbers recorded in the free page number list of the first storage area are sufficient. At this time, a free page number in the free page number list can be allocated to each new data node. If not, it means that the free page numbers recorded in the free page number list of the first storage area are insufficient. At this time, these new data nodes can be divided into a first data node set and a second data node set according to the number of free page numbers. The number of data nodes included in the first data node set is equal to the number of free page numbers, so a free page number can be allocated to each data node included in the first data node set; the data nodes of the second data node set cannot be allocated free page numbers. At this time, for any data node in the second data node set, it can first be detected whether the minimum available page number of the first storage area has exceeded the page number upper limit of the page number range corresponding to the first storage area. If it has not exceeded, the minimum available page number can be allocated to the arbitrary data node, and then the minimum available page number can be incremented. For example, assuming that the minimum available page number stored in the first storage slice is 198 and the corresponding page number range is 100-199, then for the first data node in the second data node set, since the current minimum available page number 198 does not exceed the page number upper limit 199 of the page number range, page number 198 can be allocated to the first data node, and then the minimum available page number is incremented to 199; for the second data node in the second data node set, since the current minimum available page number 199 also does not exceed the page number upper limit 199 of the page number range, page number 199 can be allocated to the second data node, and then the minimum available page number is incremented to 200; for the third data node in the second data node set, since the current minimum available page number 200 has exceeded the page number upper limit 199 of the page number range, the minimum available page number cannot be allocated to the third data node at this time, and there is no allocable free page number in the free page number list of the first storage slice, so this means that the first storage slice has no way to allocate page numbers.

[0097] If the first storage area has no way to allocate a page number, the first sub-server can select another storage area in the server system according to a set method (for example, it can be randomly selected, or selected according to the number of the storage area in ascending order), which is represented by the second storage area. Then the data of the corresponding data node is sent to the sub-server where the second storage area is located, which is represented by the second sub-server here. For example, Figure 3In the example, if storage slice 2 cannot allocate a page number, subserver 2 can determine a new storage slice (assuming it is storage slice 3), and subserver 2 will send the data of the corresponding data node to the subserver where storage slice 3 is located, that is, subserver 3. It should be noted that the determined new storage slice may also be a storage slice provided by the first subserver. In this case, the step of sending data to another subserver is not involved. The first subserver will use the same method as the first storage slice to determine whether the new storage slice (the new storage slice also has its own minimum available page number and free page number list) can allocate a page number. After the first subserver sends the data of the corresponding data node to the second subserver where the second storage slice is located, the second subserver will use the same method to determine whether the second storage slice can allocate a page number. If the page number is successfully allocated, the second subserver will return the allocated page number result to the first subserver. If the second storage area of ​​the second sub-server also cannot allocate a page number, the second sub-server will continue to send the data of the corresponding data node to the sub-server where another storage area is located (for example, the third sub-server), and so on, until the page number is successfully allocated to the data node. The successfully allocated page number will be returned to the first sub-server through each sub-server in turn.

[0098] Furthermore, determining the second storage area may include:

[0099] (1) Obtaining a global maximum page number, where the global maximum page number is the maximum value among the upper limits of the page number ranges corresponding to the respective storage areas;

[0100] (2) Searching for a storage area whose upper limit of a page number range corresponding to each of the storage areas is equal to the global maximum page number from each of the storage areas, and using the storage area as the second storage area.

[0101] In the previous description, the main server and each sub-server will maintain a global maximum page number MaxDataID. The role of the global maximum page number will be explained here. The global maximum page number represents the maximum value of the upper limit of the page number of each page number range corresponding to all storage slices. When a storage slice cannot allocate a page number, it will forward the data of the data node to other storage slices, allowing other storage slices to allocate page numbers on its behalf. Here, it can be stipulated that the data is forwarded to the storage slice with the global maximum page number (that is, the upper limit of the page number of the corresponding page number range is equal to the global maximum page number). This is because the storage slice with the global maximum page number generally stores the least data, that is, it is most likely to allocate a page number.

[0102] like Figure 5 As shown in FIG, it is a schematic diagram of the operation of determining the forwarded storage area according to the global maximum page number. Figure 5In the example, the storage capacity of each storage slice is 10, the global maximum page number MaxDataID = 59, and each sub-server maintains the global maximum page number. When storage slice 2 in sub-server 2 cannot allocate a page number, sub-server 2 can learn through interaction with each sub-server that the storage slice with the global maximum page number 59 is storage slice 6, and storage slice 6 is distributed on sub-server 1. Therefore, sub-server 2 will forward the data of the corresponding data node to sub-server 1, and sub-server 1 will allocate the page number on its behalf through storage slice 6.

[0103] Furthermore, each of the storage slices stores a target number of data pages, each of the storage slices has been numbered from small to large according to a page number range corresponding to the data pages stored therein, each of the sub-servers has completed numbering, and each of the storage slices is distributed in ascending order of numbering to each of the sub-servers arranged in ascending order of numbering for storage; after sending the data of the arbitrary data node to the second sub-server where the second storage slice is located, the method may further include:

[0104] (1) If there is no free page number in the free page number list of the second storage area, and the minimum available page number of the second storage area exceeds the global maximum page number, the second sub-server adds the global maximum page number and the target number to obtain an updated global maximum page number;

[0105] (2) The second sub-server determines the third sub-server corresponding to the updated global maximum page number, and sends the updated global maximum page number to the third sub-server to instruct the third sub-server to create a new storage area with an upper limit of the page number of the corresponding page number range equal to the updated global maximum page number, use the new storage area to allocate a corresponding page number to the arbitrary data node, and return the page number allocated to the arbitrary data node to the second sub-server.

[0106] If the second storage slice with the global maximum page number cannot allocate a page number (that is, there is no free page number in the free page number list of the second storage slice, and the minimum available page number of the second storage slice has exceeded the global maximum page number), it means that a new storage slice needs to be created. At this time, the second sub-server where the second storage slice is located adds the global maximum page number and the target number (that is, the storage capacity of each storage slice) to obtain the updated global maximum page number. Figure 3Based on the distribution pattern of storage slices shown, the second sub-server can calculate the sub-server corresponding to the updated global maximum page number (represented by the third sub-server), and then send the data of the corresponding data node and the updated global maximum page number to the third sub-server. After receiving the updated global maximum page number, the third sub-server will find that it is larger than the original global maximum page number stored locally, indicating that a new storage slice needs to be created. Therefore, the third sub-server will create a new storage slice with an upper limit of the page number range equal to the updated global maximum page number. It will then use this new storage slice to assign the corresponding page number to the corresponding data node and return the assigned page number result to the second sub-server. The second sub-server will then return the page number result to the first sub-server.

[0107] like Figure 6 The following is a schematic diagram of the operation of updating the global maximum page number and creating a new storage area. Figure 6 In the example, there is a main server and five sub-servers (sub-server 1 to sub-server 5). The initial global maximum page number is 29, and the main server and the five sub-servers all maintain the global maximum page number. Figure 6 The image above shows the state before a storage slice is created. At this point, the system has three storage slices: Storage Slice 1 (page number range: 0-9), Storage Slice 2 (page number range: 10-19), and Storage Slice 3 (page number range: 20-29). The storage capacity of each storage slice is 10. If a page number cannot be allocated in Storage Slice 1 or Storage Slice 2, the data of the corresponding data node is forwarded to Storage Slice 3, which has the global maximum page number of 29. Figure 6 The area below shows the status of the storage slice after creation. If storage slice 3 also cannot allocate a page number, sub-server 3 will add the global maximum page number 29 and the storage capacity 10 to obtain the updated global maximum page number 39, replacing the original local global maximum page number 29. In addition, based on the distribution pattern of the page number range, it can be calculated that the updated global maximum page number 39 is located on sub-server 4. Therefore, sub-server 3 will send the data of the corresponding data node and the updated global maximum page number 39 to sub-server 4. After receiving the updated global maximum page number 39, sub-server 4 finds that it is larger than the global maximum page number 29 maintained by itself, so it will create a storage slice 4 with a page number range of 30-39 and update the locally maintained global maximum page number to 39. At this point, the automatic creation of the storage slice and the update of the global maximum page number are completed.

[0108] However, at this point, the global maximum page number maintained by sub-servers 1, 2, and 5 is still 29, which is where the global maximum page number maintained on the master server comes into play. When the master server receives the page number result allocated by storage slice 4, returned by sub-server 1 or 2, it discovers that the page number result is greater than the locally maintained global maximum page number 29. At this point, the master server is informed of the newly created storage slice and updates the locally stored global maximum page number to 39. The master server then sends the updated global maximum page number 39 to each sub-server. Upon receiving the global maximum page number 39 sent by the master server, sub-server 1, 2, and 5 will each update their locally stored global maximum page number to 39, thus achieving synchronization of the global maximum page number in the system.

[0109] For the above step (5), the first sub-server will integrate the key value, hash value and corresponding page number of each new data node to obtain the corresponding data node splitting result.

[0110] 406. The first sub-server returns the data node splitting result to the main server.

[0111] The first sub-server will reply the obtained data node splitting result to the upper layer, that is, send it to the main server, and then the main server will perform the corresponding index node splitting operation.

[0112] 407. After receiving the data node splitting result, the main server performs the index node splitting operation of the Merkle B+ tree according to the data node splitting result to complete the operation of adding the key-value pair to the distributed database.

[0113] After receiving the data node splitting result (mainly including the key, page number, and hash value of the split data node) from the first child server, the master server can perform the corresponding split operation on the index node based on the data node splitting result. After the index node split is completed, the key-value pair is added to the distributed database.

[0114] like Figure 7 As shown in FIG, it is a schematic diagram of adding a key-value pair to a distributed database according to an embodiment of the present application. Figure 7In the process of Merkle B+ tree, a batch of key-value pairs are first inserted into the root node of the Merkle B+ tree, and then the key-value pairs are distributed according to the range of the index node until the key-value pairs are distributed to the index node of the last layer (leaf layer). The above operations are completed by the main server. Next, the main server will send the key-value pairs and the page number of the corresponding data node to the remote corresponding sub-server 1, and sub-server 1 will perform the key-value pair merging and data node splitting operations. During the process of performing the data node splitting operation, if the storage area 1 corresponding to the page number of the corresponding data node cannot allocate a page number for the new data node, it can determine another storage area 2 (here, random selection, selection according to the storage area number increment, or selection of the storage area with the global maximum page number as described above) and send the data of the new data node to the sub-server 2 where the storage area 2 is located. Sub-server 2 uses storage area 2 to complete the page number allocation operation and returns the obtained page number result to sub-server 1. Next, sub-server 1 returns the data node splitting result to the main server, and then the main server performs the corresponding index node splitting operation, and finally completes the addition of the key-value pair to the distributed database.

[0115] The following example illustrates the process of adding key-value pairs to the database. Figure 8 As shown, it is a Merkle B+ tree in the initial state, which includes the root node [a1, b1, c2], index node [a1], index node [b1], index node [c2], data node [a1, a2], data node [b1] and data node [c2].

[0116] Now need to Figure 8 The Merkle B+ tree shown adds three key-value pairs [a3], [b2], and [c1]. First, as Figure 9 As shown, the key-value pair distribution and merging operations are performed. Figure 9 In [a1, b1, c2], starting from the root node, key-value pairs are distributed to the nodes below. If the distribution is to the index node, the distribution continues to the next node. If the distribution is to the data node, the key-value pair is merged. For example, Figure 9 At the far right, the inserted [a3] is merged into the data node [a1, a2], the inserted [b2] is merged into the data node [b1], and the inserted [c1] is merged into the data node [c2].

[0117] After completing the key-value pair distribution and merging operations, the following is performed: Figure 10 The node splitting operation shown (including data node splitting operation and index node splitting operation). Figure 10In the process, the data nodes at the bottom are split upwards. If the data size of a data node exceeds the threshold, the data node is split to ensure that the data of each node can be written into a data page. The new nodes obtained after the split need to be assigned corresponding page numbers (the specific page number allocation method is described above). As you can see, Figure 10 The data node [a1, a2, a3] in the table is split into two data nodes [a1, a2] and [a3]. Then, the index node [a1] is also split into [a1, a3] accordingly until it reaches the root node. It can be seen that the root node is split from [a1, b1, c2] to [a1, b1, c1], which means that the index is updated. In the embodiment of the present application, the key-value pair distribution operation and the index node splitting operation are performed by the main server, and the key-value pair merging operation and the data node splitting operation are performed by the sub-server, thereby sharing the data transmission pressure generated during the key-value pair addition operation.

[0118] In the server system architecture of an embodiment of the present application, if a distributed database data query is required, the master server receives a data query request. Assuming that the key value to be queried is the key value corresponding to the target key, the master server can determine the data node to be queried based on the target key, represented here as a designated data node. The master server can obtain the page number corresponding to the designated data node, represented here as a designated page number. Then, based on the page number indexing method for sub-servers described above, the master server can find the sub-server where the data for the designated data node is located from each connected sub-server, represented here as a fourth sub-server. Next, the master server sends a key value query request carrying the target key and the designated page number to the fourth sub-server. After receiving the key value query request, the fourth sub-server finds the corresponding storage partition based on the designated page number and the maintained storage partition information, queries the data page stored in the storage partition for the page number corresponding to the designated page number, searches for the target key value corresponding to the target key from the data page, and returns the target key value to the master server. The master server then outputs the target key value, thereby completing the data query operation.

[0119] In terms of multi-version control and data rollback of the database, since the Merkle B+ tree is equivalent to saving multiple logical trees on the disk, when the data needs to be rolled back to a certain version, you only need to point the root of the tree to the root of a saved tree. If you want to save multiple logical trees, you need to set the free page number list, the to-be-released page number list, and the maximum page ID. Assuming that the database only supports two versions of data rollback, when transaction 4 is completed, it can only roll back to the state of transaction 3 at most, which is equivalent to having only two trees of transaction 3 and transaction 4 in the database. Figure 11 The figure below is a schematic diagram of performing data rollback operation on the database. Figure 11In the example, the initial page numbers of the three nodes corresponding to transaction 1 are 1, 2, and 3 respectively. At this time, the maximum page ID is 4, and the free page number list and the to-be-released page number list are both empty. After transaction 1 is completed and transaction 2 is executed, after transaction 2 updates the key value, the nodes with page numbers 2 and 3 become dirty, and the node with page number 2 is split into two nodes. Since the free page number list and the to-be-released page number list are both empty, the page number can only be allocated to each node by increasing the maximum page ID. The allocated page numbers are 4, 5, 6, and 7. At this time, the maximum page ID becomes 8, and the recycled pages 1-3 will not be released immediately (that is, they will enter the free page number list). Table), but first put it into the to-be-released page number list; when transaction 3 starts, since only two versions of data rollback are supported, the to-be-released page number of transaction 1 in the to-be-released page number list needs to be released. At this time, page numbers 1, 2, and 3 are put into the free page number list. After the update of transaction 3 is completed, node 6 becomes dirty (correspondingly, the root node 7 will also become dirty). At this time, page numbers 1 and 2 can be taken from the free page number list for allocation, and node 6 is put into the to-be-released page number list; the final result is: the free page number list stores page number 3, and the to-be-released page number list stores page numbers 6 and 7 corresponding to transaction 2. At this time, if you need to roll back to transaction 2, since the page number of the root node of transaction 2 is page number 7, the free page number list of transaction 2, and the to-be-released page number list of transaction 2 have been recorded, you only need to point the root node to page number 7, replace the current free page number list with the free page number list corresponding to transaction 2, and replace the current to-be-released page number list with the to-be-released page number list of transaction 2. Theoretically, it can support data rollback for countless versions, but the list of page numbers to be released will be longer. This technical means can be used to implement multi-version control and data rollback of the database.

[0120] When applying multi-version control and data rollback to the server system provided by the embodiment of the present application, since the main server and each sub-server save data files or index files and can record their respective maximum page IDs, free page number lists, and to-be-released page number lists, when the data is rolled back, the main server only needs to point the tree root to the root of a saved tree and send a data rollback request (including the rolled back transaction ID) to each of the sub-servers below. After receiving the data rollback request, each sub-server below replaces the local current free page number list and to-be-released page number list with the free page number list and to-be-released page number list corresponding to the rolled back transaction ID. Here, each storage area provided by a sub-server has a corresponding free page number list and to-be-released page number list, so each storage area will perform its own list replacement operation respectively.

[0121] In addition, the server system provided in the embodiments of the present application also supports horizontal expansion of sub-servers, that is, adding connected sub-servers based on the existing sub-servers. In actual operation, it is only necessary to configure the relevant information of the sub-server to be added on the main server, such as the IP address and port number, and then update the mapping information described above, and synchronize the updated mapping information to each sub-server.

[0122] In an embodiment of the present application, a main server and at least one sub-server are set up. Considering that the data nodes of the Merkle B+ tree store key-value pairs with a large amount of data, while the index nodes only store index information with a small amount of data, the data of the data nodes are divided into each sub-server for storage, wherein each sub-server is provided with at least zero storage areas, and each storage area stores a set number of data pages for storing the data of the corresponding data node. When a key-value pair needs to be added to the database, the main server will determine the data node where the key-value pair needs to be inserted, and then send the page number and key-value pair corresponding to the data node to the sub-server where the data of the data node is located. After receiving the page number and key-value pair, the sub-server will perform the key-value pair merge and data node splitting operations on the data node, and return the obtained data node splitting results to the main server. Finally, the main server performs the index node splitting operation based on the data node splitting results, thereby completing the operation of adding the key-value pair to the database. Through this setting, when operating the database, the data transmission pressure of the main server can be distributed to each sub-server, thereby reducing the data transmission pressure brought to each server.

[0123] In summary, the embodiment of the present application distributes the data transmission pressure of the main server to each sub-server. By distributing the data of the data node to each sub-server relatively evenly, it can ensure that the data reading and writing volume of each sub-server is relatively uniform. When the index node of the main server is updated, it will distribute the key-value pairs to the lower layer, and the data nodes are scattered on each sub-server. Therefore, the data node splitting operation performed by each sub-server is concurrent, thereby making the update operation concurrency of the entire database very high. In addition, the server system provided by the embodiment of the present application also supports horizontal expansion, that is, sub-servers can be easily added.

[0124] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0125] The above mainly describes a method for operating a distributed database. The following will describe an operating device for a distributed database.

[0126] See also Figure 12In one embodiment of the present application, an operating device for a distributed database applied to a master server includes:

[0127] A key-value pair acquisition module 1201 is configured to acquire key-value pairs to be added to a distributed database; wherein the distributed database is based on a Merkle B+ tree, wherein the master server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, and the total number of storage slices provided by all sub-servers is at least one, and each storage slice stores a set number of data pages, each of which is used to store data of the data nodes of the Merkle B+ tree;

[0128] The data node determination module 1202 is configured to determine, based on the key of the key-value pair, a target data node in the Merkle B+ tree into which the key-value pair is to be inserted;

[0129] The sub-server determining module 1203 is configured to determine, from the at least one sub-server, a first sub-server where the data of the target data node is located, based on a target page number corresponding to the target data node; wherein the target page number is the number of a target data page, the target data page stores the data of the target data node, and the first storage area of ​​the first sub-server stores the target data page;

[0130] The key-value pair sending module 1204 is configured to send the key-value pair and the target page number to the first sub-server, instructing the first sub-server to perform a key-value pair merge and a data node splitting operation on the target data node based on the key-value pair and the target page number, obtain a data node splitting result, and return the data node splitting result to the master server.

[0131] The index node splitting module 1205 is used to perform the index node splitting operation of the Merkel B+ tree according to the data node splitting result after receiving the data node splitting result, so as to complete the operation of adding the key-value pair to the distributed database.

[0132] In one implementation of the embodiment of the present application, each of the storage slices stores a target number of data pages, each of the storage slices has been numbered from small to large according to the page number range corresponding to the data pages stored therein, each of the sub-servers has completed numbering, and each of the storage slices is distributed in ascending order of numbering to each of the sub-servers arranged in ascending order of numbering for storage; the sub-server determination module may include:

[0133] a first modulo operation unit, configured to perform a modulo operation on the target page number and the target quantity to obtain a first value;

[0134] a second modulo operation unit, configured to perform a modulo operation on the first value and the number of each of the sub-servers to obtain a second value;

[0135] The sub-server determining unit is configured to determine the sub-server numbered with the second value among the sub-servers as the first sub-server.

[0136] Furthermore, the sub-server determination module may further include:

[0137] A mapping information acquisition unit, configured to acquire preset mapping information, wherein the mapping information records the correspondence between each preset page number range and the number of servers;

[0138] A page number range determining unit, configured to determine a target page number range in which the target page number is located within each of the preset page number ranges;

[0139] The server quantity searching unit is configured to search the mapping information for the number of servers corresponding to the target page number range as the number of each sub-server.

[0140] See also Figure 13 In an embodiment of the present application, an operating device for a distributed database applied to a first sub-server includes:

[0141] The key-value pair receiving module 1301 is configured to receive a target page number and a key-value pair to be added to a distributed database sent by a master server; wherein the distributed database is based on a Merkle B+ tree, the master server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of storage slices provided by all the sub-servers is at least 1, each storage slice stores a set number of data pages, the data pages are used to store data of data nodes of the Merkle B+ tree, the first sub-server is the sub-server where the data of the target data node determined by the master server from the at least one sub-server is located, the target page number is the number of the target data page, the target data page stores the data of the target data node, the first storage slice of the first sub-server stores the target data page, the target data node is the data node in the Merkle B+ tree determined by the master server according to the key of the key-value pair, and the key-value pair needs to be inserted;

[0142] A data node splitting module 1302 is configured to perform a key-value pair merging and a data node splitting operation on the target data node according to the key-value pair and the target page number, to obtain a data node splitting result;

[0143] The split result returning module 1303 is used to return the data node splitting result to the main server to instruct the main server to perform the index node splitting operation of the Merkel B+ tree according to the data node splitting result after receiving the data node splitting result, so as to complete the operation of adding the key-value pair to the distributed database.

[0144] In one implementation of the embodiment of the present application, the data node splitting module may include:

[0145] a storage slice determining unit, configured to determine the first storage slice from all storage slices of the first sub-server based on the target page number and storage slice information of the first sub-server; wherein the storage slice information records a page number range corresponding to each storage slice provided to the first sub-server;

[0146] a key-value pair merging unit, configured to distribute the key-value pair to the target data node stored in the first storage area, and perform a key-value pair merging operation on the target data node;

[0147] a data node splitting unit, configured to split the target data node into at least two new data nodes if the amount of data on the target data node after the key-value pair merge operation exceeds a set threshold, so that data on each of the new data nodes can be written into one of the data pages;

[0148] A page number allocation unit is configured to allocate a corresponding page number to each of the new data nodes, and write the data of each of the new data nodes into the data page corresponding to the respective page number;

[0149] The split result determination unit is used to determine the data node split result according to the key value, hash value and corresponding page number of each new data node.

[0150] Furthermore, the first sub-server records a minimum available page number and a list of free page numbers for each storage slice; wherein the minimum available page number is equal to the maximum page number currently allocated to the data node in the corresponding storage slice plus one, and the list of free page numbers records the free page numbers of the corresponding storage slice that can be currently allocated to the data node; the page number allocation unit may include:

[0151] an idle page number detection subunit, configured to detect whether the number of idle page numbers recorded in the idle page number list of the first storage area is greater than or equal to the number of each of the new data nodes;

[0152] a first page number allocation subunit, configured to allocate one free page number to each of the new data nodes if the number of free page numbers recorded in the free page number list of the first storage slice is greater than or equal to the number of the new data nodes;

[0153] The second page number allocation subunit is configured to, if the number of free page numbers recorded in the free page number list of the first storage slice is less than the number of each of the new data nodes, divide each of the new data nodes into a first data node set and a second data node set; wherein the number of data nodes included in the first data node set is equal to the number of free page numbers; allocate one free page number to each data node included in the first data node set; for any data node included in the second data node set, detect whether the minimum available page number of the first storage slice has exceeded the upper limit of the page number range corresponding to the first storage slice; if the minimum available page number of the first storage slice does not exceed the upper limit of the page number, allocate the minimum available page number of the first storage slice to the arbitrary data node and increment the minimum available page number of the first storage slice; if the minimum available page number of the first storage slice has exceeded the upper limit of the page number, determine a second storage slice, and send the data of the arbitrary data node to the second sub-server where the second storage slice is located, so as to instruct the second sub-server to allocate the corresponding page number to the arbitrary data node, and return the page number allocated to the arbitrary data node to the first sub-server.

[0154] Furthermore, the second page number allocation subunit may include:

[0155] A global maximum page number acquisition subunit is configured to acquire a global maximum page number, where the global maximum page number is a maximum value among the upper limits of the page number ranges corresponding to the respective storage areas;

[0156] The storage slice determining subunit is configured to search, from each of the storage slices, for a storage slice whose upper page number limit of the corresponding page number range is equal to the global maximum page number, as the second storage slice.

[0157] Furthermore, each of the storage slices stores a target number of data pages, each of the storage slices has been numbered from small to large according to the page number range corresponding to the data pages stored therein, each of the sub-servers has completed numbering, and each of the storage slices is distributed in ascending order of numbering to each of the sub-servers arranged in ascending order of numbering for storage; the second page number allocation sub-unit may further include:

[0158] a global maximum page number updating subunit, configured to, if there is no free page number in the free page number list of the second storage slice and the minimum available page number of the second storage slice exceeds the global maximum page number, add the global maximum page number and the target number to obtain an updated global maximum page number;

[0159] A storage slice creation sub-unit is used to determine the third sub-server corresponding to the updated global maximum page number, and send the updated global maximum page number to the third sub-server to instruct the third sub-server to create a new storage slice with an upper limit of the page number of the corresponding page number range equal to the updated global maximum page number, use the new storage slice to allocate the corresponding page number to the arbitrary data node, and return the page number allocated to the arbitrary data node to the second sub-server.

[0160] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed database operation method described in any of the above embodiments is implemented.

[0161] An embodiment of the present application further provides a computer program product, which, when executed on a server, enables the server to execute the distributed database operation method described in any one of the above embodiments.

[0162] Figure 14 Schematic diagram of a server provided by an embodiment of the present application. Figure 14 As shown, the server 14 of this embodiment includes: a processor 140, a memory 141, and a computer program 142 stored in the memory 141 and executable on the processor 140. When the processor 140 executes the computer program 142, the steps in the above-mentioned embodiments of the operating method of each distributed database are implemented, such as Figure 4 Alternatively, when the processor 140 executes the computer program 142, the functions of the modules / units in the above-mentioned device embodiments are realized, for example Figure 12 The functions of the modules 1201 to 1205 shown, or Figure 13 Functions of modules 1301 to 1303 are shown.

[0163] The computer program 142 may be divided into one or more modules / units, which are stored in the memory 141 and executed by the processor 140 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 142 in the server 14.

[0164] The processor 140 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0165] The memory 141 may be an internal storage unit of the server 14, such as a hard disk or memory of the server 14. The memory 141 may also be an external storage device of the server 14, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the server 14. Furthermore, the memory 141 may include both an internal storage unit of the server 14 and an external storage device. The memory 141 is used to store the computer program and other programs and data required by the server. The memory 141 may also be used to temporarily store data that has been output or is about to be output.

[0166] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0167] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0171] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0173] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A distributed database operation method, applied to a master server, characterized in that: The operation method includes: Obtaining a key-value pair to be added to a distributed database; wherein the distributed database is based on a Merkle B+ tree, the main server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of storage slices provided by all sub-servers is at least one, each storage slice stores a set number of data pages, and the data pages are used to store data of the data nodes of the Merkle B+ tree; Determine, according to the key of the key-value pair, a target data node in the Merkle B+ tree into which the key-value pair is to be inserted; Determining, from the at least one sub-server, a first sub-server where data of the target data node is located, based on a target page number corresponding to the target data node; wherein the target page number is a number of a target data page, the target data page stores the data of the target data node, and the first storage area of ​​the first sub-server stores the target data page; Sending the key-value pair and the target page number to the first sub-server to instruct the first sub-server to perform a key-value pair merge and a data node splitting operation on the target data node according to the key-value pair and the target page number, obtaining a data node splitting result, and returning the data node splitting result to the main server; After receiving the data node splitting result, the index node splitting operation of the Merkle B+ tree is performed according to the data node splitting result to complete the operation of adding the key-value pair to the distributed database.

2. The operating method according to claim 1, wherein: Each storage slice stores a target number of data pages, each storage slice has been numbered from small to large according to the page number range corresponding to the data pages stored therein, each sub-server has been numbered, and each storage slice is distributed in ascending order to each sub-server arranged in ascending order for storage; The determining, from the at least one sub-server according to the target page number corresponding to the target data node, a first sub-server where data of the target data node is located includes: performing a modulo operation on the target page number and the target quantity to obtain a first value; Performing a modulo operation on the first value and the number of each of the sub-servers to obtain a second value; The sub-server numbered with the second value among the sub-servers is determined as the first sub-server.

3. The operating method according to claim 2, wherein: Before performing a modulo operation on the first value and the number of each sub-server to obtain a second value, the method further includes: Obtaining preset mapping information, wherein the mapping information records the correspondence between each preset page number range and the number of servers; Determine the target page number range in which the target page number is located within each of the preset page number ranges; The number of servers corresponding to the target page number range is searched from the mapping information as the number of each sub-server.

4. A distributed database operation method, applied to a first sub-server, characterized in that: The operation method includes: Receive a target page number and a key-value pair to be added to a distributed database sent by a main server; wherein the distributed database is based on a Merkle B+ tree, the main server is connected to at least one sub-server, each sub-server is provided with at least zero storage slices, the total number of the storage slices provided by all the sub-servers is at least 1, each storage slice stores a set number of data pages, the data pages are used to store data of data nodes of the Merkle B+ tree, the first sub-server is the sub-server where the data of the target data node determined by the main server from the at least one sub-server is located, the target page number is the number of the target data page, the target data page stores the data of the target data node, the first storage slice of the first sub-server stores the target data page, the target data node is the data node in the Merkle B+ tree determined by the main server according to the key of the key-value pair, and the key-value pair needs to be inserted; Performing key-value pair merging and data node splitting operations on the target data node according to the key-value pair and the target page number to obtain a data node splitting result; 5. Return the data node splitting result to the master server to instruct the master server to perform the index node splitting operation of the Merkel B+ tree according to the data node splitting result after receiving the data node splitting result, so as to complete the operation of adding the key-value pair to the distributed database.

5. The operating method according to claim 4, characterized in that: According to the key-value pair and the target page number, Performing a key-value pair merge and a data node splitting operation on the target data node to obtain a data node splitting result, including: determining the first storage slice from all storage slices of the first sub-server according to the target page number and storage slice information of the first sub-server; wherein the storage slice information records a page number range corresponding to each storage slice provided to the first sub-server; Distributing the key-value pairs to the target data node stored in the first storage area, and performing a key-value pair merging operation on the target data node; 5. If the amount of data in the target data node after the key-value pair merge operation exceeds a set threshold, split the target data node into at least two new data nodes, so that the data of each new data node can be written into one data page; Assigning a corresponding page number to each of the new data nodes, and writing the data of each of the new data nodes into the data page corresponding to the respective page number; 0 Determine the data node splitting result according to the key value, hash value and corresponding page number of each new data node.

6. The operating method according to claim 5, characterized in that: The first sub-server records the minimum available page number and a list of free page numbers for each storage slice; wherein the minimum available page number is equal to the maximum page number currently allocated to the data node of the corresponding storage slice plus one, and the free page number list records the free page numbers of the corresponding storage slice that can be currently allocated to the data node; and the allocating a corresponding page number to each new data node includes: Detecting whether the number of free page numbers recorded in the free page number list of the first storage area is greater than or equal to the number of each of the new data nodes; If the number of free page numbers recorded in the free page number list of the first storage area is greater than or equal to the number of each of the new data nodes, allocate one free page number to each of the new data nodes; If the number of free page numbers recorded in the free page number list of the first storage slice is less than the number of each of the new data nodes, each of the new data nodes is divided into a first data node set and a second data node set; wherein the number of data nodes included in the first data node set is equal to the number of free page numbers; one free page number is allocated to each data node included in the first data node set; for any data node included in the second data node set, whether the minimum available page number of the first storage slice has exceeded the page number upper limit of the page number range corresponding to the first storage slice is detected; if the minimum available page number of the first storage slice does not exceed the page number upper limit, the minimum available page number of the first storage slice is allocated to the arbitrary data node, and the minimum available page number of the first storage slice is incremented; if the minimum available page number of the first storage slice has exceeded the page number upper limit, a second storage slice is determined, and the data of the arbitrary data node is sent to the second sub-server where the second storage slice is located, so as to instruct the second sub-server to allocate the corresponding page number to the arbitrary data node, and return the page number allocated to the arbitrary data node to the first sub-server.

7. The operating method according to claim 6, characterized in that: The determining of the second storage area includes: Obtaining a global maximum page number, where the global maximum page number is the maximum value among the upper limits of the page number ranges corresponding to the storage areas; A storage slice having a corresponding page number range with an upper limit of the page number equal to the global maximum page number is searched from each of the storage slices to serve as the second storage slice.

8. The operating method according to claim 7, wherein: Each storage slice stores a target number of data pages, each storage slice has been numbered from small to large according to the page number range corresponding to the data pages stored therein, each sub-server has been numbered, and each storage slice is distributed in ascending order to each sub-server arranged in ascending order for storage; After sending the data of the arbitrary data node to the second sub-server where the second storage area is located, the method further includes: If there is no free page number in the free page number list of the second storage area, and the minimum available page number of the second storage area exceeds the global maximum page number, the second sub-server adds the global maximum page number and the target number to obtain an updated global maximum page number; The second sub-server determines the third sub-server corresponding to the updated global maximum page number, and sends the updated global maximum page number to the third sub-server to instruct the third sub-server to create a new storage area with an upper limit of the page number of the corresponding page number range equal to the updated global maximum page number, use the new storage area to allocate the corresponding page number to the arbitrary data node, and return the page number allocated to the arbitrary data node to the second sub-server.

9. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the processor implements the operating method according to any one of claims 1 to 3, or implements the operating method according to any one of claims 4 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the operating method according to any one of claims 1 to 3, or implements the operating method according to any one of claims 4 to 8.

Images