Method and system for verifying data consistency of KV embedded database based on B + tree

Abstract

The invention is suitable for the technical field of databases, and discloses a B + tree-based KV embedded database data consistency verification method, system and device and a medium, the method comprises the following steps: obtaining change page information in a mainboard, the change page information comprising a hash value of a page; based on a hierarchical structure of a B + tree in the mainboard, sorting the change page information from top to bottom to obtain a change page list; the change page lists are synchronized to a standby board database in sequence; and comparing the hash value of the root page in the changed page list with the hash value of the root page in the standby board, and determining the consistency of the data of the main board and the standby board. According to the calculation scheme, the data transmission efficiency is high, the resource consumption is small, and the data verification speed is high.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a KV embedded database data consistency verification method, system, device and medium based on a B+ tree. Background Art

[0002] Existing embedded database consistency verification methods mainly include file replication and log replication:

[0003] File copy (such as Figure 1 As shown in the figure): The main database file is copied to the backup node regularly through the internal network (such as FTP), and the consistency of the main and backup files is ensured by verifying the file hash value. However, this method has poor real-time performance and puts a lot of pressure on the file system.

[0004] Log replication (such as Figure 2 As shown in Figure 1-1, the database operation records on the primary node are synchronized to the backup node for execution to ensure that the operations of the primary and backup nodes are completely consistent. This method requires the same CPU and disk resources as the primary node, which consumes a lot of resources and has a high demand for communication bandwidth. Summary of the invention

[0005] To solve the above problems, the present invention provides a KV embedded database data consistency verification method, system, device and medium based on B+ tree, which has high data transmission efficiency, low resource consumption and fast data consistency verification speed.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a KV embedded database data consistency verification method based on a B+ tree, comprising:

[0008] Get the changed page information in the main board, including the hash value of the page;

[0009] Based on the hierarchical structure of the B+ tree in the main board, sort the changed page information from top to bottom to obtain a changed page list;

[0010] Synchronize the changed page list to the standby board database in sequence;

[0011] Compare the hash value of the root page in the changed page list with the hash value of the root page in the slave board to determine the consistency of the data on the master and slave boards.

[0012] Furthermore, the calculation of the hash value of the page includes:

[0013] Hash calculation is performed from the leaf page to the root page from the bottom up, where the hash value of the leaf page is calculated based on the stored content; the hash value of the branch page or root page is calculated based on the hash value of its child page.

[0014] Furthermore, the list of changed pages is synchronized to the standby board database in sequence, including:

[0015] Based on the changed page information in the changed page list, a page change message of the primary and backup backup is established;

[0016] Based on the page change messages of the primary and backup backups, corresponding synchronization processing is performed.

[0017] Furthermore, establishing the page change message of the primary and backup backup also includes:

[0018] The page change messages are merged and optimized according to the page change type of the changed page information.

[0019] Furthermore, based on the page change message of the primary and backup, corresponding synchronization processing is performed, including:

[0020] The mainboard sends page change messages in sequence;

[0021] The backup board receives the page change message and performs synchronization processing according to the type of the page change message. If the type of the page change message is page addition, a new page is created in the backup board and a node association relationship in the B+ tree is established; if the type of the page change message is page modification, the page change information described in the page change message is updated to the corresponding page in the B+ tree of the backup board; if the type of the page change message is page deletion, the node association relationship of the corresponding page in the B+ tree of the backup board is released, and the page is released.

[0022] Furthermore, based on the page change message of the primary and backup, corresponding synchronization processing is performed, which also includes:

[0023] If the type of page change message received by the standby board is page addition or page change, it calculates the hash value of the corresponding changed page and verifies it with the hash value carried in the page change message. If the verification results are inconsistent, the standby board adds an identifier to the confirmation message of the main board to trigger the retransmission of the page change message.

[0024] Furthermore, the hash value of the root page in the changed page list is compared with the hash value of the root page in the backup board to determine the consistency of the primary and backup boards, wherein the compared hash value of the root page in the changed page list and the hash value of the root page in the backup board are both obtained based on the hash of the incrementally changed page. If the comparison and verification results of the two are consistent, the primary and backup boards are consistent as a whole.

[0025] In a second aspect, the present invention further provides a KV embedded database data consistency verification method system based on B+ tree, comprising:

[0026] The acquisition module is used to obtain the changed page information in the main board, and the changed page information includes the hash value of the page;

[0027] The sorting module is used to sort the changed page information from top to bottom based on the hierarchical structure of the B+ tree in the main board to obtain a list of changed pages;

[0028] The synchronization module is used to synchronize the list of changed pages to the standby board database in sequence;

[0029] The verification module is used to compare the hash value of the root page in the changed page list with the hash value of the root page in the standby board to determine the consistency of the data on the primary and standby boards.

[0030] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory;

[0031] The processor is coupled to the memory;

[0032] The processor is used to read and execute the program or instructions stored in the memory, so that the device executes the method of the first aspect.

[0033] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, which implements the method of the first aspect when executed by a processor.

[0034] The technical solution provided by the present invention has at least the following technical effects or advantages:

[0035] (1) For the KV embedded database stored in the B+ tree structure, a page-based backup solution is adopted, which has high transmission efficiency and consumes less CPU and disk resources;

[0036] (2) When backing up in pages, an efficient incremental and page hash calculation method is used to quickly verify data consistency.

[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of file replication of an embedded database consistency verification method in the background technology of the present invention;

[0040] Figure 2 A log replication schematic diagram of an embedded database consistency verification method in the background technology of the present invention;

[0041] Figure 3 Schematic diagram of the page structure of a KV embedded database based on a B+ tree in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a flow chart of a KV embedded database data consistency verification method based on a B+ tree according to an embodiment of the present invention;

[0043] Figure 5 A timing diagram of a KV embedded database data consistency verification method based on a B+ tree according to an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of primary and backup data synchronization in an embodiment of the present invention;

[0045] Figure 7 The figure is a flow chart of synchronizing the changed pages to the standby board in sequence according to the embodiment of the present invention.

[0046] Figure 8 It is a schematic diagram of the process of page retransmission when the changed pages are synchronized to the standby board in sequence in an embodiment of the present invention;

[0047] Fig. 9 A schematic diagram of a page hash value that needs to be calculated in a KV embedded database based on a B+ tree in an embodiment of the present invention;

[0048] Fig.10 It is a schematic diagram of the incremental page hash value that needs to be recalculated in the KV embedded database based on the B+ tree in an embodiment of the present invention.

[0049] Fig.11 The figure is a schematic diagram of the process of calculating the hash value of the root page in an embodiment of the present invention.

[0050] Fig.12 It is a structural schematic diagram of a KV embedded database data consistency verification method system based on B+ tree in an embodiment of the present invention;

[0051] Fig.13 The present invention is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Common KV embedded databases such as Berkeley DB and LMDB use B+ trees to store data. The B+ tree is organized in pages, and each page is a node in the B+ tree. These pages constitute the storage structure of the B+ tree. Figure 3 As shown, the nodes of the B+ tree are composed of pages and have a hierarchical structure, including root pages, branch pages, and leaf pages. The root page is located at the top level of the B+ tree and is the entry point of the entire tree. It can be a leaf page or a branch page. The branch page does not store actual data records. Its main function is indexing and navigation. It stores pointers to its child pages (leaf pages), that is, it contains multiple key-value pairs, each of which points to a child page. The leaf page is located at the bottom level of the B+ tree and stores actual data records.

[0054] The technical solution of the present invention provides a KV embedded database data consistency verification method based on B+ tree, in which the main board represents the main database, the standby board represents the standby database, and backing up the database in the main database to the standby database is the main-standby backup, and incremental data replication in units of pages is performed during the main-standby backup to synchronize data from the main board to the standby board, and incremental data replication is performed in units of pages during the main-standby backup process, and data retransmission is triggered based on the consistency verification of hash values ​​in units of pages; after the data is synchronized to the standby board, the hash value of the root page after the incremental change of the standby board data is compared with the hash value of the root page in the main board to determine the consistency of the main-standby board data.

[0055] Figure 4 A schematic diagram of a flow chart of a KV embedded database data consistency verification method based on a B+ tree according to an embodiment of the present invention;

[0056] Figure 5 A timing diagram of a KV embedded database data consistency verification method based on a B+ tree according to an embodiment of the present invention;

[0057] Combine the following Figure 4 , Figure 5 The technical solution of the present invention is described. As shown in the figure, the method comprises:

[0058] S101, obtaining the changed page information in the mainboard, wherein the changed page information includes the hash value of the page;

[0059] The database transaction on the mainboard involves operations such as adding, changing, and deleting data, which in turn will affect the page changes of the B+ tree in the mainboard. The page changes caused by data operations can be called incremental changes of the page. The incrementally changed page is the incremental data. The primary and backup backups of the incremental data in this solution are the primary and backup backups of the incrementally changed pages.

[0060] Track page changes during database transactions on the mainboard, and then obtain the changed page information. The page change type corresponds to the data operation type, including: page addition, page change, page deletion, etc. The changed pages of the B+ tree in the mainboard include: root page, branch page, and leaf page. The changed page information includes the changed page, the hash value of the page, and the type of page change.

[0061] Hash value calculation method: Start from the leaf page and perform hash calculation from bottom to top to the root page.

[0062] The hash value of the leaf page is calculated based on the stored content. The calculation formula is:

[0063]

[0064] The hash value of a branch page or root page is calculated based on the hash value of its child pages. The calculation formula is:

[0065]

[0066] Among them, PageType: the page identifier, such as 0x01, one byte, indicating that this is a leaf page; PageNumber: page number, the unique identifier of the page; NumberOfEntries: the number of entries in the page; Key and Value: the key-value pairs stored in the page; ChildHash: the hash value of the child page pointed to; ||: indicates a byte string concatenation operation;

[0067] Hash algorithms include: SHA-256, BLAKE2, RIPEMD-160, etc. You can select at least one of the hash algorithms based on different security and performance requirements.

[0068] S102, based on the hierarchical structure of the B+ tree in the main board, sort the changed page information from top to bottom to obtain a changed page list;

[0069] Based on the hierarchical structure of the B+ tree in the main board, the changed page information is sorted from top to bottom to obtain a changed page list. In the process of primary and secondary backup in page units, the top-level root page is synchronized first, and the branch page and leaf page are synchronized later, so as to build the changed B+ tree on the secondary board.

[0070] Figure 6 FIG. 1 is a schematic diagram of primary and secondary backup data synchronization in an embodiment of the present invention. Figure 6 As shown, in the database transaction on the main board, leaf page 4 and leaf page 6 of the B+ tree are tracked as change pages, the corresponding branch page 2 and branch page 3 are also change pages, and the root page is also a change page. According to the hierarchical relationship of these change pages in the B+ tree, the corresponding change page information is sorted from top to bottom to obtain a change page list.

[0071] S103, synchronizing the changed page list to the standby board database in sequence;

[0072] Based on the changed page information in the changed page list, a page change message of the primary and backup is established. The page change message is a description of the changed page information and is used for communication interaction between the main board and the backup board. The type of the page change message corresponds to the page change type of the changed page information. The page change message also carries summary information of the database transaction, including the number of pages added, changed, and deleted. When establishing the page change message, the page change message is optimized according to the page change type of the changed page information to reduce communication interaction and thus improve backup efficiency.

[0073] Optimize the page change message according to the page change type of the changed page information, including:

[0074] (1) Merge all page deletion messages in the transaction into one message;

[0075] (2) Merge messages about new page additions and page changes.

[0076] Based on the page change messages of the primary and backup servers, corresponding synchronization processing is performed, including:

[0077] The mainboard sends page change messages in sequence;

[0078] The standby board receives the page change message and performs synchronization processing according to the type of the page change message;

[0079] If the type of the page change message is page addition, a new page is created in the standby board and a node association relationship in the B+ tree is established;

[0080] If the type of the page change message is page change, the page change information described in the page change message is updated to the corresponding page in the B+ tree of the standby board;

[0081] If the type of the page change message is page deletion, the node association relationship of the corresponding page in the B+ tree of the standby board is released to release the page.

[0082] Based on the page change messages of the primary and backup backups, corresponding synchronization processing is performed, which also includes:

[0083] If the type of page change message received by the standby board is page addition or page change, it calculates the hash value of the corresponding changed page and verifies it with the hash value carried in the page change message. If the verification results are inconsistent, the standby board adds an identifier to the confirmation message of the main board to trigger the retransmission of the page change message.

[0084] Figure 7 The figure is a flow chart of synchronizing the changed pages to the standby board in sequence according to the embodiment of the present invention.

[0085] Figure 8 The figure is a schematic diagram of the process of page retransmission when the changed pages are synchronized to the standby board in sequence in an embodiment of the present invention.

[0086] S104: Compare the hash value of the root page in the changed page list with the hash value of the root page in the standby board to determine the consistency of the data on the primary and standby boards.

[0087] The hash value of the root page can reflect the changes of the entire database. By comparing the values, we can determine whether the master and slave board databases are consistent as a whole. By quickly calculating in increments, we can obtain the hash value of the root page in the change page list and the hash value of the root page in the slave board, and compare the hash values ​​of the incremental root pages to improve the efficiency of comparison and verification.

[0088] After completing the synchronization process of the primary and backup boards, recalculate the hash value of the pages in the backup board's B+ tree corresponding to the changed pages, that is, recalculate the hash value of the incrementally changed pages in the backup board, and quickly determine the hash value of the root page in the backup board's B+ tree based on the recalculated hash value of the incrementally changed pages. The hash value calculation method is the same as in step S101. Compare the hash value of the root page in the backup board with the hash value of the root page included in the list of changed pages to determine the consistency of the primary and backup board data. Fig. 9 , Fig.10As shown, hashr1 is the hash value of the root page, hashb1-hashb3 are the hash values ​​of the branch pages, and hash1-hash6 are the hash values ​​of the leaf pages; the hash values ​​of the pages involving the recalculated incremental changes are hashb2, hashb3, hash4, hash6, and hashr1. The recalculated hash value hashr1 of the standby board root page is compared with the hash value of the root page in the main board's change page list to determine the consistency of the master and standby board data. Fig.11 The figure is a schematic diagram of the process of calculating the hash value of the root page in an embodiment of the present invention.

[0089] The technical solution in the embodiments of the present invention has at least the following technical effects or advantages:

[0090] (1) For the KV embedded database stored in the B+ tree structure, a page-based backup solution is adopted, which has high transmission efficiency and consumes less CPU and disk resources;

[0091] (2) When backing up in pages, an efficient incremental and page hash calculation method is used to quickly verify data consistency.

[0092] Fig.12 A KV embedded database data consistency verification method system based on B+ tree in an embodiment of the present invention is shown in the figure. The system includes:

[0093] The acquisition module is used to obtain the changed page information in the main board, and the changed page information includes the hash value of the page;

[0094] The sorting module is used to sort the changed page information from top to bottom based on the hierarchical structure of the B+ tree in the main board to obtain a list of changed pages;

[0095] The synchronization module is used to synchronize the list of changed pages to the standby board database in sequence;

[0096] The verification module is used to compare the hash value of the root page in the changed page list with the hash value of the root page in the standby board to determine the consistency of the data on the primary and standby boards.

[0097] It should be noted that, for the sake of convenience, Fig.12 For example, only the main modules of the KV embedded database data consistency verification system structure based on B+ tree are shown. In practical applications, the system may also include modules or components not shown in the figure; the system is not limited to the above module structure, but can also be other module structures that implement the above method embodiments.

[0098] Fig.13This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. As shown in the figure, the electronic device includes: a processor and a memory;

[0099] The processor is used to read and execute programs and instructions stored in the memory, so that the electronic device executes the above method embodiment.

[0100] It should be noted that, for the sake of convenience, Fig.13 For example, only the main components of the electronic device are shown. In actual applications, the electronic device may also include components or assemblies not shown in the figure.

[0101] The present invention also provides a computer-readable storage medium storing a program or instruction. When a computer reads and executes the program or instruction, the computer executes the above method embodiment.

[0102] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A KV embedded database data consistency verification method based on B+ tree, characterized in that: include: Obtaining the changed page information in the main board, wherein the changed page information includes the hash value of the page; Based on the hierarchical structure of the B+ tree in the main board, sort the changed page information from top to bottom to obtain a changed page list; Synchronize the changed page list to the standby board database in sequence; Compare the hash value of the root page in the changed page list with the hash value of the root page in the slave board to determine the consistency of the data on the master and slave boards.

2. According to the B+ tree-based KV embedded database data consistency verification method of claim 1, it is characterized in that: The calculation of the hash value of the page includes: Hash calculation is performed from the leaf page to the root page from the bottom up, where the hash value of the leaf page is calculated based on the stored content; the hash value of the branch page or root page is calculated based on the hash value of its child page.

3. The KV embedded database data consistency verification method based on B+ tree according to claim 1 is characterized in that: The step of sequentially synchronizing the list of changed pages to the standby board database includes: Based on the changed page information in the changed page list, a page change message of the primary and backup backup is established; Based on the page change messages of the primary and backup backups, corresponding synchronization processing is performed.

4. The KV embedded database data consistency verification method based on B+ tree according to claim 3 is characterized in that: The page change message of establishing the primary and backup backup also includes: The page change messages are merged and optimized according to the page change type of the changed page information.

5. The KV embedded database data consistency verification method based on B+ tree according to claim 3 is characterized in that: The page change message based on the primary and backup backup is used to perform corresponding synchronization processing, including: The mainboard sends page change messages in sequence; The backup board receives the page change message and performs synchronization processing according to the type of the page change message. If the type of the page change message is page addition, a new page is created in the backup board and a node association relationship in the B+ tree is established; if the type of the page change message is page modification, the page change information described in the page change message is updated to the corresponding page in the B+ tree of the backup board; if the type of the page change message is page deletion, the node association relationship of the corresponding page in the B+ tree of the backup board is released, and the page is released.

6. The KV embedded database data consistency verification method based on B+ tree according to claim 3 is characterized in that: The performing of corresponding synchronization processing based on the page change message of the primary and backup backups also includes: If the type of page change message received by the standby board is page addition or page change, it calculates the hash value of the corresponding changed page and verifies it with the hash value carried in the page change message. If the verification results are inconsistent, the standby board adds an identifier to the confirmation message of the main board to trigger the retransmission of the page change message.

7. The KV embedded database data consistency verification method based on B+ tree according to claim 1 is characterized in that: The hash value of the root page in the changed page list is compared with the hash value of the root page in the backup board to determine the consistency of the main and backup boards, wherein the hash value of the root page in the changed page list and the hash value of the root page in the backup board are both obtained based on the hash of the incrementally changed page. If the comparison and verification results of the two are consistent, the main and backup boards are consistent as a whole.

8. A KV embedded database data consistency verification method system based on B+ tree, characterized in that: include: An acquisition module, used to acquire the changed page information in the main board, wherein the changed page information includes the hash value of the page; The sorting module is used to sort the changed page information from top to bottom based on the hierarchical structure of the B+ tree in the main board to obtain a list of changed pages; The synchronization module is used to synchronize the list of changed pages to the standby board database in sequence; The verification module is used to compare the hash value of the root page in the changed page list with the hash value of the root page in the standby board to determine the consistency of the data on the primary and standby boards.

9. An electronic device, characterized in that: include: Processor and memory; The processor is coupled to a memory; The processor is used to read and execute the program or instruction stored in the memory, so that the device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images