Database replication method and device, equipment and storage medium

Abstract

The invention discloses a database replication method and device, equipment and a storage medium, and relates to the field of distributed systems.The method comprises the steps that a database write-in request of a target database is obtained through a client API, nested key values are generated based on the database write-in request, and all the nested key values are combined to determine a target transaction; determining a main node and auxiliary nodes based on a distributed consistency algorithm, packaging a target transaction as an initial log entry by using the main node, synchronizing the initial log entry to the auxiliary nodes, and obtaining synchronization confirmation of each auxiliary node; wherein the synchronization confirmation is information fed back after the secondary node completes synchronization of the initial log entry; and when the number of the auxiliary nodes feeding back the synchronization confirmation is greater than a preset threshold value, submitting and marking the initial log entry to obtain a marked log entry, executing a database write-in operation in the marked log entry by using a state machine, updating a corresponding target database, and returning an operation execution result to the user side. And the consistency between the data copies is ensured.

Description

Technical Field

[0001] This invention relates to the field of distributed systems, and in particular to a database replication method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of technologies such as cloud computing, big data, and artificial intelligence, data storage and management are facing unprecedented challenges. Traditional monolithic databases cannot meet the high concurrency, high availability, and elastic scaling requirements of modern applications; therefore, distributed databases have become an inevitable choice.

[0003] Currently, various consensus protocols have been proposed in academia and industry, with Paxos and Raft being the two most representative consensus algorithms. While Paxos theoretically possesses completeness and fault tolerance, its implementation complexity is high, making it difficult to widely apply in practical systems. Compared to Paxos, Raft, with its simpler, easier-to-understand, and easier-to-implement characteristics, has become the consensus protocol commonly adopted in industry. Raft ensures data consistency across multiple replicas in a distributed system through election mechanisms, log replication, and state machine consistency. Many distributed storage systems and databases, such as etcd, HashiCorpConsul, and TiKV, have already applied the Raft protocol to their underlying replication mechanisms. These systems primarily rely on the strong consistency guarantees provided by the Raft protocol, especially when handling cross-node write operations. The Raft protocol ensures data consistency on each node through log replication and guarantees high system availability through automatic leader election.

[0004] In conclusion, ensuring consistency among data replicas is a pressing issue that needs to be addressed. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a database replication method, apparatus, device, and storage medium that can ensure consistency between data copies. The specific solution is as follows:

[0006] Firstly, this application provides a database replication method, including:

[0007] The database write request of the target database is obtained through the user-side API. Based on the database write request, a corresponding nested key value is generated, and the obtained nested key values ​​are merged to determine the target transaction.

[0008] The primary and secondary nodes are determined based on a distributed consensus algorithm. The primary node encapsulates the target transaction into an initial log entry, synchronizes the initial log entry to the secondary nodes, and obtains synchronization confirmations from each of the secondary nodes. The synchronization confirmation is the information fed back by the secondary node after it has synchronized the initial log entry.

[0009] When the number of secondary nodes that have provided the synchronization confirmation is greater than a preset threshold, the initial log entry is marked as committed to obtain a marked log entry. The database write operation in the marked log entry is executed using a state machine and the corresponding target database is updated. Then, the operation execution result is returned to the user.

[0010] Optionally, generating the corresponding nested key-value pairs based on the database write request includes:

[0011] Parse the database write request to extract the corresponding path, key, and value;

[0012] The path, key, and value are encapsulated as nested key-value pairs with a tree structure.

[0013] Optionally, the method further includes:

[0014] Store the initial log entries in the local write-ahead log;

[0015] The number and / or storage time of the initial log entries in the local write-ahead log are detected based on a preset duration.

[0016] Determine whether the obtained detection result meets the snapshot triggering conditions; wherein, the snapshot triggering conditions include any one or more of the following: the number of marked log entries reaches a preset entry number threshold, the time interval since the last snapshot file is generated reaches a preset duration, and a manual triggering command is received from the user terminal;

[0017] If the detection result meets the snapshot triggering condition, the state machine is used to generate the current snapshot file of the target database and perform persistent storage operation.

[0018] Optionally, the step of generating a current snapshot file of the target database using the state machine and performing persistent storage operations includes:

[0019] The snapshot generation interface of the state machine is invoked to serialize the storage state of the nested key-value pairs in the target database into a target binary data stream;

[0020] The target binary data stream is compressed to generate a current snapshot file and persistently stored on a local disk or remote storage system.

[0021] Optionally, the method further includes:

[0022] When a node restarts after a failure, the current snapshot file is loaded to restore the data in the target database, and the state machine is used to replay the initial log entries and the marked log entries obtained after the time point corresponding to the current snapshot file.

[0023] Optionally, the method further includes:

[0024] Monitor the operating status of the master node;

[0025] When the operating status indicates a failure of the master node, an automatic master node election operation is triggered based on the slave nodes to determine a new master node from among the slave nodes.

[0026] Optionally, the database write operation includes any one or more of the following: data insertion, data update, data deletion, transaction commit, and transaction rollback.

[0027] Secondly, this application provides a database replication apparatus, comprising:

[0028] The transaction determination module is used to obtain database write requests through the user-end API, generate corresponding nested key-value pairs based on the database write requests, and merge the obtained nested key-value pairs to determine the target transaction.

[0029] The synchronization confirmation acquisition module is used to determine the master node and slave nodes based on the distributed consensus algorithm, encapsulate the target transaction into an initial log entry using the master node, synchronize the initial log entry to the slave nodes, and obtain synchronization confirmations from each of the slave nodes; wherein, the synchronization confirmation is the information fed back by the slave node after it has synchronized the initial log entry;

[0030] The database update module is used to mark the initial log entry as committed when the number of the secondary nodes that have provided the synchronization confirmation is greater than a preset threshold, to obtain the marked log entry, to execute the database write operation in the marked log entry using a state machine and update the corresponding database, and then return the operation execution result to the user terminal.

[0031] Thirdly, this application provides an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] A processor is used to execute the computer program to implement the database replication method as described above.

[0034] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the database copying method as described above.

[0035] In summary, this application first obtains the database write request of the target database through the client API, generates corresponding nested key-value pairs based on the database write request, and merges the obtained nested key-value pairs to determine the target transaction; it then determines the master node and slave nodes based on a distributed consistency algorithm, uses the master node to encapsulate the target transaction as an initial log entry, synchronizes the initial log entry to the slave nodes, and obtains synchronization confirmations from each slave node; wherein, the synchronization confirmation is the information fed back by the slave node after synchronizing the initial log entry; when the number of slave nodes that have fed back the synchronization confirmation is greater than a preset threshold, the initial log entry is marked as committed to obtain a marked log entry, the database write operation in the marked log entry is executed using a state machine and the corresponding target database is updated, and then the operation execution result is returned to the client. As described above, this application first obtains database write requests from the target database through the client-side API, generates corresponding nested key-value pairs based on these requests, and then merges these nested key-value pairs to determine the target transaction. Next, it uses a distributed consensus algorithm to determine the master and slave nodes. The master node encapsulates the target transaction into an initial log entry, synchronizes this initial log entry to the slave nodes, and obtains synchronization confirmations from each slave node after synchronization is completed. Finally, when the number of slave nodes providing synchronization confirmations exceeds a preset threshold, the initial log entry is marked as committed to form a marked log entry. The database write operation in the marked log entry is then executed through a state machine to update the corresponding target database, and the operation execution result is returned to the client. This improves the reliability and consistency of the distributed database, enhances the atomicity of operations and the maintainability of the system, and has significant engineering application value and promising prospects for widespread adoption. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a flowchart of a database replication method disclosed in this application;

[0038] Figure 2 This is a flowchart of a specific database replication method disclosed in this application;

[0039] Figure 3 This is a schematic diagram of the system architecture of a database replication method disclosed in this application;

[0040] Figure 4 This is a schematic diagram of the structure of a database replication device disclosed in this application;

[0041] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Currently, various consensus protocols have been proposed in academia and industry, with Paxos and Raft being the two most representative consensus algorithms. While Paxos theoretically possesses completeness and fault tolerance, its implementation complexity is high, making it difficult to widely apply in practical systems. Compared to Paxos, Raft, with its simpler, easier-to-understand, and easier-to-implement characteristics, has become the consensus protocol commonly adopted in industry. Raft ensures data consistency among multiple replicas in a distributed system through election mechanisms, log replication, and state machine consistency. Many distributed storage systems and databases, such as etcd, HashiCorpConsul, and TiKV, have already applied the Raft protocol to their underlying replication mechanisms. These systems primarily rely on the strong consistency guarantees provided by the Raft protocol, especially when handling cross-node write operations. The Raft protocol ensures data consistency on each node through log replication and guarantees high system availability through automatic leader election. To address the aforementioned technical issues, this application discloses a database replication method, apparatus, device, and storage medium capable of guaranteeing consistency between data replicas.

[0044] See Figure 1 As shown, an embodiment of the present invention discloses a database replication method, including:

[0045] Step S11: Obtain the database write request of the target database through the user-side API, generate corresponding nested key values ​​based on the database write request, and merge the obtained nested key values ​​to determine the target transaction.

[0046] In this embodiment, the necessary modules are first initialized, such as the Raft node management module, state machine module, nested key-value database, and log persistence module. Then, the database write request of the target database is obtained through the client API. Next, the database write request is parsed to extract the corresponding path, key, and value. The path, key, and value are then encapsulated into a nested key-value pair with a tree structure. Specifically, the database write request is parsed according to a preset path prefix tree model to generate a key and a value, and the corresponding path is determined. The path, key, and value are then encapsulated into a triple representation (path, key, value), i.e., a nested key-value pair. Here, path represents a logical namespace, such as / user / 1001 / container / 12; key represents a specific field under the current path; and value is the corresponding data value. It can be understood that after obtaining the nested key-value pairs, the obtained nested key-value pairs are packaged into a whole, i.e., the target transaction.

[0047] Step S12: Determine the master node and slave nodes based on the distributed consensus algorithm, encapsulate the target transaction into an initial log entry using the master node, synchronize the initial log entry to the slave nodes, and obtain synchronization confirmations from each of the slave nodes; wherein, the synchronization confirmation is the information fed back by the slave node after synchronizing the initial log entry.

[0048] In this embodiment, based on the Raft protocol's inherent leader election mechanism, a unique master node is democratically elected from multiple nodes in the cluster. This master node acts as the coordinator and scheduling center for all client write requests during its term, while the remaining nodes become slave nodes. Furthermore, the master node's operational status needs to be monitored; when the operational status indicates a master node failure, an automatic master node election is triggered based on the slave nodes to determine a new master node. Specifically, when the master node crashes or experiences network isolation, the slave nodes will automatically trigger an election after a timeout to elect a new master node. This mechanism ensures that the system can recover quickly after a node failure, without affecting overall service availability.

[0049] Understandably, when a client initiates a target transaction, it is sent to the master node. Upon receiving the target transaction, the master node encapsulates it into an initial log entry with a unique, incrementing index and the current term. This entry represents the operation instructions themselves and their order. The initial log entry is then broadcast to all slave nodes in the cluster. Simultaneously, the master node appends the initial log entry to its local write-ahead log. Each slave node, upon receiving the initial log entry from the master node, performs a series of rigorous checks, including verifying that the term number in the entry is more recent than its own current term, to ensure it does not accept outdated instructions from the old leader. After successful verification, the slave node also appends the initial log entry to its local persistent storage. This step is crucial, ensuring that uncommitted logs are not lost even if a node fails and restarts. Once a slave node successfully persists the log entry securely, it immediately sends a synchronization acknowledgment message to the master node, informing it that it has successfully received and stored the initial log entry.

[0050] Step S13: When the number of secondary nodes that have provided the synchronization confirmation is greater than a preset threshold, the initial log entry is marked as committed to obtain the marked log entry. The database write operation in the marked log entry is executed using a state machine and the corresponding target database is updated. Then, the operation execution result is returned to the user terminal.

[0051] In this embodiment, the master node counts the synchronization confirmations from the slave nodes. If more than half of the nodes in the cluster return synchronization confirmations, the initial log entry is considered to have been officially committed. At this time, the master node marks the initial log entry as committed. This marking is usually manifested by updating the commit index of the entry, marking it as committed, thus obtaining the marked log entry. This commit index points to the latest log position recognized by the cluster consensus, indicating that this entry and all log entries before it have been safely replicated to a majority of nodes. Subsequently, the master node drives the state machine to execute all marked log entries sequentially from the previous commit index to the current new commit index. The state machine executes the database write operations in the marked log entries, applying the write operations to the local storage engine to complete the update of the target database, ensuring that the master node's local data state evolves forward. The database write operations can include any one or more of the following: data insertion, data update, data deletion, transaction commit, and transaction rollback. Only after the state machine successfully completes the application update does the master node finally confirm that the operation has been persisted. It then generates a successful operation execution result and returns it to the requesting client, announcing the final completion of the write operation. Simultaneously, the master node carries this latest commit index value in its subsequent heartbeat messages to each slave node. Upon receiving the heartbeat, the slave nodes compare their logs and, if they find that the index of the replicated entries is less than or equal to this commit index, they know that these entries have been committed by the cluster. This triggers their own state machines to execute the same operation instructions to asynchronously update their respective local target database replicas, ultimately achieving eventual consistency across all nodes.

[0052] As described above, this embodiment first obtains database write requests from the target database through the client API, generates corresponding nested key-value pairs based on these requests, and then merges the nested key-value pairs to determine the target transaction. Next, it uses a distributed consensus algorithm to determine the master and slave nodes. The master node encapsulates the target transaction into an initial log entry, synchronizes this initial log entry to the slave nodes, and obtains synchronization confirmations from each slave node after synchronization is completed. Finally, when the number of slave nodes providing synchronization confirmations exceeds a preset threshold, the initial log entry is marked as committed to form a marked log entry. The database write operation in the marked log entry is executed through a state machine, updating the corresponding target database, and the operation execution result is returned to the client. This improves the reliability and consistency of the distributed database, enhances the atomicity of operations and the maintainability of the system, and has significant engineering application value and promising prospects for widespread adoption.

[0053] See Figure 2As shown, to ensure the consistency and recoverability of data status, this embodiment of the invention discloses a database replication method, including:

[0054] Step S21: Store the initial log entries in the local write-ahead log.

[0055] In this embodiment, each log entry is written to the local write-ahead log before the state machine is applied, based on the WAL storage mechanism.

[0056] Step S22: Detect the number of initial log entries and / or storage time in the local write-ahead log based on a preset duration.

[0057] In this embodiment, based on a preset time period, all log entries generated in the local write-ahead log since the last snapshot file is generated are automatically detected, including both initial log entries that have not yet been committed and marked log entries that have been committed. The cumulative number of these log entries and / or the earliest storage time, i.e. the lifespan of the first log that has not been snapshotted, are detected.

[0058] Step S23: Determine whether the obtained detection result meets the snapshot triggering conditions; wherein, the snapshot triggering conditions include any one or more of the following: the number of marked log entries reaches a preset entry number threshold, the time interval since the last snapshot file is generated reaches a preset duration, and a manual triggering command is received from the user terminal.

[0059] In this embodiment, it is determined whether the obtained detection result meets the snapshot triggering conditions. The snapshot triggering conditions include any one or more of the following: the number of marked log entries reaches a preset entry threshold, the time interval since the last snapshot file generation reaches a preset duration, and a manual trigger command is received from the user. First, when the number of marked log entries reaches the preset entry threshold, for example, a cumulative total of 10,000 entries, this indicates that the database state has undergone significant changes since the last snapshot. Generating a snapshot can effectively compress the log size and prevent the write-ahead log from expanding indefinitely. Second, when the time interval since the last successful snapshot file generation reaches a preset duration, for example, more than 12 hours, it ensures that snapshots are generated periodically even if write operations are infrequent and the data volume does not reach the threshold within a certain period, reducing log replay time in case of failure. Third, receiving a manual trigger command from the user allows for the proactive creation of a snapshot backup before performing important maintenance operations, or forces data persistence under specific circumstances.

[0060] Step S24: If the detection result meets the snapshot triggering condition, the current snapshot file of the target database is generated using the state machine and persistently stored.

[0061] In this embodiment, if the detection result meets the snapshot triggering condition, the snapshot generation interface of the state machine is called to serialize the storage state of the nested key-value pairs in the target database into a target binary data stream; the target binary data stream is compressed to generate a current snapshot file and persistently stored to a local disk or remote storage system. Specifically, the snapshot() method in the state machine serializes the current snapshot file of the target database into a transmittable binary blob and hands it over to the Raft layer for persistent storage.

[0062] Furthermore, when a node restarts after a failure, the current snapshot file is loaded to restore the data in the target database. The state machine then replays the initial log entries and marked log entries obtained after the time point corresponding to the current snapshot file. Specifically, when a node restarts, it needs to load the current snapshot file and read the most recent snapshot. Subsequently, the node reads the initial log entries and marked log entries after the time point corresponding to the snapshot file. Finally, the state machine replays the operation instructions in the log entries sequentially, thereby gradually advancing the database state from the snapshot point to the latest committed state, ensuring that the node data ultimately remains consistent with other healthy nodes.

[0063] In this way, the snapshot file provides the basic state, avoiding the inefficient operation of replaying all historical logs from the initial state; while subsequent log replays ensure that the node can catch up with the latest data progress of the cluster, so that it can seamlessly rejoin the cluster and participate in the normal consensus and data synchronization process after recovery. By combining snapshot loading and log replay, the state reconstruction of the failed node is achieved efficiently.

[0064] As can be seen from the previous embodiment, this application discloses a database replication method that can guarantee the consistency between data copies. Next, we will address... Figure 3 The database replication method shown is explained in detail.

[0065] First, this application receives database write requests initiated by the client through the client API module. Subsequently, the database write request is passed to the nested key-value interface (Tree KV) module, which parses the complex write operation and generates one or more nested key-value pairs conforming to its multi-level nested path data model. The transaction processing engine module locally combines and orchestrates the nested key-value operations generated by the current request, ensuring that these operations are atomically bound together to form a complete target transaction.

[0066] Next, the Raft consensus engine module elects a master node in the cluster, while the other nodes become slave nodes. The master node's Raft consensus engine is responsible for encapsulating the target transaction into an initial log entry, which includes the term number, index, and the transaction instruction itself. The master node uses the Raft consensus engine to mark the initial log entry as persisted to its local WAL / snapshot storage module and simultaneously synchronizes the entry to all slave nodes. After receiving the initial log entry, each slave node's Raft consensus engine also persists it to its own WAL storage and then sends a synchronization confirmation message back to the master node.

[0067] It's important to know that when the master node's Raft consensus engine detects that the number of slave nodes that have provided synchronous confirmation exceeds half of the cluster, consensus is reached. The Raft consensus engine then updates its commit index, marks the initial log entry as committed, and generates a marked log entry. Subsequently, the Raft engine delivers this committed entry to the local StateMachine module.

[0068] After receiving committed log entries, the state machine module deterministically executes the transaction instructions within them in sequence. The state machine performs database write operations by calling nested key-value interfaces (Tree KV) and the transaction processing engine, thereby safely updating the target database. Once the state machine successfully applies the transaction, the client API module returns the final operation result to the user.

[0069] See Figure 4 As shown, an embodiment of the present invention discloses a database replication apparatus, comprising:

[0070] The transaction determination module 11 is used to obtain a database write request through the user-end API, generate a corresponding nested key value based on the database write request, and merge the obtained nested key values ​​to determine the target transaction.

[0071] The synchronization confirmation acquisition module 12 is used to determine the master node and slave nodes based on the distributed consensus algorithm, encapsulate the target transaction into an initial log entry using the master node, synchronize the initial log entry to the slave nodes, and obtain synchronization confirmations from each of the slave nodes; wherein, the synchronization confirmation is the information fed back by the slave node after it has synchronized the initial log entry;

[0072] The database update module 13 is used to mark the initial log entry as a commit to obtain a marked log entry when the number of the secondary nodes that have provided the synchronization confirmation is greater than a preset threshold. It then uses a state machine to execute the database write operation in the marked log entry and update the corresponding database, and finally returns the operation execution result to the user.

[0073] As described above, this application first obtains database write requests from the target database through the client-side API, generates corresponding nested key-value pairs based on these requests, and then merges these nested key-value pairs to determine the target transaction. Next, it uses a distributed consensus algorithm to determine the master and slave nodes. The master node encapsulates the target transaction into an initial log entry, synchronizes this initial log entry to the slave nodes, and obtains synchronization confirmations from each slave node after synchronization is completed. Finally, when the number of slave nodes providing synchronization confirmations exceeds a preset threshold, the initial log entry is marked as committed to form a marked log entry. The database write operation in the marked log entry is then executed through a state machine to update the corresponding target database, and the operation execution result is returned to the client. This improves the reliability and consistency of the distributed database, enhances the atomicity of operations and the maintainability of the system, and has significant engineering application value and promising prospects for widespread adoption.

[0074] In some specific implementations, the transaction determination module 11 may specifically include:

[0075] The data extraction unit is used to parse the database write request to extract the corresponding path, key and value;

[0076] A nested key-value encapsulation unit is used to encapsulate the path, key, and value into a nested key-value pair with a tree structure.

[0077] In some specific embodiments, the database replication apparatus may further include:

[0078] The entry storage module is used to store the initial log entries in the local write-ahead log;

[0079] The log detection module is used to detect the number and / or storage time of the initial log entries in the local write-ahead log based on a preset duration.

[0080] The detection result judgment module is used to determine whether the obtained detection result meets the snapshot triggering conditions; wherein, the snapshot triggering conditions include any one or more of the following: the number of marked log entries reaches a preset entry number threshold, the time interval since the last snapshot file is generated reaches a preset duration, and a manual triggering command is received from the user terminal;

[0081] The file persistence module is used to generate a current snapshot file of the target database using the state machine and perform persistent storage operations if the detection result meets the snapshot triggering condition.

[0082] In some specific implementations, the file persistence module may specifically include:

[0083] The data stream acquisition unit is used to call the snapshot generation interface of the state machine to serialize the storage state of the nested key-value pairs in the target database into a target binary data stream;

[0084] The file storage unit is used to compress the target binary data stream to generate a current snapshot file and persistently store it to a local disk or a remote storage system.

[0085] In some specific embodiments, the database replication apparatus may further include:

[0086] The entry replay module is used to load the current snapshot file to restore the data of the target database when a node restarts after a failure, and to use the state machine to replay the initial log entries and the marked log entries obtained after the time point corresponding to the current snapshot file.

[0087] In some specific embodiments, the database replication apparatus may further include:

[0088] The operation status detection module is used to monitor the operation status of the master node;

[0089] The master node determination module is used to trigger an automatic master node election operation based on the secondary nodes to determine a new master node from the secondary nodes when the running status indicates that the master node is faulty.

[0090] In some specific implementations, the database write operation includes any one or more of the following: data insertion, data update, data deletion, transaction commit, and transaction rollback.

[0091] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0092] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the database copying method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0093] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0094] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0095] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the database replication method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0096] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed database replication method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0098] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0100] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A database replication method, characterized in that, include: The database write request of the target database is obtained through the user-side API. Based on the database write request, a corresponding nested key value is generated, and the obtained nested key values ​​are merged to determine the target transaction. The primary and secondary nodes are determined based on a distributed consensus algorithm. The primary node encapsulates the target transaction into an initial log entry, synchronizes the initial log entry to the secondary nodes, and obtains synchronization confirmations from each of the secondary nodes. The synchronization confirmation is the information fed back by the secondary node after it has synchronized the initial log entry. When the number of secondary nodes that have provided the synchronization confirmation is greater than a preset threshold, the initial log entry is marked as committed to obtain a marked log entry. The database write operation in the marked log entry is executed using a state machine and the corresponding target database is updated. Then, the operation execution result is returned to the user.

2. The database replication method according to claim 1, characterized in that, The process of generating corresponding nested key-value pairs based on the database write request includes: Parse the database write request to extract the corresponding path, key, and value; The path, key, and value are encapsulated as nested key-value pairs with a tree structure.

3. The database replication method according to claim 1, characterized in that, Also includes: Store the initial log entries in the local write-ahead log; The number and / or storage time of the initial log entries in the local write-ahead log are detected based on a preset duration. Determine whether the obtained detection result meets the snapshot triggering conditions; wherein, the snapshot triggering conditions include any one or more of the following: the number of marked log entries reaches a preset entry number threshold, the time interval since the last snapshot file is generated reaches a preset duration, and a manual triggering command is received from the user terminal; If the detection result meets the snapshot triggering condition, the state machine is used to generate the current snapshot file of the target database and perform persistent storage operation.

4. The database replication method according to claim 3, characterized in that, The step of generating a current snapshot file of the target database using the state machine and performing persistent storage operations includes: The snapshot generation interface of the state machine is invoked to serialize the storage state of the nested key-value pairs in the target database into a target binary data stream; The target binary data stream is compressed to generate a current snapshot file and persistently stored on a local disk or remote storage system.

5. The database replication method according to claim 4, characterized in that, Also includes: When a node restarts after a failure, the current snapshot file is loaded to restore the data in the target database, and the state machine is used to replay the initial log entries and the marked log entries obtained after the time point corresponding to the current snapshot file.

6. The database replication method according to claim 1, characterized in that, Also includes: Monitor the operating status of the master node; When the operating status indicates a failure of the master node, an automatic master node election operation is triggered based on the slave nodes to determine a new master node from among the slave nodes.

7. The database replication method according to claim 1, characterized in that, The database write operations include any one or more of the following: data insertion, data update, data deletion, transaction commit, and transaction rollback.

8. A database replication apparatus, characterized in that, include: The transaction determination module is used to obtain database write requests through the user-end API, generate corresponding nested key-value pairs based on the database write requests, and merge the obtained nested key-value pairs to determine the target transaction. The synchronization confirmation acquisition module is used to determine the master node and slave nodes based on the distributed consensus algorithm, encapsulate the target transaction into an initial log entry using the master node, synchronize the initial log entry to the slave nodes, and obtain synchronization confirmations from each of the slave nodes; wherein, the synchronization confirmation is the information fed back by the slave node after it has synchronized the initial log entry; The database update module is used to mark the initial log entry as committed when the number of the secondary nodes that have provided the synchronization confirmation is greater than a preset threshold, to obtain the marked log entry, to execute the database write operation in the marked log entry using a state machine and update the corresponding database, and then return the operation execution result to the user terminal.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the database replication method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the database replication method as described in any one of claims 1 to 7.

Images