A distributed database multi-copy synchronization method and device

Abstract

The application discloses a distributed database multi-copy synchronization method and equipment, belongs to the field of data processing, solves the problems of low synchronization efficiency, resource waste and storage redundancy, and the method comprises the following steps: a Leader node performs compression processing on original log entries; the compressed log entries are written into a local file in a compressed format; a Follower node performs consistency checking on the compressed log entries; after the consistency checking is passed, integrity checking is performed on the compressed log entries based on compressed data and check information; after the checking is passed, the compressed log entries are stored into the local file in the compressed format; when the compressed log entries are committed and need to be applied to a state machine, the compressed data is decompressed according to a compression algorithm type identifier, and the original log data obtained by decompression is applied to the state machine. The network transmission amount is reduced, repeated compression is avoided, storage redundancy and synchronization delay are reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and device for synchronizing multiple replicas of a distributed database. Background Technology

[0002] In traditional distributed databases, when synchronizing multi-replica logs based on consensus protocols such as Raft and Paxos, the Leader node sends uncompressed raw logs to the Follower nodes. Data compression is merely a post-hoc, local, asynchronous storage optimization technique, executed independently on each node and completely separated from the protocol replication process. This architecture introduces the following significant drawbacks in real-world deployment scenarios involving cross-datacenters and wide area networks: Large amounts of uncompressed logs are transmitted directly over the network, resulting in extremely high bandwidth consumption and significantly increased latency during cross-regional replica synchronization. Each replica node independently compresses the same log data, leading to redundant waste of CPU computing resources. The slow transmission speed of large transaction logs (containing large fields or batch operations) impacts the real-time consistency of replicas. Logs must first be replicated in their original format, with compression only possible in subsequent stages, resulting in significant redundancy within a certain period of storage.

[0003] As a result, existing distributed databases, when synchronizing multiple replicas, have log replication and data compression that are independent of each other, leading to problems such as low network bandwidth utilization, redundant waste of CPU resources, high replica synchronization latency, and periodic redundancy of storage space. Summary of the Invention

[0004] To address the aforementioned issues, this application proposes a distributed database multi-replica synchronization method, applicable to systems including Leader nodes and Follower nodes. The method includes: The leader node receives write requests from clients and generates raw log entries; The original log entries are compressed to obtain compressed log entries, which include compressed data, compression algorithm type identifier, and verification information. Write the compressed log entries into the local WAL file in a compressed format; The compressed log entries are sent to the Follower nodes via log replication messages of the consensus protocol; The Follower node receives the log replication message and performs a consistency check on the compressed log entries; After the consistency check passes, the Follower node performs an integrity check on the compressed log entry based on the compressed data and the verification information; After successful verification, the Follower node stores the compressed log entry in a compressed format to the local WAL file; When the compressed log entry is submitted and needs to be applied to the state machine, the Follower node decompresses the compressed data according to the compression algorithm type identifier and applies the decompressed raw log data to the state machine.

[0005] On the other hand, embodiments of this application provide a distributed database multi-replica synchronization device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the distributed database multi-replica synchronization method described above.

[0006] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: Because the Leader node compresses the original log entries immediately after generation and sends the compressed log entries to the Follower nodes via log replication messages, the data transmitted over the network is compressed, not the uncompressed original log. This significantly reduces network traffic between replicas, improves bandwidth utilization, and is particularly beneficial in bandwidth-constrained scenarios such as cross-data center and wide area network environments, reducing replica synchronization latency.

[0007] Compression is performed when the Leader node generates the log, eliminating the need for Follower nodes to perform compression on the same log again. Compared to traditional methods where each node compresses the same log independently, this avoids the same log being compressed repeatedly by multiple replicas, thus reducing redundant CPU resource consumption.

[0008] The leader node writes compressed log entries to its local WAL file in compressed format, and the follower nodes also store compressed log entries in their local WAL files after successful verification. The logs maintain a compressed format throughout the entire process from generation to storage, avoiding the staged storage redundancy caused by storing the original logs first and then compressing them asynchronously later, which results in the simultaneous existence of uncompressed and compressed logs in traditional solutions. This helps reduce peak storage space usage and disk I / O pressure.

[0009] After the consistency check passes, the Follower node performs integrity verification on the compressed log entries based on the compressed data and verification information. Since the verification object is the compressed data itself, the Follower node does not need to decompress the compressed data during the receiving phase to complete the integrity verification, thereby avoiding the CPU overhead caused by decompression operations and reducing the instantaneous peak CPU load during the log receiving phase.

[0010] Compressed log entries are stored in compressed format until they are committed and applied to the state machine. Only when a log entry is committed and needs to be applied to the state machine does the Follower node decompress the compressed data according to the compression algorithm type identifier and apply the decompressed raw log data to the state machine. For log entries that are not committed, subsequently overwritten, or discarded, no decompression operation is required, further saving CPU resources.

[0011] The compressed log entries include a compression algorithm type identifier. When a follower node needs to apply the logs, it can use this identifier to determine the corresponding decompression method, thereby correctly restoring the original log data. Therefore, log compression and decompression processes can accurately correspond, ensuring the correctness of data recovery during multi-replica synchronization.

[0012] Compressed log entries include verification information, which Follower nodes use to verify the integrity of received compressed data and whether it has been damaged or tampered with during transmission. Only after successful verification is the compressed log entry written to the local WAL file, thus preventing corrupted data from entering the log storage and state machine, ensuring the security of the consistency protocol and the correctness of multi-replica data.

[0013] Based on this, by combining compression pre-processing, compression transmission, compression storage, decompression-free verification, and delayed decompression, it works synergistically to reduce network transmission volume, avoid repeated compression, reduce storage redundancy, and reduce peak CPU load, thereby shortening replica synchronization latency, improving real-time system consistency, and enhancing the overall performance and availability of distributed databases in multi-replica scenarios. Attached Figure Description

[0014] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a distributed database multi-replica synchronization method provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of the overall structure of a distributed database multi-replica synchronization system. Figure 3 A flowchart of log compression preprocessing for a Leader node is provided as an embodiment of this application; Figure 4 This application provides a flowchart of a Follower node's decompression verification and delayed decompression process. Figure 5 A class diagram of a log compression module provided in an embodiment of this application; Figure 6 A flowchart illustrating the generation and transmission of a compressed snapshot is provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of a distributed database multi-replica synchronization device provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0017] Figure 1 This is a flowchart illustrating a distributed database multi-replica synchronization method provided in an embodiment of this application. This method can be applied to various business domains, such as internet finance, e-commerce, instant messaging, gaming, and government services.

[0018] Distributed databases typically employ a multi-replica architecture to ensure high availability and fault tolerance. To maintain data consistency among multiple replicas, the industry commonly uses log-based consistency protocols, which mainly include: Raft protocol: An easy-to-understand and implement strong consistency protocol that ensures consistency in a distributed system through leader election and log replication mechanisms. The leader node receives write requests from clients, generates log entries, and replicates the logs to all follower nodes via AppendEntries RPC (Remote Procedure Call).

[0019] Paxos protocol: A classic distributed consensus algorithm that uses a two-phase commit process (Prepare and Accept) to ensure that multiple replicas reach a consensus on the same operation.

[0020] WAL (Write-Ahead Log): The core mechanism for databases to ensure transaction persistence and fault recovery. All data modifications must be written to the log before the actual data is updated, thus ensuring that the system can be restored to a correct state after a crash.

[0021] To reduce storage costs and improve I / O efficiency, modern database systems widely employ data compression techniques. Common compression algorithms include: LZ4: A high-speed compression algorithm with fast compression and decompression speeds, suitable for real-time scenarios, but with a relatively low compression ratio.

[0022] Snappy: A fast compression algorithm developed by Google that balances compression ratio and speed.

[0023] Zstd: A compression algorithm developed by Facebook that offers adjustable compression levels and supports high compression ratios and fast decompression.

[0024] Columnar storage compression: A dedicated compression technology used for columnar storage formats of structured data.

[0025] Currently, the mainstream multi-replica synchronization solutions, represented by Raft, generally follow the following process: The Leader receives write requests from clients, generates raw logs containing information such as transaction ID, operation type, SQL, data content, and timestamps, and writes them to the local WAL.

[0026] The leader sends logs to all followers via AppendEntries RPC. At this time, the logs are transmitted as raw, uncompressed logs.

[0027] After receiving the logs, the Follower first performs a consistency check (verifies whether the index and term number of the preceding logs match). Once the logs are confirmed to be continuous and valid, the Follower directly writes the original logs to the local WAL and then sends an ACK reply to the Leader.

[0028] Once the Leader node receives confirmation messages from a majority of Followers, it updates the commit index, marks the log entry as committed, and applies the log to the state machine (i.e., performs the actual database operation).

[0029] Follower nodes learn about the Leader's latest commit index through heartbeat messages or subsequent AppendEntries RPCs, and apply the corresponding log entries to their local state machines.

[0030] To save storage space, each node performs log compression tasks asynchronously in the background. Each node independently compresses historical logs in its local WAL file, using common compression algorithms such as LZ4, Snappy, or Zstd to compress the log data before storage.

[0031] When the log volume becomes too large, nodes generate snapshots. A node serializes the complete state of its current state machine, generates a snapshot file, and deletes old logs within the snapshot's coverage area. When a new or lagging node needs to synchronize, it achieves fast synchronization by transmitting the snapshot file.

[0032] Based on this, in existing technologies, log replication occurs at the protocol layer, and data compression occurs at the storage layer, with the two being completely decoupled in time and space.

[0033] Separating log replication and data compression into independent processes has the following technical drawbacks: Low network transmission efficiency: The log replication phase transmits uncompressed raw data, while a single database log entry typically contains SQL statements, data content, transaction information, etc., ranging in size from a few KB to several MB. When deployed across data centers or regions, large amounts of uncompressed data are transmitted directly over the network, consuming significant bandwidth and resulting in long transmission latency. The larger the data volume, the slower the transmission under limited bandwidth, the longer the replica synchronization takes, and the worse the real-time consistency of the system.

[0034] Duplicated CPU resource waste: Assuming a cluster has 1 Leader and N Followers, in the current solution, each node must compress the same log file locally. This results in the same log file being compressed N+1 times, leading to redundant consumption of CPU computing resources. The compression calculation is repeatedly performed. Algorithms such as LZ4 and Zstd inherently have CPU overhead; in high-concurrency write scenarios, this repeated compression will consume a large amount of CPU, directly impacting the overall system performance.

[0035] Storage redundancy caused by decoupling compression and replication: In existing technologies, log replication is completed first, and then compression is performed asynchronously in the background. This creates a time lag: after receiving the original logs but before compression is complete, nodes must simultaneously retain both the uncompressed and compressed logs, resulting in staged storage space redundancy. Especially in high-concurrency write scenarios, the accumulation rate of uncompressed logs is fast, and the peak storage space usage may exceed expectations, increasing disk capacity and I / O pressure.

[0036] Snapshot transmission is costly: When a new node joins the cluster or a follower node comes back online after a long period of offline failure, snapshot files need to be transmitted to quickly synchronize the state. Current technologies typically generate a snapshot first, then compress it separately, or even transmit the uncompressed snapshot directly. However, snapshots are often full data in the GB or even TB range, which is huge in size, takes a long time to transmit, consumes a lot of bandwidth, directly slows down node online and fault recovery speeds, and reduces system availability.

[0037] As a result, the existing architecture treats log replication of the consistency protocol and data compression of the storage layer as two unrelated modules, which cannot continuously leverage the advantages of compression throughout the entire process of log generation, transmission, and storage.

[0038] This application solves the aforementioned technical problems by placing the compression algorithm at the consistency protocol layer, achieving compression-aware processing throughout the entire process of log generation, transmission, and storage. Figure 1The process includes the following steps: S101: The Leader node receives write requests from clients and generates raw log entries.

[0039] The leader node, as the node currently responsible for handling write requests in the distributed database cluster, receives write operation requests from clients, such as INSERT, UPDATE, and DELETE database operation requests. The leader node generates raw log entries based on these write requests.

[0040] The original log entry is a complete description of this write operation, and typically includes one or more of the following information: log index, term number, transaction identifier, operation type, operation data, timestamp, etc.

[0041] The log index is a monotonically increasing sequence number used to identify the position of the log in the replication sequence; the term number refers to the Leader term identifier in the Raft protocol, used to identify the current term of the Leader node; the transaction identifier is used to uniquely identify this transaction; the operation type indicates the type of operation, including INSERT, UPDATE, DELETE, etc.; the operation data may include SQL statements, table names, column names, field values, etc.; and the timestamp is used to record the time when the transaction occurred.

[0042] In traditional solutions, the raw log entry would be written directly to the local WAL in uncompressed form and sent to the Follower node. However, in this solution, the raw log entry does not immediately enter the replication process after it is generated; instead, it first undergoes subsequent compression preprocessing.

[0043] S102: Compress the original log entries to obtain compressed log entries. The compressed log entries include compressed data, compression algorithm type identifier, and verification information.

[0044] In this example, compression is performed after the Leader node generates the original log entries, rather than waiting until log replication is complete before being asynchronously compressed by the storage layer. When compressing the original log entries, structured preprocessing can also be performed to improve subsequent compression efficiency.

[0045] Based on this, the original log entries are compressed, including the following steps: Step 1: Perform structured preprocessing on the raw log entries.

[0046] For example, extract duplicate fields from the original log entries, create a dictionary index, and replace the duplicate strings with the index number.

[0047] Database log entries typically include a large number of recurring fields, such as table names, column names, operation types, transaction types, and keywords in constraints. In traditional uncompressed logs, these strings appear repeatedly with each log entry, consuming a significant amount of space.

[0048] For example, in a business scenario where the user table is frequently updated, the logs may repeatedly contain strings such as the table name "users", column name "id", "name", "balance", and operation type "UPDATE". If each log entry stores all of these strings completely, there will be a large amount of duplicate and redundant data in the logs.

[0049] Based on this, dictionaries can be built according to field categories, such as table name dictionary, column name dictionary, operation type dictionary, function name dictionary, etc. For each string appearing for the first time, it will be stored in the dictionary and assigned a unique index number; for subsequent appearances of the same string, the string itself will not be stored repeatedly, but only its corresponding index number will be stored. Since the index number usually only requires one byte or a few bytes to represent, while the original string may be several to tens of bytes long, this method can significantly reduce log volume. At the same time, the dictionary index itself has a high degree of regularity, making it easier for subsequent compression algorithms to identify and compress these repeating index sequences, thereby further improving the compression ratio.

[0050] It should be noted that the dictionary can be transmitted along with the log or pre-synchronized to each Follower node to ensure that the Follower nodes can correctly reconstruct the original string based on the index number after decompression. The dictionary can be created on the Leader node and maintained along with the compressed log entries or snapshots.

[0051] For example, variable-length integer encoding can be used for numerical data to reduce the space wastage of fixed-length integers.

[0052] Database logs often contain numerical data, such as transaction IDs, log indexes, term numbers, user IDs, amounts, quantities, and timestamps. In traditional fixed-length encoding, an integer typically occupies 4 or 8 bytes, regardless of its actual numerical value. For example, the value 1 and the value 1000000 both occupy the same fixed length, resulting in a large number of high-order bytes being 0, leading to wasted space.

[0053] Variable-length integer encoding is used for numeric data. The basic idea of ​​variable-length integer encoding is that the smaller the value, the fewer bytes it occupies; the larger the value, the more bytes it occupies. For example, a value less than 128 can be represented by only 1 byte; for larger values, 2 or more bytes are used, and the most significant bit of the byte indicates whether to continue reading the next byte.

[0054] Variable-length integer encoding can significantly reduce the space occupied by small-value and high-frequency low-value fields. Since many numeric fields in database logs (such as auto-incrementing IDs, transaction sequence numbers, and log indexes) tend to increment within a small range, variable-length encoding can effectively compress this type of data. At the same time, the data encoded with variable-length encoding still retains its numerical semantics; after decompression, it can be correctly restored to its original integer value without affecting log parsing and state machine execution.

[0055] For example, deduplicating string data means storing the same string only once and reusing it through a reference pointer.

[0056] String data in database logs, such as usernames, addresses, remarks, and constants in SQL statements, often contains duplicates. For example, multiple log entries within the same transaction might contain the same username "Zhang San," or multiple operations might involve the same status value "active." In traditional storage, each occurrence of this string would result in a complete copy being stored, leading to redundancy.

[0057] Perform deduplication on string data. Specifically, maintain a string pool or string table. For the first occurrence of a string, store it completely in the string pool and assign a reference pointer or reference number; for subsequent occurrences of the same string, do not store the string content again, but only store the corresponding reference pointer or reference number.

[0058] In this method, identical strings are stored only once in the log data, and only short reference pointers need to be stored in other locations. Since reference pointers are usually small integers of fixed or variable length, they occupy far less space than the original string itself. For log data containing a large number of repeated strings, this method can significantly reduce data volume. Furthermore, the deduplicated reference numbers have high repetition and regularity, which is beneficial for further compression by subsequent compression algorithms.

[0059] During the decompression and state machine application phases, Follower nodes can restore references to their original strings based on the string pool and reference pointers, thus ensuring the complete recovery of the original log data. The string pool can be maintained and transmitted with compressed log entries, compressed snapshots, or individual synchronization messages to ensure that each replica node can correctly parse the data.

[0060] Based on this, the redundancy of log data is reduced through the above preprocessing, which makes it easier for compression algorithms to achieve higher compression ratios.

[0061] Step 2: Based on the size of the preprocessed log entries, an adaptive compression strategy is used to compress the data to obtain compressed data, and compressed log entries including compressed data, compression algorithm type identifier, and verification information are generated.

[0062] In this example, during the adaptive compression phase, the compression strategy is selected based on the size of the preprocessed log entries. Instead of applying a uniform compression method to all log entries, the system dynamically selects whether to not compress, use the first compression algorithm, or use the second compression algorithm, based on the log entry size. The corresponding compression algorithm type identifier is written into the compressed log entry so that subsequent follower nodes can accurately identify and select the appropriate decompression method during decompression. This approach allows the system to achieve a balance between compression ratio, CPU overhead, and network transmission efficiency, adapting to varying log loads in different business scenarios.

[0063] Based on this, the following steps are included: Step 21: If the log size is less than the first threshold, do not compress it, and set the compression algorithm type identifier to uncompressed.

[0064] When the size of a log entry is less than a first threshold, the log is considered a small log. For such small logs, compression itself consumes CPU resources, and the compressed data also needs to include metadata such as compression algorithm type identifier, verification information, and original data length. If the log itself is very small, the space gain from compression may not be enough to offset the overhead of compression and decompression, and the compressed data size may even increase. Therefore, small logs are not compressed; they are directly encapsulated as compressed log entries in uncompressed form, and the compression algorithm type identifier is set to uncompressed.

[0065] The uncompressed type indicates that the data portion of this log entry has not been compressed. Subsequent follower nodes, when needing to apply the log to the state machine, can use this identifier to determine that a decompression algorithm is unnecessary and can directly read and parse the data portion. This avoids unnecessary compression and decompression operations, saves CPU resources, and reduces processing latency.

[0066] The first threshold can be preset based on factors such as the actual business scenario, average log size, CPU processing power, and network bandwidth. For example, the first threshold can be set to a small value so that compression is skipped only for logs that are truly too small. For example, the first threshold could be 1KB.

[0067] Step 22: If the log size is greater than or equal to the first threshold and less than the second threshold, then the first compression algorithm is used for compression, and the compression algorithm type identifier is set to the first compression type.

[0068] Log entries are classified as medium-sized if their size is greater than or equal to a first threshold and less than a second threshold. Compression typically yields significant space gains for these log entries, but overly complex or slow compression algorithms should be avoided to prevent increased processing latency. Therefore, medium-sized log entries are compressed using the first compression algorithm, and the compression algorithm type identifier is set to the first compression type.

[0069] The first compression algorithm prioritizes compression speed, characterized by fast compression and decompression speeds while maintaining a certain compression ratio. It is suitable for scenarios with high real-time requirements and frequent log generation. By employing the first compression algorithm, compression processing can be completed in a shorter time, reducing the latency of the Leader node in generating compressed log entries and lowering the CPU overhead of the Follower nodes during subsequent decompression.

[0070] The first compression type indicates that the log entry uses the first compression algorithm. When subsequent follower nodes need to apply this log, they can select the first decompression algorithm corresponding to the first compression algorithm based on this identifier to decompress the compressed data and restore the original log data.

[0071] For example, logs greater than or equal to 1KB and less than 100KB are classified as medium-sized logs. The first compression algorithm is the LZ4 fast compression algorithm.

[0072] Step 23: If the log size is greater than or equal to the second threshold, the second compression algorithm is used for compression, and the compression algorithm type identifier is set to the second compression type; the compression speed of the first compression algorithm is higher than that of the second compression algorithm, and the compression ratio of the second compression algorithm is higher than that of the first compression algorithm.

[0073] A log entry is considered a large log if its size is greater than or equal to the second threshold. These logs typically contain large fields, batch operations, complex transactions, or large amounts of data, resulting in a large original file size and being a major source of network transmission and storage space consumption. For these logs, the priority is to maximize the compression ratio to reduce network transmission and storage space consumption, while the sensitivity to compression speed is relatively low. Therefore, the second compression algorithm is used to compress large logs, and the compression algorithm type identifier is set to the second compression type.

[0074] The second compression algorithm prioritizes compression ratio, achieving a higher compression ratio, but its compression and decompression speeds are slower than the first algorithm. By employing the second compression algorithm, the size of large logs can be significantly reduced, thereby lowering network bandwidth usage and transmission latency during cross-node transmission. This is particularly suitable for bandwidth-constrained deployment scenarios such as cross-data center and wide area network (WAN) transmissions. Furthermore, since large logs typically constitute a small percentage of the total log volume, using a compression ratio-priority algorithm will not significantly impact overall processing latency.

[0075] The second compression type indicates that the log entry uses a second compression algorithm. When subsequent follower nodes need to apply this log, they can select the second decompression algorithm corresponding to the second compression algorithm based on this identifier to decompress the compressed data and restore the original log data.

[0076] It should be noted that the first compression algorithm has a higher compression speed than the second compression algorithm, while the second compression algorithm has a higher compression ratio than the first compression algorithm. These are different compression algorithms, each suitable for different log size ranges. Through this hierarchical selection method, the system can employ the most appropriate compression strategy for logs of different sizes, avoiding the problems of over-compressing small logs and under-compressing large logs.

[0077] For example, if the log size is >100KB, it is judged as a large log, and the second compression algorithm is the Zstandard (Zstd) compression algorithm.

[0078] It should be noted that the compression algorithm type identifier mentioned above can be represented by a field, for example, using different values ​​to represent the uncompressed type, the first compression type, and the second compression type, respectively. This field can be included in the data structure of the compressed log entry and transmitted and stored along with the compressed log entry so that Follower nodes can quickly identify it during decompression.

[0079] After compression, verification information is calculated for the compressed data. This verification information can be a checksum, used by subsequent Follower nodes to verify data integrity without decompression. The original data length is also recorded for verification of length consistency after decompression. Finally, compressed log entries are generated, each including one or more of the following fields: log index, term number, compressed data, verification information, original data length, and compression algorithm type identifier. The compression algorithm type identifier indicates whether the log entry uses uncompressed, first compression, or second compression.

[0080] In this example, a compressed log entry is generated that includes compressed data, a compression algorithm type identifier, and verification information, including: Step 24: Calculate the hash value of the compressed data to obtain the checksum.

[0081] After the Leader node completes the compression process on the original log entries, compressed data is obtained. To verify whether the compressed data has been corrupted or tampered with during transmission and storage, a hash value is calculated for the compressed data, and this hash value is used as a checksum.

[0082] A hash value is a fixed-length or variable-length digest of input data calculated using a hash algorithm. Hash algorithms possess properties such as determinism, collision resistance, and avalanche effect; that is, identical input data will always produce the same hash value, different input data should ideally produce different hash values, and even small changes in the input data can lead to significant changes in the hash value. Therefore, checksums can be used to determine whether data is complete or has been modified.

[0083] The object used to calculate the hash value is the compressed data, not the original log data. In other words, the checksum is calculated based on the compressed data itself. The technical significance of this setup is that subsequent follower nodes, upon receiving compressed log entries, can directly recalculate the hash value of the received compressed data and compare it with the checksum carried in the compressed log entry, thus completing integrity verification without decompressing the compressed data. Since decompression operations typically consume significant CPU resources, especially for log entries with high compression ratios and large data volumes, the decompression overhead is even more pronounced. This implementation calculates the checksum based on the compressed data, allowing follower nodes to avoid decompression during the receiving phase, thereby reducing peak CPU load and improving synchronization efficiency.

[0084] Hash algorithms such as CRC32 and XXHash can be used. These algorithms are fast and have low resource overhead, making them suitable for real-time verification of a large number of log entries in high-concurrency write scenarios. Of course, other hash algorithms that can meet the integrity verification requirements can also be used; there are no restrictions on this.

[0085] Step 25: Record the length of the original data.

[0086] The original data length refers to the length of the original log entry before compression, that is, the number of bytes or the data size before compression.

[0087] By recording the original data length, an additional verification method is provided for the decompression process. This verification works in conjunction with the aforementioned checksum based on compressed data: the checksum verifies the integrity of the compressed data during transmission and storage; the original data length verifies whether the decompressed data has been correctly recovered. Together, they can improve the reliability and security of log synchronization without significantly increasing system overhead.

[0088] Step 26: Encapsulate the compressed data, compression algorithm type identifier, checksum, and original data length into a compressed log entry.

[0089] Compressed log entries are structured data units used for transmission and storage during log replication in a consistency protocol.

[0090] Specifically, compressed log entries include the following fields: Compressed data: The data body obtained by compressing the original log entries.

[0091] Compression algorithm type identifier: Used to indicate the type of compression algorithm used to compress the data, such as uncompressed type, first compression type, or second compression type, so that the Follower node can select the corresponding decompression algorithm when decompressing.

[0092] Checksum: A hash value calculated based on compressed data, used by Follower nodes to verify the integrity of compressed data without decompressing it; Original data length: The length of the original log entries before compression, used to verify whether the data length is consistent after decompression.

[0093] Compressed log entries may also include one or more of the following fields: log index, term number, transaction identifier, operation type, timestamp, etc. These fields can be used in processes such as consistency checks, log location, and state machine applications.

[0094] S103: Write compressed log entries to the local WAL file in compressed format.

[0095] After generating compressed log entries, the Leader node writes them to its local WAL file in compressed format. Unlike traditional solutions that write the original log first and then compress it asynchronously, this solution maintains the compressed format during the WAL writing process. Therefore, the logs stored locally on the Leader node are compressed data from the beginning, avoiding the staged storage redundancy of original and compressed logs existing simultaneously, and reducing disk space usage and I / O pressure.

[0096] S104: Send compressed log entries to the Follower nodes via log replication messages of the consensus protocol.

[0097] The leader node constructs log replication messages for the consistency protocol, such as the AppendEntriesRPC message in the Raft protocol or the corresponding replication message in the Paxos protocol. These log replication messages carry one or more compressed log entries. Because compressed data is transmitted, network traffic is significantly reduced compared to transmitting the original log; for example, network traffic is reduced by 50%-70% compared to transmitting the original log (the actual compression ratio depends on the compressibility of the log data).

[0098] For replicas deployed across data centers or wide area networks, the reduction in the amount of data transmitted directly translates to lower network latency and reduced bandwidth usage. For example, an original log size of 10MB, compressed to 3MB, reduces transmission time from 8 seconds to 2.4 seconds at 10Mbps bandwidth.

[0099] Meanwhile, since compression is performed only once on the Leader node, it avoids the CPU duplication caused by each replica node compressing the same log in the traditional scheme.

[0100] For example, the message content includes: the previous log index (prevLogIndex) and the previous log term number (prevLogTerm): used for consistency checks by the Follower; the array of compressed log entries (entries[]): containing one or more CompressedLogEntries; and the Leader's commit index (leaderCommit): informing the Follower of the current committed log position.

[0101] Based on this, the Leader node sends AppendEntries RPCs to all Follower nodes over the network. Because compressed log data is transmitted, compressed transmission significantly reduces network latency for replicas deployed across data centers or wide area networks.

[0102] S105: Follower nodes receive log replication messages and perform consistency checks on compressed log entries.

[0103] After receiving the log replication message from the Leader node, the Follower node first performs a consistency check. The consistency check is used to confirm whether the position and tenure of the log entry in the replication sequence are continuous and valid with the local log.

[0104] For example, in the Raft protocol, Follower nodes verify whether the preceding log index and preceding log term number in the log replication message match their local logs. If they do not match, the message is rejected and a failure response is returned, triggering the Leader node's log rollback mechanism.

[0105] Based on this, this step ensures the basic security of the consensus protocol and is independent of the compression process itself.

[0106] S106: After the consistency check passes, the Follower node performs an integrity check on the compressed log entries based on the compressed data and verification information.

[0107] Follower nodes perform integrity checks on the received compressed log entries. The key to this step is that the integrity check is performed directly on the compressed data, without needing to decompress it.

[0108] Based on this, the integrity verification of compressed log entries includes the following steps: Step 1: The Follower node recalculates the checksum of the received compressed data.

[0109] After the consistency check passes, the Follower node extracts compressed data from the received compressed log entries. Then, it recalculates the hash value of the compressed data using the same hash algorithm that the Leader node used to generate the checksum, resulting in a new checksum.

[0110] Since the compressed data is already a byte stream compressed by the Leader node, Follower nodes can directly run a hash algorithm on it. Hash algorithms are deterministic, meaning that the same data input will always produce the same hash value. Therefore, if the compressed data has not changed during transmission, the checksum recalculated by the Follower node should be completely consistent with the checksum calculated by the Leader node.

[0111] It's important to note that the checksum calculation is independent of the compression algorithm type. Regardless of whether the compressed log entry uses uncompressed, first-level, or second-level compression, the Follower node directly calculates the checksum on the compressed data itself, without needing to differentiate based on the compression algorithm type. Therefore, this checksum mechanism is compatible with various adaptive compression strategies and has good versatility.

[0112] Step 2: Compare the recalculated checksum with the checksum in the compressed log entry.

[0113] The checksum in a compressed log entry is a hash value calculated by the Leader node from the compressed data when generating the compressed log entry, and is encapsulated as verification information in the compressed log entry.

[0114] The comparison can be done bit-by-bit, numerically, or by other equivalent methods. If the two checksums are exactly the same, it means that the compressed data received by the Follower node is completely identical to the compressed data sent by the Leader node, and the data has not been damaged or tampered with during transmission. If the two checksums are inconsistent, it means that the compressed data may have experienced abnormalities such as bit flipping, packet loss, truncation, or tampering during transmission, and the data integrity has been compromised.

[0115] Since the checksum is calculated based on compressed data, Follower nodes do not need to decompress the compressed data during the comparison process. Therefore, the comparison operation can be completed quickly during the receiving phase without introducing additional decompression delay.

[0116] Step 3: If they match, the verification is successful; if they do not match, the log entry is rejected and a verification failure response is returned.

[0117] If the Follower node confirms that the received compressed log entries are complete and valid, it can continue with subsequent operations.

[0118] If a Follower node determines that a received compressed log entry is incomplete or corrupted, it cannot write it to its local WAL file or include it in log replication confirmation. The Follower node rejects the compressed log entry and returns a verification failure response to the Leader node. Upon receiving this verification failure response, the Leader node can take appropriate measures, such as resending the compressed log entry, triggering the log rollback mechanism, or re-performing log replication, to ensure that the multiple replicas eventually reach consensus.

[0119] By returning a verification failure response, this implementation method can promptly expose anomalies in the data transmission process, preventing corrupted log data from being incorrectly stored and applied to the state machine, thereby ensuring the security of the consistency protocol and the correctness of the multi-replica data.

[0120] In summary, when generating compressed log entries, the Leader node calculates the checksum for the compressed data and encapsulates it as verification information within the compressed log entry. Upon receiving this compressed log entry, the Follower node directly recalculates the checksum for the received compressed data and compares it with the checksum carried in the compressed log entry. The entire verification process does not require decompressing the compressed data, thus ensuring verification reliability while avoiding the CPU overhead of decompression operations.

[0121] S107: After successful verification, the Follower node will store the compressed log entries in a compressed format in the local WAL file.

[0122] Follower nodes append compressed log entries directly to their local WAL files in compressed format, instead of decompressing them first. This ensures consistency across the storage layer while saving storage space.

[0123] Follower nodes update their local log indexes and metadata, and return a success response to the Leader.

[0124] S108: When a compressed log entry is committed and needs to be applied to the state machine, the Follower node decompresses the compressed data according to the compression algorithm type identifier and applies the decompressed raw log data to the state machine.

[0125] Once the Leader node receives a successful response from a majority of Followers, it updates the commit index, marking the corresponding log entry as committed. After both the Leader and Follower nodes detect the commit index update, they begin applying the committed but not yet applied log entries to the state machine.

[0126] In traditional solutions, follower nodes often need to decompress the received logs before storing them, or store the raw logs first and then compress them in the background. The decompression and compression operations are tightly coupled with the log receiving and commit processes, easily leading to peak CPU load during the receiving phase. In this solution, however, follower nodes only perform integrity checks based on compressed data during the receiving phase and write the compressed log entries directly to their local WAL file in compressed format without decompression. Only when a log entry is confirmed by a majority vote, the leader node updates the commit index, and the log entry needs to be applied to the state machine, does the follower node read the compressed log entry from its local WAL file and perform the decompression operation.

[0127] Therefore, the decompression operation is postponed until the log data is actually needed. For uncommitted, subsequently overwritten, or discarded log entries, decompression is unnecessary, thus avoiding unnecessary CPU overhead. Meanwhile, for committed log entries, since decompression occurs during the state machine application phase, it can be closely coordinated with the state machine execution process, reducing the instantaneous CPU pressure during the receive phase.

[0128] In this example, the Follower node decompresses the compressed data, including the following steps: Step 1: Follower nodes read committed compressed log entries from their local WAL files.

[0129] When a Follower node learns through heartbeat messages, subsequent log replication messages, or commit index update notifications that the Leader node has marked a log entry as committed, the Follower node determines that the log entry needs to be applied to its local state machine. At this point, the Follower node reads the committed compressed log entry from its local WAL file.

[0130] Since the Follower nodes have already written the compressed log entries to their local WAL files in compressed format during the receiving phase, what is read here is still the compressed log entry, not the original log data. The read operation can locate the corresponding WAL record based on information such as the log index and term number, and read fields such as compressed data, compression algorithm type identifier, checksum, and original data length.

[0131] For example, Follower nodes can read committed compressed log entries sequentially according to the log index order to ensure that the state machine applies the logs in the same order as the Leader node, thereby maintaining data consistency among multiple replicas.

[0132] Step 2: Select the corresponding decompression algorithm based on the compression algorithm type identifier.

[0133] After reading a compressed log entry, the Follower node extracts the compression algorithm type identifier from it. The compression algorithm type identifier indicates the type of compression algorithm used in the compressed log entry, such as uncompressed, first compression type, or second compression type. For example, the compression_type field can be used to determine the compression algorithm type.

[0134] If the compression algorithm type is identified as uncompressed, it is determined that the data portion of the log entry has not been compressed. There is no need to call the decompression algorithm, and the data portion can be directly used as the original log data for subsequent processing.

[0135] If the compression algorithm type is identified as the first compression type, then the first decompression algorithm corresponding to the first compression algorithm is selected.

[0136] If the compression algorithm type is identified as the second compression type, then the second decompression algorithm corresponding to the second compression algorithm is selected.

[0137] Based on this, since the compression algorithm type identifier is transmitted and stored along with the compression log entry, Follower nodes can determine the decompression algorithm without additional querying or negotiation, making it compatible with multiple adaptive compression strategies and exhibiting good versatility and scalability.

[0138] Step 3: Decompress the compressed data according to the decompression algorithm, and verify whether the length of the decompressed data is consistent with the length of the original data in the compressed log entry.

[0139] The original data length is the length of the original log entry before compression, recorded by the Leader node when generating the compressed log entry. This length information is transmitted and stored along with the compressed log entry to provide additional verification during the decompression phase.

[0140] Verification can be done by comparing the number of bytes or the size of the original log data obtained after decompression with the length of the original data. If they match, it means the decompression process was correct and the data has been fully restored; if they do not match, it means that an error may have occurred during decompression, or the compressed data may have been corrupted during transmission or storage. In this case, the Follower node can reject the log entry or trigger alarms, retransmissions, or other processing.

[0141] This length verification adds a layer of data integrity verification after decompression, in addition to the checksum based on compressed data, further improving the reliability of log synchronization. Furthermore, this verification operation is only performed after decompression and does not affect the efficiency of the undecompressed verification during the receiving phase.

[0142] Step 4: If so, parse the decompressed raw log data into database operations and execute them.

[0143] If the length of the decompressed data matches the length of the original data, the Follower node confirms successful decompression and parses the decompressed original log data into specific database operations. For example, the original log data may contain information such as SQL statements, operation types, table names, column names, and field values. The Follower node parses this information into database operations such as INSERT, UPDATE, and DELETE based on the log format and executes these operations to update its local state machine.

[0144] Since the Follower node applies the logs in the same log order as the Leader node, and the log content is consistent, after performing the same operation, the state machine of the Follower node will be consistent with the state machine of the Leader node, thereby achieving data synchronization between multiple replicas.

[0145] It's worth noting that when multiple consecutive compressed log entries need to be applied, a Follower node can read multiple compressed log entries from its local WAL file at once, decompress them in batches, and then apply them to the state machine sequentially. Batch decompression improves decompression efficiency and CPU cache hit rate, reducing the overhead of frequent decompression algorithm calls. Of course, it's also possible to read, decompress, and apply entries one by one; there's no limitation on this.

[0146] After the application is completed, the Follower node can update its local applied log index to mark that the log entry has been successfully applied to the state machine so that it can continue to process the next committed log.

[0147] In this example, a snapshot generation process is triggered when the logs accumulate to a certain size (e.g., exceeding 100,000 entries or occupying more than 1GB of space). This includes the following steps: Step 1: When the snapshot triggering condition is met, the Leader node serializes the current state of the state machine to obtain snapshot data.

[0148] During the operation of a distributed database, log entries accumulate continuously. If the number of logs or the space occupied continues to grow, it will not only increase storage pressure but also prolong the log replay time when new nodes join or lagging nodes catch up. Therefore, when the logs accumulate to a certain size, it is necessary to generate snapshots to replace the old logs within the snapshot's coverage area.

[0149] When the snapshot triggering conditions are met, the Leader node initiates the snapshot generation process. The snapshot triggering conditions can be set according to the actual business scenario, and may include one or more of the following: the number of log entries exceeds a set threshold (e.g., more than 100,000), the log space occupied exceeds a set space threshold (e.g., more than 1GB), or the snapshot is triggered periodically when the system is idle.

[0150] Upon triggering, the Leader node serializes the complete state of the current state machine to obtain snapshot data. The state machine is the component within the node that actually performs database operations and stores the actual data. The serialization process converts the data structures and content currently maintained by the state machine into a storable and transmittable byte stream or data format. For example, information such as tables, indexes, transaction states, and metadata in the state machine can be encoded according to a predetermined format to form snapshot data. This snapshot data represents the complete state of the state machine up to a certain log index.

[0151] It's important to note that before serialization, the Leader node can wait for all currently committed logs to be applied to the state machine to ensure consistency between the snapshot data and the log sequence. The snapshot data typically records the last log index and term number covered by the snapshot, to help determine the transition point between the snapshot and the incremental logs later.

[0152] Step 2: Compress the snapshot data to obtain a compressed snapshot file. The compressed snapshot file includes the last log index, term number, and verification information covered by the snapshot.

[0153] The leader node compresses the snapshot data to obtain compressed snapshot files. Similar to the aforementioned log compression, snapshots are compressed during the generation phase, rather than generating uncompressed snapshots first and then compressing them separately or directly transmitting uncompressed snapshots. Therefore, snapshots maintain a compressed format from the very beginning of their generation.

[0154] Compression can employ high compression ratio algorithms. Since snapshots typically include full-state data, resulting in large data volumes, and their sensitivity to compression speed is relatively lower than their compression ratio requirements, using high compression ratio algorithms can significantly reduce snapshot size within an acceptable processing time. For example, high compression ratio algorithms can include those supporting high compression levels, such as Zstd, where the compression level can be set according to actual needs, for example, levels 3 to 5.

[0155] Compressed snapshot files include one or more of the following metadata: The last log index covered by the snapshot: This identifies the range of logs covered by the snapshot, meaning that the snapshot replaced the old logs up to this index.

[0156] Term number: Used to identify the Leader term corresponding to this snapshot, facilitating subsequent consistency checks and log integration.

[0157] Verification information: Used by subsequent receiving nodes to verify the integrity of the compressed snapshot file, such as a checksum.

[0158] Original snapshot data size: Used to verify whether the length of the decompressed data is consistent during decompression and loading.

[0159] Step 3: Save the compressed snapshot file and delete old log entries within the snapshot coverage area.

[0160] Leader nodes are saved to local storage. During saving, the compressed snapshot file remains compressed and is not decompressed. Therefore, the snapshots stored locally on the node are also compressed data, saving storage space.

[0161] Simultaneously, the Leader node deletes old log entries within the snapshot's coverage area. Since the snapshot fully records the state machine state up to the last log index, log entries prior to that index are no longer necessary for state recovery. Deleting these old logs frees up storage space, reducing disk usage and subsequent log management overhead.

[0162] It's important to note that deleting old logs does not necessarily mean data loss. If the state machine needs to be restored later, the compressed snapshot file can be loaded first, and then the incremental logs can be applied from the last log index overwritten by the snapshot. Therefore, using snapshots and incremental logs together can effectively control the log storage size while ensuring data integrity.

[0163] Step 4: When a new node joins or a follower node falls behind, a compressed snapshot file is transmitted to the new node or follower node via a snapshot installation message.

[0164] When a new node joins the cluster, or when a follower node returns to the network after a prolonged period of offline activity and is significantly behind the leader node, incremental log replication alone may not be sufficient to quickly bring it back to the current cluster state. In this case, the leader node transmits a compressed snapshot file to the new node via a snapshot install message.

[0165] Snapshot installation messages can be messages used to transport snapshots in a consensus protocol, such as the InstallSnapshot RPC message in the Raft protocol or the corresponding snapshot transport message in the Paxos protocol.

[0166] Unlike schemes that transmit uncompressed snapshots or compress them after transmission, Leader nodes directly transmit compressed snapshot files. Because the snapshot files themselves are already efficiently compressed, their size is significantly reduced, resulting in a marked decrease in transmission time and bandwidth consumption. This is particularly suitable for bandwidth-constrained scenarios such as cross-data center and wide area network environments.

[0167] Step 5: The Follower node receives the compressed snapshot file transmitted by the Leader node via the snapshot installation message.

[0168] Follower nodes receive compressed snapshot files transmitted by the Leader node via snapshot install messages. Snapshot install messages can be messages used in consensus protocols for transmitting snapshots, such as the InstallSnapshot RPC message in the Raft protocol or the corresponding snapshot transmission message in the Paxos protocol. The compressed snapshot file includes compressed snapshot data and related metadata, which may include one or more of the following: the last log index covered by the snapshot, the term number, verification information, and the original snapshot data size.

[0169] Step 6: Verify the compressed snapshot file.

[0170] This verification can be performed based on compressed snapshot data, eliminating the need to decompress the compressed snapshot file during the receiving phase, thus saving CPU resources and reducing the instantaneous CPU pressure during the receiving phase. Verification can also include checking metadata such as the last log index and term number covered by the snapshot to ensure that the snapshot is correctly integrated with the current cluster state and subsequent incremental logs.

[0171] Step 7: After verification, save the compressed snapshot file in a compressed format.

[0172] Follower nodes do not immediately decompress the compressed snapshot file; instead, they directly save the compressed snapshot file.

[0173] In this way, Follower nodes do not need to perform decompression operations during the receiving and saving phases, avoiding the CPU overhead and memory usage associated with decompression. At the same time, compressed snapshot files occupy less storage space, which helps reduce disk pressure. The saved compressed snapshot file can be decompressed later when the state machine needs to be loaded, thus achieving delayed decompression and on-demand decompression.

[0174] It's important to note that Follower nodes can record the last log index and term number covered by the snapshot while saving the compressed snapshot file, in order to determine the starting position of the incremental log later. Furthermore, Follower nodes can delete old log entries that have been overwritten by the snapshot to free up storage space.

[0175] Step 8: When the state machine needs to be loaded, decompress the compressed snapshot file and load it into the state machine.

[0176] When a follower node needs to restore the state machine to the current cluster state, such as during node startup, fault recovery, or when the state machine needs to be rebuilt, the follower node reads a saved compressed snapshot file from local storage and decompresses it. The decompressed snapshot data is then loaded into the state machine.

[0177] The loading process may include: reconstructing data structures such as tables, indexes, and transaction states in the state machine based on the snapshot data, and restoring the state machine to the state corresponding to the last log index covered by the snapshot. During the loading process, the Follower node can also verify whether the length of the decompressed snapshot data is consistent with the size of the original snapshot data recorded in the compressed snapshot file, to confirm that the decompression result is correct and complete.

[0178] Because the decompression operation is postponed until the state machine is actually loaded, Follower nodes do not need to decompress during the receive and snapshot saving phases, thus avoiding peak CPU load during the receive phase. Furthermore, for scenarios where loading the state machine is not immediately required, the compressed snapshot file can be saved in compressed format indefinitely, further saving storage space.

[0179] Step 9: After loading is complete, continue to receive incremental compressed log entries following the compressed snapshot file to complete the synchronization.

[0180] After the compressed snapshot file is loaded, the Follower node's state machine has been restored to the state corresponding to the last log index covered by the snapshot.

[0181] Follower nodes continue to receive incrementally compressed log entries sent by the Leader node, following the last log index covered by the snapshot.

[0182] These incrementally compressed log entries can be processed according to the aforementioned log compression awareness mechanism: Follower nodes receive incrementally compressed log entries, perform consistency checks and integrity verification based on compressed data, and write them to the local WAL file in compressed format after the verification is passed. When the log is committed and needs to be applied, it is decompressed according to the compression algorithm type identifier and applied to the state machine.

[0183] By loading compressed snapshots first and then applying incrementally compressed logs, Follower nodes do not need to replay each log entry from the first one. Instead, they can continue catching up from the last log index covered by the snapshot, significantly shortening node uptime and fault recovery time. Ultimately, the state machine of the Follower node will become consistent with the state machines of the Leader node and other normal Follower nodes, completing multi-replica synchronization.

[0184] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S108 will be described sequentially, but this does not mean that steps S101 to S108 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S101 to S108 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S108 can be appropriately adjusted according to actual needs.

[0185] pass Figure 1 This method, where the Leader node compresses the original log entries immediately after generation and sends the compressed log entries to the Follower nodes via log replication messages, ensures that compressed data, rather than the uncompressed original logs, is transmitted over the network. This significantly reduces network traffic between replicas, improves bandwidth utilization, and is particularly beneficial in bandwidth-constrained scenarios such as cross-data center and wide area network environments, thereby reducing replica synchronization latency.

[0186] Compression is performed when the Leader node generates the log, eliminating the need for Follower nodes to perform compression on the same log again. Compared to traditional methods where each node compresses the same log independently, this avoids the same log being compressed repeatedly by multiple replicas, thus reducing redundant CPU resource consumption.

[0187] The leader node writes compressed log entries to its local WAL file in compressed format, and the follower nodes also store compressed log entries in their local WAL files after successful verification. The logs maintain a compressed format throughout the entire process from generation to storage, avoiding the staged storage redundancy caused by storing the original logs first and then compressing them asynchronously later, which results in the simultaneous existence of uncompressed and compressed logs in traditional solutions. This helps reduce peak storage space usage and disk I / O pressure.

[0188] After the consistency check passes, the Follower node performs integrity verification on the compressed log entries based on the compressed data and verification information. Since the verification object is the compressed data itself, the Follower node does not need to decompress the compressed data during the receiving phase to complete the integrity verification, thereby avoiding the CPU overhead caused by decompression operations and reducing the instantaneous peak CPU load during the log receiving phase.

[0189] Compressed log entries are stored in compressed format until they are committed and applied to the state machine. Only when a log entry is committed and needs to be applied to the state machine does the Follower node decompress the compressed data according to the compression algorithm type identifier and apply the decompressed raw log data to the state machine. For log entries that are not committed, subsequently overwritten, or discarded, no decompression operation is required, further saving CPU resources.

[0190] The compressed log entries include a compression algorithm type identifier. When a follower node needs to apply the logs, it can use this identifier to determine the corresponding decompression method, thereby correctly restoring the original log data. Therefore, log compression and decompression processes can accurately correspond, ensuring the correctness of data recovery during multi-replica synchronization.

[0191] Compressed log entries include verification information, which Follower nodes use to verify the integrity of received compressed data and whether it has been damaged or tampered with during transmission. Only after successful verification is the compressed log entry written to the local WAL file, thus preventing corrupted data from entering the log storage and state machine, ensuring the security of the consistency protocol and the correctness of multi-replica data.

[0192] By combining compression pre-processing, compression transmission, compression storage, decompression-free verification, and delayed decompression, it works synergistically to reduce network transmission volume, avoid redundant compression, reduce storage redundancy, and reduce peak CPU load, thereby shortening replica synchronization latency, improving real-time system consistency, and enhancing the overall performance and availability of distributed databases in multi-replica scenarios.

[0193] In summary, by placing compression at the consistency protocol layer, logs maintain compression awareness throughout the entire process of generation, transmission, verification, storage, and application. This reduces network transmission volume, avoids redundant CPU compression, reduces storage redundancy, lowers peak CPU usage during the receiving phase, and shortens synchronization latency without affecting consistency.

[0194] Furthermore, the key points are as follows: The log compression pre-processing mechanism at the consistency protocol layer differs from traditional solutions where logs are first generated, copied, or written to disk in their original format before compression by the storage layer. Instead, the log content is structured and compressed immediately after log entries are generated on the Leader node. Specifically, this includes: dictionary-based processing of repetitive information such as table names, field names, and operation types; variable-length encoding for numeric fields; and reference representation of duplicate data. Subsequently, an appropriate compression algorithm is selected based on log characteristics, generating compressed log entries containing information such as log index, term number, transaction identifier, compression algorithm type, length before and after compression, and checksum. In this way, logs exist in compressed format during the consistency protocol replication process, avoiding redundant processing and additional resource consumption caused by the protocol layer copying the original logs and the storage layer compressing them again.

[0195] Regarding the integrity verification mechanism for compressed logs: After receiving the compressed logs, the Follower node can directly calculate the checksum based on the compressed data and compare it with the checksum field sent by the Leader to determine whether the logs have been damaged or tampered with during transmission.

[0196] Adaptive Compression Strategy: Instead of applying a fixed compression method to all logs, the system dynamically selects a processing strategy based on the size or content characteristics of log entries. For example, smaller logs may not be compressed to avoid the compression overhead outweighing the benefits; medium-sized logs can use a faster compression algorithm; and larger logs can use an algorithm with a higher compression ratio to reduce the amount of data transmitted across nodes. Through this adaptive strategy, the system can achieve a good balance between compression ratio, CPU overhead, and network transmission efficiency, adapting to changes in log load under different business scenarios.

[0197] Delayed decompression and on-demand decompression mechanism: After receiving and verifying compressed logs, Follower nodes can directly write the compressed logs to their local WAL or log storage files without immediate decompression. Decompression and parsing are only performed on logs that have been committed and need to be applied to the state machine or database execution layer. For uncommitted, subsequently overwritten, or discarded logs, no decompression operation is required. This mechanism reduces the instantaneous CPU pressure during the log receiving phase and avoids unnecessary decompression of invalid logs. Furthermore, during batch log commits, multiple compressed logs can be decompressed in batches, improving processing efficiency.

[0198] An integrated mechanism for compressed snapshots extends the aforementioned compression concept to the snapshot generation, transmission, and storage processes. When generating a snapshot, nodes can directly compress state machine data to create a compressed snapshot file. When a new node joins, a lagging node catches up, or a fault recovers, the compressed snapshot is transmitted via the snapshot installation interface. Receiving nodes can directly save the compressed snapshot and decompress it only when actually loading or restoring the state machine. This design adopts a unified compression approach for log replication and snapshot synchronization, reducing the disconnect between log synchronization and snapshot synchronization in engineering implementation and lowering network transmission overhead during large-scale data recovery.

[0199] More intuitively, Figure 2 This is a schematic diagram of the overall structure of a distributed database multi-replica synchronization system provided in an embodiment of this application.

[0200] exist Figure 2 This paper demonstrates the overall architecture of a distributed database multi-replica synchronization system based on log compression awareness. The system includes clients, a Leader node, and multiple Follower nodes. After receiving a write request from a client, the Leader node generates log entries and performs pre-compression processing (including structured preprocessing, adaptive compression strategy selection, checksum calculation), then transmits the compressed logs to all Follower nodes over the network. Follower nodes receive the compressed logs, perform uncompressed verification, and store them in compressed format in their local WAL file after successful verification. When the logs are committed, each node decompresses them as needed and applies them to the state machine.

[0201] In summary, this technology addresses the problems of low network efficiency, CPU waste, temporary storage redundancy, and slow snapshot synchronization in existing technologies, achieving the following technical effects: Reduce the amount of data transmitted between replicas by 50%-70%, thereby reducing bandwidth consumption and cross-regional synchronization latency; avoid repeated compression of the same log data on each replica node, thereby reducing CPU computing overhead; shorten replica synchronization latency by 30%-50%, thereby improving real-time consistency of the system; optimize storage space utilization, thereby reducing temporary space occupation of logs during the replication phase; accelerate snapshot transmission and new node catch-up speed, thereby improving system availability and fault recovery capabilities.

[0202] Figure 3 This is a flowchart of a Leader node's log compression preprocessing step provided in an embodiment of this application.

[0203] Figure 3 This describes the pre-compression processing flow of the Leader node when generating log entries. The process begins with receiving a client write request, first generating a raw log entry containing complete transaction information, and then proceeding to the adaptive compression strategy determination stage. Based on log size, three processing paths are used: small logs (<1KB) are encapsulated directly without compression; medium logs (1KB-100KB) use LZ4 fast compression; and large logs (>100KB) use the Zstd high compression ratio algorithm. After compression, the checksum is calculated, and finally, the logs are encapsulated into a CompressedLogEntry structure and written to the local WAL file, subsequently sent to Follower nodes via AppendEntriesRPC.

[0204] Figure 4 This is a flowchart of a Follower node with no decompression verification and delayed decompression provided in an embodiment of this application.

[0205] Figure 4 This demonstrates the processing flow of Follower nodes after receiving compressed logs, showcasing two core mechanisms: no decompression verification and delayed decompression. The Follower first receives the AppendEntries RPC and performs a consistency check (verifying prevLogIndex and prevLogTerm). After the check passes, it proceeds to the integrity verification stage.

[0206] The key innovation lies in the fact that the Follower directly recalculates the checksum of the compressed data and compares it with the received checksum. This entire process eliminates the need for data decompression, thus saving CPU overhead. After successful verification, the compressed log is directly written to the local WAL file in compressed format. Only when the log receives majority confirmation and is marked as committed by the Leader does the Follower read the compressed log from the WAL, select the appropriate decompression algorithm based on the compression_type, and apply the decompressed data to the state machine. This delayed decompression strategy postpones the decompression operation until the log data is actually needed, avoiding peak CPU load during the receiving phase.

[0207] Figure 5 This is a class diagram of a log compression module provided in an embodiment of this application.

[0208] Figure 5 The structure and relationships between compressed log entries, compression strategies, algorithm types, the WAL manager, and the state machine in the log compression module are described using a class diagram. A compressed log entry contains six core fields: log index, term number, compressed data byte stream, checksum, original data size, and compression algorithm type identifier, as well as three core methods: compression, checksum, and decompression. The compression strategy is responsible for implementing adaptive compression logic, including two methods: selecting a compression algorithm based on data size and executing adaptive compression. The two classes are connected by a dependency relationship, indicating that when a compressed log entry performs a compression operation, it calls the compression strategy's methods to determine which compression algorithm to use. The WAL manager manages the compressed log entries, providing operations such as appending, reading, and compressing log entries. When performing operations such as state application or snapshot loading, the state machine can use the decompressed data from the compressed log entries to achieve the storage, reading, decompression, and state application of log data.

[0209] Figure 6 This is a flowchart illustrating the generation and transmission of a compressed snapshot, as provided in an embodiment of this application.

[0210] Figure 6 This document describes the complete process of the compressed snapshot mechanism, including four stages: snapshot generation, compression, transmission, and loading. When a node's logs accumulate to a trigger threshold (e.g., exceeding 100,000 entries or occupying more than 1GB of space), the snapshot generation process is initiated. The node first performs a complete serialization of its current state machine to obtain the raw snapshot data. Then, it applies the Zstd high compression ratio algorithm (compression level 3-5) for compression, generating a compressed snapshot file containing metadata such as lastIncludedIndex, lastIncludedTerm, and checksum. After snapshot generation, the node deletes old logs within the snapshot's coverage area to free up space. When a new node joins or a follower falls significantly behind, the leader transmits the compressed snapshot file via InstallSnapshot RPC. The receiving node verifies the checksum upon receipt; if verification is successful, it saves the compressed snapshot and decompresses and loads it into its state machine when needed. After snapshot loading is complete, the node continues to receive incremental logs to achieve complete synchronization. Throughout the entire process, the snapshot remains in compressed format, significantly reducing transmission time and storage space consumption.

[0211] Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0212] Figure 7A schematic diagram of the structure of a distributed database multi-replica synchronization device provided in this application embodiment includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the distributed database multi-replica synchronization method described above.

[0213] Some embodiments of this application provide a distributed database multi-replica synchronization non-volatile computer storage medium storing computer-executable instructions that can execute the aforementioned distributed database multi-replica synchronization method.

[0214] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0215] The devices and methods provided in this application are one-to-one correspondences. Therefore, the devices also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices will not be repeated here.

[0216] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A method for synchronizing multiple replicas in a distributed database, characterized in that, Applied to a system including Leader nodes and Follower nodes, the method includes: The leader node receives write requests from clients and generates raw log entries; The original log entries are compressed to obtain compressed log entries, which include compressed data, compression algorithm type identifier, and verification information. Write the compressed log entries into the local WAL file in a compressed format; The compressed log entries are sent to the Follower nodes via log replication messages of the consensus protocol; The Follower node receives the log replication message and performs a consistency check on the compressed log entries; After the consistency check passes, the Follower node performs an integrity check on the compressed log entry based on the compressed data and the verification information; After successful verification, the Follower node stores the compressed log entry in a compressed format to the local WAL file; When the compressed log entry is submitted and needs to be applied to the state machine, the Follower node decompresses the compressed data according to the compression algorithm type identifier and applies the decompressed raw log data to the state machine.

2. The method according to claim 1, characterized in that, The original log entries are compressed to obtain compressed log entries, specifically including: The original log entries are preprocessed using a structured approach; Based on the size of the preprocessed log entries, an adaptive compression strategy is used to compress the data to obtain compressed data, and compressed log entries are generated that include the compressed data, compression algorithm type identifier, and verification information.

3. The method according to claim 2, characterized in that, Based on the size of the preprocessed log entries, an adaptive compression strategy is used to compress the data, resulting in compressed data, specifically including: If the log size is less than the first threshold, it will not be compressed, and the compression algorithm type will be set to uncompressed. If the log size is greater than or equal to the first threshold and less than the second threshold, then the first compression algorithm is used for compression, and the compression algorithm type identifier is set to the first compression type; If the log size is greater than or equal to the second threshold, the second compression algorithm is used for compression, and the compression algorithm type identifier is set to the second compression type; the compression speed of the first compression algorithm is higher than that of the second compression algorithm, and the compression ratio of the second compression algorithm is higher than that of the first compression algorithm.

4. The method according to claim 2, characterized in that, The original log entries are subjected to structured preprocessing, including at least one of the following: Extract duplicate fields from the original log entries to create a dictionary index, and replace duplicate strings with index numbers; Variable-length integer encoding is used for numerical data; Remove duplicate references from string data.

5. The method according to claim 2, characterized in that, Generate compressed log entries that include the compressed data, compression algorithm type identifier, and verification information, specifically including: Calculate the hash value of the compressed data to obtain the checksum; Record the length of the original data; The compressed data, the compression algorithm type identifier, the checksum, and the original data length are encapsulated into a compressed log entry.

6. The method according to claim 1, characterized in that, The Follower node performs integrity verification on the compressed log entries based on the compressed data and the verification information, specifically including: The Follower node recalculates the checksum of the received compressed data; The recalculated checksum is compared with the checksum in the compressed log entry; If they match, the verification is considered successful. If there is a discrepancy, the log entry to be compressed will be rejected and a verification failure response will be returned.

7. The method according to claim 1, characterized in that, The Follower node decompresses the compressed data according to the compression algorithm type identifier, specifically including: The Follower node reads the committed compressed log entries from the local WAL file; Select the corresponding decompression algorithm based on the compression algorithm type identifier; According to the decompression algorithm, the compressed data is decompressed, and it is verified whether the length of the decompressed data is consistent with the length of the original data in the compressed log entry. If so, parse the decompressed raw log data into database operations and execute them.

8. The method according to claim 1, characterized in that, The method further includes: When the snapshot triggering condition is met, the Leader node serializes the current state of the state machine to obtain snapshot data; The snapshot data is compressed to obtain a compressed snapshot file, which includes the last log index covered by the snapshot, the term number, and verification information. Save the compressed snapshot file and delete old log entries within the snapshot coverage area; When a new node joins or a follower node falls behind, the compressed snapshot file is transmitted to the new node or follower node via a snapshot installation message.

9. The method according to claim 8, characterized in that, The method further includes: The Follower node receives a compressed snapshot file transmitted by the Leader node via a snapshot installation message; The compressed snapshot file is verified; After successful verification, the compressed snapshot file is saved in a compressed format. When the state machine needs to be loaded, the compressed snapshot file is decompressed and loaded into the state machine; After loading is complete, continue receiving incremental compressed log entries following the compressed snapshot file to complete synchronization.

10. A distributed database multi-replica synchronization device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a distributed database multi-replica synchronization method according to any one of claims 1-9.

Images