Distributed system snapshot method, apparatus, device, and medium

Abstract

The application discloses a distributed system snapshot method and device, equipment and medium, relates to the technical field of distributed system snapshot, and comprises the following steps: generating corresponding event objects based on state change operations of each node in the distributed system; in response to a snapshot trigger condition, each node creates a local incremental checkpoint respectively, and records the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks that have occurred state changes recorded through event objects since the last snapshot; collecting the local incremental checkpoint and the current logical timestamp of each node to determine a global consistency time point according to the current logical timestamp, and generating snapshot metadata to determine the dependency relationship between the current snapshot and the last snapshot based on the dependency pointer pointing to the last snapshot in the snapshot metadata. The application solves the problems that the full snapshot occupies memory and bandwidth, it is difficult to support high-frequency snapshots, and the incremental snapshot recovery needs to traverse long-chain incremental data, which is low in efficiency.

Description

Technical Field

[0001] This invention relates to the field of distributed system snapshot technology, and in particular to distributed system snapshot methods, apparatus, devices and media. Background Technology

[0002] Distributed systems achieve high scalability and processing power by distributing computing tasks across multiple nodes. However, issues such as node failures and network latency pose challenges to ensuring the consistency and reliability of the system state. Therefore, periodically creating globally consistent snapshots of the entire system has become a key technology for scenarios such as fault recovery and state rollback.

[0003] Traditional distributed snapshot technologies, such as the Chandy-Lamport algorithm, coordinate the global state by sending tag messages to all nodes. However, this synchronization process introduces significant pauses and delays, leading to a severe degradation in system performance. Furthermore, full snapshots consume substantial storage space and network bandwidth, making it difficult to support high-frequency snapshot requirements.

[0004] While incremental snapshot technology can reduce the amount of data per snapshot, the current management of dependencies between snapshots is complex, requiring traversal of long-chain incremental data during recovery, resulting in low efficiency. Log-based snapshot technology can provide fine-grained change records, but recovery requires replaying a large amount of historical logs, which is time-consuming. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, device, and medium for snapshotting distributed systems, which can solve the problems of full snapshots occupying memory and bandwidth, making it difficult to support high-frequency snapshots, and the low efficiency of incremental snapshot recovery requiring traversal of incremental data over long links.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a distributed system snapshot method, comprising: Based on the state change operations of each node in the distributed system, corresponding event objects are generated. In response to the snapshot triggering condition, each node creates a local incremental checkpoint and records the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks whose state has changed since the last snapshot, recorded by the event object. Collect local incremental checkpoints and current logical timestamps of each node to determine the global consistency time point based on the current logical timestamp, and generate snapshot metadata to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers to the previous snapshot in the snapshot metadata.

[0007] Optionally, based on the state change operations of each node in the distributed system, corresponding event objects are generated, including: In response to a state change operation occurring on any node in a distributed system, an event object corresponding to the state change operation is generated. The event object is asynchronously published to the event bus and persistently stored in the event log; The event object carries an event identifier, an event type, a current node identifier, and a logical timestamp of the current node. The logical timestamp is used to maintain a vector clock for the causal order of event objects in the distributed system.

[0008] Optionally, in response to snapshot triggering conditions, each node creates a local incremental checkpoint, including: The snapshot coordinator initiates the snapshot operation based on a preset time interval or event quantity threshold, and broadcasts the snapshot start message to each node. After each node receives the start snapshot message, it obtains the current logical timestamp of each node and creates a local incremental checkpoint.

[0009] Optionally, the globally consistent time point is the minimum of the current logical timestamps reported by all nodes.

[0010] Optionally, snapshot metadata includes snapshot identifier, global consistency time point, storage location of local incremental checkpoints on each node, and dependency pointers.

[0011] Optional, distributed system snapshot methods also include: Get a full snapshot and use it as the base snapshot; The local incremental checkpoints are applied sequentially from the base snapshot in the order they were created, and the event objects recorded in the event log that are not reflected in the state corresponding to the last local incremental checkpoint are re-executed to restore the state of the distributed system.

[0012] Optional, distributed system snapshot methods also include: If any node fails to create a local incremental checkpoint or times out during the snapshot period, the snapshot coordinator marks the current snapshot round as a failure, abandons the current snapshot round, and starts the next snapshot round.

[0013] In a second aspect, the present invention provides a distributed system snapshot device, comprising: The object generation module is used to generate corresponding event objects based on the state change operations of each node in the distributed system. The incremental checkpoint creation module is used to create local incremental checkpoints on each node in response to snapshot triggering conditions, and to record the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks that have undergone state changes since the last snapshot, recorded by the event object. The dependency determination module is used to collect local incremental checkpoints and current logical timestamps of each node for snapshot collection, so as to determine the global consistency time point based on the current logical timestamp, and generate snapshot metadata, so as to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers pointing to the previous snapshot in the snapshot metadata.

[0014] Thirdly, the present invention provides an electronic device, comprising: Memory, used to store computer programs; A processor for executing computer programs to implement the steps of the distributed system snapshot method disclosed above.

[0015] Fourthly, the present invention provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed distributed system snapshot method.

[0016] This invention provides a method for generating corresponding event objects based on the state change operations of each node in a distributed system; in response to snapshot triggering conditions, each node creates a local incremental checkpoint and records the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks whose state changes have occurred since the previous snapshot, recorded through the event object; the local incremental checkpoints and current logical timestamps of each node are collected to determine the global consistency time point based on the current logical timestamp, and snapshot metadata is generated to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers pointing to the previous snapshot in the snapshot metadata.

[0017] As can be seen from the above technical solution, by generating event objects corresponding to the state change operations of nodes, the change tracking basis is provided for local incremental checkpoints, avoiding blind scanning of the full state. Furthermore, by creating local incremental checkpoints, the problem of high storage overhead of full snapshots is solved. At the same time, by identifying changes through event objects, full scanning is avoided. In this way, each snapshot only saves the changed data blocks, significantly reducing storage space and network bandwidth consumption, and supporting high-frequency snapshots. Next, the current logical timestamp is recorded to provide a causal order basis for subsequent asynchronous coordination, avoiding global synchronization barriers. The relative order of snapshots can be determined without pausing business, reducing the impact on system performance. Attached Figure Description

[0018] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A flowchart of a distributed system snapshot method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a specific distributed system snapshot method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of a distributed system snapshot device provided in an embodiment of the present invention; Figure 4 This is a diagram of an electronic device provided as an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0021] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0022] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Distributed systems achieve unprecedented scalability and processing power by distributing computing and storage tasks across a large number of nodes. However, the complexity of the system also increases dramatically, with node failures, network partitions, and data inconsistencies becoming the norm rather than the exception. To ensure system reliability and data consistency, periodically creating a globally consistent snapshot of the entire system is a crucial technique. This snapshot can be used for state recovery after system failures, data auditing, state rollback, and system migration.

[0024] Traditional snapshot techniques, such as the Chandy-Lamport algorithm, coordinate the global state by sending marker messages to all processes. While this guarantees consistency, the synchronization process introduces significant pauses and delays. In systems with massive amounts of data and frequent state updates, taking a global snapshot can take several minutes or even longer, during which system performance degrades drastically. Furthermore, full snapshots consume substantial amounts of storage space and network bandwidth.

[0025] In modern financial transactions, real-time monitoring, online games, and other scenarios, system states change rapidly, requiring sufficiently fine-grained snapshots and a high enough frequency to capture critical system moments. Therefore, researching a low-overhead, high-performance, and high-frequency distributed snapshot method has significant theoretical and practical value.

[0026] To address this, the present invention provides a distributed system snapshot scheme that combines state change recording (event sourcing) with periodic state persistence (incremental checkpointing). Through a decentralized coordination mechanism and asynchronous pipeline operation, it achieves high-performance, low-latency snapshot acquisition.

[0027] like Figure 1 As shown, the present invention provides a distributed system snapshot method, including: Step S11: Generate corresponding event objects based on the state change operations of each node in the distributed system.

[0028] In this embodiment, in response to a state change operation occurring on any node in the distributed system, an event object corresponding to the state change operation is generated. The event object is asynchronously published to the event bus and persistently stored in the event log. Each event object carries an event identifier, event type, current node identifier, and the logical timestamp of the current node. The logical timestamp is used to maintain a vector clock for the causal order of event objects in the distributed system. It can be understood that the distributed system includes various nodes, an event bus, a snapshot coordinator, and a distributed object store. Each application node in the distributed system immediately generates an immutable event object when executing business logic and causing a change in its local state. Specifically, when a node receives an external request or internal message and completes a state change operation, the event generator inside the node creates an event object based on that state change operation. The event object should contain at least: a globally unique event ID, an event identifier to distinguish different events, an event type to describe the type of state change, such as "key-value pair update" or "counter increment"; a current node identifier, a logical clock value maintained by the current node at the time of the state change, i.e., a logical timestamp; and payload data that records the specific content of the change, such as key, old value, and new value.

[0029] Once an event object is generated, the node does not block subsequent business processing. Instead, it asynchronously sends the event object to the distributed event bus. The event bus uses a publish / subscribe pattern, allowing all components that need to listen for events (such as other nodes, snapshot coordinators, and log collectors) to subscribe to relevant events. The asynchronous publishing process is as follows: the node places the event object into its local send queue, and a background thread independently sends the events in the queue in batches or individually to the event bus. The send operation does not wait for confirmation, and the node continues to process subsequent requests. This asynchronous non-blocking mechanism ensures that the impact of event recording operations on normal request processing is minimized.

[0030] When the event bus receives an event object, it persists it to a distributed log storage (such as Apache Kafka, a distributed file system, or a log directory in object storage). Each event is appended to the corresponding node's or global event log file during persistence, and a write timestamp is recorded. Only after persistence is complete is the event considered committed and available for subsequent snapshot recovery or state rollback. By storing the event log in a reliable backend storage, the system state can be reconstructed by replaying events from the log, even if all nodes fail simultaneously.

[0031] To maintain the causal order of events in a distributed system, each node maintains a logical clock. In this invention, a vector clock is used as the implementation of the logical timestamp. Each node i maintains a vector VC[i] of length N (N being the total number of nodes), where VC[i][j] represents the number of events that node i knows have occurred at node j. When a node generates an event due to a state change, it increments the value of its own dimension in its vector clock by 1 and uses this vector as the logical timestamp of the event. When a node receives an event from another node, it updates its own vector according to the partial order rule of the vector clock: taking the maximum value in each dimension and then incrementing its own dimension by 1. In this way, the causal order of any two events can be determined by comparing their vector clocks: if all dimensions of the vector clock of event A are less than or equal to the vector clock of event B, and at least one dimension is strictly less than, then A is a causal predecessor of B; otherwise, they are concurrent events. This vector clock mechanism provides a precise causal order basis for snapshot coordination, conflict detection, and state merging.

[0032] Consider a distributed system with three nodes (A, B, and C) as an example. Initially, the vector clock of each node is [0,0,0]. When node A performs a state change operation (writing the key-value pair "x=1"), A generates event E1 with a vector clock of [1,0,0], and asynchronously publishes E1 to the event bus and persists it. Subsequently, node B receives an external request to update its state and generates event E2 with a vector clock of [0,1,0]. If node A depends on an event previously issued by node B when processing subsequent requests, A will merge the vector clocks after receiving B's event, thus ensuring the transitivity of causal order. The event bus and persistent logs guarantee the reliable propagation and storage of these events, providing a complete and ordered change record for subsequent incremental snapshots and fault recovery.

[0033] Step S12: In response to the snapshot triggering condition, each node creates a local incremental checkpoint and records the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks whose state has changed since the last snapshot, recorded by the event object.

[0034] In this embodiment, the snapshot coordinator initiates a snapshot operation based on a preset time interval or an event count threshold, and broadcasts a snapshot start message to each node. Upon receiving the snapshot start message, each node obtains its current logical timestamp and creates a local incremental checkpoint. It is understood that one or more snapshot coordinator instances are configured in the distributed system to manage the initiation and coordination of global snapshots. The snapshot coordinator automatically triggers snapshot operations according to a preset scheduling strategy. This scheduling strategy includes, but is not limited to, time interval triggering and event count threshold triggering. Time interval triggering means the coordinator maintains a timer, and a new snapshot operation is initiated every preset time interval (e.g., 1 second, 500 milliseconds, or less). The length of the time interval can be dynamically configured according to the system's recovery point objective (RPO) requirements. Event count threshold triggering means the coordinator listens for state change events on the event bus, and immediately triggers a snapshot operation when the cumulative number of events occurring globally or on a single node reaches a preset threshold (e.g., 10,000 changes). When the triggering condition is met, the coordinator assigns an incrementing sequence number (e.g., the Nth snapshot) to this round of snapshots and begins the snapshot initiation process. The snapshot coordinator generates a "Start Snapshot" message, denoted as Begin_Snapshot_N. This message contains at least the following fields: snapshot sequence number N, start timestamp (the coordinator's local time), and the current epoch of the global logical clock (if maintained). The coordinator then publishes the Begin_Snapshot_N message to the event bus. Because the event bus uses a publish / subscribe model, all application nodes subscribed to snapshot-related topics receive the message in real time. The broadcast operation itself is asynchronous; the coordinator can continue subsequent processing without waiting for node confirmation, thus avoiding blocking.

[0035] Each application node in the distributed system subscribes to the snapshot control message channel upon startup. When a node receives the Begin_Snapshot_N message, it immediately reads the current value of its own maintained logical clock (or vector clock) as the node's "snapshot logical timestamp" at the moment of this snapshot, i.e., records the current logical timestamp. This timestamp will be used for subsequent global consistency determination.

[0036] It's important to note that for scenarios requiring strong consistency, nodes can briefly pause processing new state change events to ensure that subsequently serialized states accurately correspond to the logical timestamps of the records. Since the entire pause process typically takes only microseconds to milliseconds, its impact on system throughput is negligible. For eventual consistency scenarios, this step can be omitted.

[0037] After obtaining the current logical timestamp, each node independently creates its own local incremental checkpoint. The specific steps are as follows: Each node maintains a dirty data bitmap, recording which data blocks in memory have been modified since the last successful snapshot (i.e., the (N-1)th snapshot). This bitmap is updated in real-time by previous state change events (recorded via event sourcing). Each node only traverses the data blocks marked as dirty in the dirty data bitmap, serializing them into binary format to form an incremental checkpoint file, i.e., a local incremental checkpoint. This file does not contain unchanged data blocks, so its size is much smaller than a full checkpoint. The node asynchronously uploads the generated incremental checkpoint file to a distributed object storage (e.g., Amazon S3, HDFS, MinIO, etc.) and obtains the storage address (e.g., URL or object key). The upload operation runs in parallel with the business processing thread, without blocking normal requests. The node saves the storage address of this checkpoint, its serialized size, and the previously recorded current logical timestamp in its local cache, ready to send it to the coordinator.

[0038] For example, a distributed system consists of three nodes A, B, and C, with a preset snapshot interval of 1 second. The coordinator initiates the 5th snapshot at time T0, broadcasting the Begin_Snapshot_5 message to the event bus. Nodes A, B, and C receive this message almost simultaneously: Node A's current logical clock value is [105, 98, 102]. It pauses new event processing for 0.5 milliseconds, reads the dirty data bitmap, and finds that three data blocks have been modified since the 4th snapshot. It serializes these blocks and uploads them to object storage, obtaining the address "s3: / / snapshots / app / 5 / A.inc". Node B's current logical clock value is [104, 100, 101]. Similarly, it creates an incremental checkpoint and uploads it. Node C's current logical clock value is [106, 99, 103]. It performs the same operation. Subsequently, each node sends a completion message (containing its own logical clock value and checkpoint address) back to the coordinator for determining the global consistency time point. Throughout the process, the node's normal business processing is only paused for a few microseconds at the moment the snapshot is started, and the overall performance is almost unaffected.

[0039] In this embodiment, if any node fails to create a local incremental checkpoint or times out during the snapshot period, the snapshot coordinator marks this round of snapshots as failed and abandons it, then starts the next round of snapshots. It is understood that the snapshot coordinator also implements a snapshot failure handling mechanism to ensure that the entire system's snapshot process does not get blocked or wait when individual nodes experience temporary failures or network timeouts, thus guaranteeing the high-frequency continuity of snapshots. Specifically, when the snapshot coordinator starts the Nth round of snapshots according to a preset time interval or event quantity threshold and broadcasts the start snapshot message to all nodes, the coordinator enters a state of waiting to collect completion messages from each node. The coordinator sets a timeout timer for each round of snapshots, and the timeout period can be dynamically configured based on the system's average snapshot completion time (e.g., 1.5 times the preset snapshot period or an absolute time limit). During a snapshot period, if any of the following occurs, resulting in an anomaly during node internal serialization, a disk write error, insufficient memory causing the node to fail to create a local incremental checkpoint, a network partition, or a slow response from the object storage service, preventing the node from successfully uploading the checkpoint file and sending a completion message within the timeout period (i.e., the node timed out of checkpoint upload, the node process crashed, or the node itself failed to respond to any messages), the coordinator determines that this round of snapshots has failed. The coordinator marks the Nth round of snapshots as "failed" in persistent metadata (e.g., a metadata database or snapshot metadata file) and may selectively record the reason for the failure (e.g., "Node A timed out" or "Node B checkpoint creation anomaly"). The marked snapshot will not be used for any subsequent fault recovery or state rollback operations. Simultaneously, the coordinator abandons this round of snapshots; that is, it will not attempt to force completion, partial recovery, or retry this round of snapshots. After abandonment, the coordinator will clean up the temporary state related to this round of snapshots (e.g., the temporarily stored completion message count). Then, the coordinator will not wait or sleep due to the failure of this round of snapshots. Instead, it will start the N+1th round of snapshots according to the preset scheduling policy (e.g., immediately or at the original time interval). The new round of snapshots will rebroadcast the start snapshot message, and each node will try to create a local incremental checkpoint again. Since the failure of the previous round did not affect the node's event logging and normal business processing, the new round of snapshots can be executed normally.

[0040] It's important to note that even if a snapshot fails, the system's state integrity and data consistency remain unaffected. This is because the event log is continuously recording data; all state change operations on each node are generated as immutable events and asynchronously persisted to the event log. Snapshots serve only as a means to accelerate state recovery, not as the sole source of data. When a snapshot fails, the system can still start from the last successfully completed snapshot (the (N-1)th snapshot) and, combined with subsequent changes recorded in the event log, fully restore the system state. Therefore, the strategy of discarding failed snapshots and immediately starting the next snapshot round is both safe and efficient.

[0041] Through the above-mentioned snapshot failure handling mechanism, while ensuring consistency, the snapshot process blockage or system stoppage caused by temporary failure of individual nodes is effectively avoided, thereby supporting the continuous operation of high-frequency snapshots at the second or even sub-second level, and improving the fault tolerance and availability of the distributed system.

[0042] In this embodiment, the snapshot failure handling includes: if any node fails to create a local incremental checkpoint or times out during the snapshot period, the snapshot coordinator will retry the node a limited number of times; if the node still fails or times out after the retry, the current snapshot round is marked as a failure and abandoned, and the next snapshot round is started. It is understood that a limited retry strategy is introduced to balance snapshot success rate and system response speed. Specifically, when the snapshot coordinator does not receive a completion message from a node within a preset timeout period, or receives a failure indication from the node, the coordinator does not immediately abandon the current snapshot round, but initiates a retry for the node. The retry method involves the coordinator resending the start snapshot message to the node, and the node retrying to create a local incremental checkpoint and upload it. The number of retries is configurable, and can be configured to 2 to 3 times. The timeout for each retry can be increased progressively (e.g., 1 second for the first retry, 2 seconds for the second retry) to cope with instantaneous network jitter or brief periods of high node load. If a node fails to create a checkpoint after reaching the maximum number of retries, the coordinator determines that this round of snapshots has failed, marks the node as a faulty node, records the failure information, then abandons this round of snapshots and immediately starts the next round of snapshots. Meanwhile, the system relies on continuously recorded event logs to ensure state integrity and does not affect normal business operations.

[0043] This approach effectively addresses temporary faults such as occasional network jitter and instantaneous increases in node load, preventing the entire snapshot round from failing due to brief anomalies and thus improving the overall success rate in high-frequency snapshot scenarios. The number of retries and timeout are configurable, typically adding only millisecond-level latency without affecting the second-level snapshot frequency target. If a retry still fails, the system promptly abandons the attempt and starts the next round, avoiding infinite waiting or blocking. Combined with the event log fallback mechanism, this ensures continuous system availability.

[0044] Step S13: Collect the local incremental checkpoints and current logical timestamps of each node to determine the global consistency time point based on the current logical timestamp, and generate snapshot metadata to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers pointing to the previous snapshot in the snapshot metadata.

[0045] In this embodiment, after the snapshot coordinator broadcasts the start snapshot message, each node independently creates a local incremental checkpoint and records the current logical timestamp at the time of creation. After each node completes checkpoint creation, it sends a completion message (named Complete_Snapshot_N) to the snapshot coordinator. This completion message includes at least a node identifier representing the node ID that sent the message, a unique access path (URL or object key) representing the node's access from object storage after uploading the checkpoint file (i.e., the incremental checkpoint storage address), and the current logical timestamp (the logical clock value read by the node when creating the checkpoint; for vector clocks, this is the vector maintained by the node). After starting this round of snapshots, the snapshot coordinator enters a collection state, maintaining a list of completed nodes. Upon receiving a completion message from a node, it records the logical timestamp and checkpoint address it carries. The coordinator continues to wait until it receives completion messages from all nodes or the preset timeout expires. In this way, the coordinator obtains the state snapshot positions of all nodes globally in this snapshot and their corresponding logical timestamps.

[0046] After collecting completion messages from all nodes (or processing received messages after a timeout), the snapshot coordinator determines a globally consistent time point based on the current logical timestamps reported by each node. In this invention, the globally consistent time point is the minimum value among the current logical timestamps reported by all nodes. If the system uses a scalar logical clock, the coordinator directly compares the logical timestamp values ​​reported by each node and takes the minimum as the globally consistent time point T. If the system uses a vector clock, the coordinator compares the globally minimum dimension of each node's vector (taking the minimum value reported by all nodes for each dimension) to form a globally consistent vector, which can be converted into a scalar representation or directly used as the consistency point. The physical meaning of this minimum value T is that all events that occurred before time T (events with logical timestamps less than or equal to T on all nodes) have been fully captured in this round of snapshots. This is because each node records its current logical timestamp when creating a checkpoint, which represents the sequence number of the events processed by that node. Taking the minimum value ensures that no events after T are mistakenly included, and all events before T are reflected in their respective states. Therefore, this globally consistent point in time ensures the causal consistency of the snapshot, satisfying the feasibility requirement of global snapshots in distributed systems.

[0047] After determining the global consistency time point, the snapshot coordinator generates corresponding snapshot metadata for this round of snapshots and persists it to a reliable metadata database (a metadata directory of a distributed key-value store, relational database, or object store). The snapshot metadata includes at least the unique ID of this round of snapshots (snapshot sequence number N or UUID), i.e., the snapshot identifier, the global consistency time point, the storage location of incremental checkpoints for each node (a mapping table with node ID as the key and checkpoint address as the value), and a dependency pointer to the previous snapshot (the identifier of the previous successful snapshot (the (N-1)th snapshot) or its metadata storage address). In addition, the metadata may optionally include auxiliary information such as the snapshot creation timestamp, snapshot status (e.g., "successful"), the size of each node's checkpoints, and checksums.

[0048] In this embodiment, when restoring using snapshots, a full snapshot is obtained and used as the base snapshot. Local incremental checkpoints are applied sequentially from the base snapshot, following the snapshot creation time order. Event objects recorded in the event log but not reflected in the state corresponding to the last local incremental checkpoint are also re-executed to restore the distributed system state. It can be understood that the dependency pointers in the snapshot metadata explicitly establish the dependency relationship between the current snapshot and the previous snapshot. Specifically, for the Nth snapshot, its metadata dependency pointer points to the metadata location of the (N-1)th snapshot (e.g., storage path or database primary key). When the system needs to perform state restoration, it can start from the latest snapshot (e.g., the Nth snapshot) and backtrack to the (N-1)th snapshot, the (N-2)th snapshot, and so on, until the base snapshot (full snapshot). During restoration, the steps are reversed, starting from the base snapshot and applying incremental checkpoints sequentially according to the snapshot creation time order. Through this chained dependency pointer design, this scheme achieves ordered association between incremental snapshots, avoiding the overhead of maintaining additional indexes or scanning all snapshots. Simultaneously, the dependency pointers ensure that each snapshot depends only on its direct predecessor, facilitating snapshot cleanup, merging, and version management.

[0049] Taking a system with three nodes (A, B, and C) as an example, after completing the second round of snapshots (full baseline snapshot), the third round of snapshots is currently underway. After each node creates an incremental checkpoint, the logical timestamps reported to the coordinator are: A:105, B:100, and C:102, respectively. The coordinator takes the minimum value of 100 as the global consistency time point T. Then, it generates metadata for the third snapshot, which includes T=100, the checkpoint addresses of each node (s3: / / snap / app / 3 / A.inc, etc.), and a dependency pointer snap-2 pointing to the second snapshot. This dependency pointer indicates that the third snapshot saves only the changed data blocks based on the second snapshot, and the second snapshot must be applied before the third snapshot is applied during recovery. In this way, the system can manage the incremental snapshot chain with minimal metadata overhead and support efficient point-in-time state recovery.

[0050] As can be seen from the above technical solution, by generating event objects corresponding to the state change operations of nodes, the change tracking basis is provided for local incremental checkpoints, avoiding blind scanning of the full state. Furthermore, by creating local incremental checkpoints, the problem of high storage overhead of full snapshots is solved. At the same time, by identifying changes through event objects, full scanning is avoided. In this way, each snapshot only saves the changed data blocks, significantly reducing storage space and network bandwidth consumption, and supporting high-frequency snapshots. Next, the current logical timestamp is recorded to provide a causal order basis for subsequent asynchronous coordination, avoiding global synchronization barriers. The relative order of snapshots can be determined without pausing business, reducing the impact on system performance.

[0051] like Figure 2 As shown, this invention also provides a specific distributed system snapshot method, with the following detailed process: First, a continuous event recording process, which is the normal request processing flow of the system and also the process of preparing basic data for the snapshot, including: Receiving and processing requests: Application nodes receive external requests and execute business logic. Updating memory state: Business logic may cause changes in the memory state of nodes. Generating events: After a state change, the node immediately generates a corresponding "state change event". This event contains detailed information about the change (such as the Key, old value, and new value in the Key-Value pair) and is assigned a logical timestamp by the current node. Asynchronously publishing events: The generated event is asynchronously published to the event bus. This operation is asynchronous and non-blocking to ensure that the impact on normal request processing is minimized. This process records the complete history of state changes. In the event of a system failure, even without the latest snapshot, the state can be reconstructed by replaying the event logs.

[0052] Then the periodic snapshot capture process begins. This is the process of actively creating consistent snapshots, triggered periodically by the snapshot coordinator (e.g., once per second). Triggering a snapshot cycle: The snapshot coordinator starts a new snapshot cycle, Snapshot_N, based on a preset time interval (e.g., 1 second) or an event count threshold. It first increments the epoch of the global logical clock, then broadcasts the Begin_Snapshot_N message to the event bus. Node reception and preparation: All application nodes subscribe to and receive the Begin_Snapshot_N message.

[0053] Furthermore, a local incremental checkpointing process is created. To obtain an accurate local view of the state, nodes can temporarily pause processing new events (typically on the microsecond level). For many eventually consistent systems, this step can be omitted in exchange for higher availability. The node serializes its in-memory state. A key optimization is that it only serializes data blocks modified since the last successful snapshot, Snapshot_{N-1}. This is achieved by maintaining a dirty data bitmap. The serialized incremental checkpoint file is asynchronously uploaded to a distributed object storage (such as S3 or HDFS) and assigned a unique file address.

[0054] Then, a completion notification is sent. Once the node has completed uploading the local checkpoint, it sends a Complete_Snapshot_N message to the snapshot coordinator, which contains the maximum logical timestamp of the node when the checkpoint was created.

[0055] Next, global consistency is confirmed through the coordinator. First, the coordinator collects Complete_Snapshot_N messages from all nodes. Upon receiving completion messages from all nodes, or after a timeout, the coordinator determines a globally consistent time point T based on the timestamps reported by all nodes. T is typically defined as the smallest timestamp among all node completion messages, ensuring that all events before T are included in this snapshot.

[0056] Then, the coordinator writes the metadata of this snapshot Snapshot_N to a reliable metadata database. The metadata includes: snapshot ID, global consistency time point T, storage addresses of all node checkpoint files, and pointers to the parent snapshot Snapshot_{N-1} (forming a chain structure).

[0057] Once Snapshot_N is confirmed, earlier snapshots (such as Snapshot_{N-3}) can be safely cleaned up according to the retention policy. The system continues to operate normally, preparing for the next snapshot cycle.

[0058] Thanks to incremental checking and asynchronous pipelines, this method supports very high snapshot frequencies (on the order of seconds) while having minimal impact on latency and throughput for normal business requests. The recovery process does not require replaying all historical events. Simply loading the latest baseline snapshot and applying a small number of subsequent incremental snapshots quickly restores the system to its most recent consistent state. Incremental snapshots significantly reduce storage space consumption and lower costs. The architecture is completely decentralized, with nodes operating independently. Adding or removing nodes does not significantly impact the snapshot mechanism itself, and the system is easily horizontally scalable.

[0059] like Figure 3As shown, the present invention also discloses a distributed system snapshot device, comprising: Object generation module 11 is used to generate corresponding event objects based on the state change operations of each node in the distributed system; The incremental checkpoint creation module 12 is used to create local incremental checkpoints for each node in response to snapshot triggering conditions, and to record the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks that have undergone state changes since the last snapshot, recorded by the event object. The dependency determination module 13 is used to collect local incremental checkpoints and current logical timestamps of each node in the snapshot collection, so as to determine the global consistency time point based on the current logical timestamp, and generate snapshot metadata, so as to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers pointing to the previous snapshot in the snapshot metadata.

[0060] Therefore, the entire snapshot process is designed as an asynchronous pipeline. Event logging, checkpoint uploading, and coordinator confirmation are all non-blocking steps. This allows snapshot operations to run in parallel with business processes to the greatest extent possible, achieving seamless integration with business performance. Logical / vector clocks are used to replace complex distributed locks or two-phase commit protocols. The coordinator does not need to block all nodes at the start of a snapshot; instead, it determines a globally consistent point in time after collecting completion messages. Each snapshot forms a dependency chain with the previous snapshot, and recovery starts directly from the most recent complete baseline snapshot through the dependency chain.

[0061] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the distributed system snapshot method disclosed in any of the foregoing embodiments. Furthermore, the electronic device in this embodiment may specifically be an electronic computer.

[0062] In this embodiment, the power supply 23 is used to provide operating voltage for various hardware devices on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0063] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0064] The operating system 221 is used to manage and control the various hardware devices on the electronic device and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the distributed system snapshot method executed by the electronic device as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0065] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned distributed system snapshot method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0067] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0068] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0069] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0070] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A snapshot method for a distributed system, characterized in that, include: Based on the state change operations of each node in the distributed system, corresponding event objects are generated. In response to the snapshot triggering condition, each node creates a local incremental checkpoint and records the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks whose state has changed since the last snapshot, recorded by the event object. Collect the local incremental checkpoints and the current logical timestamps of each node to determine the global consistency time point based on the current logical timestamps, and generate snapshot metadata to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers to the previous snapshot in the snapshot metadata.

2. The distributed system snapshot method according to claim 1, characterized in that, The generation of corresponding event objects based on the state change operations of each node in the distributed system includes: In response to a state change operation occurring on any node in the distributed system, an event object corresponding to the state change operation is generated; The event object is asynchronously published to the event bus and persistently stored in the event log; The event object carries an event identifier, an event type, a current node identifier, and a logical timestamp of the current node. The logical timestamp is used to maintain a vector clock for the causal order of event objects in the distributed system.

3. The distributed system snapshot method according to claim 1, characterized in that, In response to the snapshot triggering condition, each node creates a local incremental checkpoint, including: The snapshot coordinator initiates the snapshot operation based on a preset time interval or event quantity threshold, and broadcasts the snapshot start message to each node. After each node receives the start snapshot message, it obtains the current logical timestamp of each node and creates a local incremental checkpoint.

4. The distributed system snapshot method according to claim 1, characterized in that, The globally consistent time point is the minimum value among the current logical timestamps reported by all nodes.

5. The distributed system snapshot method according to claim 1, characterized in that, The snapshot metadata includes the snapshot identifier, the global consistency time point, the storage location of the local incremental checkpoints of each node, and dependency pointers.

6. The distributed system snapshot method according to claim 1, characterized in that, Also includes: Obtain a full snapshot and use it as a baseline snapshot; The local incremental checkpoints are applied sequentially from the baseline snapshot in the order of their creation time, and the event objects recorded in the event log that are not reflected in the state corresponding to the last local incremental checkpoint are re-executed to restore the state of the distributed system.

7. The distributed system snapshot method according to any one of claims 1 to 6, characterized in that, Also includes: If any node fails to create a local incremental checkpoint or times out during the snapshot period, the snapshot coordinator marks the current snapshot round as a failure, abandons the current snapshot round, and starts the next snapshot round.

8. A distributed system snapshot device, characterized in that, include: The object generation module is used to generate corresponding event objects based on the state change operations of each node in the distributed system. The incremental checkpoint creation module is used to create local incremental checkpoints for each node in response to snapshot triggering conditions, and to record the current logical timestamp when the node creates the local incremental checkpoint; the local incremental checkpoint contains data blocks that have undergone state changes since the last snapshot, recorded by the event object; The dependency determination module is used to collect the local incremental checkpoints and the current logical timestamps of each node in the snapshot collection, so as to determine the global consistency time point based on the current logical timestamps, and generate snapshot metadata, so as to determine the dependency relationship between the current snapshot and the previous snapshot based on the dependency pointers pointing to the previous snapshot in the snapshot metadata.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the distributed system snapshot method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the distributed system snapshot method as described in any one of claims 1 to 7.

Images