Method for production fault reproduction based on multi-level deterministic snapshots and temporal capsules

Abstract

The application discloses a production fault reproduction method and system based on multi-level deterministic snapshots and time capsules. The method comprises the following steps: deploying a lightweight snapshot agent on each node of a production system to build a three-layer snapshot collection system; capturing non-deterministic events in real time and forming event trajectories; configuring fault triggering rules, automatically triggering the time capsule mechanism when detecting abnormal indicators, and completing multi-level snapshot collection and event trajectory archiving; compressing, encrypting and incrementally storing data to an independent cluster; when reproducing a fault, reconstructing the production environment in a sandbox environment, recovering the state layer by layer in the order of dependent service level, system level and process level; starting the playback engine, injecting non-deterministic event trajectories according to the original time sequence through system call hijacking, and combining time virtualization technology to realize deterministic playback. The application can capture the full context at the fault moment with extremely low performance overhead, realize deterministic reproduction of production faults in an isolated environment, and greatly improve fault positioning efficiency.

Description

Technical Field

[0001] This invention relates to the field of software system operation and maintenance and fault diagnosis technology, and in particular to a method for reproducing production faults based on multi-level deterministic snapshots and time capsules. Background Technology

[0002] In the production and operation of large-scale distributed software systems, fault reproduction has always been a core challenge for developers and operations personnel. When anomalies or failures occur in the production environment, engineers often need to spend hours or even days locating the root cause of the problem, and the most time-consuming part of this is fault reproduction. Quickly and accurately reproducing production faults is of great significance for shortening the mean time to recovery (MTTR) and ensuring system stability.

[0003] In existing technologies, production fault diagnosis mainly relies on the following solutions: (1) Log analysis method: This method involves collecting log information from the system during runtime and combining it with manual analysis to infer the cause of the failure. However, log information is often discrete and incomplete, only recording preset key nodes and unable to fully reconstruct the complete context of the system's runtime. For occasional failures or failures triggered by boundary conditions, logs often cannot provide sufficient information.

[0004] (2) Traffic replay method: By recording network traffic in the production environment and replaying it in the test environment, the fault scenario can be simulated. However, traffic replay can only restore information at the network input layer and cannot restore the internal state of the system (such as memory state, cached data, thread scheduling state), so it cannot guarantee the deterministic reproduction of the fault.

[0005] (3) Debugger Attach: This method involves attaching a debugger to the production environment to diagnose problems in real time. However, this method can have a significant impact on the performance of the production system and may even cause the system to crash, making it extremely risky to use in a production environment.

[0006] The core shortcomings of existing technologies are: First, they cannot fully capture the entire runtime context of the system, including non-deterministic factors such as memory state, thread scheduling, random numbers, and time; second, they cannot achieve consistent snapshots of the state of multiple nodes in a distributed system; third, snapshot collection incurs too much performance overhead on production systems and cannot be deployed routinely; and fourth, they cannot guarantee the determinism of the execution path during playback, resulting in "the same input cannot reproduce the same problem".

[0007] Therefore, there is an urgent need for a technical solution that can capture the full context of a production environment failure with low performance overhead and achieve deterministic reproduction in an isolated environment. Summary of the Invention

[0008] In order to solve the problems existing in the prior art, the present invention provides the following technical solution.

[0009] The first aspect of this invention provides a method for reproducing production failures based on multi-level deterministic snapshots and time capsules, comprising: In response to a fault, a time capsule mechanism is automatically triggered. The time capsule mechanism includes: automatically collecting multi-level snapshots and sealing non-deterministic event trajectories within a preset time window of the fault occurrence to generate a fault time capsule; wherein, the multi-level snapshots include a system of process-level snapshots, system-level snapshots, and dependent service-level snapshots; the non-deterministic event trajectories are generated by capturing non-deterministic events in real time and serializing and storing them in timestamp order. The fault time capsules are compressed, encrypted, and versioned before being stored in a separate time capsule storage cluster. Incremental snapshot technology is used to store only the data that differs from the baseline snapshot. In response to a request to reproduce the fault, a runtime environment consistent with the production environment is automatically constructed in an isolated sandbox cluster based on the environmental metadata in the fault time capsule, and the consistency of the environment is ensured by content addressing hash verification. In the sandbox environment, snapshot states are restored layer by layer in the order of dependent service level, system level, and process level. Among them, the restoration of dependent service level snapshots is used to ensure the consistency of the state of external dependencies, the restoration of system level snapshots is used to ensure the consistency of kernel and network states, and the restoration of process level snapshots is used to ensure that the runtime state of the process is restored to the time when the failure occurred. By activating the replay engine, deterministic replay is performed based on the restored process state; by invoking hijacking technology, the captured non-deterministic event trajectories are injected into the execution process in the original time sequence, ensuring that all non-deterministic factors are consistent with the production environment; and by using time virtualization technology, the time elapsed perceived by the process is consistent with the original execution.

[0010] Preferably, the multi-level snapshots are collected in an asynchronous incremental manner: a full baseline snapshot is collected during system off-peak periods, and incremental snapshots are continuously collected; the snapshot collection process utilizes the operating system's resource isolation mechanism to isolate resources, ensuring that the CPU utilization of the production process is controlled within a preset threshold.

[0011] Preferably, eBPF technology is used to intercept system calls at the kernel level and capture the nondeterministic events in real time; for managed languages, nondeterministic events at the virtual machine level are captured through bytecode injection or interpreter hooks.

[0012] Preferably, a multi-level buffer mechanism is used to trigger the time capsule mechanism: a circular buffer is maintained in memory to continuously retain the trajectory of non-deterministic events within the most recent preset time period; when the triggering condition is met, the data in the circular buffer is immediately written to disk, and a snapshot is triggered to ensure that the complete context before and after the failure is captured.

[0013] Preferably, the triggering includes one or more combinations of the following triggering methods: threshold triggering, i.e., the error rate or response time exceeds a preset threshold; log triggering, i.e., the occurrence of error logs containing specific keywords; event triggering, i.e., process crash, memory overflow or system abnormality; and manual triggering.

[0014] Preferably, a globally consistent snapshot protocol is used to restore the snapshot state: a distributed clock synchronization and two-phase commit mechanism are used to ensure that the snapshots of multiple nodes are consistent in logical time; for cross-node network communication, packet capture and replay technology is used to restore the interaction timing between nodes.

[0015] Preferably, the deterministic replay includes: ensuring that concurrent programs achieve instruction-level deterministic replay by controlling at least one fine-grained factor among thread scheduling order, memory allocation address, and garbage collection triggering timing.

[0016] Preferably, the method further includes: verifying the consistency of the execution path based on the fingerprint information of the replay process and the fingerprint information of the original execution; if the verification fails, automatically rewinding and replaying to ensure the determinism of the replay result; the fingerprint information includes at least one of function call sequence, memory access pattern, and output result. Preferably, the method further includes: on the basis of deterministic playback, supporting at least one debugging operation among breakpoint setting, single-step execution, memory viewing and variable modification; while ensuring that the debugging operation does not destroy the determinism of the playback, and can be rolled back to any point in time for re-execution at any time.

[0017] A second aspect of the present invention provides a production failure reproduction system based on multi-level deterministic snapshots and time capsules, comprising: The multi-level snapshot acquisition module is used to deploy lightweight agents on each node of the production system to collect three-level snapshot data: process level, system level, and dependent service level. The nondeterministic event capture module is used to capture nondeterministic events during process execution in real time through system call interception technology, forming a nondeterministic event trajectory; The time capsule triggering module is used to automatically trigger the collection and sealing of time capsules when abnormal indicators are detected, according to preset fault triggering rules; The time capsule storage module is used to compress, encrypt, incrementally store, and manage versions of snapshot data and non-deterministic event trajectories. The sandbox environment management module is used to automatically build an isolated sandbox consistent with the production environment and provide environment consistency verification. The state recovery engine is used to restore multi-level snapshots in the order of dependent service level, system level, and process level, restoring the system state at the time of failure. A deterministic replay engine is used to achieve deterministic program execution based on the restored state through non-deterministic event injection and time virtualization.

[0018] The present invention has the following significant technical advantages over the prior art: (1) Deterministic reproduction: By combining multi-level snapshots, non-deterministic event trajectories and time virtualization, the complete context of the production failure can be accurately restored in an isolated environment, achieving true deterministic reproduction and solving the core pain point of "cannot be reproduced" in traditional solutions.

[0019] (2) Extremely low performance overhead: By adopting technologies such as asynchronous incremental snapshots, eBPF kernel-level interception, and resource isolation, the performance impact on the production system is controlled within 5%, and it can be deployed in the production environment normally, rather than being temporarily enabled only during troubleshooting.

[0020] (3) Complete context coverage: The three-layer snapshot system covers the complete context from the internal state of the process to the state of the system kernel and then to the state of external dependencies. Nondeterministic event capture covers all factors that may cause differences in execution paths, such as random numbers, time, and thread scheduling, to ensure that nothing is missed.

[0021] (4) Distributed system support: Through global consistency snapshot protocol and distributed clock synchronization, it supports fault reproduction of multi-node distributed systems and solves the problem of state consistency in distributed environment.

[0022] (5) Intelligent triggering and automatic sealing: The time capsule mechanism can automatically trigger data collection when a fault occurs without manual intervention, ensuring that the fault scene is completely preserved; even if the fault causes the process to crash, the complete context before the crash can be preserved through the circular buffer.

[0023] (6) Efficient storage and management: Incremental snapshot technology and compression algorithm greatly reduce storage overhead. Version management supports comparative analysis of multiple fault scenarios, which facilitates problem tracing and regression verification.

[0024] (7) User-friendly debugging experience: Based on deterministic playback, it provides complete interactive debugging capabilities, allowing engineers to debug production faults as if they were debugging local programs, greatly improving fault location efficiency and reducing mean time to recovery (MTTR) by more than 60%. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall system architecture provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a multi-level snapshot acquisition architecture provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the time capsule triggering and sealing process provided in this embodiment of the invention; Figure 4 This is a schematic diagram illustrating the working principle of a deterministic playback engine provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of a distributed system consistency snapshot provided in an embodiment of the present invention. Detailed Implementation

[0026] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0027] Example 1: System Overall Architecture like Figure 1 As shown, this embodiment provides a production failure reproduction system based on multi-level deterministic snapshots and time capsules. The overall architecture of the system includes two parts: the production environment side and the time capsule reproduction environment side.

[0028] The following components are deployed on the production environment side: (1) Multi-level snapshot acquisition module, which is deployed on each business node of the production system and runs as a lightweight user-mode agent. It is responsible for collecting three-level snapshot data at the process level, system level and dependent service level. This module adopts an asynchronous incremental acquisition method, collects full baseline snapshots during the system's low-peak period, and then continuously collects incremental snapshots. It uses Linux's Cgroup and Namespace mechanisms to isolate resources and ensure that the CPU utilization rate of the production process is controlled within 5%. (2) Nondeterministic event capture module, which intercepts system calls at the kernel level based on eBPF technology. It captures nondeterministic events such as random number generation, system time acquisition, network packet arrival order, thread scheduling switching, signal processing, and file I / O results during process execution in real time, and serializes and stores these events in the order of timestamps to form nondeterministic event trajectories. For managed languages ​​such as Java and Python, this module also captures nondeterministic events at the virtual machine level through bytecode injection or interpreter hooks. (3) Time capsule trigger module, which is configured with a variety of fault triggering rules. When the monitoring system detects abnormal indicators, it automatically triggers the time capsule mechanism. This module adopts a multi-level buffering mechanism, maintaining a circular buffer in memory to continuously retain the non-deterministic event trajectory of the most recent N minutes; when the triggering condition is met, the data in the circular buffer is immediately written to disk, and a snapshot is triggered to ensure that the complete context before and after the failure is captured. (4) The time capsule storage module is responsible for compressing, encrypting and versioning the collected multi-level snapshot data and non-deterministic event trajectory, and storing them in an independent time capsule storage cluster. It adopts incremental snapshot technology to store only the difference from the baseline snapshot, which greatly reduces storage overhead.

[0029] The Time Capsule reproduction environment is equipped with the following components: (1) Sandbox environment management module. When a fault needs to be reproduced, it automatically builds a running environment that is completely consistent with the production environment in the isolated sandbox cluster based on the environment metadata in the Time Capsule. This includes the operating system version, software dependencies, configuration parameters, feature switches, network topology, etc., and ensures the consistency of the environment through content addressing hash verification. (2) State recovery engine. In the sandbox environment, it restores the snapshot state layer by layer in the order of dependent service level, system level, and process level, ensuring that the state of each layer is accurately restored to the time when the fault occurred. (3) Deterministic replay engine. After startup, it continues to execute on the basis of the restored process state. Through system call hijacking technology, it accurately injects the captured nondeterministic event trajectory into the execution process in the original time order, ensuring that all nondeterministic factors are completely consistent with the production environment. Through time virtualization technology, it makes the time flow perceived by the process completely consistent with the original execution. (4) Execution path verification module (optional): During playback, fingerprint information of key execution paths (such as function call sequence, memory access mode, output results, etc.) is collected in real time and compared with the fingerprint of the original execution. If there is an inconsistency, the system will automatically roll back and replay. (5) Interactive debugging module (optional): Based on deterministic playback, it provides debugging capabilities such as breakpoint setting, single-step execution, memory viewing, and variable modification. It supports root cause location of faults and ensures that debugging operations will not destroy the determinism of playback. It can roll back to any point in time and re-execute at any time.

[0030] Example 2: Multi-level snapshot acquisition system like Figure 2 As shown, this embodiment details the specific implementation of the multi-level snapshot acquisition system. The multi-level snapshot acquisition system comprises three levels: process level, system level, and dependent service level. (1) Process-level snapshot: Used to capture the complete runtime state of the target process. Specifically, it includes: a complete image of the process's memory space, read from the process's virtual memory content via the / proc / [pid] / mem interface; CPU register state, obtained by calling the ptrace(PTRACE_GETREGS) system call to get the values ​​of all general-purpose registers and floating-point registers; thread scheduling state, obtained by reading the status information of each thread in the / proc / [pid] / task / directory to get the thread's scheduling policy, priority, running status, etc.; file descriptor table, obtained by reading the / proc / [pid] / fd / directory to get all file descriptors opened by the process and the file information they point to. The process-level snapshot uses copy-on-write technology, freezing the process for only a very short time (milliseconds) during the acquisition, and then asynchronously copying the memory pages to ensure that the impact on the production process is minimized.

[0031] (2) System-level snapshots: These are used to capture operating system-level state information. Specifically, they include: kernel network connection status (obtained by reading the five-tuple information of all network connections from / proc / net / tcp, / proc / net / udp, etc.); routing tables and ARP caches (obtained through the netlink interface); file system status (obtained by reading superblock information to get the file system's mount point, block size, usage, etc.); and system kernel parameters (obtained by reading parameters under / proc / sys / to get the currently effective kernel configuration). System-level snapshots also use an incremental approach, recording only the differences from the previous snapshot.

[0032] (3) Service-level snapshots: These are used to capture the consistent state of external services that a process depends on. Specifically, this includes: database state, obtained through database snapshot mechanisms (such as MySQL's FLUSH TABLES WITH READ LOCK + LVM snapshot, or MongoDB's db.fsyncLock() + file system snapshot); cache service state, exported through Redis's BGSAVE or Memcached's stats command; and message queue state, obtained through Kafka's consumer offset snapshot or RabbitMQ's queue state. The key to relying on service-level snapshots is ensuring consistency with process-level snapshots in logical time.

[0033] In terms of data acquisition scheduling, this embodiment adopts an asynchronous incremental approach: first, a full baseline snapshot is collected during system off-peak hours (such as 2-4 AM), and then incremental snapshots are continuously collected at configurable intervals (default 5 minutes). Incremental snapshots only record memory pages that have changed since the last snapshot (identified by the Soft Dirty Bit), newly added / disconnected network connections, modified files, etc. The snapshot agent process is restricted to independent CPU and memory control groups using Linux's Cgroup mechanism, ensuring that its CPU utilization does not exceed 5% and memory usage does not exceed 512MB.

[0034] Example 3: Time Capsule Triggering and Sealing Process like Figure 3 As shown, this embodiment describes in detail the triggering and sealing process of the time capsule.

[0035] The Time Capsule triggering module is configured with various fault triggering rules, including: (a) Threshold triggering: triggered when the error rate of the monitoring system exceeds a preset threshold (e.g., 5%) or the response time exceeds a preset threshold (e.g., 500ms); (b) Log triggering: triggered when error logs containing specific keywords (e.g., "OOM", "Segmentation Fault", "Panic") appear in the log system; (c) Event triggering: triggered when abnormal events such as process crashes, OOM Killer triggers, and kernel Panic are detected; (d) Manual triggering: operations and maintenance personnel can manually trigger Time Capsule collection through command-line tools or management interfaces. These triggering methods can be used individually or in combination.

[0036] The core mechanism of the time capsule is multi-level buffering: a circular buffer is maintained in memory, which continuously retains the non-deterministic event traces of the most recent N minutes (30 minutes by default). The circular buffer adopts a lock-free design and uses CAS (Compare-And-Swap) operation to implement a multi-producer single-consumer mode to ensure performance during high-concurrency writes. When the triggering condition is met, the system executes the following steps: (1) immediately writes the data in the circular buffer from memory to disk to local storage; (2) simultaneously triggers multi-level snapshot collection to freeze the memory state of the target process; (3) packages the non-deterministic event traces and multi-level snapshot data into a time capsule; (4) compresses (using the LZ4 algorithm), encrypts (using the AES-256-GCM algorithm), and manages the time capsule data (using timestamps and sequence numbers as version identifiers); (5) transmits the encrypted time capsule data to an independent time capsule storage cluster.

[0037] The time capsule storage cluster adopts a distributed storage architecture and supports horizontal scaling. Each time capsule contains the following metadata: capsule ID, generation timestamp, trigger reason, list of involved nodes, snapshot version number, event track size, and environmental metadata (operating system version, software dependency version, configuration parameter hash, etc.). Incremental snapshot technology is used for storage, storing only the differences from the baseline snapshot. Deduplication is achieved through content-addressable storage, ensuring that identical data blocks are stored only once.

[0038] Example 4: Deterministic Replay Engine like Figure 4 As shown, this embodiment describes in detail the working principle of the deterministic playback engine.

[0039] The workflow of a deterministic playback engine is as follows: The first step is environment reconstruction: The sandbox environment management module automatically builds the runtime environment in an isolated sandbox cluster based on the environment metadata in the time capsule. The specific process includes: pulling the corresponding operating system image from the image repository based on the operating system version number in the environment metadata; installing all dependent software and their specified versions using the package manager based on the software dependency list; restoring all configuration files from the production environment based on configuration parameter hashes; and rebuilding the same network topology in the sandbox based on the network topology information. After the build is complete, each component in the environment is verified using content addressing hashes to ensure complete consistency with the production environment.

[0040] The second step is state recovery: the state recovery engine restores the snapshot state layer by layer in the order of dependent service level, system level, and process level. First, it restores the dependent service level state: reconstructing the state of external dependent services such as databases, caches, and message queues to the state at the time of the failure based on the dependent service level snapshots. Then, it restores the system level state: restoring the kernel network connection state, routing table, file system state, etc., based on the system level snapshots. Finally, it restores the process level state: restoring the process's memory space, register state, thread scheduling state, and file descriptor table based on the process level snapshots, accurately restoring the process to its runtime state at the time of the failure. During the recovery process, the consistency of the state at each level is ensured by comparing the logical timestamps of each snapshot.

[0041] The third step, deterministic replay: After the deterministic replay engine starts, it continues execution based on the restored process state. The replay engine intercepts all system calls of the process using system call hijacking techniques (such as LD_PRELOAD or seccomp-BPF), injecting the captured non-deterministic event traces precisely in the original chronological order. Specifically, when the process calls gettimeofday() to get the time, the replay engine returns the time value recorded during the original execution, not the current time; when the process calls read() to read network data, the replay engine returns the network packets captured during the original execution, not the actual network data; when a thread scheduling switch occurs, the replay engine switches according to the original execution scheduling order; when the process calls getrandom() to get a random number, the replay engine returns the random number sequence recorded during the original execution. Through time virtualization technology, the replay engine maintains a virtual clock, making the time elapsed perceived by the process completely consistent with the original execution.

[0042] For concurrent programs, the replay engine employs instruction-level deterministic guarantees: by controlling fine-grained factors such as thread scheduling order (using a deterministic scheduler instead of the operating system scheduler), memory allocation address (using a custom memory allocator to ensure that the allocated address is consistent with the original execution), and garbage collection triggering timing (for managed languages, triggering GC at the precise location where GC occurs in the original execution), it ensures that even highly concurrent programs can achieve instruction-level deterministic replay.

[0043] The fourth step, execution path verification (optional): During replay, the execution path verification module collects fingerprint information of key execution paths in real time, including function call sequences (recording call stack changes through function entry / exit instrumentation), memory access patterns (recording access patterns through memory page access bits), and output results (return values ​​and output data of system calls). This fingerprint information is compared item by item with the fingerprint collected during the original execution. If inconsistencies are found, the system automatically reverts to the previous checkpoint and replays the data, ensuring the determinism of the replay results.

[0044] Example 5: Distributed System Consistency Snapshot like Figure 5 As shown, this embodiment describes in detail the global consistency snapshot protocol for distributed systems.

[0045] In distributed systems, the states of multiple nodes may have causal relationships. If each node takes snapshots independently, inconsistencies may arise between the snapshots (for example, node A's snapshot records the action of sending a message to node B, while node B's snapshot has not yet recorded receiving that message). To address this issue, this embodiment employs a globally consistent snapshot protocol: (1) Distributed clock synchronization: The hybrid logical clock (HLC) mechanism is adopted. Each node maintains a logical clock and carries the clock value when transmitting messages to ensure that the snapshots of each node can be aligned in logical time.

[0046] (2) Two-phase snapshot submission protocol: The snapshot acquisition process is divided into two phases—the first phase is the preparation phase: the time capsule triggering module sends a snapshot preparation request to all relevant nodes. After receiving the request, each node pauses message sending (but does not pause message receiving), records the current logical clock value, and returns an acknowledgment; the second phase is the execution phase: after all nodes confirm, the triggering module sends a snapshot execution command to each node, and each node simultaneously acquires a local snapshot and attaches the logical clock value. In this way, the snapshots of all nodes are consistent in logical time.

[0047] (3) Cross-node network communication restoration: Cross-node network communication that occurred during or before snapshot acquisition is restored using packet capture and replay technology. Specifically, in the production environment, each node captures all incoming and outgoing network packets using tcpdump or eBPF, recording the packet content, send / receive timestamps, source / destination addresses, and other information. During playback, the playback engine replays these network communications in the sandbox environment according to the original time sequence based on the captured network packet records, ensuring that the interaction time sequence between nodes is consistent with the production environment.

[0048] (4) Message queue state consistency: For scenarios involving asynchronous communication via message queues (such as Kafka), snapshots need to ensure that messages that have not yet been consumed in the message queue still exist during replay. In this embodiment, the consumption offset of each consumer is recorded as part of the snapshot, and the message queue is restored to the state at the time of failure based on the offset during replay.

[0049] Example 6: Interactive Diagnosis In addition to deterministic playback, this system also provides interactive debugging capabilities. The interactive debugging module supports the following operations: (1) Breakpoint setting: Supports setting breakpoints at specified function entry points, specified lines of code, and specified memory addresses. When playback reaches the breakpoint, it will automatically pause, and engineers can view the current memory status, variable values, call stack, and other information. Breakpoint setting is achieved by inserting INT3 instructions (x86 architecture) or BRK instructions (ARM architecture) at the target address.

[0050] (2) Single-step execution: Supports two modes: instruction-level single-step and source code-level single-step. Instruction-level single-step is achieved by setting the TF (Trap Flag) flag; source code-level single-step is achieved by mapping source code lines to instruction address ranges through debugging information (DWARF) and setting temporary breakpoints at the beginning of the next source code line.

[0051] (3) Memory viewing: Supports viewing the contents of any memory address and displaying them in multiple formats (hexadecimal, ASCII, structure, etc.). This is achieved by reading the process memory space restored in the sandbox environment, without affecting the playback process.

[0052] (4) Variable modification: Supports modifying variable values ​​while the system is paused. The modified value will affect the subsequent execution path, but the system records the state before modification through a snapshot mechanism to ensure that it can revert to the point before modification at any time and re-execute, ensuring that debugging operations will not destroy the determinism of playback.

[0053] (5) Time travel debugging: Supports jumping forward or backward to any point in time during execution. Jumping forward is achieved by restarting execution from the specified point in time through the replay engine; rolling backward is achieved by restoring to a previously saved checkpoint snapshot. This allows engineers to freely travel before and after the failure to accurately locate the root cause of the problem.

[0054] Example 7: Complete Workflow The following is a complete fault reproduction case to illustrate the workflow of this invention: Suppose that an e-commerce platform's order service experiences intermittent timeout failures in its production environment, occurring 2-3 times per day, each lasting approximately 5 minutes before automatically recovering. Traditional troubleshooting methods are unable to pinpoint the root cause. With the system of this invention, the workflow is as follows: Step 1 (Routine Deployment): Deploy a lightweight snapshot agent and a nondeterministic event capture module on all production nodes of the order service. The snapshot agent works asynchronously and incrementally, continuously collecting incremental snapshots after collecting baseline snapshots during off-peak periods, with CPU utilization controlled at 3-5%. The nondeterministic event capture module continuously captures system calls at the kernel level via eBPF, writing the nondeterministic event trajectory to a circular buffer in memory (retaining data from the most recent 30 minutes).

[0055] Step Two (Automatic Trigger): Configure the fault triggering rules as "response time exceeds 500ms for 1 minute" and "error log contains the keyword 'timeout'". When the fault occurs again, the monitoring system detects a spike in response time to 800ms and automatically triggers the time capsule mechanism. The traces of nondeterministic events in the most recent 30 minutes in the circular buffer are immediately written to disk, and multi-level snapshot collection is triggered simultaneously. The process is frozen within milliseconds and a memory snapshot is completed. The entire triggering and collection process has less than 5% impact on the production system.

[0056] Step 3 (Storage Management): The Time Capsule storage module compresses the collected snapshot data and event traces using LZ4 and AES-256 encryption. It then uses incremental storage to save only the parts that differ from the baseline snapshot (the actual incremental data volume is about 15% of the total data) and transfers the Time Capsule to an independent storage cluster.

[0057] Step 4 (Environment Reconstruction): The operations and maintenance personnel select the time capsule in the management interface, and the system automatically builds a running environment in the sandbox cluster that is completely consistent with the production environment, ensuring environment consistency through content addressing hash verification.

[0058] Step 5 (State Recovery): The state recovery engine recovers layer by layer in the order of dependent service level (restoring MySQL and Redis to the state at the time of failure), system level (restoring network connection and file system state), and process level (restoring process memory and thread state), accurately restoring the process to the runtime state at the time of failure.

[0059] Step Six (Deterministic Replay): The deterministic replay engine starts and continues execution based on the restored process state. Through system call hijacking, captured non-deterministic events are injected in their original chronological order—the original time value is returned when the process calls `gettimeofday()`, the original data packet is returned when the process reads data from the network, and the original thread switching occurs during thread scheduling. The execution path verification module compares the fingerprints of the replay path with the original path in real time to confirm that the replay results are consistent.

[0060] Step Seven (Interactive Diagnosis): During the deterministic replay, engineers set breakpoints, pausing at the critical function before the failure occurred. By examining memory and variable values, they discovered the root cause—a concurrent Map operation experienced a data race at a specific timeframe, leading to the loss of some order data. After confirming the root cause, the engineers fixed the code and verified the repair effect through time capsule replay again; the failure no longer recurred. The entire fault localization process was shortened from several days to less than 2 hours, reducing the mean time to repair (MTTR) by more than 60%.

[0061] Example 8: System Device Example This embodiment provides a production failure reproduction system based on multi-level deterministic snapshots and time capsules, including the following modules: (1) Multi-level snapshot acquisition module: A lightweight agent deployed on production nodes to collect three-level snapshot data at the process level, system level, and dependent service level. This module includes a process-level snapshot unit (capturing the memory state, register state, thread scheduling state, and file handle state of the target process), a system-level snapshot unit (capturing the operating system kernel state, network connection state, and file system state), and a dependent service-level snapshot unit (capturing the consistency state of external dependent services such as databases, caches, and message queues).

[0062] (2) Nondeterministic event capture module: Based on eBPF and system call interception technology, it captures all nondeterministic events during process execution in real time and forms an event trajectory. This module intercepts system calls at the kernel level without modifying the target process code; for managed languages ​​such as Java and Python, it also captures nondeterministic events at the virtual machine level through bytecode injection or interpreter hooks.

[0063] (3) Time Capsule Trigger Module: Based on preset fault rules, the module intelligently triggers the acquisition and storage of time capsules to ensure that the fault context is not lost. This module adopts a circular buffer mechanism to continuously retain the trajectory of non-deterministic events within the most recent preset time in memory; when a fault is triggered, the buffer data is immediately written to disk to ensure that the complete context before and after the fault is preserved. Fault triggering rules include one or more combinations of threshold triggering, log triggering, event triggering, and manual triggering.

[0064] (4) Time Capsule Storage Module: Responsible for the compression, encryption, incremental storage, and version management of snapshot data and event traces. It adopts the LZ4 compression algorithm and the AES-256-GCM encryption algorithm, achieves data deduplication through content-addressed storage, and uses incremental snapshot technology to store only the differences from the baseline snapshot.

[0065] (5) Sandbox Environment Management Module: Automatically builds an isolated sandbox consistent with the production environment and provides environment consistency verification. It builds the runtime environment based on the environment metadata in the time capsule, including the operating system version, software dependencies, configuration parameters and network topology, and verifies environment consistency through content addressing hashing.

[0066] (6) State recovery engine: Restores multi-level snapshots layer by layer in the order of dependent service level, system level, and process level to accurately restore the system state at the time of failure. For distributed systems, a global consistency snapshot protocol is adopted, and the snapshots of multiple nodes are kept consistent in logical time through distributed clock synchronization and two-phase commit mechanism.

[0067] (7) Deterministic Replay Engine: Based on the restored state, deterministic program execution is achieved through nondeterministic event injection and time virtualization. Nondeterministic event trajectories are injected in the original time sequence through system call hijacking technology, and time virtualization technology makes the time elapsed perceived by the process consistent with the original execution. For concurrent programs, deterministic replay at the instruction level is achieved by controlling the thread scheduling order, memory allocation address, and garbage collection trigger timing.

[0068] The system may also include an execution path verification module and an interactive debugging module as optional components. The execution path verification module collects fingerprint information of key execution paths in real time during playback, compares it with the fingerprint of the original execution, and automatically rolls back and replays if inconsistencies are found. The interactive debugging module provides debugging capabilities such as breakpoint setting, single-step execution, memory viewing, and variable modification on top of deterministic playback, and supports rewinding to any point in time for re-execution.

[0069] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for reproducing production failures based on multi-level deterministic snapshots and time capsules, characterized in that, include: In response to a fault, a time capsule mechanism is automatically triggered. The time capsule mechanism includes: automatically collecting multi-level snapshots and sealing non-deterministic event trajectories within a preset time window of the fault occurrence to generate a fault time capsule; wherein, the multi-level snapshots include a system of process-level snapshots, system-level snapshots, and dependent service-level snapshots; the non-deterministic event trajectories are generated by capturing non-deterministic events in real time and serializing and storing them in timestamp order. The fault time capsules are compressed, encrypted, and versioned before being stored in a separate time capsule storage cluster. Incremental snapshot technology is used to store only the data that differs from the baseline snapshot. In response to a request to reproduce the fault, a runtime environment consistent with the production environment is automatically constructed in an isolated sandbox cluster based on the environmental metadata in the fault time capsule, and the consistency of the environment is ensured by content addressing hash verification. In the sandbox environment, snapshot states are restored layer by layer in the order of dependent service level, system level, and process level. Among them, the restoration of dependent service level snapshots is used to ensure the consistency of the state of external dependencies, the restoration of system level snapshots is used to ensure the consistency of kernel and network states, and the restoration of process level snapshots is used to ensure that the runtime state of the process is restored to the time when the failure occurred. By activating the replay engine, deterministic replay is performed based on the restored process state; by invoking hijacking technology, the captured non-deterministic event trajectories are injected into the execution process in the original time sequence, ensuring that all non-deterministic factors are consistent with the production environment; and by using time virtualization technology, the time elapsed perceived by the process is consistent with the original execution.

2. The production failure reproduction method according to claim 1, characterized in that, The multi-level snapshots are collected using an asynchronous incremental approach: a full baseline snapshot is collected during system off-peak periods, and incremental snapshots are continuously collected; the snapshot collection process utilizes the operating system's resource isolation mechanism to ensure that the CPU utilization of the production process is controlled within a preset threshold.

3. The production failure reproduction method according to claim 1, characterized in that, eBPF technology is used to intercept system calls at the kernel level and capture the nondeterministic events in real time; for managed languages, nondeterministic events at the virtual machine level are captured through bytecode injection or interpreter hooks.

4. The production failure reproduction method according to claim 1, characterized in that, A multi-level buffering mechanism is used to trigger the time capsule mechanism: a circular buffer is maintained in memory to continuously retain the trajectory of non-deterministic events within the most recent preset time period; when the triggering condition is met, the data in the circular buffer is immediately written to disk, and a snapshot is triggered to ensure that the complete context before and after the failure is captured.

5. The production failure reproduction method according to claim 1, characterized in that, The triggering includes one or more combinations of the following triggering methods: threshold triggering, i.e., the error rate or response time exceeds a preset threshold; log triggering, i.e., the appearance of an error log containing a specific keyword; event triggering, i.e., process crash, memory overflow or system abnormality; and manual triggering.

6. The production failure reproduction method according to claim 1, characterized in that, A globally consistent snapshot protocol is used to restore the snapshot state: a distributed clock synchronization and two-phase commit mechanism are adopted to ensure that the snapshots of multiple nodes are consistent in logical time; for cross-node network communication, packet capture and replay technology is used to restore the interaction timing between nodes.

7. The production failure reproduction method according to claim 1, characterized in that, The deterministic replay includes ensuring that concurrent programs achieve instruction-level deterministic replay by controlling at least one fine-grained factor among thread scheduling order, memory allocation address, and garbage collection triggering timing.

8. The production failure reproduction method according to claim 1, characterized in that, The method further includes: verifying the consistency of the execution path based on the fingerprint information of the replay process and the fingerprint information of the original execution; if the verification fails, the process is automatically rolled back and replayed to ensure the determinism of the replay result; the fingerprint information includes at least one of function call sequence, memory access mode and output result.

9. The method for reproducing production failures according to claim 1, characterized in that, The method further includes: on the basis of deterministic playback, supporting at least one of the following debugging operations: breakpoint setting, single-step execution, memory viewing, and variable modification; while ensuring that the debugging operation does not destroy the determinism of the playback, and can be rolled back to any point in time for re-execution at any time.

10. A production failure reproduction system based on multi-level deterministic snapshots and time capsules, characterized in that, include: The multi-level snapshot acquisition module is used to deploy lightweight agents on each node of the production system to collect three-level snapshot data: process level, system level, and dependent service level. The nondeterministic event capture module is used to capture nondeterministic events during process execution in real time through system call interception technology, forming a nondeterministic event trajectory; The time capsule triggering module is used to automatically trigger the collection and sealing of time capsules when abnormal indicators are detected, according to preset fault triggering rules; The time capsule storage module is used to compress, encrypt, incrementally store, and manage versions of snapshot data and non-deterministic event trajectories. The sandbox environment management module is used to automatically build an isolated sandbox consistent with the production environment and provide environment consistency verification. The state recovery engine is used to restore multi-level snapshots in the order of dependent service level, system level, and process level, restoring the system state at the time of failure. A deterministic replay engine is used to achieve deterministic program execution based on the restored state through non-deterministic event injection and time virtualization.

Images