A dynamic backup and recovery method and system for full-stack data protection

Abstract

This invention relates to the field of data management technology and proposes a dynamic backup and recovery method and system for full-stack data preservation. The steps include: classifying full-stack data, which includes application logs, service program logs, operation logs, application configurations, service program configurations, critical operating system configurations, database data, and running application status data; setting differentiated backup strategies and cycles based on data type and business importance; performing automated backups and scheduling and managing backup tasks according to preset backup strategies; and when data recovery is required, matching the recovery process according to the backup type and performing the recovery operation. This invention unifies logs, configurations, databases, and running applications into a unified management framework and adopts an optimal strategy from backup and migration to recovery by automatically matching data types, ensuring the integrity, availability, and service continuity of business data.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and more specifically, to a dynamic backup and recovery method and system for full-stack data preservation. Background Technology

[0002] With the widespread adoption of cloud computing, big data, and microservice architectures, modern information systems are rapidly evolving towards distributed and continuous delivery. In this context, data has become a core asset, and its security and integrity are directly related to business continuity and the survival and development of enterprises.

[0003] Traditional data backup and recovery technologies mainly revolve around static data backup of file systems and databases. They typically employ a strategy that combines regular full backups with incremental backups, supplemented by physical media such as tapes and disks or early network storage for data preservation.

[0004] However, with cloud-native and containerization technologies becoming mainstream, traditional backup solutions have undergone fundamental changes in application architecture. Monolithic applications are now broken down into numerous microservices, dynamically orchestrated and run in containers. Traditional backup methods struggle to effectively capture the instantaneous memory state, network connectivity, and session information of running containers, leading to the loss of application state after recovery and compromising business continuity. Secondly, the complexity of system components and data types has increased dramatically. Logs, configuration files, databases, application code, and runtime states collectively constitute the full stack of data, with varying backup strategies and recovery targets, lacking unified management. Furthermore, to meet the demands of agile development and testing, production and testing environments often require frequent interaction. Traditional isolation methods are insufficient, easily leading to data corruption or accidental impacts on production systems.

[0005] While existing technologies include snapshot-based virtual machine backup, log-based database recovery, and CRIU (Checkpoint / Restore In Userspace)-based container checkpointing, each has its limitations: virtual machine snapshots have coarse granularity and high resource consumption; traditional backup tools have weak support for running applications, long recovery times, and significant impact on business interruptions; backup strategy management is fragmented, lacks automation and intelligence, relies on manual operation, is prone to errors, and is inefficient; cross-environment data migration often uses simple file copying, lacks end-to-end integrity verification and breakpoint resume mechanisms, and makes it difficult to guarantee data consistency.

[0006] In summary, there is an urgent need for a dynamic backup and recovery solution that can adapt to cloud-native architectures, cover all data types, achieve automated and intelligent management, ensure extremely low recovery time and recovery point targets, and safely and efficiently migrate data in complex network environments, so as to ensure the resilience and continuity of modern digital businesses when facing failures or changes. Summary of the Invention

[0007] In view of this, the present invention proposes a dynamic backup and recovery method and system for full-stack data preservation to solve the problems existing in the prior art.

[0008] To achieve the above objectives, this invention proposes a dynamic backup and recovery method for full-stack data preservation, comprising: The full-stack data is categorized, including application logs, service program logs, operation logs, application configuration, service program configuration, critical operating system configuration, database data, and running application status data. Set differentiated backup strategies and cycles based on data type and business importance; Perform automated backups and schedule and manage backup tasks according to preset backup policies; When data recovery is required, the recovery process is matched according to the backup type and the recovery operation is performed.

[0009] Furthermore, the specific details of setting differentiated backup strategies and cycles include: For log data, perform scheduled incremental backups and use encryption and multiple replica mechanisms for data storage; For configuration data, version management is performed through the configuration management center, and any changes will automatically trigger a backup snapshot; For database data, a combination of full backup and real-time transaction log streaming backup is used; For running applications, back up the application's static program files and dependency packages, and capture and save the application's running state.

[0010] Furthermore, the process of capturing and saving the running state of the application includes: The user-space process of the target container is suspended by a signal, and the eBPF program mount point used for this capture is registered with the kernel. The registered eBPF program is dynamically loaded into the Linux kernel, attached to kernel trace points and probes, and listens for and records system calls and their parameters that occur in the specified process tree after the capture time, forming an incremental log. By accessing kernel data structures, the real-time kernel state of the process can be read and serialized; Copy the incremental log and the real-time kernel state from kernel space to user space buffer; Send a security interrupt signal to the target process to trigger a pre-implanted security handling function to dump the contents of the user space heap, stack, and shared memory region, and collect container metadata; The kernel captures state and incremental logs, user space memory dumps, and container metadata, which are then integrated, serialized, encrypted, and compressed to generate snapshot files and stored.

[0011] Furthermore, the scheduling and management of backup tasks according to a preset backup strategy is specifically implemented using an event-driven workflow engine based on Kubernetes CronJob and Argo Workflows, including: Define the backup policy as a Kubernetes custom resource BackupPolicy and store it. The deployment policy controller monitors the BackupPolicy resource and publishes a backup task trigger event when the execution time window is reached; The event is captured by the workflow event source, and a predefined Argo workflow template is instantiated to execute the backup task. If the execution fails, it is automatically retried or an alarm is triggered according to the retry policy. Based on the retention policy in BackupPolicy, expired backup files are automatically cleaned up through a separate cleanup workflow.

[0012] Furthermore, the method also includes a secure migration step for backup data, specifically including: Initialize the Restic repository structure on the target side using the same encryption key as the source side; Read the encrypted data block packets and their hash tree structure associated with the selected backup set from the source repository and transmit them through an encrypted channel; During transmission, a checkpoint mechanism based on a list of data packet IDs is used to resume transmission from where it was interrupted, and content hash verification is performed on each data block that arrives at the target. After the transfer is complete, the index and metadata files of the source backup set are transferred to the target repository, and the target engine reconstructs a backup snapshot view that is consistent with the source.

[0013] Furthermore, the process of performing the recovery operation includes: Receive recovery requests and parse backup snapshot metadata to identify the backup type; If it is identified as a snapshot of a running application, a container checkpoint recovery process based on CRIU is triggered; After the state injection is complete, a continuation signal is sent to the container instance to perform a health check; After the health check passes, update the traffic configuration to switch to the newly restored container instance.

[0014] Furthermore, the CRIU-based container checkpoint recovery process includes: Locate and extract the encrypted checkpoint image file, decrypt and unpack it, and execute the checkpoint injection command in a blank container instance to reconstruct the process tree state that is completely consistent with the time of saving.

[0015] On the other hand, to achieve the above objectives, this invention proposes a dynamic backup and recovery system for full-stack data preservation, comprising: The data processing module is used to classify the data across the entire stack. The strategy management module is used to set and manage differentiated backup strategies and cycles based on data type and business importance; The backup execution engine is used to automatically schedule and execute backup tasks according to the backup policy; The data migration module is used to perform encrypted and secure migration of backup data; The recovery execution engine is used to match and execute the data recovery process based on the backup type. A visual management interface is provided to offer functions such as policy configuration, task monitoring, recovery operations, and resource viewing.

[0016] Furthermore, the backup execution engine specifically includes an event-driven workflow engine based on Kubernetes and Argo Workflows; The data migration module is implemented based on the Restic warehouse structure and a data block-level verification mechanism. The recovery execution engine employs CRIU-based container checkpoint recovery technology for runtime application snapshots.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention employs eBPF technology to capture the kernel-level real-time state of running containerized applications. It can directly read and serialize the core states of the process, such as registers, memory mappings, and file descriptors, and record incremental system call logs. This enables a complete backup of the application's running state, significantly reducing the interference of the backup process on business processes, shortening the application recovery time to the second level, and maintaining the continuity of memory sessions and incomplete transactions, thus achieving lossless business recovery.

[0018] This invention employs a Restic content addressing-based encrypted block mechanism for backup data migration. The system uses encrypted data blocks as transmission units and constructs a hash tree for integrity verification. Combined with checkpoints in the data packet ID list, it achieves precise breakpoint resumption, ensuring mathematical consistency of data during migration, improving the efficiency and reliability of cross-environment migration, and effectively supporting efficient off-site disaster recovery and data distribution.

[0019] This invention utilizes an event-driven workflow engine based on Kubernetes and Argo Workflows to declare backup policies as observable cloud-native resources. This enables full-process automation and resilience assurance of backup tasks, from scheduling, execution, and retrying to lifecycle management. It significantly reduces operational complexity, optimizes resource usage by automatically cleaning up expired backups through policy-driven processes, and improves the reliability and manageability of the entire backup system. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the dynamic backup and recovery method for full-stack data preservation according to the present invention. Detailed Implementation

[0021] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] This embodiment proposes a dynamic backup and recovery method for full-stack data preservation, such as... Figure 1 As shown, it includes the following steps: Based on the data type and business importance, differentiated backup strategies and cycles are set. Specifically, for log data, the system performs scheduled incremental backups and uses encrypted storage and multiple replica mechanisms to ensure the integrity and auditability of the logs.

[0023] For configuration data, version management is carried out through the configuration management center. Any change will automatically trigger a backup snapshot to ensure that the configuration is traceable and can be rolled back quickly.

[0024] Database backup employs a combination of full backups and real-time log streaming backups. The system periodically performs full database backups while continuously capturing database transaction logs. Backup data is stored in a distributed storage system, and all backup files are encrypted. When recovery is required, the system uses point-in-time precise recovery, capable of restoring the database to a specified state within minutes.

[0025] For running applications, the system not only backs up their static program files and dependencies, but also saves the application's runtime state through container snapshot technology. This technology can capture dynamic data, including memory state, session information, and runtime configuration, thereby enabling application restarts within seconds and ensuring business continuity.

[0026] Specifically, when using container snapshot technology to save the state of an application, the administrator selects the target container or Pod to be captured in its running state via the interface. After receiving the instruction, the system sends a request to the backup agent service on the node where the target container resides. The agent service first suspends signal transmission for all user-space processes within the target container, ensuring that the process tree is relatively static at the moment of capture. Simultaneously, the agent service registers a dedicated eBPF program mount point with the kernel for this capture.

[0027] The registered eBPF program is dynamically loaded into the Linux kernel and attached to key kernel trace points and probes. Once running, the program works by directly intercepting and copying the critical runtime state of the process from the kernel event stream. It performs two main tasks: first, it listens for and records all system calls and their parameters occurring after a specified process tree at that moment, forming an incremental log of pending operations; second, it accesses kernel data structures to directly read and serialize the process's real-time kernel state, including the register contents of all threads, the virtual memory mapping table, and the file descriptor table pointing to kernel objects such as open files and network sockets.

[0028] The above data is copied from kernel space to the user space buffer held by the proxy service.

[0029] After capturing the transient kernel-mode state, the agent service sends a security interrupt signal to the target process. Subsequently, the process enters a pre-implanted security handling function. This function, in cooperation with the agent service, rapidly dumps the contents of the user-space heap, stack, and shared memory regions it uses through a pre-established high-performance memory channel. Simultaneously, the agent service collects complete container metadata from the container runtime and cgroup files, including namespaces, cgroup hierarchy and restrictions, mount point information, and security context.

[0030] The proxy service integrates and serializes the three parts of data collected—kernel-captured state and incremental logs, user-space memory dumps, and container metadata. The system uses a custom, versionable, structured format for packaging. During packaging, all data is encrypted and compressed.

[0031] Finally, the generated single snapshot file is uploaded to the object storage repository for runtime backups, and the index and associated static program backup version information are recorded in the backup metadata database.

[0032] In a preferred embodiment, this method implements strict resource isolation between the production and testing environments. Isolation measures cover multiple levels, including physical storage, computing resources, network channels, access permissions, and data instances. Virtualization and network policies ensure that operations in the testing environment do not affect the security and stability of the production system.

[0033] Furthermore, the execution and management of backup tasks are highly automated. The system automatically schedules backup jobs according to preset policies and automatically retryes or issues alerts when a task fails. The lifecycle of backup tasks is automatically managed by the system, which periodically cleans up expired backup files based on retention policies to optimize storage resource usage.

[0034] Specifically, the system uses an event-driven workflow engine based on KubernetesCronJob and ArgoWorkflows to automate the scheduling, execution, and lifecycle management of backup jobs. All backup policies are abstracted and defined as custom resources within the cluster.

[0035] The specific steps are as follows: Administrators configure backup policies in the management interface, converting the policy content into a custom Kubernetes resource named BackupPolicy. This custom resource specifies the backup target, execution cycle, number of retries and intervals after task failure, associated retention policy, and corresponding alarm receiving group. The CRD is then committed and persistently stored in the Kubernetes etcd database.

[0036] Deploy a dedicated policy controller to continuously monitor the status of all BackupPolicy resources. When the execution time window defined in the policy is reached, the controller publishes a formatted backup task trigger event to the internal Kubernetes Event resource and listens for this bus through the workflow event source deployed by ArgoWorkflows. Upon capturing the event, it immediately instantiates a predefined parameterized Argo workflow template, transforming the scheduled request into a specific workflow execution instance.

[0037] The instantiated workflow strictly follows the steps defined in the template. The first step in the predefined steps is a pre-check to verify the availability of the target system and storage. The core backup execution step runs as a separate container, with the image containing backup tools such as Restic and Velero. The ArgoWorkflows engine manages the lifecycle of this container and records detailed logs. If this step container exits with a non-zero code, the engine automatically retryes the step according to the retry policy defined in BackupPolicy. If the retry exhaustion fails, the workflow status is marked as failed, and an alert step is automatically initiated. This step container calls the integrated alerting platform API to send an alert message containing the task ID and an error log summary to the specified receiving group.

[0038] For each successfully completed backup workflow, a dedicated lifecycle manager service is invoked in the final post-processing step. This service receives the snapshot ID and timestamp of this backup. Internally, the manager maintains a metadata index of all backup snapshots and continuously evaluates them according to the retention policy in the source BackupPolicy.

[0039] In addition, the system has a dedicated cleanup workflow that requests the lifecycle manager to list all expired backup snapshot IDs that have exceeded the retention policy period. For each expired snapshot, the cleanup workflow securely invokes the backup tool's native delete command to remove it from the storage repository and synchronously updates the metadata index. The entire cleanup process runs as a standard workflow and is also covered by monitoring and alerting mechanisms.

[0040] In a preferred embodiment, the system provides a visual backup management interface, through which administrators can view the execution progress, historical records, and storage resource usage of backup tasks in real time. Simultaneously, administrators can dynamically adjust backup strategies, including backup frequency, storage path, and retention period; all operations take effect immediately.

[0041] The secure migration function for backup data supports scenarios such as system upgrades, platform replacements, and disaster recovery drills. The system provides a variety of secure transmission protocols to choose from and implements integrity verification and breakpoint resumption mechanisms during the migration process to ensure data integrity during migration.

[0042] Specifically, the administrator selects the backup set to be migrated through a visual interface and specifies the target storage location. The system generates a globally unique migration task ID and initializes a brand new Restic repository structure on the target end that uses the same encryption key.

[0043] The migration engine reads all data packets related to the backup set from the source repository. Restic works by first splitting the file data into variable-length blocks, with each block's content using the BLAKE2b hash algorithm to generate a unique identifier. The data blocks are encrypted using AES-256-CTR mode before storage. During migration, the system transmits the encrypted data block packets and their associated hash tree structure. The transmission channel uses bidirectional TLS 1.3 encryption, forming a double-encrypted data transmission within the encrypted tunnel.

[0044] The transmission process employs a strict checkpoint mechanism, including: after each batch of data packets is successfully transmitted and verified, the system records a persistent checkpoint in the migration task status, containing a list of successfully transmitted data packet IDs. If the transmission process is interrupted due to network disruption, upon recovery, the engine compares the data packet list in the source repository with the existing data packet list in the target repository, queries the last successful checkpoint, and only transmits the missing data packets, achieving precise breakpoint resumption. Upon arrival at the target, each data packet is immediately hashed using the same encryption key and hash algorithm, and compared with the hash value in the data packet metadata. Any inconsistency immediately marks the data packet as corrupted and triggers retransmission, ensuring bit-level consistency for each data block.

[0045] After all data packets have been transmitted and verified, the system transfers the final files, including the snapshot index and metadata of the source backup set, to the target repository. The target repository's Restic engine uses the received index files to completely reconstruct a backup snapshot view in the new repository that is identical to the one from the source. The system then executes a final snapshot integrity verification command to confirm that the target backup set can be read and restored normally, updates the migration task status to success, and records the final audit log.

[0046] Understandably, this invention achieves security, reliability, and end-to-end integrity in the backup data migration process under complex network environments. By using encrypted data blocks as the smallest transmission and verification unit and combining them with hash trees, it ensures absolute mathematical consistency of data content from source to destination.

[0047] When data recovery is required, the administrator selects the target application or configuration snapshot to be recovered on the unified management interface and confirms the recovery time point. The system receives the recovery request, automatically parses the metadata of the backup snapshot, identifies its backup type as "running application snapshot", and then triggers the container checkpoint recovery process based on CRIU.

[0048] The system invokes the checkpoint recovery engine integrated into the container orchestration platform. This engine first locates the encrypted checkpoint image file storage location based on backup metadata, extracts the encrypted archive file from a specific namespace in the distributed object storage, and performs decryption and unpacking operations in memory. The recovery engine then executes the checkpoint injection command in a pre-prepared blank container instance.

[0049] CRIU technology, with the support of the operating system kernel, serializes the complete state of a running process tree, including memory pages, register contents, open file descriptors, signal handler states, and namespace information, into a series of disk image files. During recovery, the system precisely reconstructs the process tree memory mapping in a new container environment, perfectly matching the state at the time of saving, restoring all execution contexts and setting the process state to runnable. This allows applications to resume execution from the precise point in time where they were interrupted, preserving session data, cached state, and unfinished business transactions in memory, thus achieving data recovery and business continuity.

[0050] After the recovery engine completes state injection, it immediately sends a resume signal to the container instance. The system automatically executes a predefined health check script to verify the application service's port listening, internal state, and connections to dependent components. Upon successful verification, the system updates the service mesh or load balancer configuration, seamlessly switching production traffic to the newly recovered container instance, and records the final state and time of this recovery in the audit log. The entire recovery process is transparent, and administrators monitor it in real time until business operations are fully restored.

[0051] Understandably, the above methods effectively ensure the integrity, availability, and service continuity of business data by classifying full-stack data, formulating automated backup strategies, implementing environment isolation, providing a visual operation interface and secure migration capabilities, constructing a dynamic backup and recovery system that supports rapid recovery throughout the entire data lifecycle.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A dynamic backup and recovery method for full-stack data preservation, characterized in that, include: The full-stack data is categorized, including application logs, service program logs, operation logs, application configuration, service program configuration, critical operating system configuration, database data, and running application status data. Set differentiated backup strategies and cycles based on data type and business importance; Perform automated backups and schedule and manage backup tasks according to preset backup policies; When data recovery is required, the recovery process is matched according to the backup type and the recovery operation is performed.

2. The method according to claim 1, characterized in that, The specific provisions regarding setting differentiated backup strategies and cycles include: For log data, perform scheduled incremental backups and use encryption and multiple replica mechanisms for data storage; For configuration data, version management is performed through the configuration management center, and any changes will automatically trigger a backup snapshot; For database data, a combination of full backup and real-time transaction log streaming backup is used; For running applications, back up the application's static program files and dependency packages, and capture and save the application's running state.

3. The method according to claim 2, characterized in that, The process of capturing and saving the running state of the application includes: The user-space process of the target container is suspended by a signal, and the eBPF program mount point used for this capture is registered with the kernel. The registered eBPF program is dynamically loaded into the Linux kernel, attached to kernel trace points and probes, and listens for and records system calls and their parameters that occur in the specified process tree after the capture time, forming an incremental log. By accessing kernel data structures, the real-time kernel state of the process can be read and serialized; Copy the incremental log and the real-time kernel state from kernel space to user space buffer; Send a security interrupt signal to the target process to trigger a pre-implanted security handling function to dump the contents of the user space heap, stack, and shared memory region, and collect container metadata; The kernel captures state and incremental logs, user space memory dumps, and container metadata, which are then integrated, serialized, encrypted, and compressed to generate snapshot files and stored.

4. The method according to claim 1, characterized in that, The scheduling and management of backup tasks according to a preset backup strategy is specifically implemented using an event-driven workflow engine based on Kubernetes CronJob and Argo Workflows, including: Define the backup policy as a Kubernetes custom resource BackupPolicy and store it. The deployment policy controller monitors the BackupPolicy resource and publishes a backup task trigger event when the execution time window is reached; The event is captured by the workflow event source, and a predefined Argo workflow template is instantiated to execute the backup task. If the execution fails, it is automatically retried or an alarm is triggered according to the retry policy. Based on the retention policy in BackupPolicy, expired backup files are automatically cleaned up through a separate cleanup workflow.

5. The method according to claim 1, characterized in that, The method also includes a secure migration step for backup data, specifically including: Initialize the Restic repository structure on the target side using the same encryption key as the source side; Read the encrypted data block packets and their hash tree structure associated with the selected backup set from the source repository and transmit them through an encrypted channel; During transmission, a checkpoint mechanism based on a list of data packet IDs is used to resume transmission from where it was interrupted, and content hash verification is performed on each data block that arrives at the target. After the transfer is complete, the index and metadata files of the source backup set are transferred to the target repository, and the target engine reconstructs a backup snapshot view that is consistent with the source.

6. The method according to claim 1, characterized in that, The process of performing the recovery operation includes: Receive recovery requests and parse backup snapshot metadata to identify the backup type; If it is identified as a snapshot of a running application, a container checkpoint recovery process based on CRIU is triggered; After the state injection is complete, a continuation signal is sent to the container instance to perform a health check; After the health check passes, update the traffic configuration to switch to the newly restored container instance.

7. The method according to claim 6, characterized in that, The CRIU-based container checkpoint recovery process includes: Locate and extract the encrypted checkpoint image file, decrypt and unpack it, and execute the checkpoint injection command in a blank container instance to reconstruct the process tree state that is completely consistent with the time of saving.

8. A dynamic backup and recovery system for full-stack data preservation, characterized in that, The system for implementing the method as described in any one of claims 1 to 7 comprises: The data processing module is used to classify the data across the entire stack. The strategy management module is used to set and manage differentiated backup strategies and cycles based on data type and business importance; The backup execution engine is used to automatically schedule and execute backup tasks according to the backup policy; The data migration module is used to perform encrypted and secure migration of backup data; The recovery execution engine is used to match and execute the data recovery process based on the backup type. A visual management interface is provided to offer functions such as policy configuration, task monitoring, recovery operations, and resource viewing.

9. The system according to claim 8, characterized in that, The backup execution engine specifically includes an event-driven workflow engine based on Kubernetes and Argo Workflows; The data migration module is implemented based on the Restic warehouse structure and a data block-level verification mechanism. The recovery execution engine employs CRIU-based container checkpoint recovery technology for runtime application snapshots.

Images