A cloud database fault self-recovery system and method based on a resource arrangement engine

Abstract

The application belongs to the technical field of cloud computing and database, and particularly relates to a cloud database fault self-recovery system and method based on a resource arrangement engine, which comprises the following steps: collecting node operation indexes through a monitoring agent deployed on a database node; when the node operation indexes deviate from preset threshold values, analyzing fault types and affected data ranges by a fault diagnosis service to generate a fault diagnosis report; receiving the fault diagnosis report by a resource arrangement engine; sending an isolation instruction to a data processing engine by the resource arrangement engine, wherein the isolation instruction comprises an isolation strategy type and the affected data range; executing an isolation operation by the data processing engine; dynamically allocating virtual computing resources for each data subset by the resource arrangement engine, and sending a reconstruction task scheduling instruction to the data processing engine; and resuming the database service after the resource arrangement engine receives a verification pass signal. The application can shorten fault recovery time, improve data logical consistency, and ensure the security of original data.

Description

Technical Field

[0001] This invention belongs to the field of cloud computing and database technology, specifically relating to a cloud database fault self-healing system and method based on a resource orchestration engine. Background Technology

[0002] Cloud database services have become a core infrastructure for enterprises. Existing fault self-healing technologies can be mainly divided into three categories: 1. Traditional solutions based on master-slave failover (such as MySQL MHA), which cannot handle multi-node failures or data corruption; 2. Recovery solutions based on log replay (such as PostgreSQL PITR), which have excessively long recovery times for TB-level data and cannot handle log corruption; 3. Scheduling solutions based on container orchestration (such as Kubernetes Operator), which only solve node-level failures and cannot handle data shard corruption or cross-node transaction inconsistency issues.

[0003] Problems with existing technology: However, existing technologies have the following problems: 1. Static recovery strategies are inefficient, and full recovery is still performed when a small amount of data is corrupted; 2. Data reconstruction ignores transaction dependencies, and insufficient parallelism can easily lead to lock conflicts; 3. The verification mechanism only relies on basic checksums and cannot detect data inconsistencies at the logical level; 4. Resource scheduling and data operations are deeply coupled, which violates the principles of cloud-native architecture and results in poor system scalability. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud database fault self-healing system and method based on a resource orchestration engine, which can shorten fault recovery time, improve data logical consistency, and ensure the security of original data.

[0005] The specific technical solution adopted by this invention is as follows: A cloud database fault self-healing method based on a resource orchestration engine includes the following steps: The monitoring agent deployed on the database node collects node operation metrics in real time. When a node's performance metrics deviate from a preset threshold, the fault diagnosis service analyzes the fault type and the range of affected data, and generates a fault diagnosis report. The resource orchestration engine receives fault diagnosis reports and determines the isolation strategy type based on preset strategies; The resource orchestration engine sends isolation commands to the data processing engine. The isolation commands include the isolation policy type and the range of data affected. The data processing engine performs isolation operations: If a lightweight isolation strategy is adopted, the affected data range is replicated, and the difference data items are identified and isolated on the replicated data to generate multiple independent data subsets. If a snapshot isolation strategy is used, the storage service is invoked to create a read-only snapshot copy, and the boundaries of similar data items are marked in the metadata; The resource orchestration engine dynamically allocates virtual computing resources for each data subset and sends reconstruction task scheduling instructions to the data processing engine. The data processing engine performs data reconstruction on the allocated resources and returns the reconstruction status to the resource orchestration engine; The data processing engine reassembles the reconstructed subset of data to generate a recovered dataset and performs consistency verification. Once the resource orchestration engine receives the verification pass signal, it resumes the database service.

[0006] According to another aspect of the present invention, the resource orchestration engine determines the isolation policy type by: Receive data discrepancy metrics pre-calculated by the data processing engine; Based on the comparison between the data difference index and the preset threshold, a lightweight isolation strategy or a snapshot isolation strategy is selected.

[0007] According to another aspect of the present invention, the resource orchestration engine dynamically allocates virtual computing resources including: Based on the size of the data subset and the priority of the reconstruction task, an independent container instance is allocated to each subset, and a CPU / memory quota limit is set.

[0008] According to another aspect of the present invention, the resource orchestration engine dynamically adjusts the preset threshold, including: Obtain historical fault recovery records, which contain the correspondence between historical data differences and recovery times; A recovery time prediction model is constructed based on the aforementioned correspondence; When the predicted recovery time of the lightweight isolation strategy exceeds the service level agreement threshold, the preset threshold for data difference is automatically reduced, triggering the snapshot isolation strategy in advance.

[0009] According to another aspect of the present invention, the data processing engine performs data reconstruction including: Analyze the transaction dependencies between different data subsets and generate a reconstruction task dependency graph; The parallel reconstruction order is determined based on the reconstruction task dependency graph, prioritizing the reconstruction of data subsets with no dependencies. When a rebuild conflict is detected, the rebuild task for the conflicting subset is paused, and execution resumes after the dependencies are recalculated.

[0010] According to another aspect of the present invention, a cloud database fault self-healing system based on a resource orchestration engine is also provided, comprising: A distributed database cluster, consisting of multiple database nodes; The monitoring agent module is deployed on the database node and is used to collect node operation metrics; The fault diagnosis service module, connected to the monitoring agent module, is used to generate fault diagnosis reports; The resource orchestration engine, as an independent scheduling service, connects to the fault diagnosis service module and is used for: Receive fault diagnosis reports; Generate isolation policy instructions and resource allocation schemes; Schedule reconstruction tasks; The data processing engine module connects the resource orchestration engine and the distributed storage, and is used to perform data isolation, reconstruction and verification. The resource management interface connects the resource orchestration engine and the cloud infrastructure for implementing resource allocation.

[0011] According to another aspect of the present invention, the data processing engine module includes: Lightweight isolation units are used to perform one-to-one replication and differential data removal; The snapshot management unit is used to call the snapshot API of the storage service; The consistency verification unit is used to execute data verification algorithms.

[0012] According to another aspect of the present invention, the consistency verification unit performs multi-level verification: First-level verification: Perform checksums and comparisons on each reconstructed subset of data; Second layer of verification: Perform transaction log replay verification on the recombined recovery dataset; The third layer of verification: Before restoring the database service, perform shadow traffic tests to compare the consistency of responses between the old and new services.

[0013] According to another aspect of the present invention, a cloud database fault self-healing device based on a resource orchestration engine is also provided, comprising: The fault detection unit is used to monitor the operating status of the cloud database in real time. The resource scheduling unit, as a dedicated hardware acceleration module, is connected to the fault detection unit and is used to execute resource allocation decisions; A task coordination unit, connected to the resource scheduling unit, is used to generate task scheduling instructions; The data processing unit, connected to the task coordination unit, is used to perform data isolation and reconstruction; The verification control unit, connected to the data processing unit, is used to trigger the consistency verification process.

[0014] According to another aspect of the present invention, the data processing unit includes a hardware acceleration module: The hardware acceleration module uses an FPGA chip to implement a data difference calculation circuit, which processes the hash operations of multiple data blocks in parallel. The hardware acceleration module directly accesses the storage device through the DMA channel, bypassing CPU memory copying and reducing the latency of data isolation operations.

[0015] According to another aspect of the present invention, an electronic device is also provided, the electronic device including a memory and a processor; the memory is used to store a program; the processor executes the program to implement the method described in any one of the foregoing.

[0016] According to another aspect of the present invention, a computer-readable storage medium is also provided, the storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the preceding embodiments.

[0017] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the method described in any one of the preceding embodiments.

[0018] The technical effects achieved by this invention are as follows: This invention improves the efficiency and flexibility of cloud database fault self-healing by decoupling resource scheduling and data operations and introducing a dynamic strategy selection mechanism. The resource orchestration engine is positioned as an independent scheduling service, responsible only for decision-making and coordination, without directly manipulating business data, adhering to the separation of concerns characteristic of cloud-native architecture. Simultaneously, a dynamic threshold adjustment mechanism is employed. The resource orchestration engine builds a recovery time prediction model based on historical fault recovery records. When the predicted recovery time of the lightweight isolation strategy exceeds the service level agreement threshold, it automatically lowers the preset threshold for data discrepancies, triggering the snapshot isolation strategy in advance. This allows for intelligent selection of the optimal recovery strategy based on the severity of the fault, avoiding the low recovery efficiency caused by fixed thresholds, improving service availability, and reducing operational costs.

[0019] This invention addresses the issues of insufficient parallelism and imperfect verification mechanisms in data reconstruction by employing a dependency-aware parallel reconstruction mechanism and a multi-layered verification system. The data processing engine generates a reconstruction task dependency graph by parsing the operation sequences in the transaction log, prioritizing the execution of data subsets without dependencies. Upon detecting reconstruction conflicts, it automatically pauses conflicting tasks and recalculates dependencies, improving the stability and parallel efficiency of the reconstruction process. Furthermore, to address the shortcomings of existing solutions that rely solely on checksum verification and cannot detect logical data inconsistencies, this invention utilizes a three-layer progressive verification mechanism: the first layer verifies physical data integrity, the second layer verifies logical consistency through transaction log replay, and the third layer verifies the correctness of business semantics by performing traffic testing in a shadow environment. These three layers of verification are executed sequentially, and the process terminates upon failure at any stage, ensuring the quality of data recovery. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a diagram showing how the resource orchestration engine in this invention determines the isolation strategy type. Figure 3 This is a flowchart of the method for dynamically adjusting preset thresholds in the resource orchestration engine of this invention; Figure 4 This is a flowchart of the data reconstruction method executed by the data processing engine in this invention; Figure 5 This is a schematic diagram of the cloud database fault self-healing system structure in this invention; Figure 6 This is a schematic diagram of the data processing engine module structure in this invention; Figure 7 This is a structural diagram of the consistency verification unit in this invention; Figure 8 This is a schematic diagram of the cloud database fault self-healing device in this invention. Detailed Implementation

[0021] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0022] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] According to an embodiment of the present invention, a method embodiment of a cloud database fault self-healing method based on a resource orchestration engine is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] like Figure 1 As shown, a cloud database fault self-healing method based on a resource orchestration engine includes the following steps: S1. Real-time collection of node operation metrics through monitoring agents deployed on database nodes; S2. When a node's operating indicators deviate from a preset threshold, the fault diagnosis service analyzes the fault type and the range of affected data, and generates a fault diagnosis report. S3. The resource orchestration engine receives fault diagnosis reports and determines the isolation strategy type according to preset strategies. S4. The resource orchestration engine sends an isolation command to the data processing engine. The isolation command includes the isolation policy type and the range of data affected. S5, the data processing engine performs isolation operations: If a lightweight isolation strategy is adopted, the affected data range is replicated, and the difference data items are identified and isolated on the replicated data to generate multiple independent data subsets. If a snapshot isolation strategy is used, the storage service is invoked to create a read-only snapshot copy, and the boundaries of similar data items are marked in the metadata; S6, the resource orchestration engine dynamically allocates virtual computing resources to each data subset and sends reconstruction task scheduling instructions to the data processing engine; S7. The data processing engine performs data reconstruction on the allocated resources and returns the reconstruction status to the resource orchestration engine. S8, the data processing engine, reassembles the reconstructed subset of data to generate a recovered dataset and performs consistency verification; S9. Once the resource orchestration engine receives the verification pass signal, the database service is restored.

[0025] In step S1, the monitoring agent is deployed as a lightweight process at the operating system layer of each database node. The monitoring agent continuously collects key operating metrics, including CPU utilization, memory utilization, disk I / O throughput, network latency, connection pool status, and transaction processing rate. The monitoring agent uses a circular buffer to store the metric data. It only reports to the fault diagnosis service when the metric fluctuation exceeds the standard deviation threshold, thus avoiding false alarms caused by network jitter. The monitoring agent communicates with the database kernel through shared memory to reduce the impact of the collection process on database performance.

[0026] In step S2, after receiving the abnormal indicators reported by the monitoring agent, the fault diagnosis service first performs multi-dimensional correlation analysis. For hardware faults (such as a sudden increase in disk I / O error rate), the fault diagnosis service obtains the underlying storage health status by calling the storage driver interface. For software faults (such as an increase in transaction timeout rate), the fault diagnosis service parses the database kernel logs to locate the specific abnormal thread or lock contention point. After determining the fault type, the fault diagnosis service further analyzes the affected data range: for node-level faults, the affected range is all data shards carried by that node; for shard-level faults, the affected range is determined by analyzing the shard routing table; for transaction-level faults, the affected range is traced back to the relevant data pages through the transaction dependency graph. The fault diagnosis report contains structured data, clearly identifying the fault type code, the degree of impact, the list of affected data shards, and the time window.

[0027] In step S3, after receiving the fault diagnosis report, the resource orchestration engine determines the isolation strategy type based on a preset strategy. The preset strategy uses a decision tree model: when the amount of data affected by the fault is less than the set value of the total data volume, and the fault type is a transient error, a lightweight isolation strategy is selected; when the amount of data affected by the fault is greater than the set value of the total data volume, or the fault type is a persistent error (such as storage media damage), a snapshot isolation strategy is selected. The resource orchestration engine maintains a strategy configuration table and supports dynamic updates of strategy parameters. The strategy decision-making process is completed in memory, avoiding disk I / O latency.

[0028] In step S4, the resource orchestration engine generates standardized isolation instructions. Each isolation instruction includes a protocol version number, a unique instruction identifier, an isolation policy type identifier, a description of the affected data range, and a timeout period. Isolation instructions are transmitted via the gRPC protocol to ensure reliable and sequential transmission. The resource orchestration engine maintains a state machine for each isolation instruction, tracking the status of each stage: sending, acknowledgment, execution, and completion. If no acknowledgment response is received from the data processing engine within the timeout period, the resource orchestration engine automatically resends the instruction. If the number of retries exceeds a set limit, the alarm level is escalated.

[0029] In step S5, the data processing engine executes corresponding operations according to the isolation instructions. Under the lightweight isolation strategy, the data processing engine adopts a block processing mechanism: the affected data range is divided into blocks of varying sizes (e.g., 128MB, 256MB, 512MB, etc.), and an independent memory buffer is allocated to each block. Data copying is completed via asynchronous DMA transfer, and the original data remains in a read-only state. Checksum calculation is performed incrementally: the CRC32 value is recalculated only for modified pages (marked by the dirty bit in the page table). Difference data items are marked with bitmap indexes (1 indicates valid, 0 indicates invalid), and valid data items are reassembled into contiguous data subsets based on their original offsets. The number of subsets does not exceed twice the number of affected shards to avoid excessive sharding and scheduling overhead. Under the snapshot isolation strategy, the data processing engine calls the standard API of the cloud storage service to create a read-only snapshot copy. After the snapshot is created, the data processing engine parses the data shard topology information and marks the boundaries of similar data items in the metadata according to the range of shard key hash values, forming logically isolated data subsets. Both strategies ensure that the original data storage structure remains in a read-only state and no write operations are performed.

[0030] In step S6, the resource orchestration engine dynamically allocates virtual computing resources, including: allocating independent container instances to each subset based on the size of the data subset and the priority of the reconstruction task, and setting CPU / memory quota limits. The resource orchestration engine allocates virtual computing resources to each data subset, and the reconstruction task priority is divided into three levels: 1. Core business tables (such as user accounts and transaction records) are high priority; 2. Non-core business tables are medium priority; 3. Logs / archived data are low priority. Priorities are pre-configured through the data dictionary, and high-priority tasks can preempt resource quotas from low-priority tasks. The allocation algorithm comprehensively considers four factors: data subset size, reconstruction complexity, node load balancing, and task priority. For high-priority data subsets, multi-core CPUs and large memory containers are allocated; for low-priority subsets, lightweight containers are allocated. The resource orchestration engine maintains the resource pool status and prioritizes physical nodes with sufficient idle resources. The reconstruction task scheduling instruction includes resource allocation details, reconstruction algorithm identifier, dependency description, and priority identifier. Instructions are sent asynchronously through a message queue to ensure sequentiality in high-concurrency scenarios. The resource orchestration engine sets an independent resource quota limit for each rebuild task to prevent a single task from consuming excessive resources.

[0031] In step S7, the data processing engine performs a reconstruction operation within the allocated container resources. The reconstruction process consists of three stages: first, verifying the integrity of the data subset and repairing the damaged index structure; then, applying the transaction log to roll forward incomplete transactions; and finally, performing data format standardization. During the reconstruction process, the data processing engine monitors resource usage in real time. When CPU utilization consistently exceeds 90% or memory usage exceeds 80% of the quota, it automatically adjusts the reconstruction strategy to reduce resource consumption. The reconstruction status includes five states: initialization, running, paused, completed, and failed. Status changes are communicated to the resource orchestration engine in real time via callback functions, ensuring the scheduler is aware of the latest execution progress.

[0032] In step S8, the data processing engine reassembles the reconstructed data subsets into a complete dataset. The reassembly process employs a two-phase commit protocol: the first phase collects metadata from all subsets and verifies version compatibility; the second phase performs a physical merge to generate a unified data layout. The data processing engine triggers a consistency verification process, with verification results including three states: pass, partial failure, and complete failure. In the case of partial failure, only the failed subsets are reprocessed, without affecting the subsets that have already passed verification.

[0033] In step S9, after the resource orchestration engine receives the verification signal for all data subsets, it executes the service recovery process. The recovery process includes: updating the node status in the service registry; refreshing the client connection pool configuration; and gradually increasing traffic weight to the newly recovered node. The resource orchestration engine sets an observation period after recovery, during which key indicators are continuously monitored. Only when the indicators stabilize beyond a preset time window is the fault self-healing process officially completed. The entire recovery process is transparent to upper-layer applications; client connections are not interrupted, and clients only perceive a brief increase in latency.

[0034] As an optional embodiment, refer to Figure 2 As shown, the resource orchestration engine determines the isolation policy types including: S101, Data difference index pre-calculated by the received data processing engine; S102. Based on the comparison between the data difference index and the preset threshold, select either the lightweight isolation strategy or the snapshot isolation strategy.

[0035] Based on the above, when determining the isolation strategy type, the resource orchestration engine can receive a data discrepancy indicator pre-calculated by the data processing engine. This indicator is generated by comparing the checksum difference between the original data and the real-time replica, reflecting the proportion of abnormal data within the affected data range. The resource orchestration engine compares the received data discrepancy indicator with a dynamically configured preset threshold. When the discrepancy is below the threshold, a lightweight isolation strategy is selected; when the discrepancy reaches or exceeds the threshold, a snapshot isolation strategy is selected. The preset threshold is automatically adjusted based on historical fault recovery records and the current system load status, ensuring the adaptability of strategy selection. This allows the resource orchestration engine to make decisions based on the precise degree of data anomaly, avoiding misjudgments due to fixed thresholds and improving the efficiency and reliability of fault self-healing. The calculation of the data discrepancy indicator is completed asynchronously in the background by the data processing engine, without affecting the scheduling performance of the resource orchestration engine.

[0036] As an optional implementation, the resource orchestration engine dynamically allocates virtual computing resources by: allocating independent container instances to each subset based on the size of the data subset and the priority of the reconstruction task, and setting CPU / memory quota limits.

[0037] Based on the above, when dynamically allocating virtual computing resources, the resource orchestration engine comprehensively considers two key factors: the size of the data subset and the priority of the reconstruction task. The size of the data subset is determined by the number of storage bytes occupied by the computation subset, and the priority of the reconstruction task is graded according to the degree of impact of the fault and the importance of the business. The resource orchestration engine allocates an independent container instance for each data subset to ensure resource isolation between reconstruction tasks and avoid mutual interference. During the allocation process, the resource orchestration engine queries the current available resource pool status and prioritizes deploying container instances on physical nodes with sufficient resources. For large data subsets or high-priority tasks, more CPU cores and memory resources are allocated; for small subsets or low-priority tasks, the minimum necessary resources are allocated. At the same time, strict CPU and memory quota limits are set for each container instance to prevent individual reconstruction tasks from excessively consuming system resources and affecting the normal operation of other critical services. The resource quota limit is dynamically adjusted according to the overall system load to ensure that basic service quality can still be maintained under high load conditions.

[0038] As an optional embodiment, refer to Figure 3 As shown, the resource orchestration engine dynamically adjusts preset thresholds through the following steps: S201. Obtain historical fault recovery records, which contain the correspondence between historical data differences and recovery times; S202. Construct a recovery time prediction model based on the correspondence relationship; S203. When the predicted recovery time of the lightweight isolation strategy exceeds the service level agreement threshold, the preset threshold for data difference is automatically reduced to trigger the snapshot isolation strategy in advance.

[0039] Based on the above, the resource orchestration engine can dynamically adjust the preset thresholds used for strategy selection. The engine continuously collects and stores historical fault recovery records, which include the actual data variance indices and corresponding recovery time data for each fault recovery process. The resource orchestration engine uses this historical data to build a recovery time prediction model. This model employs time series analysis to establish a functional relationship between data variance and expected recovery time.

[0040] At runtime, the resource orchestration engine uses a predictive model to estimate the recovery time required to implement a lightweight isolation strategy under the current failure condition. When the predicted recovery time exceeds the service level agreement (SLA) threshold configured in the system, the resource orchestration engine automatically reduces the preset threshold for data variance, triggering a snapshot isolation strategy even in cases of minor data anomalies. This dynamic adjustment mechanism ensures that the system can automatically optimize strategy selection within SLA constraints, avoiding recovery time exceeding limits due to insisting on using a lightweight isolation strategy. The predictive model is periodically updated with new historical data to adapt to changes in system load and business model evolution, maintaining predictive accuracy.

[0041] As an optional embodiment, refer to Figure 4 As shown, the data processing engine performs data reconstruction through the following steps: S301. Analyze the transaction dependencies between each data subset and generate a reconstruction task dependency graph; S302. Determine the parallel reconstruction order based on the reconstruction task dependency graph, and prioritize the reconstruction of data subsets with no dependencies. S303. When a reconstruction conflict is detected, pause the reconstruction task of the conflicting subset, recalculate the dependencies, and then resume execution.

[0042] Based on the above, when performing data reconstruction, the data processing engine first analyzes the transaction dependencies between different data subsets. This analysis involves parsing the operation sequences in the transaction logs to identify data read / write dependencies between different data subsets, particularly cross-subset transaction references and lock contention. Based on the analysis results, the data processing engine generates a reconstruction task dependency graph in the form of a directed acyclic graph (DAG), where nodes represent data subsets and edges represent the directions of dependencies. The reconstruction task scheduler determines the parallel reconstruction order based on the dependency graph, prioritizing the execution of nodes with an in-degree of zero (i.e., data subsets without prerequisite dependencies) and dynamically releasing the execution rights of subsequent tasks after the prerequisite tasks are completed.

[0043] During the reconstruction process, the data processing engine monitors the task execution status in real time and maintains a global lock table, recording the lock status of each data subset. When a reconstruction task requires cross-subset operations, a two-phase locking protocol is used: In the first phase, locks are acquired in ascending order of subset ID; if any lock acquisition fails, the acquired lock is released and a retry is performed; in the second phase, the locks are released after reconstruction. Conflict detection is based on the lock wait graph. When a circular wait is detected, the task with the shortest execution time is selected for rollback to ensure maximum overall system progress. The rollback task is added to the retry queue and re-executed after resource release. At this time, the engine immediately pauses the reconstruction tasks of all conflicting subsets, re-analyzes the dependencies under the current data state, updates the reconstruction task dependency graph, and resumes execution according to the new dependency order. This avoids deadlock problems in parallel reconstruction, improves the stability and efficiency of the reconstruction process, and ensures the integrity and consistency of data reconstruction.

[0044] As an optional embodiment, refer to the appendix Figure 5 This paper provides a cloud database fault self-healing system based on a resource orchestration engine, including: A distributed database cluster, consisting of multiple database nodes; The monitoring agent module is deployed on the database node and is used to collect node operation metrics; The fault diagnosis service module connects to the monitoring agent module and is used to generate fault diagnosis reports; The resource orchestration engine, as an independent scheduling service, connects to the fault diagnosis service module for: Receive fault diagnosis reports; Generate isolation policy instructions and resource allocation schemes; Schedule reconstruction tasks; The data processing engine module connects the resource orchestration engine and distributed storage, and is used to perform data isolation, reconstruction and verification. The resource management interface connects the resource orchestration engine and cloud infrastructure for implementing resource allocation.

[0045] As described above, the cloud database fault self-healing system includes multiple collaborative functional modules. The distributed database cluster consists of multiple physical or virtual database nodes, each node carries a portion of the data shards, and the nodes are interconnected through a network. The monitoring agent module is deployed as a lightweight daemon process on each database node, continuously collecting basic operating indicators such as CPU, memory, I / O, and network, as well as database-specific indicators such as transaction latency and lock wait time. The collection frequency can be dynamically adjusted to balance performance overhead.

[0046] The fault diagnosis service module runs as an independent microservice with a built-in decision tree model: the root node is the CPU utilization threshold (greater than 90%), the first-level branches are I / O latency (greater than 500ms) and network packet loss rate (greater than 5%), and the leaf nodes correspond to specific fault type codes (e.g., 0x01 = disk failure, 0x02 = memory leak). The decision tree is automatically updated monthly using historical fault data, and the fault diagnosis service identifies the affected data range based on the decision tree results. The resource orchestration engine is the core scheduling component of the system, deployed as an independent service process. It does not directly manipulate business data, but is only responsible for decision-making and coordination. After receiving the fault diagnosis report, the resource orchestration engine generates isolation policy instructions and detailed resource allocation schemes, continuously monitors the execution status of reconstruction tasks, and dynamically adjusts the scheduling strategy based on feedback.

[0047] Furthermore, the data processing engine module communicates with the resource orchestration engine through a standard API interface, handling specific execution tasks including data isolation, reconstruction, and verification. While the data processing engine module directly accesses the distributed storage system, all operations are performed on replicas or snapshots to ensure the security of the original data. The resource management interface, acting as a bridge between the system and the underlying cloud infrastructure, encapsulates the differences in resource management APIs across different cloud platforms (such as AWS, Alibaba Cloud, and private clouds), translating the abstract resource requirements of the resource orchestration engine into concrete infrastructure operation instructions, thus achieving cross-cloud platform compatibility.

[0048] As an optional embodiment, refer to the appendix Figure 6 The data processing engine module includes: Lightweight isolation units are used to perform one-to-one replication and differential data removal; The snapshot management unit is used to call the snapshot API of the storage service; The consistency verification unit is used to execute data verification algorithms.

[0049] Based on the above, the data processing engine module consists of three functional units. The lightweight isolation unit uses memory-mapped file technology to achieve one-to-one data copying. By comparing and verifying each page and identifying the differences in data items, the differences in data items are logically marked rather than physically deleted, thus preserving the integrity of the original data structure. Copy-on-write is used during the copying process to ensure that the original data is not affected.

[0050] Furthermore, the snapshot management unit encapsulates the snapshot interfaces of mainstream cloud storage services, unifies API calls of different storage backends through the adapter pattern, maintains the snapshot lifecycle, including creation, status query, mounting and cleanup operations, supports asynchronous snapshot creation to avoid blocking the main thread, and automatically records snapshot metadata, including timestamp, data range and consistency point information, when creating a snapshot.

[0051] Furthermore, the consistency verification unit employs a layered verification mechanism. The bottom layer uses CRC32 or SHA256 algorithms to verify data block integrity, the middle layer verifies data logical consistency through transaction log replay, and the consistency verification unit only verifies infrastructure layer consistency: 1. Physical layer (checksum); 2. Transaction layer (WAL log replay); 3. Interface layer (schema compatibility). The consistency verification unit supports incremental verification, focusing on verifying only the reconstructed subset of data. Verification results are output in a structured report format, including error location and type information, for use by the resource orchestration engine for decision-making. Business semantic verification is completed by the application layer's health check interface.

[0052] As an optional embodiment, refer to the appendix Figure 7 The consistency verification unit performs multi-level verification: First-level verification: Perform checksums and comparisons on each reconstructed subset of data; Second layer of verification: Perform transaction log replay verification on the recombined recovery dataset; The third layer of verification: Before restoring the database service, perform shadow traffic tests to compare the consistency of responses between the old and new services.

[0053] Based on the above, the consistency verification unit adopts a three-layer progressive verification mechanism to ensure the integrity and correctness of data recovery; The first layer of verification is performed at the data subset level. A checksum is calculated for each reconstructed subset and compared with the baseline checksum of the original data to quickly identify data corruption at the physical level.

[0054] The second layer of verification is performed after the dataset is combined. It reconstructs the data state by replaying the transaction logs, verifies the atomicity and consistency of the transactions, and ensures that the data operation logic across subsets is correct.

[0055] The third layer of verification is performed in the shadow environment: historical read requests are captured using a traffic recording tool (with de-identification processing), replayed on the restored instance, and the result set is compared.

[0056] An alarm is triggered when the difference rate exceeds a predetermined value, prompting operations personnel to intervene and analyze the data. The replay process is conducted in a separate VPC, physically isolated from the production network. The three-layer verification is executed sequentially; failure at any layer terminates the subsequent process and returns detailed error information. The verification process takes place in an isolated environment, without affecting production services. Only after successful verification is the resource orchestration engine allowed to trigger the service recovery process.

[0057] As an optional embodiment, refer to the appendix Figure 8 It also provides a cloud database fault self-healing device based on a resource orchestration engine, including: The fault detection unit is used to monitor the operating status of the cloud database in real time. The resource scheduling unit, as a dedicated hardware acceleration module, is connected to the fault detection unit and is used to execute resource allocation decisions; A task coordination unit, connected to the resource scheduling unit, is used to generate task scheduling instructions; The data processing unit, connected to the task coordination unit, is used to perform data isolation and reconstruction; The verification control unit, connected to the data processing unit, is used to trigger the consistency verification process.

[0058] According to the above device, the cloud database fault self-healing device adopts a hardware acceleration architecture design. The fault detection unit works in collaboration with hardware sensors and software agents to collect the operating status parameters of the database nodes in real time, including hardware indicators such as CPU temperature, memory bandwidth utilization, and I / O queue depth, and software indicators such as transaction processing rate and number of connections.

[0059] The resource scheduling unit is implemented using a dedicated FPGA chip. It allocates DMA windows through SR-IOV virtualization technology. The operating system pre-allocates contiguous physical memory pages as a shared buffer. The FPGA accesses the buffer address through the PCIe BAR register. The data exchange volume for each scheduling decision is less than 4KB, avoiding TLB refresh overhead. The decision response time is less than 50μs.

[0060] The task coordination unit, as a hybrid software-firmware module, receives the decision results from the resource scheduling unit, generates standardized task scheduling instructions, and distributes them to each execution unit via a high-speed interconnect bus. The data processing unit includes a dedicated data processing chip, supporting hardware-accelerated data copying, verification, calculation, and data reassembly operations. This unit is directly connected to the storage device via an NVMe interface for efficient data transmission. The verification control unit is responsible for coordinating the consistency verification process, triggering verification operations at each layer, and summarizing the verification results. This unit has a built-in state machine controller to ensure the timing correctness of the verification process. Communication between units employs a combination of hardware signal lines and shared memory to maintain data consistency while ensuring low latency. The entire device interacts with external systems through a unified control bus.

[0061] As an optional embodiment, the data processing unit includes a hardware acceleration module: The hardware acceleration module uses an FPGA chip to implement the data difference calculation circuit, which processes the hash operations of multiple data blocks in parallel. The hardware acceleration module accesses the storage device directly through the DMA channel, bypassing CPU memory copying and reducing the latency of data isolation operations.

[0062] As described above, the hardware acceleration module in the data processing unit is implemented based on field-programmable gate array (FPGA) technology. The hardware acceleration module includes a dedicated data difference calculation circuit, which is designed as a parallel processing architecture and can perform hash operations on multiple data blocks simultaneously, significantly improving data comparison efficiency.

[0063] The hardware acceleration module establishes a direct connection with the storage device through a direct memory access channel (DMI), completely bypassing the CPU's memory copying process during data transfer, effectively eliminating context switching and data transport overhead in traditional software processing. In data isolation operations, raw data flows directly from the storage device into the FPGA via the DMA channel for real-time hash calculation, and the calculation result is fed back to the control unit via a hardware interrupt mechanism. The entire data processing flow is completed at the hardware level, avoiding frequent switching between operating system kernel mode and user mode, significantly reducing processing latency. The FPGA configuration of the hardware acceleration module is stored in SPI Flash and includes both SHA256 and CRC32 core circuits. At runtime, the configuration register switching algorithm is achieved via PCIe: writing 0x01 enables SHA256, and writing 0x02 enables CRC32. The configuration switching time is less than 10μs, eliminating the need to reload the bitstream. The hardware acceleration module communicates with the main system via a standard PCIe interface, ensuring compatibility and maintainability.

[0064] Furthermore, the hardware acceleration module of the data processing unit is based on the Xilinx Ultrascale+ architecture and contains 16 parallel hash computation units. Each unit supports a 64-bit data width, a clock frequency of 250MHz, and a theoretical throughput of 4GB / s. The DMA controller adopts a scatter-gather mode, supporting the transfer of non-contiguous memory regions with a maximum transfer granularity of 1MB. The FPGA and NVMe SSD are directly connected via the CCIX protocol, bypassing the CPU memory controller, with an end-to-end latency of less than 50μs.

[0065] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0066] According to another aspect of the present invention, an electronic device is also provided, the electronic device including a memory and a processor; the memory is used to store a program; the processor executes the program to implement the method of any of the foregoing.

[0067] According to another aspect of the present invention, a computer-readable storage medium is also provided, the storage medium storing a computer program that, when executed by a processor, implements the method of any of the foregoing.

[0068] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the method described in any of the foregoing.

[0069] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A cloud database fault self-healing method based on a resource orchestration engine, characterized in that, Includes the following steps: The monitoring agent deployed on the database node collects node operation metrics in real time. When a node's performance metrics deviate from a preset threshold, the fault diagnosis service analyzes the fault type and the range of affected data, and generates a fault diagnosis report. The resource orchestration engine receives fault diagnosis reports and determines the isolation strategy type based on preset strategies; The resource orchestration engine sends isolation commands to the data processing engine. The isolation commands include the isolation policy type and the range of data affected. The data processing engine performs isolation operations: If a lightweight isolation strategy is adopted, the affected data range is replicated, and the difference data items are identified and isolated on the replicated data to generate multiple independent data subsets. If a snapshot isolation strategy is used, the storage service is invoked to create a read-only snapshot copy, and the boundaries of similar data items are marked in the metadata; The resource orchestration engine dynamically allocates virtual computing resources for each data subset and sends reconstruction task scheduling instructions to the data processing engine. The data processing engine performs data reconstruction on the allocated resources and returns the reconstruction status to the resource orchestration engine; The data processing engine reassembles the reconstructed subset of data to generate a recovered dataset and performs consistency verification. Once the resource orchestration engine receives the verification pass signal, it resumes the database service.

2. The cloud database fault self-healing method based on a resource orchestration engine according to claim 1, characterized in that, The resource orchestration engine determines the isolation strategy type, including: Receive data discrepancy metrics pre-calculated by the data processing engine; Based on the comparison between the data difference index and the preset threshold, a lightweight isolation strategy or a snapshot isolation strategy is selected.

3. The cloud database fault self-healing method based on a resource orchestration engine according to claim 1, characterized in that, The resource orchestration engine dynamically allocates virtual computing resources, including: Based on the size of the data subset and the priority of the reconstruction task, an independent container instance is allocated to each subset, and a CPU / memory quota limit is set.

4. The cloud database fault self-healing method based on a resource orchestration engine according to claim 1, characterized in that: The resource orchestration engine dynamically adjusts preset thresholds, including: Obtain historical fault recovery records, which contain the correspondence between historical data differences and recovery times; A recovery time prediction model is constructed based on the aforementioned correspondence; When the predicted recovery time of the lightweight isolation strategy exceeds the service level agreement threshold, the preset threshold for data difference is automatically reduced, triggering the snapshot isolation strategy in advance.

5. The cloud database fault self-healing method based on a resource orchestration engine according to claim 1, characterized in that: The data processing engine performs data reconstruction including: Analyze the transaction dependencies between different data subsets and generate a reconstruction task dependency graph; The parallel reconstruction order is determined based on the reconstruction task dependency graph, prioritizing the reconstruction of data subsets with no dependencies. When a rebuild conflict is detected, the rebuild task for the conflicting subset is paused, and execution resumes after the dependencies are recalculated.

6. A cloud database fault self-healing system based on a resource orchestration engine, running the method as described in any one of claims 1 to 5, characterized in that, include: A distributed database cluster, consisting of multiple database nodes; The monitoring agent module is deployed on the database node and is used to collect node operation metrics; The fault diagnosis service module is connected to the monitoring agent module and is used to generate fault diagnosis reports; The resource orchestration engine, as an independent scheduling service, connects to the fault diagnosis service module and is used for: Receive fault diagnosis reports; Generate isolation policy instructions and resource allocation schemes; Schedule reconstruction tasks; The data processing engine module connects the resource orchestration engine and the distributed storage, and is used to perform data isolation, reconstruction and verification. The resource management interface connects the resource orchestration engine and the cloud infrastructure for implementing resource allocation.

7. A cloud database fault self-healing system based on a resource orchestration engine according to claim 6, characterized in that, The data processing engine module includes: Lightweight isolation units are used to perform one-to-one replication and differential data removal; The snapshot management unit is used to call the snapshot API of the storage service; The consistency verification unit is used to execute data verification algorithms.

8. A cloud database fault self-healing system based on a resource orchestration engine according to claim 7, characterized in that, The consistency verification unit performs multi-level verification: First-level verification: Perform checksums and comparisons on each reconstructed subset of data; Second layer of verification: Perform transaction log replay verification on the recombined recovery dataset; The third layer of verification: Before restoring the database service, perform shadow traffic tests to compare the consistency of responses between the old and new services.

9. A cloud database fault self-healing device based on a resource orchestration engine, corresponding to the system as described in any one of claims 6 to 8, characterized in that, include: The fault detection unit is used to monitor the operating status of the cloud database in real time. The resource scheduling unit, as a dedicated hardware acceleration module, is connected to the fault detection unit and is used to execute resource allocation decisions; The task coordination unit, connected to the resource scheduling unit, is used to generate task scheduling instructions; The data processing unit, connected to the task coordination unit, is used to perform data isolation and reconstruction; The verification control unit, connected to the data processing unit, is used to trigger the consistency verification process.

10. A cloud database fault self-healing device based on a resource orchestration engine according to claim 9, characterized in that, The data processing unit includes a hardware acceleration module: The hardware acceleration module uses an FPGA chip to implement a data difference calculation circuit, which processes the hash operations of multiple data blocks in parallel. The hardware acceleration module directly accesses the storage device through the DMA channel, bypassing CPU memory copying and reducing the latency of data isolation operations.

Images