Cloud native architecture fault recovery method and system based on multi-dimensional data

Abstract

The invention discloses a cloud native architecture fault recovery method and system based on multi-dimensional data. The method comprises the steps that multi-dimensional heterogeneous data such as indexes, logs and call chains are collected in real time, time sequence alignment is carried out, and standardized event data are generated; dynamically constructing a fault characteristic spectrum based on event data, and accurately positioning a fault root cause by iteratively calculating a root cause score; and for the candidate self-healing strategy, combined with the situation information of the fault characteristic spectrum, carrying out situation self-adaptive risk assessment from multiple dimensions such as service availability, data consistency, resource consumption and the like. According to the scheme of the invention, the deep insight of complex associated faults is realized by constructing the dynamic graph, and the safety and success rate of automatic operation are remarkably improved by a situation-adaptive risk assessment mechanism.

Description

Technical Field

[0001] This invention belongs to the field of cloud computing and intelligent operation and maintenance, and specifically relates to a cloud-native architecture fault recovery method and system based on multi-dimensional data. Background Technology

[0002] With the rapid development of cloud computing technology, cloud-native architecture, centered on microservices, containerization, and dynamic orchestration, has become the mainstream model for building modern application systems. This architecture endows applications with unprecedented agility, elasticity, and scalability. However, the complexity of the system has also increased dramatically, with hundreds or thousands of ephemeral and dynamic service instances intertwined, posing unprecedented challenges to traditional system operations and maintenance (O&M) models.

[0003] Currently, to ensure the stability of cloud-native systems, operations and maintenance (O&M) personnel typically deploy a combined monitoring system. For example, they use monitoring systems to track basic performance metrics such as CPU and memory, leverage log aggregation systems to collect and retrieve massive amounts of business and system logs, and utilize application performance monitoring (APM) tools to track complex call relationships between distributed services. However, the data sources generated by these monitoring tools are often independent, forming data silos. When a failure occurs, anomaly information from different dimensions is scattered across various platforms. O&M personnel need to rely on their personal experience, spending a significant amount of time manually correlating and troubleshooting across platforms. In time-sensitive failure scenarios, this is not only inefficient but also highly prone to misjudgment due to incomplete information.

[0004] Furthermore, existing technologies have significant limitations in the depth of fault diagnosis and the security of automated operations. Traditional monitoring systems often use static threshold-based alarms. When faced with cascading failures involving multiple services caused by a single underlying problem, they often generate a large number of derivative alarms, known as an alarm storm. This not only fails to help operations personnel focus on the problem but also masks the true root cause, making accurate fault location extremely difficult. Regarding automated repair, existing self-healing solutions are usually based on simple scripts with preset rules, such as performing a restart operation when a service is unresponsive. This one-size-fits-all repair logic lacks a deep understanding of the fault context and a comprehensive assessment of the potential risks of the repair operation. It cannot distinguish the drastically different business impacts of restarting a stateless edge service versus restarting a core stateful database. Rashly executing automated operations may lead to data loss or prolonged interruption of core functions, making the repair risks difficult to control. In addition, these solutions generally lack a closed-loop mechanism for automatically verifying the effects of the repair actions and performing rollbacks after failures, resulting in a lack of reliable quality assurance throughout the automation process.

[0005] In summary, existing O&M technologies are insufficient in their capabilities for data fusion and analysis, deep root cause analysis, and secure and controllable automated processing when faced with the highly dynamic nature of cloud-native systems, the complex dependencies between components, and the stringent requirements for business continuity. Therefore, the industry urgently needs a new automated O&M approach that can integrate multi-dimensional data, intelligently analyze fault propagation paths, and make secure decisions to meet the challenges of the cloud-native era. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a cloud-native architecture fault recovery method and system based on multi-dimensional data. This addresses the technical problems of difficulty in adapting to the high dynamism and complexity of cloud-native systems, the existence of data silos, inaccurate root cause localization, high risks of automated repair, and lack of proactive prevention capabilities.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution.

[0008] This invention first discloses a cloud-native architecture fault recovery method based on multidimensional data, which includes the following steps: Step 1: Collect multidimensional heterogeneous data in the cloud-native architecture in real time, and perform time-series alignment on the multidimensional heterogeneous data based on the preset time-series alignment triplet to obtain standardized event data; Step 2: Construct a dynamic fault feature map based on the standardized event data, and perform fault diagnosis and root cause localization based on the fault feature map to identify potential fault root causes; Step 3: Match the root cause of the fault with the self-healing strategy. After matching the target self-healing strategy, initiate a situation-adaptive risk assessment to calculate the total risk score of the target self-healing strategy by analyzing the situation information of the fault feature map. Step 4: Based on the risk level corresponding to the total risk score, determine whether to execute the target self-healing strategy, and perform closed-loop verification or rollback after execution.

[0009] The present invention further includes the following preferred embodiments: The time-aligned triple includes: a service identifier, a container instance identifier, and a timestamp; the time-alignment of the multidimensional heterogeneous data based on the preset time-aligned triple further includes: The raw data records from different sources are uniformly parsed and mapped into standardized events containing the service identifier, container instance identifier, and timestamp.

[0010] The root cause localization further includes: Calculate the endogeneity anomaly degree of each node in the fault feature map, whereby the endogeneity anomaly degree is calculated from the node’s own anomaly features. By combining the influence weights that characterize the probability of fault propagation between nodes, the root cause score of each node is updated through iterative calculation, and the node with the highest root cause score is identified as the root cause of the fault.

[0011] The iterative calculation of the root factor score satisfies the following relationship: ; in, For nodes In the Root cause score in rounds of iteration For nodes The degree of endogenous abnormality, The damping coefficient is... Pointing to a node upstream node In the Root cause score in rounds of iteration For the node To the node The influence weight.

[0012] The situation-adaptive risk assessment further includes: The potential impact of the self-healing strategy under the current fault characteristic map is quantified from three dimensions: service availability, data consistency, and resource consumption, so as to obtain the risk score of each dimension. Based on the context of the fault feature map, the risk weights of the three dimensions are dynamically calculated; The total risk score is obtained by weighted summation of the risk scores from the three dimensions.

[0013] The dynamic calculation of the risk weights of the three dimensions further includes analyzing the topological importance and attribute characteristics of nodes in the fault feature map; When the fault feature map contains state service nodes or core nodes in the topology, the risk weights of the data consistency dimension or service availability dimension are increased accordingly.

[0014] The weight of the data consistency dimension The following relationship must be satisfied: ; in, This is the current fault feature map. For its set of nodes, For nodes Status score, For nodes The centrality score, This is the normalization factor.

[0015] The closed-loop verification further includes: After the self-healing strategy is executed, a set of key performance indicators are continuously monitored within a preset verification window. If the key performance indicators have not recovered to the normal baseline by the end of the verification window, a predefined rollback mechanism will be automatically triggered.

[0016] The method further includes: Analyze historical indicator data using time series forecasting models to predict the probability of failure within future time windows; When the probability of failure exceeds the warning threshold, steps 2 to 4 are actively triggered to generate and evaluate preventive measures.

[0017] This invention also discloses a cloud-native architecture fault recovery system based on multidimensional data, utilizing the aforementioned cloud-native architecture fault recovery method based on multidimensional data, comprising: The data acquisition and alignment module is used to acquire multidimensional heterogeneous data in the cloud-native architecture in real time, and perform time-series alignment on the multidimensional heterogeneous data based on a preset time-series alignment triplet to obtain standardized event data. The diagnosis and localization module is used to construct a dynamic fault feature map based on the standardized event data, and to perform fault diagnosis and root cause localization based on the fault feature map in order to identify potential fault root causes. The risk assessment module is used to match the root cause of the failure with the self-healing strategy. After matching the target self-healing strategy, the scenario-adaptive risk assessment is initiated to calculate the total risk score of the target self-healing strategy by analyzing the scenario information of the failure feature map. The decision-making and execution module is used to determine whether to execute the target self-healing strategy based on the risk level corresponding to the total risk score, and to perform closed-loop verification or rollback after execution.

[0018] Accordingly, this application also discloses a terminal, including a processor and a storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the aforementioned cloud-native architecture fault recovery method based on multidimensional data.

[0019] Accordingly, this application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned cloud-native architecture fault recovery method based on multidimensional data.

[0020] The beneficial effects of this invention are that, compared with the prior art, this invention provides a cloud-native architecture fault recovery method and system based on multi-dimensional data, which can diagnose complex faults in cloud-native architecture in real time and accurately, and conduct a rigorous, situation-adaptive risk assessment before performing self-healing operations, effectively avoiding the risks that may be caused by blind automated operations. At the same time, through a predictive early warning mechanism, the operation and maintenance capabilities are upgraded from post-repair to pre-prevention, which significantly enhances the stability and reliability of the system. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall functional architecture of the cloud-native architecture fault recovery system based on multidimensional data according to the present invention; Figure 2 This is an overall flowchart of the cloud-native architecture fault recovery method based on multidimensional data in this invention; Figure 3 This is a schematic diagram of the multidimensional heterogeneous data time-series alignment process of the present invention; Figure 4 This is a schematic diagram of the structure of the dynamic fault feature map of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0023] The embodiments described in this application are merely some, not all, embodiments of the present invention. Based on the spirit of the present invention, other embodiments obtained by those skilled in the art without inventive effort are all within the protection scope of the present invention.

[0024] To address the shortcomings of existing technologies, this invention proposes a cloud-native architecture fault recovery method and system based on multi-dimensional data. The aim is to establish an automated fault handling system that integrates real-time diagnosis, risk perception, and closed-loop self-healing to improve the accuracy, security, and efficiency of system operation and maintenance, and to automatically handle complex distributed system faults in cloud-native environments.

[0025] See Figure 1 As shown, the cloud-native architecture fault recovery method based on multi-dimensional data disclosed in this invention includes the following steps: Step 1: Collect multidimensional heterogeneous data in the cloud-native architecture in real time, and perform time-series alignment on the multidimensional heterogeneous data based on the preset time-series alignment triplet to obtain standardized event data.

[0026] By deploying lightweight agents on various nodes in a cloud-native environment, multi-dimensional heterogeneous data such as metrics, logs, call chains, and network flows are collected in real time. To break down barriers between different data sources, this invention introduces a time-series alignment mechanism. Specifically, through a preset time-series alignment triplet containing a service identifier, container instance identifier, and timestamp, all collected raw data records from different sources and formats (such as metrics, logs, call chains, and network flows) are parsed and mapped into a unified, standardized event. This process transforms scattered, isolated data points into a structured event stream with time as the axis, enabling cross-entity correlation analysis, providing a solid data foundation for subsequently building a global fault view.

[0027] Step 2: Construct a dynamic fault feature map based on the standardized event data, and perform fault diagnosis and root cause localization based on the fault feature map to identify potential fault root causes.

[0028] The system state is abstracted as a dynamic fault feature graph. Nodes in this graph represent entities such as services and instances, edges represent dynamic dependencies between entities, and node attributes are composed of standardized event data. The graph's topology is not static but dynamically generated by employing a graph neural network model to learn and analyze call chain data and network flow data in real time, thus accurately reflecting the instantaneous changes in the system topology.

[0029] In root cause localization, this invention employs an improved iterative root cause score calculation method. First, the endogenous anomaly degree of each node in the graph is calculated. This value is quantified from the node's own anomalous characteristics (such as sudden changes in indicators, error logs, etc.), representing the probability that the node itself has a problem. Then, by combining the influence weights characterizing the probability of fault propagation between nodes, the root cause score of each node is updated through iterative calculation.

[0030] Specifically, a node's root cause score is the sum of its own endogenous outlier score and the influence score propagated from upstream nodes. Its iterative calculation satisfies the following relationship: ; in, For nodes In the Root cause score in rounds of iteration For nodes The degree of endogenous abnormality, To balance the damping coefficient between endogenous anomalies and the effects of propagation, Pointing to a node upstream node In the Root cause score in rounds of iteration For the node To the node The influence weights are determined. Through multiple iterations until convergence, the node with the highest root cause score is identified as the most likely root cause of the failure.

[0031] Step 3: Match the root cause of the fault with the self-healing strategy. After matching the target self-healing strategy, initiate a situation-adaptive risk assessment to calculate the total risk score of the target self-healing strategy by analyzing the situation information of the fault feature map.

[0032] After a candidate self-healing strategy is matched, the system does not execute it immediately, but instead initiates a scenario-adaptive risk assessment process. This process quantifies the potential impact of executing the self-healing strategy from three dimensions: service availability, data consistency, and resource consumption.

[0033] In specific embodiment 4, the core content of the self-healing strategy includes: Source: Primarily derived from a pre-defined strategy library, generated based on historical fault data, expert knowledge, and open-source cloud-native practices (such as Kubernetes self-healing mechanisms), supporting real-time dynamic adjustments.

[0034] Storage: Pre-stored in databases, including time-series databases (such as Prometheus), full-text search databases (such as Elasticsearch), and relational databases (such as MySQL), for efficient retrieval and execution.

[0035] Matching rules: Root cause score iterative calculation and pattern scheduler are used. Matching strategies are based on data integrity (such as full-dimensional or missing rollback), combined with thresholds, probabilities and contextual weights to ensure fit.

[0036] Specific content includes automated actions such as node isolation, instance restart, resource scaling, and configuration rollback, emphasizing risk assessment to avoid secondary failures, and supporting interpretable output such as anomaly scores and propagation paths.

[0037] The importance (i.e., risk weight / risk score) of each dimension in this invention is dynamic rather than fixed. Risk weights are dynamically calculated by analyzing the current fault feature map context. For example, when the fault feature map includes stateful service nodes such as databases, or involves core nodes in the topology, the risk weights of the data consistency or service availability dimensions are automatically increased, making subsequent decisions more prudent. The weight of the data consistency dimension is used as an example. For example, its calculation can satisfy the following relationship:

[0038] in, This is the current fault feature map. For its set of nodes, For nodes The state score characterizes whether it is a stateful service. For nodes Centrality score in the topology. To ensure that the sum of all weights equals 1, a normalization factor is used. Finally, the total risk score is obtained by weighted summation of the risk scores for each dimension.

[0039] Step 4: Based on the risk level corresponding to the total risk score, determine whether to execute the target self-healing strategy, and perform closed-loop verification or rollback after execution.

[0040] Risk scores are categorized into three levels: low, medium, and high. For low-risk operations, a self-healing strategy is automatically implemented; for medium-risk operations, the process is paused and manual approval is requested; for high-risk operations, the self-healing strategy is directly blocked and an alternative solution is sought.

[0041] After the strategy is executed, the system enters a closed-loop verification phase to ensure the operation is effective and harmless. Within a preset verification window, a set of key performance indicators are continuously monitored. If the indicators fail to recover to the normal baseline within the window period, a predefined rollback mechanism is automatically triggered to undo the executed operations, thereby preventing the escalation or prolongation of the fault.

[0042] Furthermore, the method described in this invention may also include a predictive early warning step. By utilizing time-series prediction models such as Long Short-Term Memory (LSTM) networks, historical indicator data is analyzed to predict the probability of a failure occurring within a future time window. When this probability exceeds an early warning threshold, steps 2-4 above can be proactively triggered to generate and evaluate preventative response plans in advance, achieving a shift from passive response to proactive defense.

[0043] In a specific implementation environment, the method of this invention can be applied to a cloud-native computing environment managed based on container orchestration technology. This environment includes one or more master nodes responsible for cluster management and control, and multiple worker nodes used to actually run containerized instances that host business applications. These applications are organized through a microservice architecture, with complex call and dependency relationships between services, and communicate through service discovery and load balancing mechanisms. Data generated by applications and systems, such as performance metrics, logs, and call chain information, are distributed across various nodes and containers. The cloud-native architecture fault recovery system described in this invention is deployed in such an environment to monitor, diagnose, and self-heal the health status of the entire cluster.

[0044] A cloud-native architecture fault recovery system can include the following modules: data acquisition and alignment, diagnosis and localization, risk assessment, decision-making and execution, and predictive early warning. These modules work together to form a complete process from data input to closed-loop fault handling.

[0045] The data acquisition and alignment module forms the data foundation of the entire system. This module continuously captures multidimensional, heterogeneous data from the runtime environment by deploying lightweight acquisition agents on all worker nodes of the cluster. This includes application performance metrics, structured and unstructured log data, call chain data conforming to open standards, and network flow data reflecting actual network interactions between services. After acquiring the raw data, the core task of this module is time-series alignment, uniformly mapping all data records to standardized event data containing a time-series alignment triplet: service identifier, container instance identifier, and timestamp. This module eliminates semantic gaps between different data sources, providing unified and synchronized data input for subsequent correlation analysis and graph construction.

[0046] In one embodiment of the invention, the data acquisition and alignment module deploys a resource-controlled acquisition agent on each worker node. This agent captures network flow data and system-level metrics with low performance overhead using techniques such as kernel-level packet filtering. Simultaneously, it subscribes to events during container runtime through standard interfaces and collects metrics and call chain data in accordance with specifications such as open-source monitoring and alerting protocols and open telemetry standards. To control its own resource consumption, the operation of this agent is strictly limited by the control group mechanism provided by the operating system.

[0047] The specific process for time alignment is as follows: For any original data record, the system extracts key information from its metadata or content. For example, the service identifier can be obtained from the container instance's tag; the container instance identifier can be obtained directly from the container runtime environment; and the timestamp uses the node system clock synchronized via a network time protocol to ensure the uniformity of time across the entire cluster. Subsequently, this information, along with the original data payload, is encapsulated into a standardized data object, which is then pushed to a high-throughput message queue as standardized event data for consumption by downstream modules.

[0048] The diagnosis and localization module receives standardized event data from other modules and is responsible for intelligent fault identification and root cause tracing. Based on a stream processing engine, this module transforms real-time event streams into a dynamic fault feature graph. This graph uses services and instances as nodes, service calls and network connections as edges, and event data as dynamic attributes of nodes and edges. To ensure the real-time nature of the graph topology, this module employs a graph neural network model, continuously learning and updating the dependencies between services by analyzing call chains and network flow data. When a fault occurs, this module executes a root cause localization algorithm on the graph, iteratively calculating the root cause score for each node, following the following formula:

[0049] in, Representative node In the Root cause score in rounds of iteration The intrinsic anomaly degree is calculated based on the node's own anomaly characteristics. To balance the damping coefficient between endogenous and propagational effects, upstream node In the root cause score of the previous round, This represents the faulty node. propagation to nodes The probability of impact is weighted. Ultimately, the node with the highest root cause score is identified as a potential root cause of the failure and is passed to the next module along with its associated failure information.

[0050] The diagnostic and localization module revolves around a native graph database and a stream processing framework. The stream processing job consumes standardized event data from the message queue in real time and transforms it into operation instructions for the graph database. For example, a new call chain data will trigger a graph topology update to create or update edges representing inter-service call relationships; a metric data will update the attributes of the corresponding node in the graph.

[0051] To calculate the influence weights in the root cause localization formula This invention employs a graph neural network model. Its implementation consists of two stages: offline training and online inference. In the offline training stage, historical data from normal and fault periods are used to train a graph attention network model or a graph sampling and aggregation network model. The model's task is to learn a function whose input is any two connected nodes. and Given the features and edge attributes, the output represents the fault originating from... spread to The probability value. During the online inference phase, the trained model is deployed in the diagnosis and localization module, and dynamically calculates the influence weight of each edge in the graph based on the real-time constructed fault feature map. .

[0052] Endogenous anomaly degree of nodes It is a comprehensive score, derived by weighting out anomaly data from multiple dimensions associated with the node. For example, for indicator data, standard scores or the Three Sigma rule can be used to calculate the degree of deviation from the normal baseline; for log data, the growth rate of error or warning level logs per unit time can be statistically analyzed. It can be represented as:

[0053] in Weights for different data types, For the corresponding data type in the node Abnormal scores.

[0054] The iterative calculation of root cause scores will continue until the change (e.g., L2 norm) of the root cause score vectors of all nodes between two adjacent iterations is less than a preset minimum threshold. At this point, the algorithm is considered to have converged.

[0055] The risk assessment module receives the identified root causes of the failure and matches candidate self-healing strategies from a pre-defined strategy library. Before execution, this module initiates a context-adaptive risk assessment. Unlike using fixed risk standards, this module dynamically assesses operational risk based on the context of the current failure feature map. Using methods such as Monte Carlo simulation, this module quantifies the potential impact of the self-healing strategy across three dimensions: service availability, data consistency, and resource consumption, and calculates the risk score for each dimension. Simultaneously, it analyzes the topology and node attributes of the failure map, dynamically adjusting the risk weights for the three dimensions. For example, the weight of the data consistency dimension... The calculation can satisfy the following relationship:

[0056] in, Refers to the current fault feature map. For its set of nodes, To represent nodes State score indicating whether it is a stateful service For nodes The centrality score, This is a normalization factor. By weighting and summing the risk scores of each dimension with their dynamic weights, a total risk score that accurately reflects the current situation is obtained.

[0057] In the risk assessment module, the quantification of self-healing strategies is achieved through Monte Carlo simulations. Specifically, the system first captures a snapshot of the current fault characteristic map. For a specific self-healing strategy, such as "restarting a container instance of service A," the simulation process defines this operation as a set of probabilistic events. For example, the unavailability duration of this instance is modeled as a random variable following a log-normal distribution, and the probability of its upstream caller encountering failure when calling this instance is also set based on historical data. The system will run thousands of such simulations, sampling all random variables and extrapolating the system's state evolution in each run. Finally, all the results are statistically analyzed; for example, the 95th percentile of the service availability loss duration is taken as the final potential impact value. To achieve a conservative and robust risk assessment. In calculating adaptive weights... At that time, the state score of the node This can be determined by examining the specific tags of the container instance; for example, an instance with a tag indicating the database type... A value of 1 indicates centrality, otherwise 0. This is achieved by running standard network link analysis algorithms in real time on the current fault feature map snapshot. In this way, the structural and semantic features of the map are transformed into specific numerical values, which are then substituted into the weight calculation formula, thereby enabling the risk assessment model to adapt to fault scenarios. Among them, adaptive weights It refers to the dynamic weights of various dimensions in risk assessment, used to balance dimensions such as service availability, data consistency, and resource consumption; but is not limited to the above. Specific examples, The weights of the data consistency dimension are calculated through graph contexts (such as node state and centrality) to ensure the adaptability of the evaluation.

[0058] The decision-making and execution module receives the total risk score from the module and classifies the risk into three levels: low, medium, and high, based on preset risk thresholds. For low-risk strategies, execution is automatic; for medium-risk strategies, execution is paused and awaits manual confirmation; for high-risk strategies, execution is blocked and a safer alternative is sought. At the execution level, this module interacts with the cloud platform's control plane's application programming interface (API) to translate self-healing instructions (such as restarting container instances or adjusting service traffic policies) into actual operations. After the operation is completed, the module immediately initiates a closed-loop verification mechanism, monitoring key performance indicators within a preset verification window. If the indicators fail to recover, a rollback procedure is automatically triggered to undo the operation, ensuring the system's eventual consistency and stability.

[0059] In the decision-making and execution module, the policy execution engine relies on direct interaction with the cloud-native platform's control plane. This engine sends instructions to the platform's application programming interface (API) server by calling the officially provided client library. For example, the "restart container instance" operation corresponds to deleting a specified container group object; the "traffic switching" operation corresponds to modifying the selector field of a service object; and the "scale up" operation corresponds to updating the expected replica number field of a horizontal container group autoscalor object.

[0060] The key performance indicators monitored by the system are typically predefined service level targets, such as a "99.9% request success rate" or "95% request response time less than 200 milliseconds." The normal baseline is dynamically calculated by averaging the indicator data over the most recent stable time window. The rollback mechanism predefines the reverse operation corresponding to each self-healing strategy. For example, for a traffic switching operation, the rollback operation restores the selector of the service object to its state before execution.

[0061] In a further embodiment, the predictive early warning module, operating in parallel, elevates the operational mode from passive response to proactive defense. This module utilizes time-series prediction models such as Long Short-Term Memory (LSTM) networks to continuously analyze historical metrics data of critical services to predict potential performance bottlenecks or failures in the near future. When the predicted failure probability exceeds a set early warning threshold, the module proactively triggers a simulated diagnostic and risk assessment process, generating and evaluating preventative measures in advance to provide decision support for operations personnel or enable proactive intervention under specific conditions.

[0062] The predictive early warning module is based on a trained Long Short-Term Memory (LSTM) network model. The model's input is a multivariate time-series vector, which at each time step contains multiple performance metrics for a key service, such as queries per second, average response time, and CPU utilization. The model's output is a predicted sequence of all metric values ​​for this vector within a future time window (e.g., the next hour). These predicted values ​​are compared to preset health thresholds. If one or more metrics are predicted to continuously exceed the threshold, a failure risk is identified, and the confidence level represents the failure probability. The model is trained offline, using a large amount of historical metric data exported from a time-series database.

[0063] To further illustrate the present invention, two specific embodiments will be provided below to demonstrate the application process of the method. These embodiments aim to illustrate how the present invention addresses both passively triggered complex faults and proactively predicted potential risks in a real-world cloud-native environment.

[0064] In the first embodiment, the scenario is set as a financial-grade payment system running in a containerized environment. This system comprises two core microservices: "Order Service" and "Inventory Service." During a high-concurrency transaction scenario, the system experienced a failure where some user payment requests remained unresponsive for an extended period.

[0065] First, the data acquisition and alignment module captured abnormal signals from multiple sources when the fault occurred. It detected that both the order service and inventory service container instances were outputting numerous error logs containing the phrase "timeout for acquiring distributed lock"; simultaneously, the monitoring system showed a sharp increase in request response time metrics and a jump in error rate metrics from zero for these two services. The module quickly processed these heterogeneous raw data records, unifying them into standardized event data with precise timestamps, service identifiers, and container instance identifiers, and then injected them into the real-time data stream.

[0066] Subsequently, the diagnosis and localization module consumes this event data in real time. Due to the surge in error logs, the endogenous anomalies of the order service and inventory service nodes in the fault feature map are... The calculated value is very high. More importantly, through analysis of the call chain data, this module discovered that while the order service holds the lock on resource A, it is waiting for the inventory service to release the lock on resource B; while the inventory service holds the lock on resource B, it is waiting for the order service to release the lock on resource A. This circular waiting relationship forms a clear loop structure on the dynamically constructed fault feature map. After iterative calculation, the root cause localization algorithm calculates the root cause score. Ultimately, the focus was on these two service nodes and the distributed lock service node they both depended on, thus pinpointing the root cause of the failure as "a distributed deadlock caused by resource contention between the order service and the inventory service."

[0067] After locating the root cause, the system matched a self-healing strategy from the strategy library: "Forcefully release the locks held by the transaction and restart the service instance corresponding to the transaction, triggering its transaction rollback and retry mechanism." At this point, the risk assessment module intervened. Since both the order service and the inventory service are marked as stateful services that process core transactions, their statefulness scores are... Very high. The weight of the data consistency dimension is significant when calculating risk weights. The value was adaptively calculated to be relatively high. This module, through Monte Carlo simulation, estimated that forcibly releasing the lock and restarting has a certain probability of causing a very small portion of transaction states to fail to roll back correctly, thus potentially leading to data inconsistency. After comprehensive calculation, the overall risk score of this self-healing strategy was rated as "medium risk".

[0068] Finally, the decision-making and execution module received the "medium risk" assessment result. According to preset rules, the system did not automatically execute the strategy. Instead, it pushed an alert containing a complete fault analysis process (including a deadlock loop diagram), a suggested execution strategy, and its risk assessment report to the on-duty operations engineer. After confirming the analysis conclusions and weighing the risks, the engineer clicked "Approve Execution" through the interface. The module then called the distributed lock service's interface to release the specified lock and sent instructions to the container orchestration platform's control plane to safely restart the relevant container instances for the order service. After the strategy was executed, the module entered a closed-loop verification phase, continuously monitoring the error rate and response time of the order and inventory services. Within a few minutes of the verification window, all indicators returned to normal baseline levels, and the system confirmed that the fault had been successfully resolved.

[0069] In the second embodiment, the scenario is set as an e-commerce platform whose "product recommendation service" is about to face a surge in traffic during a large-scale promotional event. This scenario focuses on illustrating the application of the predictive early warning module.

[0070] Hours before the major promotional event began, the predictive alert module's built-in Long Short-Term Memory (LSTM) network model continuously analyzed historical and real-time performance metrics of the product recommendation service. This model combined current metric growth trends with historical traffic patterns from similar events last year (e.g., last year's similar events), predicting that in 30 minutes, the service's query rate per second would exceed the CPU and memory capacity limits of its container instance, leading to significantly longer response times and severely impacting user experience. The model's output probability of future failures exceeded the preset alert threshold.

[0071] At this point, the system did not wait for the actual failure to occur, but instead proactively triggered a simulated failure handling process. After receiving the warning, the diagnosis and location module identified the potential root cause of the failure as "insufficient computing resources for the product recommendation service to handle the expected load." The system then matched two preventative strategies: Strategy A was to "horizontally scale up the service and increase the number of container instances," and Strategy B was to "enable a pre-configured, higher-level secondary cache layer."

[0072] Next, the risk assessment module conducted a context-adaptive risk assessment for both strategies. For strategy A, the assessment results showed that its impact on service availability and data consistency was minimal, with the main risk being increased costs in the resource consumption dimension, and the overall risk score was rated as "low risk". For strategy B, although the resource consumption risk was even lower, its risk score in the data consistency dimension was slightly higher due to the potential introduction of temporary inconsistencies between cached data and master data, and the final overall risk score was rated as "medium risk".

[0073] Based on the assessment results, the decision-making and execution module recommended Strategy A, which had a lower overall risk score, to the operations team and provided a clear suggestion on the amount of capacity to be increased. Because this operation was rated as "low-risk," the system could even be configured to execute automatically in this scenario. After confirmation by the operations personnel, the module invoked the cloud platform interface to smoothly complete the capacity expansion operation for the product recommendation service. When the traffic surge of the promotional event arrived, the service, with its pre-prepared sufficient resources, stably handled all requests, successfully averting a potential major failure.

[0074] The beneficial effects of this invention are that, compared with the prior art, this invention provides a cloud-native architecture fault recovery method and system based on multi-dimensional data, which has the following technical effects: 1. It enables rapid and accurate fault location, significantly improving diagnostic efficiency.

[0075] By constructing a dynamic fault feature map, previously isolated metrics, logs, call chains, and other multi-dimensional data are effectively integrated in time and space, forming a holistic view reflecting the real-time status of the system. Based on this, through iterative calculation of root cause scores, the root cause and secondary effects of faults can be distinguished from complex alarm phenomena, effectively avoiding the interference of alarm storms. This reduces the traditional troubleshooting process, which relies on manual experience and takes tens of minutes or even hours, to minutes or even seconds.

[0076] 2. Improved the safety and reliability of automated self-healing operations.

[0077] Before implementing any self-healing strategy, a context-adaptive risk assessment is performed. This assessment mechanism dynamically adjusts the risk weights for different dimensions such as service availability and data consistency by combining contextual information revealed by the current fault feature map (such as the importance and state of the faulty node), and quantifies the potential impact of the strategy. This overcomes the secondary disasters such as data loss or core business interruption that may result from the one-size-fits-all repair logic in traditional solutions, ensuring the controllability and security of automated operations.

[0078] 3. A complete automated closed loop from execution to verification has been built, enhancing the system's self-healing resilience.

[0079] This invention addresses the shortcomings of existing self-healing solutions that "end immediately after execution" by introducing closed-loop verification and automatic rollback mechanisms. After the self-healing strategy is completed, the system continuously monitors key indicators within a preset verification window to objectively confirm whether the fault has been truly resolved. If verification fails, the system automatically performs a rollback operation, restoring the system to its stable state before execution. This ensures that even if the automated decision is imperfect, it can be quickly corrected, preventing erroneous operations from causing continuous damage to the system.

[0080] 4. It provides a comprehensive and unified view of the state of cloud-native systems, fundamentally solving the problem of data silos.

[0081] The data acquisition and alignment module transforms information scattered across different monitoring platforms into a standardized event stream by uniformly encapsulating all heterogeneous raw data into time-series aligned triples containing service, instance, and timestamp information. This process provides a high-quality, unified data foundation for subsequent in-depth, cross-dimensional data correlation analysis (such as building fault maps), enabling the system to comprehensively understand the entire fault picture, rather than just seeing scattered anomalies.

[0082] 5. It has enabled a shift from reactive fault response to proactive predictive maintenance.

[0083] The predictive early warning module continuously analyzes historical and real-time indicator data, identifying potential abnormal trends and health deterioration patterns through time-series prediction models. Before a fault has a real impact on business operations, this module can trigger the diagnostic process in advance, transforming operations and maintenance from reactive remediation to proactive intervention. This not only effectively avoids or mitigates losses caused by faults but also represents a significant improvement in intelligent operations and maintenance capabilities.

[0084] This invention can be a system, method, and / or computer program product. This invention also discloses a cloud-native architecture fault recovery system based on multidimensional data, which is based on the aforementioned cloud-native architecture fault recovery method based on multidimensional data, comprising: The data acquisition and alignment module is used to acquire multidimensional heterogeneous data in the cloud-native architecture in real time, and perform time-series alignment on the multidimensional heterogeneous data based on a preset time-series alignment triplet to obtain standardized event data. The diagnosis and localization module is used to construct a dynamic fault feature map based on the standardized event data, and to perform fault diagnosis and root cause localization based on the fault feature map in order to identify potential fault root causes. The risk assessment module is used to match the root cause of the failure with the self-healing strategy. After matching the target self-healing strategy, the scenario-adaptive risk assessment is initiated to calculate the total risk score of the target self-healing strategy by analyzing the scenario information of the failure feature map. The decision-making and execution module is used to determine whether to execute the target self-healing strategy based on the risk level corresponding to the total risk score, and to perform closed-loop verification or rollback after execution.

[0085] Based on the spirit of this invention, those skilled in the art will readily conceive of a computer program product derived from the aforementioned cloud-native architecture fault recovery method based on multidimensional data. The computer program product may include a computer-readable storage medium on which computer-readable program instructions are loaded to cause a processor to implement various aspects of this disclosure. That is, this application also includes a terminal comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the aforementioned cloud-native architecture fault recovery method based on multidimensional data.

[0086] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0087] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0088] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A cloud-native architecture fault recovery method based on multidimensional data, characterized in that, Includes the following steps: Step 1: Collect multidimensional heterogeneous data in the cloud-native architecture in real time, and perform time-series alignment on the multidimensional heterogeneous data based on the preset time-series alignment triplet to obtain standardized event data; Step 2: Construct a dynamic fault feature map based on the standardized event data, and perform fault diagnosis and root cause localization based on the fault feature map to identify potential fault root causes; Step 3: Match the root cause of the fault with the self-healing strategy. After matching the target self-healing strategy, initiate a situation-adaptive risk assessment to calculate the total risk score of the target self-healing strategy by analyzing the situation information of the fault feature map. Step 4: Based on the risk level corresponding to the total risk score, determine whether to execute the target self-healing strategy, and perform closed-loop verification or rollback after execution.

2. The method according to claim 1, characterized in that, The time-aligned triple includes a service identifier, a container instance identifier, and a timestamp; The step of performing time-series alignment of the multidimensional heterogeneous data based on preset time-series alignment triples further includes: The raw data records from different sources are uniformly parsed and mapped into standardized events containing the service identifier, container instance identifier, and timestamp.

3. The method according to claim 2, characterized in that, The root cause localization further includes: Calculate the endogeneity anomaly degree of each node in the fault feature map, whereby the endogeneity anomaly degree is calculated from the node’s own anomaly features. By combining the influence weights that characterize the probability of fault propagation between nodes, the root cause score of each node is updated through iterative calculation, and the node with the highest root cause score is identified as the root cause of the fault.

4. The method according to claim 3, characterized in that, The iterative calculation of the root factor score satisfies the following relationship: ; in, For nodes In the Root cause score in rounds of iteration For nodes The degree of endogenous abnormality, The damping coefficient is... Pointing to a node upstream node In the Root cause score in rounds of iteration For the node To the node The influence weight.

5. The method according to claim 4, characterized in that, The situation-adaptive risk assessment further includes: The potential impact of the self-healing strategy under the current fault characteristic map is quantified from three dimensions: service availability, data consistency, and resource consumption, so as to obtain the risk score of each dimension. Based on the context of the fault feature map, the risk weights of the three dimensions are dynamically calculated; The total risk score is obtained by weighted summation of the risk scores from the three dimensions.

6. The method according to claim 5, characterized in that, The dynamic calculation of the risk weights of the three dimensions further includes analyzing the topological importance and attribute characteristics of nodes in the fault feature map; When the fault feature map contains state service nodes or core nodes in the topology, the risk weights of the data consistency dimension or service availability dimension are increased accordingly.

7. The method according to claim 6, characterized in that, The weight of the data consistency dimension The following relationship must be satisfied: ; in, This is the current fault feature map. For its set of nodes, For nodes Status score, For nodes The centrality score, This is the normalization factor.

8. The method according to claim 7, characterized in that, The closed-loop verification further includes: After the self-healing strategy is executed, a set of key performance indicators are continuously monitored within a preset verification window. If the key performance indicators have not recovered to the normal baseline by the end of the verification window, a predefined rollback mechanism will be automatically triggered.

9. The method according to claim 8, characterized in that, Also includes: Analyze historical indicator data using time series forecasting models to predict the probability of failure within future time windows; When the probability of failure exceeds the warning threshold, steps 2 to 4 are actively triggered to generate and evaluate preventive measures.

10. A cloud-native architecture fault recovery system based on multidimensional data, characterized in that, include: The data acquisition and alignment module is used to acquire multidimensional heterogeneous data in the cloud-native architecture in real time, and perform time-series alignment on the multidimensional heterogeneous data based on a preset time-series alignment triplet to obtain standardized event data. The diagnosis and localization module is used to construct a dynamic fault feature map based on the standardized event data, and to perform fault diagnosis and root cause localization based on the fault feature map in order to identify potential fault root causes. The risk assessment module is used to match the root cause of the failure with the self-healing strategy. After matching the target self-healing strategy, the module initiates a situation-adaptive risk assessment to calculate the total risk score of the target self-healing strategy by analyzing the situation information of the failure feature map. The decision-making and execution module is used to determine whether to execute the target self-healing strategy based on the risk level corresponding to the total risk score, and to perform closed-loop verification or rollback after execution.

11. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the cloud-native architecture fault recovery method based on multidimensional data according to any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the cloud-native architecture fault recovery method based on multidimensional data as described in any one of claims 1-10.

Images