Method, system and device for recommending a fault drill scenario and readable storage medium

Abstract

The application discloses a fault drill scene recommendation method, system, device and readable storage medium. The method comprises the following steps: constructing an architecture graph of a business cluster according to multi-source data, obtaining a fault risk type of each node, determining a target fault risk type of a business component, matching in a preset rule library, obtaining a fault drill scene, and generating a drill scheme in combination with historical drill data to update the fault risk type of the node and / or the preset rule library. The application ensures the pertinence of the fault drill scene recommendation by outputting the risk label of the constructed architecture graph, improves the efficiency of drill design by reusing the historical drill data, and finally optimizes the architecture graph and the rule library of the rich scene recommendation based on the drill effect of the target drill scheme, further improves the recommendation accuracy of the fault drill scene, and ensures the stability of the business cluster under the subsequent micro-service architecture.

Description

Technical Field

[0001] This application relates to the field of microservices technology, specifically to a recommended method, system, device, and readable storage medium for fault simulation scenarios. Background Technology

[0002] Microservice architecture is an architectural form that breaks down a monolithic application into independently deployed small service nodes. With the explosive growth of internet businesses, large-scale microservice architectures exhibit typical characteristics such as massive numbers of nodes, complex call chains (e.g., hundreds of millions of calls per day), and cross-cluster deployment. Therefore, to ensure the stable operation of microservice architecture, it is necessary to conduct advance drills and designs for potential failure scenarios in business clusters deployed with microservice architecture, thereby reducing the possibility of business crashes due to sudden failures.

[0003] The current lack of systematic analysis of architectural vulnerabilities in microservice architectures leads to insufficient simulation results when designing fault scenarios, which in turn affects subsequent online business. Summary of the Invention

[0004] This application provides a recommended method, system, device, and readable storage medium for fault drill scenarios, aiming to improve the design effectiveness of fault drill scenarios under microservice architecture.

[0005] Firstly, this application provides a method for recommending fault simulation scenarios, applied to a microservice architecture, the method comprising: An architecture graph of the business clusters is constructed based on multi-source data from multiple business clusters deployed in a microservice architecture. Graph analysis is then performed on the architecture graph to obtain the fault risk type of each node in the architecture graph. Based on the fault risk type of each node in the architecture graph, the target fault risk type associated with the target business component in the business cluster is determined, and the target fault risk type is matched in a preset rule base to obtain the target fault drill scenario of the target business component. Based on the target fault simulation scenario, a target simulation plan for the target business component is generated from historical simulation data; and Based on the exercise results of the target exercise scheme, the fault risk types of each node in the architecture graph and / or the preset rule base are updated to generate a recommended fault exercise scenario for the next business component based on the updated fault risk types and / or the updated preset rule base.

[0006] In one embodiment of this application, the multi-source data includes service registration information and link tracing information of the service cluster; The step of constructing the architecture graph of the business clusters based on multi-source data from multiple business clusters deployed in a microservice architecture includes: Based on the service registration information of the multiple business clusters, nodes in the business clusters are extracted, and heterogeneous nodes of the architecture graph of the business clusters are generated according to the nodes and their node attribute information. Based on the link tracing information of the multiple business clusters, the call information between nodes in the business clusters is extracted, and directed edges between corresponding heterogeneous nodes in the architecture graph are generated according to the call information.

[0007] In one embodiment of this application, the method further includes: In response to changes in the version identifier information in the node attribute information of heterogeneous nodes in the architecture graph, and / or changes in the hash value of the directed edge in the architecture graph, the architecture graph of the business cluster is incrementally updated.

[0008] In one embodiment of this application, the fault risk type includes a node risk type used to indicate the importance of a node; The graph analysis processing of the architecture graph yields the failure risk types of each node in the architecture graph, including: Based on the calling relationships between nodes in the architecture graph, a target node is determined for each node, and the weight of the target node is iteratively updated based on the weight of the target node. After the iterative update, nodes whose updated weights exceed a preset weight threshold are identified as risk nodes with node risk type in the architecture graph. The initial weight of each node is determined based on the node attribute information of each node.

[0009] In one embodiment of this application, the failure risk type includes an edge dependency risk type used to indicate the importance of dependencies between nodes; The graph analysis processing of the architecture graph yields the failure risk types of each node in the architecture graph, including: Identify strongly connected components in the architecture graph; if the strongly connected component contains multiple cross-cluster nodes, determine that the nodes in the strongly connected component are risk nodes with a first edge dependency risk type in the architecture graph; and / or The call chain depth of each node in the architecture graph is determined based on breadth-first search, and nodes whose call chain depth exceeds a preset depth threshold are identified as risk nodes with a second edge dependency risk type in the architecture graph.

[0010] In one embodiment of this application, the multi-source data further includes monitoring indicator data in the service cluster, and the fault risk type includes a capacity risk type used to indicate node capacity risk; The graph analysis processing of the architecture graph yields the failure risk types of each node in the architecture graph, including: Obtain monitoring metric data for each node in the architecture graph, wherein the monitoring metric data is used to indicate the load status of each node; Based on the monitoring index data of each node, time-series prediction is performed to obtain the predicted load status of each node, so as to identify the nodes with overload status as risk nodes with capacity risk in the business cluster.

[0011] In one embodiment of this application, the multi-source data further includes change record data in the business cluster, and the fault risk type includes a change risk type used to indicate the risk of node changes; The graph analysis processing of the architecture graph yields the failure risk types of each node in the architecture graph, including: Obtain change record data for each node in the architecture graph, as well as the number of downstream target nodes of each node; Based on the weighted result of the change frequency in the change record data and the number of downstream target nodes, the change risk score of each node in the architecture graph is determined, and nodes whose change risk scores exceed a preset score threshold are identified as risk nodes with change risk type in the microservice architecture.

[0012] In one embodiment of this application, the step of matching the target fault risk type in a preset rule base to obtain the target fault drill scenario for the target business component includes: The target fault simulation scenario for the target business component is obtained by matching the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base.

[0013] In one embodiment of this application, the step of matching the target fault simulation scenario of the target business component in the preset rule base according to the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component includes: The target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component are matched in the preset rule base to obtain the first fault drill scenario of the target business component. Based on the historical drill data of the reference business component associated with the target business component, a second fault drill scenario for the target business component is generated; The target fault simulation scenario is determined from the first fault simulation scenario and the second fault simulation scenario.

[0014] In one embodiment of this application, the reference service component associated with the target service component includes: A first business component having the same architectural characteristics and / or the same business level as the target business component; and / or A second business component whose graph embedding vector in the architecture graph exceeds a preset similarity threshold with the target business component, wherein the graph embedding vector of the business component is determined based on the node embedding vector of the node in the architecture graph.

[0015] In one embodiment of this application, generating the target exercise plan for the target business component from historical exercise data based on the target fault exercise scenario includes: Structured data extraction is performed on historical exercise data to construct a knowledge graph of the historical exercise data, and a mapping relationship is established between the knowledge graph and the architecture graph. Based on the mapping relationship between the knowledge graph and the architecture graph, feature vectors of each of the historical training data are generated through a trained graph neural network model; Based on the vector similarity between the feature vectors of the target fault drill scenario and the historical drill data, a target drill scheme matching the target fault drill scenario is determined from the historical drill data.

[0016] In one embodiment of this application, updating the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme includes: If the exercise score used to indicate the exercise effect of the target exercise scheme does not exceed the first preset score threshold, the risk identification threshold used to identify the fault risk type of each node in the architecture graph is adjusted to update the fault risk type of each node in the architecture graph. If the exercise score used to indicate the exercise effect of the target exercise scheme exceeds a second preset score threshold, the matching rules in the preset rule base are updated based on the target exercise scheme.

[0017] In one embodiment of this application, the method further includes: The target exercise plan is used as historical exercise data to update the knowledge graph of historical exercise data.

[0018] Secondly, this application also provides a recommendation system for fault simulation scenarios, applied to a microservice architecture, the system comprising: The architecture analysis module is used to construct an architecture graph of the business clusters based on multi-source data of multiple business clusters deployed in a microservice architecture, and to perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph. The scenario recommendation module is used to determine the target fault risk type associated with the target business component in the business cluster based on the fault risk type of each node in the architecture graph, and to match the target fault risk type in a preset rule base to obtain the target fault drill scenario of the target business component. The knowledge base module is used to generate a target exercise plan for the target business component from historical exercise data based on the target fault exercise scenario; and The optimization module is used to update the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme, so as to generate a recommended fault exercise scenario for the next business component based on the updated fault risk type and / or the updated preset rule base.

[0019] Thirdly, this application provides a computer device, the computer device comprising: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the recommended method of fault simulation scenarios as described in any of the preceding claims.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the recommended method for fault simulation scenarios as described in any of the preceding claims.

[0021] The fault drill scenario recommendation method provided in this application constructs a collaborative architecture of three engine modules. First, it uses multi-source data from multiple business clusters deployed in a microservice architecture to build an architecture graph of the business clusters. Then, it combines graph analysis processing technology to identify the risks of nodes in the graph and obtain fault risk types. Next, based on the fault risk types of the nodes, it determines the target fault risk types of the target business components in the business clusters. Based on the target fault risk types, it matches them in a preset rule base to obtain the recommended target fault drill scenarios for the target business components. At the same time, it reuses historical drill data to generate corresponding target drill plans. Based on the execution effect of the drill plans, it updates the fault risk types of each node in the architecture graph and / or the preset rule base to improve the recommendation effect of fault drill scenarios for subsequent business components. The solution provided in this application ensures the relevance of fault drill scenario recommendations by constructing risk labels output by the architecture graph. At the same time, it improves the efficiency of drill design by reusing historical drill data. Finally, it optimizes the architecture graph and enriches the rule base for scenario recommendations based on the drill effect of the target drill plan, further improving the accuracy of fault drill scenario recommendations and ensuring the stability of business clusters under the subsequent microservice architecture. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A system architecture diagram is provided for an embodiment of this application to recommend a fault simulation scenario for a movie. Figure 2 A flowchart illustrating the steps of a recommended method for a fault simulation scenario provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the steps for obtaining fault risk types by performing graph analysis on an architecture graph, as provided in an embodiment of this application. Figure 4a This is a schematic diagram illustrating the process of constructing a weighted directed heterogeneous graph of a business cluster in the architecture analysis module provided in this application embodiment; Figure 4b This is a schematic diagram illustrating the process of constructing risk type labels from nodes in the identification architecture graph provided in an embodiment of this application; Figure 5a This is a schematic diagram illustrating a process for generating candidate scenes, provided in an embodiment of this application. Figure 5bA schematic diagram illustrating the steps for scoring fault drill scenarios provided in this application embodiment; Figure 6 A flowchart illustrating the steps for constructing a knowledge graph of historical exercise data is provided in this embodiment of the application. Figure 7 A simplified structural diagram of a recommendation system for a fault simulation scenario provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0026] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0027] To clearly understand the recommended method for fault simulation scenarios provided in this application's embodiments, the relevant application scenarios of this method are first explained. Specifically, the recommended method for fault simulation scenarios provided in this application is typically applied in microservice architectures, where a monolithic application is split into independently deployed small service nodes. With the explosive growth of internet industry businesses, large-scale microservice architectures, such as multiple business clusters deployed in a microservice architecture, often exhibit typical characteristics such as massive numbers of nodes (tens of thousands of nodes), complex call chains (e.g., hundreds of millions of calls per day), and cross-cluster deployment. To ensure the stable operation of microservice architectures, it is often necessary to design and conduct advance simulations of potential fault scenarios in the business clusters, thereby reducing the abnormal problem of business crashes due to sudden failures.

[0028] However, current drill design in microservice architectures largely relies on human experience, resulting in problems such as poor risk identification, poor drill scenario recommendation, and low efficiency in drill design due to low experience reuse. To address these technical issues, this application provides a method, system, device, and readable storage medium for recommending fault drill scenarios. By constructing a collaborative architecture of three engine modules, it achieves data interoperability between "node risk identification," "fault scenario recommendation," and "drill scheme reuse." In particular, by combining forward-driven and reverse-optimization, it effectively improves the recommendation accuracy of fault drill scenarios.

[0029] For a better understanding of the above, please refer to [link / reference]. Figure 1 , Figure 1 This application provides a system architecture diagram for a method of recommending fault drill scenarios for movies. Specifically, it includes an architecture vulnerability analysis engine 110 (hereinafter referred to as the architecture analysis module in subsequent embodiments for ease of description), a drill scenario recommendation engine 120 (hereinafter referred to as the scenario recommendation module in subsequent embodiments for ease of description), and a drill knowledge base module 130 (hereinafter referred to as the knowledge base module in subsequent embodiments for ease of description). The architecture analysis module constructs an architecture graph by integrating multi-source data from multiple business clusters deployed in a microservice architecture. Using graph processing technology, it identifies risk nodes within the architecture graph and provides risk labels to the scenario recommendation module. The scenario recommendation module, based on the provided risk labels, matches corresponding fault scenarios in a preset rule base and provides them to the knowledge base module. The knowledge base module, combining historical drill data, provides corresponding drill plans. Based on the execution effect of the drill plan, it updates and optimizes the risk labels provided by the architecture analysis module and the preset rule base in the scenario recommendation module. Specific embodiments will be described below.

[0030] Please see Figure 2 , Figure 2 This is a flowchart illustrating the steps of a recommended method for a fault simulation scenario provided in an embodiment of this application. This method is typically used for processing business clusters under a microservice architecture, specifically including steps S210 to S240: S210, construct an architecture graph of the business clusters based on multi-source data of multiple business clusters deployed in a microservice architecture, and perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph.

[0031] In the embodiments of this application, the above steps are typically performed by the architecture vulnerability analysis engine 110, i.e., the architecture analysis module.

[0032] In one embodiment of this application, a business cluster is typically used to implement a specific service provided by a vendor, such as a payment cluster, a logistics cluster, a membership cluster, etc., and these business clusters are usually interconnected. Multiple business clusters deployed in a microservice architecture are typically split into independently deployed small nodes according to business hierarchy, such as microservices, middleware, databases, etc.

[0033] Furthermore, the multi-source data of the business cluster can typically be collected by lightweight data acquisition programs deployed in various business clusters for collaborative work. The collected multi-source data is usually used to construct an architecture graph that can describe the overall architecture of the business cluster. For example, in one embodiment, the multi-source data of the business cluster can typically include service registration information and link tracing information. Service registration information can be understood as metadata registered by a service instance with a registry center, enabling service consumers to discover, locate, and invoke the service instance. Link tracing information can be understood as data used to record and display the complete path and detailed information of a request calling multiple services. Therefore, the service registration information and link tracing information based on the multi-source data can correspond to nodes and edges in the business cluster, thereby constructing an architecture graph that describes the overall architecture of the business cluster. Specifically, in one embodiment of this application, the architecture graph can be a directed heterogeneous graph. That is, in one embodiment, constructing the architecture graph of the business cluster based on the multi-source data of multiple business clusters deployed in a microservice architecture includes: Based on the service registration information of the multiple business clusters, nodes in the business clusters are extracted, and heterogeneous nodes of the architecture graph of the business clusters are generated according to the nodes and their node attribute information. Based on the link tracing information of the multiple business clusters, the call information between nodes in the business clusters is extracted, and directed edges between corresponding heterogeneous nodes in the architecture graph are generated according to the call information.

[0034] To facilitate understanding of the above content, the following will be described in conjunction with specific embodiments. Specifically, in one embodiment of this application, service registration information can be obtained from service registration centers, such as Nacos (a dynamic service discovery, configuration management, and service management platform), Eureka (a service discovery framework focused on dynamically locating middleware services in cloud environments and supporting load balancing and failover), and ZooKeeper (a distributed application coordination service), based on a collection program (such as an Agent cluster). The service registration information may specifically include the type of a service node, such as a microservice, middleware, or database. Furthermore, the service registration information may also include the service node's business level, such as core or non-core; its business cluster affiliation, such as a payment cluster or a logistics cluster; its Service Level Objective (SLO), i.e., the SLO level, which indicates the version identifier for incremental updates; and its regional information, indicating the physical deployment location, etc. In this way, for each service node, the information extracted from the service registration information can be used as the node attribute information of that node, thus serving as a heterogeneous node in the architecture graph of the business cluster. Here, heterogeneous nodes mean that each node in the architecture graph has different attribute information, which is used to indicate the unique node type and function of the service node.

[0035] Furthermore, in one embodiment of this application, link tracing information can be extracted from a link tracing system, such as SkyWalking (an application performance monitoring system that provides integrated observation capabilities including distributed tracing, service topology analysis, performance indicator monitoring and visualization) and Pinpoint (a program for providing application performance management for distributed systems), based on a data collection program. Typically, by extracting the call paths from the link tracing information, edges between service nodes can be constructed sequentially, thus serving as directed edges between corresponding heterogeneous nodes in the architecture graph. Furthermore, based on the link tracing information, the edge attribute information of each directed edge can also be determined. For example, it may include call frequency, indicating the call frequency or number of calls between service nodes; business link affiliation, indicating the business cluster to which the calling service belongs; and edge attribute information may also include hash verification information for verifying edge updates. The hash verification information and the node's version identifier will be used for incremental updates to the architecture graph of the business cluster. For example, in one possible implementation, the method further includes: In response to changes in the version identifier information in the node attribute information of heterogeneous nodes in the architecture graph, and / or changes in the hash value of the directed edge in the architecture graph, the architecture graph of the business cluster is incrementally updated.

[0036] In other words, incremental updates to the graph are triggered by change events. Through a dual verification mechanism of "node version identifier (recording the timing of component deployment / configuration changes) + edge hash verification (verifying the integrity of dependencies)," incremental synchronization is only triggered when the component version number is updated and / or the edge hash value changes. Compared with the existing full update solution, this can reduce the amount of data transmission by more than 90%, and the update latency is ≤500ms, ensuring the real-time performance of the constructed architecture graph.

[0037] Furthermore, the above embodiments use a directed heterogeneous graph as an example for illustration. In fact, in some embodiments, the architectural graph can also be a weighted directed heterogeneous graph, that is, different weights can be assigned to nodes and / or edges in the graph. For example, in one possible implementation, taking nodes as an example, weights can be set based on information such as business level (e.g., core or non-core), cluster affiliation, and SLO level in the node attributes. In particular, for core nodes with a business level of core, their weight can be set to 1.5 times. Similarly, the weights of edges can also be set appropriately based on attributes such as cross-cluster marking, call frequency, and business link affiliation. This application does not limit the specific scheme for constructing a weighted directed heterogeneous graph, but by constructing a weighted graph, the subsequent identification effect of node failure risks can be further improved.

[0038] Of course, the aforementioned solution is merely an example of constructing an architecture graph using partial multi-source data. In reality, multi-source data can include more information based on actual needs. For example, monitoring metrics such as CPU, memory, and network utilization can be extracted from monitoring systems like Prometheus (an operations and maintenance monitoring system that stores information in time-series data format) to monitor the load status of nodes. Furthermore, multi-source data can also include data from continuous delivery / continuous deployment pipelines—that is, all status, performance, and result information generated in the automated process from program submission to production deployment. Change log data indicating node changes can typically be extracted to confirm the impact of frequent node changes on downstream nodes. It should be noted that this multi-source information can enrich the construction of the business cluster's architecture graph, facilitating the identification of more types of fault risks in the architecture graph through graph analysis. Specific implementation schemes will be described in subsequent related embodiments.

[0039] Building upon the aforementioned solutions, agent clusters can continuously collect relevant multi-source data and transmit it to a graph database via a distributed messaging system, such as Kafka (a high-throughput distributed publish-subscribe messaging system). This allows for the construction of an architecture graph within the graph database, identifying potential fault risk types for nodes. Furthermore, considering that the architecture graphs of tens of business clusters typically contain hundreds of thousands of nodes and millions of edges, a distributed graph database, such as JanusGraph (a distributed graph database engine), can be used to improve graph processing efficiency. This allows for sharded storage of nodes and edges according to business clusters or physical deployment regions. Each shard operates independently and supports horizontal scaling, thus avoiding storage bottlenecks for large-scale data. Moreover, the Kafka distributed messaging system can achieve high throughput of hundreds of thousands of TPS (Transactions Per Second), effectively preventing the loss of collected multi-source data and ensuring the correctness of the architecture graph.

[0040] After constructing the architecture graph of the business cluster using the methods described above, the graph processing algorithms or related analysis models deployed in the architecture vulnerability analysis engine can accurately identify the types of potential failure risks of key nodes in the business cluster architecture. In other words, based on graph analysis of the architecture graph, the failure risk types of each node in the architecture graph can be obtained. Specifically, to facilitate understanding of the above, the following will be explained in conjunction with specific embodiments.

[0041] In one embodiment of this application, the fault risk type typically includes a node risk type indicating the importance of a node. Furthermore, in some embodiments of this application, the fault risk type may also include an edge dependency risk type indicating the importance of dependencies between nodes. Of course, the fault risk type may also include both the node risk type and the dependency risk type simultaneously. For details, please refer to... Figure 3 , Figure 3 This diagram illustrates a step-by-step flowchart for obtaining fault risk types by performing graph analysis on an architecture graph, specifically including steps S310 to S330: S310, based on the calling relationship between each node in the architecture graph, determine the target node pointing to each node, and iteratively update the weight of the target node pointing to it based on the weight of the target node. After the iterative update is completed, the node whose updated weight exceeds the preset weight threshold is determined as a risk node with node risk type in the architecture graph.

[0042] In one embodiment of this application, the importance score of each node in the architecture graph can be calculated based on a ranking algorithm. Specifically, taking a feasible ranking algorithm as an example, the importance score of each node in the architecture graph depends on the weights of other target nodes pointing to that node. For example, if the target nodes pointing to node A include nodes B, C, and D, then the updated weight of node A is obtained by summing the weights of nodes B, C, and D. The weights of nodes B, C, and D can usually be determined based on the current weight of node B and the number of other nodes it points to. For example, if the current weight of node B is 0.5, and it points to both node A and node D, that is, it points to two nodes, then the weight from node B to node A is 0.25. Therefore, using the aforementioned sorting algorithm, after setting the initial weights of each node—for example, based on the node attribute information mentioned above, such as whether it is a core business, SLO level, or cluster affiliation—the weights of each node are iteratively updated using the sorting algorithm until a preset condition is met. For example, the iteration can be terminated by judging whether the Tolerance has reached a preset threshold used to confirm whether the iteration result is "good enough." In one possible implementation, the Tolerance could be set to 10. -6 In other words, when the difference in weight between a node before and after the iteration update does not exceed this value, the iteration termination condition can be considered met, thus ending the iteration process. It can be understood that since the weight of each node in the architecture graph depends on the sum of the weights of other nodes pointing to that node, the node's weight can be used to indicate the node's importance score, that is, the degree of impact of a single node on the business, thereby accurately locating single points of failure (SPOF) in the core business chain, i.e., risky nodes with node risk types.

[0043] For example, in one possible implementation, nodes whose updated weights exceed a preset weight threshold can be identified as risk nodes with node risk type in the architecture graph. Here, the preset weight threshold can be a fixed value, or it can be determined based on the statistical results of the weights of each node. For instance, the preset weight threshold can be set as the average weight of each node plus twice the standard deviation. That is, when the weight of a node is higher than this weight threshold, the node can be marked as a single point of failure (SPOF), thus becoming a risk node with node risk type. Of course, in some embodiments of this application, the preset weight threshold can also be dynamically updated based on the training results of subsequent training plans. The specific update process will be described in subsequent related embodiments.

[0044] The above-described scheme is merely a ranking method for calculating node importance scores to identify single points of failure (SPOF) in the core business chain. In reality, based on actual needs, to avoid the continuous accumulation of weights for dangling nodes without any outgoing links during weight updates, a damping coefficient d (usually less than 1) can be introduced to achieve iterative weight updates. The weights allocated to other target nodes pointing to that node are calculated only based on the probability of the damping coefficient. The proportions outside the damping coefficient are obtained from the entire "common pool." Specifically, the formula for updating weight RP is as follows: RP(i)=(1-d) / N+d×RP(j) Where RP(i) is the weight RP score of the i-th node, N is the number of nodes in the architecture graph, d is the damping coefficient, and RP(j) is the sum of the assigned weights of all other nodes pointing to the i-th node. In one possible implementation, the damping coefficient can be set to 0.85.

[0045] The aforementioned solution can identify single-point risks in the architecture graph, thereby outputting corresponding risk labels to provide subsequent fault scenario recommendations.

[0046] S320, Identify strongly connected components in the architecture graph. If the strongly connected components contain multiple cross-cluster nodes, determine that the nodes in the strongly connected components are risk nodes with a first edge dependency risk type in the architecture graph.

[0047] In addition to identifying single points of risk—that is, identifying single points with high importance scores and significant impact on business—the call relationships between nodes, especially those involving cross-cluster or long call chains (dependencies), can also be used to locate core bottleneck nodes within these dependencies. Specifically, this will be explained below with reference to concrete examples.

[0048] Specifically, in one embodiment of this application, strongly connected components in an architecture graph can be identified using the Tarjan algorithm (a linear-time algorithm for solving strongly connected components of a directed graph). A strongly connected component refers to a subgraph in the architecture graph in which each node can reach another node through directed edges in the graph (the path must include the path from node u to node v and from node v to node u), and there are no other nodes in the architecture graph that can be extended to this subgraph without breaking the bidirectional reachability of the subgraph, i.e., the largest connected subgraph that meets the bidirectional reachability requirement. Based on this, if a strongly connected component contains multiple cross-cluster nodes, then the nodes in the strongly connected component can be identified as risk nodes with the first-side dependency risk type in the architecture graph. That is, they are key nodes with core bottlenecks in the call relationship (dependency relationship). Specifically, the first-side dependency risk type can be circular dependency risk, that is, there may be multiple nodes that directly or indirectly depend on each other, which may lead to a cascading amplification of system failures (such as the performance degradation of node A will be passed on through the call relationship and eventually back to A, causing a chain of crashes of service nodes in the dependency relationship) or performance bottleneck risk.

[0049] S330, determine the call chain depth of each node in the architecture graph based on breadth-first search, and identify nodes whose call chain depth exceeds a preset depth threshold as risk nodes with a second side dependency risk type in the architecture graph.

[0050] Of course, in addition to the first type of edge dependency risk mentioned above, namely nodes with circular dependency risk, for the risk of cascading performance degradation of on-chain service nodes due to long call chains, some embodiments of this application also provide a method to determine the call chain depth of each node in the architecture graph through breadth-first search, such as layered BFS (Breadth-First Search), and identify nodes whose call chain depth exceeds a preset depth threshold. For example, nodes with a call chain depth exceeding 10 layers are identified as having the second type of edge dependency risk in the architecture graph, namely risk nodes marked as long call chain risk.

[0051] Of course, the methods described above are merely based on graph analysis of the architecture graph to identify key bottleneck nodes within it. Specifically, distributed parallel computing can be implemented using Spark GraphX ​​(a distributed graph processing framework for graph computation, primarily used for the storage, manipulation, and analysis of large-scale graph data), thereby completing computations on hundreds of thousands of nodes within minutes.

[0052] Building upon the aforementioned foundations, and based on actual needs, the types of failure risks can include even more risk types, which are typically determined by relying on more multi-source data. The following will illustrate this with specific examples.

[0053] Specifically, in one embodiment of this application, the multi-source data further includes monitoring indicator data from the business cluster. In this case, the fault risk type also includes a capacity risk type indicating node capacity risk. Then, the step of performing graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph includes: Obtain monitoring metric data for each node in the architecture graph, wherein the monitoring metric data is used to indicate the load status of each node; Based on the monitoring index data of each node, time-series prediction is performed to obtain the predicted load status of each node, so as to identify the nodes with overload status as risk nodes with capacity risk in the business cluster.

[0054] Specifically, the Agent cluster can obtain monitoring metrics data for each node from a monitoring system, such as the Prometheus federated cluster. These monitoring metrics data can typically be used to indicate the load status of each node, and usually include CPU, memory, and network utilization, etc.

[0055] Building upon the aforementioned foundation, by collecting monitoring indicator data for each node within a preset historical period—for example, historical monitoring indicator data from the most recent week or half a month—and inputting this data into a trained neural network model with time-series prediction capabilities, such as LSTM (Long Short-Term Memory), it becomes possible to predict the load status of nodes within a preset future period. For instance, this could predict the resource utilization of nodes within the next 24 hours, such as CPU, memory, or network utilization, and identify nodes that may be overloaded, marking them as risk nodes with capacity risks within the business cluster. Of course, based on actual scenario requirements, different capacity thresholds can be set for core or non-core nodes to determine whether a node is overloaded; this application embodiment does not impose such limitations.

[0056] Furthermore, in another embodiment of this application, the multi-source data also includes change record data in the business cluster. In this case, the fault risk type includes a change risk type used to indicate the risk of node changes. That is, the graph analysis processing of the architecture graph to obtain the fault risk type of each node in the architecture graph includes: Obtain change record data for each node in the architecture graph, as well as the number of downstream target nodes of each node; Based on the weighted result of the change frequency in the change record data and the number of downstream target nodes, the change score of each node in the architecture graph is determined, and nodes with change scores exceeding a preset score threshold are identified as risk nodes with change risk type in the microservice architecture.

[0057] In this application, considering that a changed node usually affects downstream nodes, i.e., other nodes that directly or indirectly depend on it, especially when nodes change frequently, it is very easy to affect downstream core nodes. Therefore, in one embodiment of this application, the change record data of the node can be read from the CD (Continuous Delivery) pipeline data through the Agent cluster, and the change risk score of the node can be obtained based on the change record data and the weighted result of the number of downstream target nodes, such as the number of downstream core nodes. Nodes whose change risk scores exceed the preset score threshold are identified as risk nodes with change risk type in the microservice architecture.

[0058] Generally, the more frequent the changes (i.e., the higher the frequency) and the more downstream target nodes there are, the higher the change risk score, indicating that the node has a high-frequency change risk. By identifying this node and recommending fault simulation scenarios, cascading failures caused by changes can be avoided. For example, in one possible implementation, if the number of changes exceeds 3 within 24 hours, the node can be considered to have a high-frequency change risk. Based on this, by further determining the number of downstream target nodes, the final change risk score of the node can be frequency × 10 + number of downstream core nodes × 15. This change score can be used as feature information of the node's change risk type and passed to the scenario recommendation module for recommending fault simulation scenarios.

[0059] Of course, the above-mentioned solution is only a feasible implementation scheme for constructing an architecture graph and identifying the types of node failure risks in the architecture graph. Any conventional improvements made by those skilled in the art based on this solution should be considered within the scope of protection claimed by the technical solution of this application.

[0060] Furthermore, in one embodiment of this application, the risk type identified by any of the foregoing methods can be provided as a corresponding risk type label to the scenario recommendation module for recommending fault drill scenarios.

[0061] To clearly understand the execution flow of the architecture analysis module provided in the embodiments of this application, please refer to... Figure 4a and Figure 4b ,in, Figure 4a This diagram illustrates the process of constructing a weighted directed heterogeneous graph of a business cluster in the architecture analysis module. Specifically, it includes the following steps: (1) Multi-source data collection, such as obtaining service registration information from service registration centers such as Nacos, Eureka, and ZooKeeper (ZK) through Agent clusters, obtaining link tracing information from link tracing systems such as SkyWalking and Pinpoint, obtaining indicator monitoring data from monitoring coefficients such as Prometheus, and reading node change record data from CD (Continuous Delivery) pipeline data; (2) The weighted directed heterogeneous graph is constructed by transmitting the message to a graph database, such as the distributed graph database JanusGraph, through a distributed messaging system, such as the Kafka message queue; (3) Extract node information from service registration information, such as services, middleware or databases, and then add corresponding node attributes to the nodes based on the service registration information, including but not limited to business level (core / non-core), SLO level, weight (for core nodes, the weight needs to be set to 1.5 times), version number of node change and availability zone (physical deployment area). (4) Extract edge information from the link tracing information, that is, the call (dependency) relationship between nodes. If there is a call data of node A calling node B, add a directed edge from node A to node B, and add the attribute information of the edge according to the link tracing information, including but not limited to call frequency, cross-cluster mark, business link affiliation and edge hash verification. (5) After the architecture graph is completed, the node version number and edge hash are used for double verification. Synchronization is triggered only when the node version number changes or the edge hash value changes. The incremental update reduces the amount of data transmission and reduces the update delay. (6) The distributed graph database JanusGraph is sharded and stored according to business clusters or availability zones for subsequent identification of the failure risk types of nodes in the architecture graph.

[0062] and Figure 4b This further illustrates a flowchart of the process for identifying nodes in the architecture graph to construct risk type labels. It uses different modules to identify different risk types for nodes, specifically including the following: I. Single-point risk identification module (using PageRank ranking algorithm) (1) Initialize algorithm parameters, including: damping coefficient, for example, it can be set to 0.85. In addition, it can also include the initial weight of each node, such as setting information based on the node's attributes such as SLO level, business level, etc., and tolerance used to indicate the iteration termination condition of the algorithm. (2) Introduce business level weight optimization, set the weight of core nodes to a multiple of the original weight, such as 1.5 times, to improve the algorithm's attention to core nodes; (3) Iterative computation can be performed using Spark Graphx distributed parallel computation, where hundreds of thousands of nodes can complete iterative computation in minutes to obtain the weight RP score of each node; (4) Calculate the risk identification threshold and identify single point of risk (SPOF) based on the weighted RP score and the risk threshold. The risk identification threshold can be set as the mean of the node weighted RP score + 2 times the standard deviation, and nodes with RP scores exceeding the risk identification threshold are marked as single point of risk (SPOF).

[0063] II. Dependency Risk Identification Module (1) Identify strongly connected components (SCCs) in the architecture graph using the Tarjan algorithm. (2) Detect the number of nodes in the strongly connected component, and determine whether the node is across the cluster when there are multiple nodes, that is, when the number of nodes is greater than 1. When a strongly connected component across the cluster is detected, it is marked as having a first-edge dependency risk type, that is, having a circular dependency risk. (3) Calculate the call chain depth of the node using the BFS (Breadth-First Search) algorithm, and when the call chain depth reaches a preset depth threshold, for example, more than 10 layers, mark it as having a second-side dependency risk type, that is, having a long call chain risk.

[0064] III. Capacity Risk Identification Module (Based on Long Short-Term Memory Network LSTM) (1) Load preset historical data, such as historical monitoring data within 7 days, which usually includes load indicators such as CPU, memory, and network utilization. (2) Input the loaded data into the Long Short-Term Memory (LSTM) network to predict the resource utilization rate for a preset period, such as 24 hours. (3) Set differentiated thresholds based on the business level of the node (such as core or non-core). For example, the threshold for core nodes can be set to 75%, and the threshold for non-core nodes can be set to 85%. (4) Mark nodes whose predicted maximum resource utilization rate exceeds the corresponding threshold as nodes with capacity overflow risk; IV. Modification of the High-Frequency Risk Identification Module (1) Load the node's change records within 24 hours and detect the frequency of node changes; (2) Filter out nodes with a change frequency of ≥3 times, and analyze the downstream core dependencies of the node, that is, the number of downstream core nodes that directly or indirectly call the node. (3) The impact score of the change is calculated by weighted algorithm, specifically: frequency x10 + number of downstream cores x15. The higher the frequency or the more downstream cores, the greater the impact of the node change. (4) Mark it as a high-risk change and characterize the level of risk by scoring the impact of the change.

[0065] After the analysis and processing of the architecture graph is completed through the solutions provided by any of the aforementioned modules, the risk types of each node can be further summarized, and the final comprehensive risk score can be calculated in a weighted manner. Then, the risk identification results associated with the node, i.e., risk identification labels, can be output. Specifically, the label can usually include the specific risk type, risk score, and other detailed descriptions. For example, as a possible implementation, the risk scores identified by each module can be identified as sequence vectors.

[0066] To facilitate understanding of the above content, the following analysis will be conducted in conjunction with specific business scenarios. Specifically, data from the service registry (ZK), tracing (PinPoint), and CD pipelines of 10 business clusters are collected via an Agent cluster. This data is then transmitted to the JanusGraph distributed graph database via Kafka to construct a weighted directed heterogeneous graph with 100,000 nodes and 5 million edges, with an incremental update latency of 420ms.

[0067] Storage and Computation Layer: JanusGraph distributed graph database is used, and node and edge data are stored in shards according to business clusters; Spark cluster (Master + 10 Workers) supports parallel graph computing to ensure graph construction efficiency; Risk identification results: The PageRank algorithm identified the order-service node in the payment cluster as having a PageRank score of 0.15 (more than twice the standard deviation of the mean), indicating a single point of failure in the core business. The Tarjan algorithm detected a circular dependency across two clusters on this node. Combined with change data, it was found that there were 5 changes in the past 24 hours, involving 4 downstream core nodes. Finally, it was marked as a composite risk of "single point of failure + circular dependency + high-risk change", with a score of 89.8, which verified the risk identification capability of the engine.

[0068] The architecture vulnerability analysis engine provided in this application integrates multiple data sources and various graph algorithms to achieve integrated identification of multiple types of risks, with an identification accuracy of ≥90% and an omission rate of less than 5%, providing a precise basis for exercise design.

[0069] S220, based on the fault risk type of each node in the architecture diagram, determine the target fault risk type associated with the target business component in the business cluster, and match it in a preset rule base according to the target fault risk type to obtain the target fault drill scenario of the target business component.

[0070] Based on the aforementioned construction of the architecture map and the output of the fault risk type of each node in the architecture map, in the embodiments of this application, the exercise scenario recommendation engine 120, that is, the scenario recommendation module, will further determine the target fault risk type associated with the target business component in the business cluster based on the fault risk type output by the architecture analysis module.

[0071] Specifically, target business components in a business cluster are typically deployed on one or more nodes in a microservice architecture. Therefore, the fault risk type associated with the target business component in the business cluster can be determined based on the fault risk type of the nodes involved. This is then further matched against a pre-built rule base to obtain the target fault drill scenario that matches the target fault risk type. It can be understood that the pre-built rule base typically stores the association between different fault risk types and their corresponding fault drill scenarios.

[0072] Of course, in addition to the fault risk type labels output by the architecture analysis module, the scenario recommendation module can further integrate more information about the target business components to achieve matching in the preset rule base, thereby obtaining the corresponding fault drill scenario.

[0073] For example, in one possible implementation, the step of matching the target fault risk type in a preset rule base to obtain the target fault drill scenario for the target business component includes: The target fault simulation scenario for the target business component is obtained by matching the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base.

[0074] Specifically, in one embodiment of this application, a preset rule base describes fault drill scenarios with specific fault risk types corresponding to different component types under different architectural features. For example, in one possible implementation, the preset rule base may include rules for matching different network fault modes. For instance, in one possible implementation, for microservice or middleware business components, when they have cross-cluster dependency risks and have "multi-region" or "cross-AZ (cross-availability zone)" architectural features, network fault features such as "cross-region network latency" and "cross-region network partitioning" may often occur. Alternatively, for microservice business components, when they have single-point risks and belong to the core layer, network fault features such as "network packet loss" may occur. Of course, the above-described embodiments are based on network fault modes as examples. In fact, in addition to network fault modes, other fault modes may also be included, such as component fault modes or data fault modes. For example, taking component failure modes as an example, when the component type is a microservice, and it has both single point of failure risk and capacity overflow risk, and has the architectural characteristics of being deployed on a Kubernetes cluster, then it can be considered that the component may experience component failures such as random termination of Pods (the smallest unit deployed and managed in a Kubernetes cluster) or Pod CPU burnout. Alternatively, for a component type of database, when it has single point of failure risk and the architectural characteristics are master-slave architecture, then the component may experience component failures such as database master-slave failover or database connection exhaustion.

[0075] Of course, the solutions provided above are only some possible implementations. In fact, based on actual needs, the preset rule base can also include more fault modes related to the component type of the business component (such as microservices, middleware, or databases), the fault risk types associated with the component (such as single point of failure risk, circular dependency risk, capacity overflow risk, etc. output by the architecture analysis module), and the architectural characteristics of the component (such as master-slave architecture, core layer, or deployment on a certain type of cluster). This allows for association with the target fault drill scenario corresponding to the fault mode, enabling advance fault drills for the corresponding component. In particular, the preset rule base can be further enriched and supplemented based on historical drill data.

[0076] Of course, the aforementioned solution only recommends target fault drill scenarios based on matching within a preset rule base. In fact, besides matching the corresponding first fault drill scenario from the preset rule base based on the relevant characteristics of the business component, in some embodiments of this application, since the drill database module stores historical related drill data, other fault drill scenarios that may be related to the target business component can be further provided based on this drill data. For example, for a reference business component with a specific correlation to the target business component, a second fault drill scenario for the target business component can be generated based on its historical drill data, thereby facilitating the subsequent selection of the target fault drill scenario from the first fault drill scenario and the second fault drill scenario, for example, based on multi-dimensional matching scores. That is, in one embodiment of this application, the step of matching the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base to obtain the target fault drill scenario for the target business component includes: The target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component are matched in the preset rule base to obtain the first fault drill scenario of the target business component. Based on the historical drill data of the reference business component associated with the target business component, a second fault drill scenario for the target business component is generated; The target fault simulation scenario is determined from the first fault simulation scenario and the second fault simulation scenario.

[0077] The implementation scheme for matching the target business component's first fault simulation scenario in a preset rule base based on the target fault risk type, the target business component's component type, and the target business component's architectural characteristics will not be elaborated upon in this embodiment.

[0078] Based on the historical drill data of the reference business components associated with the target business component, a second fault drill scenario for the target business component is generated. Here, the reference business components include two types: the first business component recommended by the collaborative filtering layer that has the same architectural features and / or the same business level, or the second business component with similar embedding vectors obtained by the graph embedding matching layer, i.e., the GraphSAGE algorithm. Specifically, for the sake of understanding the above content, they will be explained separately below.

[0079] Specifically, in one embodiment of this application, the first business component refers to other business components that have the same architectural characteristics and / or the same business level as the target business component. In particular, the first business component can be other business components with the same architectural characteristics and the same business level as the target business component, such as those belonging to the core business level of different business clusters and all deployed on a Kubernetes cluster. Thus, based on collaborative filtering, a first business component similar to the target business component can be obtained, and historical exercise data of this first business component can be further extracted from the subsequent exercise database to generate the corresponding second fault exercise scenario.

[0080] Of course, the above collaborative filtering only filters similar business components based on architectural features and business layers. In fact, the GraphSAGE algorithm can aggregate the neighbor information of nodes to obtain embedding vectors that describe the structural characteristics of each node in the graph. These embedding vectors can usually describe the similarity of the nodes' structures in the graph architecture. Generally, the closer the embedding vectors are, the more similar the structures of the two nodes in the graph are, and thus they have more similar properties. Therefore, based on the nodes involved in the target business component, the node embedding vectors obtained by the GraphSAGE algorithm are fused to obtain the graph embedding vector of the target business component, that is, the component embedding vector. On this basis, by using IVF_FLAT vector quantization technology, the second business component whose graph embedding vector exceeds a preset similarity threshold with the graph embedding vector of the target business component can be retrieved. The node embedding vectors obtained by the GraphSAGE algorithm usually need to rely on pre-trained model training. This method can usually be executed in a training knowledge base, and this application embodiment does not limit this.

[0081] After selecting reference business components associated with the target business component through any of the aforementioned methods, a second fault drill scenario for the target business component can be generated based on the historical drill data of the reference business components. In particular, in some embodiments of this application, the fault drill scenario can be selected from historical drill data with ≥3 execution times and an average effect score ≥75 points as the second fault drill scenario.

[0082] After recommending the first and second fault drill scenarios as described above, a series of candidate fault drill scenarios can be output based on merging and deduplication. For each candidate fault drill scenario, a multi-dimensional scoring model can be used to quantitatively evaluate the candidate fault drill scenario, thereby selecting the fault drill scenario with higher priority as the target fault drill scenario.

[0083] Specifically, in some embodiments of this application, the multi-dimensional scoring model can achieve quantitative evaluation of candidate fault simulation scenarios through at least one of the following dimensions, which may include: Architecture fit, risk relevance, effectiveness of historical drills, implementation difficulty, and controllability of business impact.

[0084] The architecture matching score can be calculated by checking deployment type matching or cross-cluster / cross-AZ feature matching. For example, in one possible implementation, the base score for architecture matching is 60 points. If the deployment type matches, such as "Kubernetes" appearing in the fault simulation scenario name and in the business component's architecture features, an additional score, such as 20 points, can be added. Similarly, if "cross-cluster" or "cross-AZ" appears in the fault simulation scenario name and in the business component's architecture features, a pre-defined branch can be added, also with 20 points. The final score is then the architecture matching score. Of course, the above calculation of architecture matching score only provides some possible methods; alternative calculation methods to indicate whether the deployment type of a business component matches the fault simulation scenario are also feasible.

[0085] The risk correlation score is used to indicate whether the fault simulation scenario is targeted at the identified risk types. Specifically, similar to architecture matching, the base score for risk correlation is 50 points. If a component's risk label indicates a single point of failure, and the fault simulation scenario name contains "kill" or "failover," a certain score can be added, for example, 30 points. Similarly, if a component's risk label indicates a circular dependency risk, and the fault simulation scenario name contains "network" or "partition," or if a component's risk label indicates a capacity overflow risk, and the fault simulation scenario name contains "CPU" or "memory," a certain score can also be added. The final score (not exceeding 100) is the risk correlation score. Of course, the above calculation of risk correlation only provides some possible calculation methods. Schemes based on other calculation methods to indicate whether the risk type of a business component matches the fault simulation scenario are also feasible. This application embodiment does not limit this.

[0086] Furthermore, regarding the effectiveness of historical drills, the average historical performance score for the fault drill scenario can be extracted from the subsequent drill knowledge base. For implementation difficulty, different scores can be set based on different fault drill scenarios. For example, for simple scenarios such as Pod failures and delayed injection, the implementation difficulty score can be set to 90; for general scenarios such as network partitions and database failures, the implementation difficulty score can be set to 75; and for complex scenarios such as data consistency and distributed transactions, the implementation difficulty score can be set to 60. Regarding the controllability of business impact, this can be based on whether it is a core business or whether there are protection mechanisms such as degradation / rate limiting protection in scenarios belonging to core businesses. For non-core businesses or core businesses with protection mechanisms such as degradation / rate limiting protection, the impact controllability score is higher.

[0087] Building upon the aforementioned foundation, by assigning different weights to the scores of different dimensions and then summing the scores of each dimension using weighted averages, a final score can be obtained to indicate the priority of fault drill scenarios. For example, in one possible implementation, the weight of architecture matching degree can be set to 0.35, the weight of risk correlation degree can be set to 0.25, the weight of historical drill effectiveness can be set to 0.2, the weight of implementation difficulty can be set to 0.1, and the weight of business impact controllability can be set to 0.1. Of course, the dimensions and corresponding weights provided above are only one possible implementation. Based on actual needs, users can choose more or fewer dimensions and set corresponding weights for each dimension for calculation. This application embodiment does not impose any limitations on this.

[0088] Furthermore, in some embodiments of this application, the screening threshold can be dynamically adjusted according to the cluster size. For example, for fault drill scenarios of large-scale clusters, the score needs to exceed 75 points, while for fault drill scenarios of small and medium-sized clusters, the score needs to exceed 70 points, thereby ensuring that resources are prioritized for high-value drill scenarios.

[0089] For a clearer understanding of the above, please refer to [link / reference]. Figure 5a and Figure 5b , Figure 5a and Figure 5b The diagrams illustrate the steps in the scenario recommendation module for filtering candidate fault simulation scenarios and scoring these scenarios. Details are provided below.

[0090] Please see Figure 5a , Figure 5a This is a schematic diagram of a candidate scene generation process provided in an embodiment of this application, specifically including the following steps.

[0091] (1) Input the target component information, which usually includes the component's identification information (component_id), component type (type), cluster (cluster), and physical deployment area (zone); (2) Extract the corresponding risk labels from the architecture vulnerability analysis engine, including but not limited to the aforementioned SinglePoint, CircularDependency, CapacityRisk and ChangeRisk. (3) Extract the architectural context information of the components, including cluster, zone, business level and features, and input them into the subsequent candidate scenario generation module.

[0092] Specifically, the candidate scene generation module typically includes multiple levels of filtering, including rule-based reasoning, collaborative filtering, and graph embedding matching, which will be explained below.

[0093] For fault scenario recommendations based on rule-based reasoning layers, the following are included: By matching triples (component type, risk type, architecture feature) in the preset rule base, the corresponding failure mode rules are obtained, thereby generating corresponding rule reasoning candidate scenarios. For fault scenario recommendations based on collaborative filtering layers, the following are included: Analyze historical exercise data of similar components, such as components with the same component type, the same business level, and the same architectural characteristics; The execution frequency and effect score of historical exercise data are obtained by statistical analysis, and data that meets the preset conditions are selected. For example, scenarios with an execution frequency of more than three times and an average effect score of more than 75 points are selected as collaborative filtering candidate scenarios. For fault scenario recommendations based on graph embedding matching layers, the following are included: The node embedding vectors of the architecture graph are generated based on the GraphSAGE algorithm, which supports IVF_FLAT vector quantization and enables millisecond-level retrieval of hundreds of thousands of components. The embedding vector of the target component is generated based on the node embedding vector, and the vector similarity is retrieved to filter out a certain number of similar components with a similarity threshold of more than 0.75, such as the top 20 similar components in similarity ranking. Historical drill scenarios of similar components are extracted and graph embedding candidate scenarios are generated. In specific embodiments, the selection can also be based on execution frequency and effect score.

[0094] After merging the different candidate scenarios obtained above and removing duplicates, they are further input into the subsequent multi-dimensional scoring model for candidate scenario screening.

[0095] For details, please refer to Figure 5b , Figure 5b The flowchart illustrating the steps for scoring a fault drill scenario provided in this application embodiment includes the following steps: (1) Traverse each candidate scenario and calculate the architecture matching score based on deployment type matching and cross-cluster / cross-AZ feature matching; (2) The risk correlation score is calculated based on the degree of risk label matching, that is, whether the exercise scenario matches the identified risks; (3) Obtain historical performance scores from the exercise knowledge base to calculate the historical exercise effectiveness score; (4) Calculate the implementation difficulty score based on the implementation difficulty of the fault drill scenario, such as simple scenario (Pod failure), medium scenario (network partition), and complex scenario (transaction consistency); (5) Determine the business controllability impact score based on whether it is a non-core business or a core business with a protection mechanism; (6) Calculate the final score based on the score of each dimension and the corresponding weight, and select the corresponding score threshold for filtering according to the cluster rules. For example, the score for large-scale clusters should be more than 75 points, and the score for small and medium-sized clusters should be more than 70 points. (7) Sort by score in descending order and output the top 5 fault drill scenarios as high-value target fault drill scenarios, which can then be used to determine the corresponding drill plan in conjunction with the drill knowledge base.

[0096] To facilitate understanding of the above content, the following will be explained in conjunction with specific business. For the target component delivery-service in the logistics cluster (core business, deployed across 3 clusters), the architecture vulnerability analysis engine outputs the risk label as "cross-cluster dependency bottleneck". Candidate scenario generation: The rule reasoning layer matches three scenarios, "cross-AZ network latency", "PodRandomKill", and "cross-cluster network partition", from 50+ rule bases; the collaborative filtering layer analyzes historical exercise data of similar core services to supplement the "PodRandomKill + cross-cluster dependency interruption" combined scenario. Scoring and Selection: Based on the multi-dimensional scoring model, the "PodRandomKill + Cross-Cluster Network Partition" combined scenario scored 79.2 (Architecture Matching 0.35×88 + Risk Correlation 0.25×92 + Historical Validity 0.2×95 + Implementation Difficulty 0.1×85 + Controllability 0.1×88), making it a high-priority fault drill scenario; Output: {targetComponent: "delivery-service", candidateScenes: ["PodRandomKill+Cross-cluster network partition"], score: 79.2, riskTag: "Cross-cluster dependency bottleneck", architectureContext: "Core business, deployed across 3 clusters"}.

[0097] The solution provided in this application uses a hybrid recommendation mechanism of "rule reasoning + collaborative filtering" and a multi-dimensional scoring model to select high-priority fault drill scenarios, improving the scenario matching accuracy to over 85% and enabling it to adapt to microservice architectures of different scales.

[0098] S230, Based on the target fault drill scenario, generate the target drill plan for the target business component from historical drill data.

[0099] In one embodiment of this application, after the scenario recommendation module outputs the target fault simulation scenario that needs to be simulated, the experience of historical simulation data will be further reused to generate a target simulation plan that matches the target business component.

[0100] Specifically, to improve the effectiveness of generating exercise plans, in one embodiment of this application, the aforementioned plan can be executed through an exercise knowledge base, i.e., a knowledge base module. This knowledge base module structures historical exercise data (including fault scenarios, execution processes, recovery plans, and effect evaluations) and constructs a knowledge graph, thereby achieving accurate recommendation and reuse of similar cases and corresponding recovery plans. For further details, please refer to [link to relevant documentation]. Figure 6 , Figure 6 A flowchart illustrating the steps for constructing a knowledge graph of historical training data, as provided in this application embodiment, specifically includes steps S610 to S630: S610, perform structured data extraction on the historical exercise data to construct a knowledge graph of the historical exercise data, and establish a mapping relationship between the knowledge graph and the architecture graph.

[0101] In one embodiment of this application, by extracting structured data from the entire historical exercise process, a knowledge graph containing a quadruple of "fault scenario - architecture features - recovery plan - effect evaluation" can be constructed, thereby further extracting the corresponding core triple information, including <fault scenario, association, architecture features>, <recovery plan, adaptation, fault scenario>, <exercise case, coverage, risk type>, etc.

[0102] For example, in one embodiment, historical exercise data can be loaded and entity information such as exercise case identification information, fault scenarios, recovery solutions, architectural features, risk types, and performance scores can be extracted to construct nodes as a knowledge graph, including "exercise case (identification information), fault scenarios, recovery solutions, architectural features, and risk types". Corresponding edges are added to the triples, such as fault scenarios associating with architectural features (adding an edge from the fault scenario to the first attribute of the corresponding architectural feature), recovery solutions as adapting to fault scenarios (adding an edge from the recovery solution to the second attribute of the corresponding fault scenario), exercise cases covering risk types, exercise cases containing fault scenarios, exercise cases using recovery solutions, etc.

[0103] S620, based on the mapping relationship between the knowledge graph and the architecture graph, feature vectors of each of the historical training data are generated through a trained graph neural network model.

[0104] Building upon the aforementioned foundation, after constructing a knowledge graph based on historical training data, a mapping relationship between the knowledge graph and the architecture graph can be further established, thereby calculating the feature vector of the historical training data based on the trained graph neural network model GraphSAGE.

[0105] Specifically, in one implementation of this application, a mapping can be established between entities such as "component type," "architectural features," and "business level" in the knowledge graph and the node attributes of the architecture graph output by the architecture vulnerability analysis engine, thereby achieving a two-way linkage between "architectural query → case recommendation" and "case analysis → risk tag reinforcement." Based on this, and through training the graph neural network model GraphSAGE, all nodes can be further generated, including node embedding vectors of the architecture graph in the architecture analysis module and embedding vectors of the knowledge graph in the exercise knowledge base module, to accurately characterize the structural features of cases and architectures. The former, i.e., the node embedding vectors of the architecture graph in the architecture analysis module, can be used for candidate scenario matching in the graph embedding layer, while the latter, i.e., the embedding vectors of the knowledge graph, is mainly used for matching similar cases.

[0106] S630, based on the vector similarity between the feature vectors of the target fault drill scenario and the historical drill data, determine the target drill scheme that matches the target fault drill scenario from the historical drill data.

[0107] Building upon the aforementioned foundation, the feature vector of the target fault drill scenario (usually including the embedding vector of the target business component) obtained by the graph neural network model GraphSAGE is compared with the feature vector of historical drill data to calculate the similarity. This allows for the selection of several cases with the highest similarity. The recovery plan in these cases is then automatically adapted and adjusted in conjunction with the architectural features of the target scenario (such as region, network, and cluster type) to generate a customized drill plan, thereby greatly shortening the design cycle for new scenario drills.

[0108] Specifically, to facilitate understanding of the above content, taking the newly added domestic multi-zone deployment of member clusters (20,000 nodes, covering three data centers in different regions), with the core service being member-service, a contingency plan for "cross-regional network latency" needs to be designed. Similarity calculation: The architectural feature vector of the member-service (including features such as zone affiliation and cross-regional dependency) is generated by the GraphSAGE algorithm, and the similarity is calculated with the historical case vectors in the exercise knowledge base. The exercise case of "cross-regional network latency" of the domestic inventory cluster is matched, and the similarity is 0.85, 0.82 and 0.79 respectively. Case reuse and contingency plan adaptation: Recommend recovery solutions that match the cases (cross-regional disaster recovery switching, dynamic adjustment of leased line bandwidth), and combine the characteristics of cross-regional deployment of member clusters (cross-regional leased line latency differences) and network characteristics (inter-regional leased line bandwidth limitations) to automatically adjust the disaster recovery switching threshold and leased line bandwidth configuration parameters, thereby generating customized drill contingency plans; Implementation results: The design cycle for new scenario drills was shortened from 72 hours to 12 hours, and the accuracy rate of contingency plan adaptation reached 89%, which verified the ability of the knowledge base module to reuse experience in data center scenarios in multiple regions across China. The solution provided in this application constructs a structured reuse system of "historical exercise data - knowledge graph - graph embedding vector", achieves millisecond-level matching of similar scenarios through graph embedding algorithm, achieves Top-K case recommendation accuracy of ≥80%, and shortens the design cycle of new scenario exercises by 80%.

[0109] S240, based on the exercise effect of the target exercise scheme, update the fault risk type of each node in the architecture diagram and / or the preset rule base.

[0110] In one embodiment of this application, recommended fault drill scenarios for the next business component can be generated based on the updated fault risk type and / or the updated preset rule base.

[0111] Based on the aforementioned solution, after determining the target training plan for the target business component and conducting the training, the training process can be further statistically analyzed, and the fault risk type of each node in the architecture diagram and / or the preset rule base can be updated according to the training effect.

[0112] Specifically, in one embodiment of this application, updating the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme includes: If the exercise score used to indicate the exercise effect of the target exercise scheme does not exceed the first preset score threshold, the risk identification threshold used to identify the fault risk type of each node in the architecture graph is adjusted to update the fault risk type of each node in the architecture graph. If the exercise score used to indicate the exercise effect of the target exercise scheme exceeds a second preset score threshold, the matching rules in the preset rule base are updated based on the target exercise scheme.

[0113] Specifically, in one embodiment of this application, in order to quantitatively evaluate the effectiveness of fault drills under a microservice architecture, thereby determining the optimal recovery plan in advance and mitigating sudden risks to online business, a drill score (DS) can be constructed. This score is obtained by weighted summation of multiple evaluation dimensions, thus objectively reflecting the performance of the drill plan in the three core aspects of "recovery speed," "business stability," and "operational reliability." Specifically, in one embodiment, the multiple evaluation dimensions involved in determining the drill score will be explained in detail below.

[0114] In one embodiment, the exercise score can be determined based on the mean-time-to-restore-system (MTTR) of the system services. This can be achieved through a tracing system or a monitoring time-series library, monitoring the average duration from the start of a fault injection until all affected microservices return to normal operation (meeting a preset availability threshold, such as interface P99 latency ≤ 1.2 times the baseline value). This average duration describes the system's recovery speed. Specifically, the shorter the mean recovery time, the more effective the exercise scheme is in terms of "recovery speed," and the higher the exercise score can be set.

[0115] In another embodiment, the exercise score can also be determined based on the degree of deviation of core business indicators (such as request success rate and transaction volume) during the exercise. It can be obtained by calculating the maximum deviation of core business indicators (such as request success rate, core transactions, and order payment throughput) from the normal baseline value through a business monitoring dashboard or an APM (Application Performance Monitoring) system. For example, if the baseline request success rate is 99.99%, and it drops to a minimum of 95% during the exercise, the deviation is 4.99%. This degree of deviation describes the performance of the exercise plan in the "business stability" dimension. Specifically, the lower the deviation value, the more effective the exercise plan is in the "business stability" dimension, and the higher the exercise score can be set.

[0116] In another embodiment, the exercise score can also be determined based on the execution accuracy of the preset recovery plan. For example, the proportion of steps that are correctly executed on the first attempt to the total number of steps can be determined by exercise execution logs and operation audit records. These steps typically include operations such as fault location, traffic switching, rate limiting and degradation, instance restart, and configuration rollback. This proportion also reflects the performance of the exercise plan in the "operational reliability" dimension. Specifically, the higher the proportion, the more effective the exercise plan is in the "operational reliability" dimension, and the higher the exercise score can be set.

[0117] Of course, in addition to the factors provided above, in some embodiments, the exercise score can also be determined based on more evaluation dimensions. For example, it can also be determined by the fault detection timeliness, that is, the time taken from the occurrence of the fault to the triggering of the monitoring alarm and then to the automatic or manual confirmation of the fault type by obtaining the alarm system timestamp. To a certain extent, it can reflect the observability capability of the exercise plan.

[0118] Building upon the aforementioned foundation, further normalized numerical processing is applied to convert the performance of the exercise plan across various evaluation dimensions, such as "recovery speed," "business stability," and "operational reliability," into corresponding scores. Then, by weighting each dimension branch according to given weights, the final exercise score of the exercise plan can be obtained and recorded in the database along with the exercise plan for subsequent processing.

[0119] For example, in one embodiment of this application, historical case performance data in the exercise knowledge base module can feed back into the architecture vulnerability analysis engine to optimize risk identification thresholds and algorithm parameters. Simultaneously, it enriches the rule base and collaborative filtering samples of the intelligent recommendation engine for exercise scenarios, improving recommendation accuracy. For instance, in one embodiment, if the exercise score for the exercise performance does not exceed 60 points, the risk identification threshold for each node's fault risk type can be increased to reduce false alarms. Conversely, if the exercise score exceeds 85 points, it can be added to the rule base, thereby enriching the collaborative filtering samples and improving recommendation accuracy.

[0120] Furthermore, in one embodiment of this application, the method further includes: The target exercise plan is used as historical exercise data to update the knowledge graph of historical exercise data.

[0121] To facilitate understanding of the above, taking the stability assurance process of 10 business clusters (total of 80,000 nodes) as an example, the following closed-loop execution steps are involved: The architecture vulnerability analysis engine identified three high-risk scenarios in the order cluster. The intelligent recommendation engine for the exercise scenario pushed four suitable scenarios. After the exercise was executed, the results were stored in the knowledge base. The knowledge base then fed back and optimized the risk identification threshold, reducing the false alarm rate by 5%.

[0122] The fault drill scenario recommendation method provided in this application constructs a collaborative architecture of three engine modules. First, it uses multi-source data from multiple business clusters deployed in a microservice architecture to build an architecture graph of the business clusters. Then, it combines graph analysis processing technology to identify the risks of nodes in the graph and obtain fault risk types. Next, based on the fault risk types of the nodes, it determines the target fault risk types of the target business components in the business clusters. Based on the target fault risk types, it matches them in a preset rule base to obtain the recommended target fault drill scenarios for the target business components. At the same time, it reuses historical drill data to generate corresponding target drill plans. Based on the execution effect of the drill plans, it updates the fault risk types of each node in the architecture graph and / or the preset rule base to improve the recommendation effect of fault drill scenarios for subsequent business components. The solution provided in this application ensures the relevance of fault drill scenario recommendations by constructing risk labels output by the architecture graph. At the same time, it improves the efficiency of drill design by reusing historical drill data. Finally, it optimizes the architecture graph and enriches the rule base for scenario recommendations based on the drill effect of the target drill plan, further improving the accuracy of fault drill scenario recommendations and ensuring the stability of business clusters under the subsequent microservice architecture.

[0123] Based on the fault simulation scenario recommendation method provided in the embodiments of this application, this application also provides a simplified structural diagram of a fault simulation scenario recommendation system, specifically including: The architecture analysis module 710 is used to construct an architecture graph of the business clusters based on multi-source data of multiple business clusters deployed in a microservice architecture, and to perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph. The scenario recommendation module 720 is used to determine the target fault risk type associated with the target business component in the business cluster based on the fault risk type of each node in the architecture graph, and to match the target fault risk type in a preset rule base to obtain the target fault drill scenario of the target business component. Knowledge base module 730 is used to generate a target exercise plan for the target business component from historical exercise data based on the target fault exercise scenario; and The optimization module 740 is used to update the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme, so as to generate a recommended fault exercise scenario for the next business component based on the updated fault risk type and / or the updated preset rule base.

[0124] In one embodiment of this application, the multi-source data includes service registration information and link tracing information of the business cluster. The architecture analysis module 710 is used to construct an architecture graph of the business cluster based on the multi-source data of multiple business clusters deployed in a microservice architecture, including: The architecture analysis module 710 is used to extract nodes in the business clusters based on the service registration information of the multiple business clusters, and generate heterogeneous nodes of the architecture graph of the business clusters according to the nodes and their node attribute information. The architecture analysis module 710 is used to extract call information between nodes in the business clusters based on the link tracing information of the multiple business clusters, and generate directed edges between corresponding heterogeneous nodes in the architecture graph according to the call information.

[0125] In one embodiment of this application, the architecture analysis module 710 is further configured to incrementally update the architecture graph of the business cluster in response to changes in the version identifier information in the node attribute information of heterogeneous nodes in the architecture graph, and / or changes in the hash value of the directed edge in the architecture graph.

[0126] In one embodiment of this application, the fault risk type includes a node risk type indicating the importance of a node. The architecture analysis module 710 is used to perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph, including: The architecture analysis module 710 is used to determine the target node pointing to each node based on the calling relationship between each node in the architecture graph, and to iteratively update the weight of the target node based on the weight of the target node. After the iterative update is completed, the node whose updated weight exceeds the preset weight threshold is identified as a risk node with node risk type in the architecture graph. The initial weight of each node is determined based on the node attribute information of each node.

[0127] In one embodiment of this application, the failure risk type includes an edge dependency risk type used to indicate the importance of dependencies between nodes; the architecture analysis module 710 is used to perform graph analysis processing on the architecture graph to obtain the failure risk type of each node in the architecture graph, including: The architecture analysis module 710 is used to identify strongly connected components in the architecture graph. When the strongly connected component contains multiple cross-cluster nodes, the nodes in the strongly connected component are determined to be risk nodes with a first edge dependency risk type in the architecture graph. The call chain depth of each node in the architecture graph is determined based on breadth-first search, and nodes whose call chain depth exceeds a preset depth threshold are identified as risk nodes with a second edge dependency risk type in the architecture graph.

[0128] In one embodiment of this application, the multi-source data further includes monitoring indicator data in the service cluster, and the fault risk type includes a capacity risk type used to indicate node capacity risk; the architecture analysis module 710 is used to perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph, including: The architecture analysis module 710 is used to obtain monitoring indicator data of each node in the architecture graph, and the monitoring indicator data is used to indicate the load status of each node. The architecture analysis module 710 is used to perform time-series prediction based on the monitoring index data of each node to obtain the predicted load status of each node, so as to identify the nodes with overload status as risk nodes with capacity risk type in the business cluster.

[0129] In one embodiment of this application, the multi-source data further includes change record data in the business cluster, and the fault risk type includes a change risk type used to indicate the risk of node changes; the architecture analysis module 710 is used to perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph, including: The architecture analysis module 710 is used to obtain change record data of each node in the architecture graph, as well as the number of downstream target nodes of the node. The architecture analysis module 710 is used to determine the change risk score of each node in the architecture graph based on the weighted result of the change frequency in the change record data and the number of downstream target nodes, and to identify nodes whose change risk scores exceed a preset score threshold as risk nodes with change risk type in the microservice architecture.

[0130] In one embodiment of this application, the scenario recommendation module 720 is used to match the target fault risk type in a preset rule base to obtain the target fault drill scenario for the target business component, including: The scenario recommendation module 720 is used to match the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base to obtain the target fault drill scenario of the target business component.

[0131] In one embodiment of this application, the scenario recommendation module 720 is used to match the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base to obtain the target fault drill scenario of the target business component, including: The scenario recommendation module 720 is used to match the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base to obtain the first fault drill scenario of the target business component. The scenario recommendation module 720 generates a second fault simulation scenario for the target business component based on the historical simulation data of the reference business component associated with the target business component. The scenario recommendation module 720 determines the target fault simulation scenario from the first fault simulation scenario and the second fault simulation scenario.

[0132] In one embodiment of this application, the reference service component associated with the target service component includes: A first business component having the same architectural characteristics and / or the same business level as the target business component; and / or A second business component whose graph embedding vector in the architecture graph exceeds a preset similarity threshold with the target business component, wherein the graph embedding vector of the business component is determined based on the node embedding vector of the node in the architecture graph.

[0133] In one embodiment of this application, the knowledge base module 730 is used to generate a target exercise plan for the target business component from historical exercise data based on the target fault exercise scenario, including: The knowledge base module 730 is used to extract structured data from historical exercise data to construct a knowledge graph of the historical exercise data and establish a mapping relationship between the knowledge graph and the architecture graph. The knowledge base module 730 is used to generate feature vectors for each of the historical training data based on the mapping relationship between the knowledge graph and the architecture graph through a trained graph neural network model. The knowledge base module 730 is used to determine a target exercise scheme that matches the target fault exercise scenario from the historical exercise data based on the vector similarity between the feature vectors of the target fault exercise scenario and the historical exercise data.

[0134] In one embodiment of this application, the knowledge base module 730 is used to update the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme, including: The knowledge base module 730 is used to adjust the risk identification threshold for identifying the fault risk type of each node in the architecture graph when the exercise score indicating the exercise effect of the target exercise scheme does not exceed the first preset score threshold, so as to update the fault risk type of each node in the architecture graph. The knowledge base module 730 is used to update the matching rules in the preset rule base based on the target exercise scheme when the exercise score indicating the exercise effect of the target exercise scheme exceeds a second preset score threshold.

[0135] In one embodiment of this application, the knowledge base module 730 is further configured to use the target exercise plan as historical exercise data to update the knowledge graph of the historical exercise data.

[0136] In some embodiments of this application, the method for recommending fault simulation scenarios can be implemented as a computer program, which can be implemented in, for example... Figure 8 The system runs on the computer device shown. The computer device's memory can store the various program modules that make up the recommendation system that constitutes this fault simulation scenario, for example... Figure 7The diagram shows an architecture analysis module 710, a scenario recommendation module 720, a knowledge base module 730, and an optimization module 740. The computer program comprised of these modules causes the processor to execute the steps in the fault simulation scenario recommendation method described in the various embodiments of this application.

[0137] For example, Figure 8 The computer equipment shown can be used as follows Figure 7 The architecture analysis module 710 in the recommended installation system for the fault simulation scenario shown executes step S210. The computer device can execute step S220 through the scenario recommendation module 720. The computer device can execute step S230 through the knowledge base module 730, and so on. The computer device includes a processor, memory, and network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements a recommended method for a fault simulation scenario.

[0138] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In some embodiments of this application, a computer device is provided, including one or more processors; memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to perform the following steps: An architecture graph of the business clusters is constructed based on multi-source data from multiple business clusters deployed in a microservice architecture. Graph analysis is then performed on the architecture graph to obtain the fault risk type of each node in the architecture graph. Based on the fault risk type of each node in the architecture diagram, the target fault risk type associated with the target business component in the business cluster is determined, and the target fault drill scenario of the target business component is obtained by matching the target fault risk type in the preset rule base. Based on the target fault simulation scenario, a target simulation plan for the target business components is generated from historical simulation data; and Based on the training results of the target training scheme, the fault risk types and / or preset rule bases of each node in the architecture diagram are updated, so as to generate recommended fault training scenarios for the next business component based on the updated fault risk types and / or updated preset rule bases.

[0140] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to perform the following steps: An architecture graph of the business clusters is constructed based on multi-source data from multiple business clusters deployed in a microservice architecture. Graph analysis is then performed on the architecture graph to obtain the fault risk type of each node in the architecture graph. Based on the fault risk type of each node in the architecture diagram, the target fault risk type associated with the target business component in the business cluster is determined, and the target fault drill scenario of the target business component is obtained by matching the target fault risk type in the preset rule base. Based on the target fault simulation scenario, a target simulation plan for the target business components is generated from historical simulation data; and Based on the exercise results of the target exercise plan, the fault risk types and / or preset rule bases of each node in the architecture diagram are updated, so as to generate recommended fault exercise scenarios for the next business component based on the updated fault risk types and / or updated preset rule bases.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above provides a detailed description of the recommended methods, systems, devices, and storage media for fault simulation scenarios provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for recommending fault simulation scenarios, applied to a microservice architecture, characterized in that, The method includes: An architecture graph of the business clusters is constructed based on multi-source data from multiple business clusters deployed in the microservice architecture, and graph analysis is performed on the architecture graph to obtain the fault risk type of each node in the architecture graph. Based on the fault risk type of each node in the architecture graph, the target fault risk type associated with the target business component in the business cluster is determined, and the target fault risk type is matched in a preset rule base to obtain the target fault drill scenario of the target business component. Based on the target fault simulation scenario, a target simulation plan for the target business component is generated from historical simulation data; and Based on the exercise results of the target exercise scheme, the fault risk types of each node in the architecture graph and / or the preset rule base are updated to generate a recommended fault exercise scenario for the next business component based on the updated fault risk types and / or the updated preset rule base.

2. The method according to claim 1, characterized in that, The multi-source data includes service registration information and link tracing information of the business cluster; The step of constructing the architecture graph of the business clusters based on multi-source data from multiple business clusters deployed in a microservice architecture includes: Based on the service registration information of the multiple business clusters, nodes in the business clusters are extracted, and heterogeneous nodes of the architecture graph of the business clusters are generated according to the nodes and their node attribute information. Based on the link tracing information of the multiple business clusters, the call information between nodes in the business clusters is extracted, and directed edges between corresponding heterogeneous nodes in the architecture graph are generated according to the call information.

3. The method according to claim 2, characterized in that, The fault risk types include node risk types used to indicate node importance; The graph analysis processing of the architecture graph yields the failure risk types of each node in the architecture graph, including: Based on the calling relationships between nodes in the architecture graph, a target node is determined for each node, and the weight of the target node is iteratively updated based on the weight of the target node. After the iterative update, nodes whose updated weights exceed a preset weight threshold are identified as risk nodes with node risk type in the architecture graph. The initial weight of each node is determined based on the node attribute information of each node.

4. The method according to claim 2, characterized in that, The failure risk types include edge dependency risk types used to indicate the importance of dependencies between nodes; The graph analysis processing of the architecture graph yields the failure risk types of each node in the architecture graph, including: Identify strongly connected components in the architecture graph. If the strongly connected components contain multiple cross-cluster nodes, determine that the nodes in the strongly connected components are risk nodes with a first edge dependency risk type in the architecture graph. and / or The call chain depth of each node in the architecture graph is determined based on breadth-first search, and nodes whose call chain depth exceeds a preset depth threshold are identified as risk nodes with a second edge dependency risk type in the architecture graph.

5. The method according to claim 1, characterized in that, The step of matching the target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component in the preset rule base to obtain the target fault drill scenario for the target business component includes: The target fault risk type, the component type of the target business component, and the architectural characteristics of the target business component are matched in the preset rule base to obtain the first fault drill scenario of the target business component. Based on the historical drill data of the reference business component associated with the target business component, a second fault drill scenario for the target business component is generated; The target fault simulation scenario is determined from the first fault simulation scenario and the second fault simulation scenario.

6. The method according to claim 1, characterized in that, The step of generating the target exercise plan for the target business component based on the target fault exercise scenario and from historical exercise data includes: Structured data extraction is performed on historical exercise data to construct a knowledge graph of the historical exercise data, and a mapping relationship is established between the knowledge graph and the architecture graph. Based on the mapping relationship between the knowledge graph and the architecture graph, feature vectors of each of the historical training data are generated through a trained graph neural network model; Based on the vector similarity between the feature vectors of the target fault drill scenario and the historical drill data, a target drill scheme matching the target fault drill scenario is determined from the historical drill data.

7. The method according to any one of claims 1 to 6, characterized in that, The update of the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme includes: If the exercise score used to indicate the exercise effect of the target exercise scheme does not exceed the first preset score threshold, the risk identification threshold used to identify the fault risk type of each node in the architecture graph is adjusted to update the fault risk type of each node in the architecture graph. If the exercise score used to indicate the exercise effect of the target exercise scheme exceeds a second preset score threshold, the matching rules in the preset rule base are updated based on the target exercise scheme.

8. A recommendation system for fault simulation scenarios, applied to a microservice architecture, characterized in that, The system includes: The architecture analysis module is used to construct an architecture graph of the business clusters based on multi-source data of multiple business clusters deployed in a microservice architecture, and to perform graph analysis processing on the architecture graph to obtain the fault risk type of each node in the architecture graph. The scenario recommendation module is used to determine the target fault risk type associated with the target business component in the business cluster based on the fault risk type of each node in the architecture graph, and to match the target fault risk type in a preset rule base to obtain the target fault drill scenario of the target business component. The knowledge base module is used to generate a target exercise plan for the target business component from historical exercise data based on the target fault exercise scenario; and The optimization module is used to update the fault risk type of each node in the architecture graph and / or the preset rule base based on the exercise effect of the target exercise scheme, so as to generate a recommended fault exercise scenario for the next business component based on the updated fault risk type and / or the updated preset rule base.

9. A computer device, characterized in that, The computer device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the recommended method for fault simulation scenarios as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the recommended method for the fault simulation scenario as described in any one of claims 1 to 7.

Images