Business scenario fault drilling method and device, computer device, and storage medium

Abstract

The present disclosure relates to a business scenario fault drilling method and device, computer equipment and storage medium. The method comprises: obtaining business scenario information of a target business scenario; wherein the business scenario information contains the calling relationship of at least one technical component relied on by the target business scenario; sending the business scenario information to a pre-trained AI model, and obtaining a fault drilling scheme for the target business scenario generated by the AI model; executing the fault drilling scheme, injecting the fault corresponding to the fault drilling scheme into the technical component relied on by the service running the target business scenario; after injecting the fault, collecting the running data of the target business scenario, and analyzing the running data based on the preset business rules to determine the dependence strength of the service on the technical component, which is used to represent the influence degree of the fault of the technical component on the core function of the target business scenario. The present disclosure can make the fault drilling conform to the real business risk.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, and storage medium for conducting fault simulations in business scenarios. Background Technology

[0002] In the field of system stability and fault tolerance verification, existing technical solutions typically rely on pre-configured static fault scenario libraries for generalized fault injection testing, or on fixed rule engines to determine the dependencies between technical indicators of services. Verification frameworks built using these methods have relatively fixed test scenario coverage and are subject to delays in updates. As the complexity of distributed system architectures and business processes continues to increase, these static and general verification methods struggle to automatically generate fault test cases closely related to specific business contexts. This results in insufficient verification coverage of real-world business risks, and the resource utilization efficiency of the verification process needs optimization. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, computer equipment, and storage medium for fault simulation in business scenarios that can reflect real business risks, in order to address the above-mentioned technical problems.

[0004] Firstly, this disclosure provides a method for fault simulation in a business scenario, the method including:

[0005] Obtain business scenario information for the target business scenario; wherein, the business scenario information includes the call relationships of at least one technical component on which the target business scenario depends; Business scenario information is input into the AI ​​model to generate a fault drill plan for the target business scenario; The fault simulation plan is executed to inject faults into the technical components that the services running the target business scenario depend on. After a fault is injected, the operational data of the target business scenario is collected, and the operational data is analyzed based on business rules to determine the service's dependence on the technical components. The dependence strength is used to characterize the impact of the technical component's failure on the core functions of the target business scenario.

[0006] Secondly, this disclosure also provides a business scenario fault drill device, the device comprising: The business scenario information acquisition module is used to acquire business scenario information of the target business scenario; wherein, the business scenario information includes the call relationship of at least one technical component on which the target business scenario depends; The fault drill plan acquisition module is used to input business scenario information into the AI ​​model to generate a fault drill plan for the target business scenario. The fault injection execution module is used to execute fault drill plans and inject faults into the technical components that the services of the target business scenario depend on. The dependency strength analysis and judgment module is used to collect the operation data of the target business scenario after a fault is injected, and analyze the operation data based on preset business rules to determine the dependency strength of the service on the technical components. The dependency strength is used to characterize the degree of impact of the fault of the technical components on the core functions of the target business scenario.

[0007] Thirdly, this disclosure also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0008] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0009] The business scenario fault simulation method, apparatus, computer equipment, and storage medium disclosed herein acquire business scenario information containing specific technical component call relationships, enabling the fault simulation target to be anchored to the technical components upon which specific business functions depend, ensuring that the starting point of the simulation is the real business architecture and processes. Furthermore, by leveraging AI models to analyze business scenario information and potential fault modes of technical components, customized fault simulation schemes are dynamically generated, ensuring that the injected faults are not merely general technical anomalies, but risk scenarios closely integrated with specific business contexts. By implementing this fault injection through the target technical component, the virtual risks generated by the AI ​​model can be transformed into observable actual system disturbances, completing the technical implementation from theoretical deduction to practical verification. Thus, the fault simulation system can analyze the operational data collected during the fault period based on preset, well-defined business rules (rather than simply technical indicators) and determine the dependency strength, thereby directly aligning the evaluation criteria for the simulation effect to the question of "whether it affects core business functions." In this way, the entire process of fault drills—including input (business scenario), design (AI solution), execution (precise injection), and evaluation (business rule determination)—is closely linked to real business risks. This ensures that fault drills can effectively reveal and verify weaknesses that truly threaten business continuity, making fault drills more closely aligned with real business risks. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies of this disclosure will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a scenario-dependent fault simulation system provided in an embodiment of the present disclosure; Figure 2 A flowchart illustrating a business scenario fault drill method provided in this embodiment of the disclosure; Figure 3 This is a schematic flowchart illustrating the steps for obtaining business scenario information according to an embodiment of the present disclosure; Figure 4 This is a flowchart illustrating the steps involved in generating a fault simulation exercise scheme, as provided in an embodiment of this disclosure. Figure 5 This is a flowchart illustrating the steps of executing a fault drill scheme according to an embodiment of the present disclosure; Figure 6 A flowchart illustrating the steps for determining the dependence of a service on technical components, as provided in an embodiment of this disclosure; Figure 7 A schematic diagram of the interactive process of a fault drill provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the structure of a business scenario fault simulation device provided in an embodiment of the present disclosure; Figure 9 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present disclosure. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0013] In one exemplary embodiment, the business scenario fault simulation method provided in this disclosure can be applied to, for example, Figure 1The illustrated scenario-based fault simulation system includes a terminal 101 that communicates with a server 102 via a network. A data storage system can store the data that the server 102 needs to process. This data storage system can be integrated onto the server 102 or located on a cloud or other network server. The terminal 101 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The server 102 can include a scenario platform and an AI model service. The AI ​​model deployed in the AI ​​model service can be a pre-trained large language model based on the Transformer architecture. This model is trained using supervised fine-tuning (SFT) using massive amounts of official documentation of technical components, historical system fault reports, operation and maintenance knowledge base text, and business scenario description data. This enables it to understand the semantics of business scenarios, parse technical dependencies, and infer and generate fault simulation solutions that fit the scenario.

[0014] In one exemplary embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a scenario-dependent fault drill provided in an embodiment of the present disclosure, such as... Figure 2 As shown, a method for fault simulation in a business scenario is provided, which can be applied to... Figure 1 The following explanation uses a fault simulation system as an example, including the following steps S201 to S204. Wherein: S201. Obtain business scenario information of the target business scenario; wherein, the business scenario information includes the call relationship of at least one technical component on which the target business scenario depends.

[0015] The target business scenario refers to a functional unit with a clear business meaning that requires stability verification, such as "online payment" or "user membership identity query." Business scenario information refers to a set of metadata describing the relationship between the business logic and technical implementation of that scenario. Technical components refer to application components (software) located between the operating system and the application, providing general services, such as Redis caching, MySQL databases, and RocketMQ message queues. Call relationships refer to the interactions between the service and one or more technical components during the execution of business logic, including requests, read / write operations, and message passing.

[0016] For example, the fault simulation system can query or receive a detailed definition of a specified target business scenario from the scenario platform that stores and manages business metadata via interface calls. This definition may include the scenario name, a list of core business interfaces involved in the target business scenario, and the service and technical component paths traversed by these interface calls, obtained through distributed tracing technology.

[0017] As an example, the fault simulation system can obtain information about the "user member identity query" business scenario. For instance, it can determine that the interface for this business scenario is / api / getUserType. By analyzing the link data, it can extract that the interface call link contains "member service", "Redis cluster cluster-01" and "MySQL instance", thereby determining the dependency of this business scenario on the two technical components of Redis and MySQL.

[0018] Optionally, by acquiring contextualized information about business scenarios from a business perspective and clarifying the technical dependencies of the business scenario information, the fault drill system can accurately associate abstract business risks with specific technical components, enabling subsequent fault drills to directly verify the processes supporting real business operations, thereby ensuring that the drill objectives are consistent with real business continuity risks.

[0019] S202. Input business scenario information into the AI ​​model to generate a fault drill plan for the target business scenario.

[0020] Here, the AI ​​model refers to a large language model pre-trained with massive amounts of technical documents, failure cases, and business knowledge, capable of semantic understanding and logical reasoning. The failure drill plan refers to a structured set of instructions that specifies in detail what kind of failure to inject, to which target to inject, under what conditions to inject, and how to verify the effect.

[0021] For example, the fault simulation system can take the acquired structured (with a preset structure format) business scenario information (such as scenario name and a list of dependent technical components) as input and submit it to the inference engine of the AI ​​model. The AI ​​model can then automatically generate a customized fault simulation plan based on its built-in knowledge and understanding of the business context.

[0022] As an example, the AI ​​model can generate a fault simulation plan based on the "user membership identity query" business scenario and its dependence on Redis: the fault type is "simulated Redis cluster network interruption", the injection target is "Redis cluster A", the duration is "5 minutes", and it is recommended to observe indicators such as "member query interface response time" and "order success rate".

[0023] Optionally, by introducing AI models to intelligently generate drill plans based on business scenarios, the fault drill system can get rid of its dependence on manually preset static scripts and can dynamically generate test cases that fit the scenario for ever-changing business combinations and architectures. This allows the generated fault scenarios to simulate risks caused by specific business chain dependencies more flexibly and intelligently.

[0024] S203. Execute the fault drill plan and inject faults into the technical components on which the service of the target business scenario depends.

[0025] Execution refers to transforming text or structured schemes generated by AI models into specific operations that can be triggered in a real-world operating environment. Fault injection refers to simulating pre-defined abnormal states on specified technical components using pre-defined technical means.

[0026] For example, the fault simulation system can execute fault simulation plans generated by parsing the AI ​​model and convert the plans into configuration instructions that can be recognized by the underlying chaos engineering tools. The execution engine of the fault simulation system can initiate fault injection experiments on the container or host where the target technology component resides by calling the application programming interface of the chaos engineering tools.

[0027] As an example, the execution engine can instruct the chaos engineering tools to implement a network rule with a 100% packet loss rate for 5 minutes on a specified Redis cluster node, based on the fault simulation plan, thereby simulating a fault state in which the Redis cluster network is completely isolated.

[0028] Optionally, through the execution of automated fault drill schemes, the fault drill system can safely and accurately reproduce fault scenarios generated by AI models in production or simulation environments, achieving seamless integration from intelligent decision-making to actual action, and enabling the reliable implementation and verification of drill schemes designed for real business risks.

[0029] S204. After the fault is injected, the operation data of the target business scenario is collected, and the operation data is analyzed based on the preset business rules to determine the service's dependence on the technical components. The dependence strength is used to characterize the degree of impact of the technical component's failure on the core functions of the target business scenario.

[0030] Among these, operational data can refer to various technical and business metrics collected during fault injection. Pre-defined business rules can refer to core standards defined in advance within the scenario platform to differentiate the level of business impact, such as "a payment success rate below 99.9% indicates damage to core functions." Dependency strength can refer to a classification result, such as "strong dependency" or "weak dependency," used to quantify the impact of the technical component failure on the business.

[0031] For example, during the experiment, the fault simulation system can continuously collect data on the success rate, response time, and health status of dependent technical components of the target business interface through integrated monitoring components. After the experiment, the analysis module of the fault simulation system can obtain the business rules corresponding to the business scenario from the scenario platform, and compare the collected core business indicators with the health thresholds in the rules to automatically determine the dependency strength.

[0032] As an example, after a Redis failure, the system observed a 50% increase in the response time of the "Member Query Interface," but the "Member Order Success Rate" remained above 99.8%. Based on the rule that "the order success rate is not lower than 99.5%," the system can determine that Redis is a weak dependency in this scenario, meaning its failure did not cause any interruption to core business operations.

[0033] Optionally, by determining dependency strength based on preset business rules (rather than purely technical indicators), the fault simulation system can directly align the evaluation criteria for simulation effectiveness with the actual success criteria of the business. This ensures that the simulation conclusions directly reflect whether a component failure will affect the business, guaranteeing that optimization efforts always revolve around ensuring the security of real business operations.

[0034] In this embodiment, by acquiring business scenario information containing the call relationships of specific technical components, the target of the fault drill is anchored to the technical components upon which specific business functions depend, ensuring that the starting point of the drill is the real business architecture and processes. Furthermore, by leveraging AI models to analyze business scenario information and potential fault modes of technical components, a customized fault drill plan is dynamically generated. This ensures that the injected faults are general technical anomalies, while the injected faults are risk scenarios closely integrated with a specific business context. By implementing this fault injection through the target technical component, the virtual risks generated by the AI ​​model can be transformed into observable actual system disturbances, completing the technical implementation from theoretical deduction to practical verification. Thus, the fault drill system can analyze the operational data collected during the fault period based on preset, well-defined business rules (rather than simply technical indicators) and determine the dependency strength, thereby directly aligning the evaluation criteria for the drill effect to the question of "whether it affects core business functions." In this way, the entire process of fault drills—including input (business scenario), design (AI solution), execution (precise injection), and evaluation (business rule determination)—is closely linked to real business risks. This ensures that fault drills can effectively reveal and verify weaknesses that truly threaten business continuity, making fault drills more closely aligned with real business risks.

[0035] In one exemplary embodiment, the method further includes: The determination of dependency strength is fed back to the AI ​​model, which is then used to optimize the generation of fault simulation schemes based on dependency strength.

[0036] Feedback operations refer to forming samples from the "input (business scenario) - output (dependency strength)" results of a complete drill. Optimization operations refer to the AI ​​model using these feedback samples to adjust its internal parameters or generation logic, making future fault drill plans more accurate.

[0037] For example, after generating a verification report, the fault drill system can encapsulate the business scenario information of this drill, the original solution generated by AI, and the final determination of the dependency strength into a training sample and send it to the training service of the AI ​​model.

[0038] As an example, the AI ​​model might initially think that "Redis network outage" has a significant impact on the "member query" scenario. However, after multiple drills, the feedback consistently indicates a "weak dependency." Upon receiving this feedback, the AI ​​model can learn this correlation and adjust its strategy when generating solutions for similar scenarios in the future. For example, it could prioritize generating "network latency" faults rather than "complete outages," or more accurately predict their business impact.

[0039] In this embodiment, by establishing a feedback loop from verification results to the AI ​​model, the fault simulation system enables the AI ​​to continuously learn from real business impact data. This allows the AI ​​model's ability to generate fault scenarios to continuously evolve, becoming increasingly accurate in predicting and simulating fault scenarios that match the actual business risk levels, thus creating a virtuous cycle of continuous improvement.

[0040] In one exemplary embodiment, the AI ​​model optimizes the generation of fault simulation schemes based on dependency strength, including: The verification results containing dependency strength are used as training samples to incrementally train the AI ​​model; Based on the trained AI model, update the adjustment strategy parameters for generating fault simulation plans.

[0041] Incremental training refers to a machine learning method that fine-tunes an existing model using newly generated data. Adjusting strategy parameters refers to the weights of the model's internal control scheme generation tendencies, such as injecting preference settings for different fault types or different target user groups.

[0042] For example, after accumulating a certain number of feedback samples, the AI ​​model training service can initiate an incremental training process. During training, the AI ​​model's weights for assessing the risk level of various faults in specific business scenarios can be adjusted based on the labels of the actual dependency strength in the samples.

[0043] As an example, the strategy parameters inside the AI ​​model may change after training: for example, for the core trading interface, extreme failures such as database connection errors injected in historical feedback can easily cause strong dependence, so the probability of generating extreme failures such as database connection errors injected can be reduced; at the same time, controllable delay strategies can both verify resilience and are relatively safe, so the weight of injecting controllable delay strategies can be increased.

[0044] In this embodiment, by incrementally training the AI ​​model based on real-world drill results, the fault simulation system can continuously optimize its fault generation strategy. This ensures that the AI ​​model can dynamically adapt to changes in business architecture and deepen its understanding, enabling it to generate drill plans that more intelligently balance validation effectiveness and business risks, thus making fault simulations more closely aligned with real-world business scenarios.

[0045] In one exemplary embodiment, Figure 3 This is a flowchart illustrating the steps for obtaining business scenario information according to an embodiment of the present disclosure. In step S201, obtaining the business scenario information of the target business scenario may include: S301. Query the list of interfaces and call chains associated with the target business scenario from the scenario platform; S302. Extract the identification information of at least one technical component that the target business scenario depends on from the call chain.

[0046] Here, "scenario platform" refers to a system that independently stores metadata and topology relationships for a business scenario. "Interface list" refers to one or more application programming interfaces (APIs) involved in implementing the business scenario. "Call chain" refers to the complete service and component path a request traverses, recorded through tracing technology. "Identification information" refers to identifiers that uniquely identify technical component instances, such as "Redis@cluster-01".

[0047] For example, the fault simulation system can obtain detailed information about a target business scenario by calling the application programming interface (API) provided by the scenario platform and passing in its unique identifier. The returned data can include key interfaces in that scenario and their call chain diagram. The fault simulation system can analyze this chain diagram, identify all technical components that are not part of the business service, and categorize them as technical component dependencies.

[0048] As an example, when querying a business scenario involving asset settlement, its core interface / api / settlement can be obtained, and its call chain can be parsed as: settlement service, risk control service, and Redis (rule cache). Based on this, the fault simulation system can extract the dependent technical components as "risk control service" and "Redis cluster A".

[0049] In this embodiment, by actively pulling structured business-technology mapping information from the scenario platform, the fault simulation system can automatically and accurately construct a correlation graph between business functions and underlying technology dependencies. This allows the system to directly locate key technology nodes carrying real business flows, ensuring that the selection of simulation targets stems from the actual business architecture, thereby enabling fault injection to accurately target potential risk points in the business chain.

[0050] In one exemplary embodiment, business scenario information is input into an AI model to generate a fault simulation plan for a target business scenario, including: Based on business scenario information, the AI ​​model analyzes the dependency of the target business scenario on at least one technical component and generates a fault simulation plan for the target business scenario.

[0051] Among these, business scenario information refers to a set of metadata describing a specific business process, including the scenario name, associated core business interfaces, interface priorities, and call paths obtained through tracing analysis. Dependencies refer to the interactive relationships between the business scenario and specific technical components during execution, such as calls, read / write operations, or message passing. A fault drill plan can be a structured set of instructions that specifies under what conditions, which technical components, and what faults are injected, along with the expected observed metrics.

[0052] For example, the fault simulation system takes the acquired business scenario information (including scenario name, associated interfaces, and a list of middleware in the call chain) as input and submits it to the AI ​​model's inference engine. After receiving this information, the AI ​​model, in conjunction with its built-in technical documentation knowledge base, analyzes the role of each technical component in the business scenario (e.g., whether Redis is used as a cache or persistent storage) and the expected level of business dependence on them. Based on this analysis, the AI ​​model automatically generates a fault simulation plan highly matched to the scenario, clearly defining the target, type, parameters, and scope of fault injection.

[0053] As an example, the fault simulation system can input information from a "user membership identity query" scenario into the AI ​​model, including the core interface ` / api / getUserType` associated with this scenario and the Redis cluster A and MySQL database B in its call chain. After analysis, the AI ​​model might conclude that Redis primarily serves as a cache in this scenario, and its failure could increase response time, but core business logic can still be completed through the database. Therefore, a fault scenario involving runtime Redis network latency can be generated and configured to only apply to high-value users. Simultaneously, considering the core position of this interface in the chain, the AI ​​model can also generate a fault scenario where the MySQL connection fails at startup, to verify the service's startup resilience under extreme conditions.

[0054] In this embodiment, by enabling the AI ​​model to proactively analyze its dependencies on underlying technical components based on business scenario information, a customized fault simulation plan is generated. This transforms fault injection from a random or generic technical test into a risk verification deeply integrated with the business context. Consequently, the generated fault simulation plan accurately reflects the real business impact that may occur when a specific technical component fails in a specific business scenario, ensuring that the simulation activities always revolve around real business vulnerabilities and effectively improving the alignment between fault simulation and business risk management.

[0055] In one exemplary embodiment, the AI ​​model analyzes the dependency of a target business scenario on at least one technical component based on business scenario information, including: The AI ​​model obtains the potential failure modes of at least one technical component by parsing the official documentation of the technical component; and generates a failure drill plan for the target business scenario based on business scenario information and potential failure modes; the failure types described in the official documentation include at least one of the following dimensions: node dimension, network dimension, and configuration dimension.

[0056] In this context, "official documentation" refers to explanatory materials released by the technology component provider or community. "Potential failure modes" refers to typical abnormal situations that the component may experience, as described in the documentation. Node-level failures focus on the instance itself (e.g., downtime), network-level failures focus on communication problems (e.g., latency, packet loss), and configuration-level failures focus on behavioral anomalies caused by misconfigurations.

[0057] For example, before generating a solution, the AI ​​model can retrieve or access the official documentation of the technical components it has learned. It can extract common failure modes of the technical component across different dimensions from the documentation. Considering the role of the technical component in the current business scenario (e.g., whether it's used for caching or persistence), it selects one or more of the most relevant and likely to cause business risks, incorporating them into the generated solution. By directly parsing potential failure modes from the official documentation of the technical components, it ensures that the generated failure scenarios are highly typical and realistic, comprehensively covering all common anomalies acknowledged by the technical component provider. This method overcomes the omissions, delays, or subjectivity that may exist in traditional solutions relying on human experience bases, enabling failure drills to always verify the most authoritative and most likely risks.

[0058] As an example, for Redis caching, the AI ​​model can parse from the documentation that network partitioning is a common cause of "CLUSTERDOWN" errors. Combined with the "member query" business scenario, the AI ​​model can generate a "network-level" fault simulation plan: simulating network partitioning in the Redis cluster and observing the application's degradation capabilities.

[0059] In this embodiment, by driving the AI ​​model to extract fault patterns from authoritative documents, the typicality and realism of the generated fault scenarios can be ensured. Filtering based on business context ensures that the injected faults are not merely general anomalies, but rather the risks most likely to occur to specific dependent components under their specific business roles. This allows the drills to effectively verify performance in the face of real, typical faults.

[0060] In one exemplary embodiment, Figure 4 This is a flowchart illustrating the steps involved in generating a fault simulation exercise scheme according to an embodiment of this disclosure, as shown below. Figure 4 As shown, generating a fault drill plan for the target business scenario can include: S401. Generate a startup failure scenario to verify whether the service can successfully start or scale up in the event of a technical component failure; and / or, S402. Generate a runtime fault scheme to verify whether the business functions can be degraded when the service fails in the event of a technical component failure.

[0061] Among them, the startup failure scenario refers to a scenario simulating that the technical components it depends on are unavailable during service instance initialization, startup, or horizontal scaling. The runtime failure scenario refers to a scenario simulating that the technical components it depends on suddenly fail while the service is running stably and providing services.

[0062] For example, AI models can design verification schemes for the same business scenario and technical component dependencies from two lifecycle stages. For startup failures, the fault drill scheme can focus on the service process's tolerance to unavailable dependencies during connection initialization and configuration loading. For runtime failures, the fault drill scheme can focus on the service's resilience and degradation capabilities to sudden dependency failures while continuously processing requests.

[0063] As an example, for the Redis dependency of the membership service, the AI ​​model can generate two fault drill scenarios: a startup fault drill scenario, which creates a Redis connection failure when deploying a new instance or scaling up, to verify whether the service can skip cache dependencies and complete the startup; and a runtime fault drill scenario, which injects Redis network latency during peak service periods to verify whether query requests can automatically degrade to the database without affecting core transactions.

[0064] In this embodiment, by distinguishing and covering two key scenarios—"startup" and "runtime"—the fault simulation system can perform resilience testing throughout the entire service lifecycle. This allows fault simulations to more comprehensively map real-world operational risks: covering both sudden dependency failures during online service runtime and the risks that may be encountered during remedial operations such as scaling up and restarting after a failure, thus providing more comprehensive business continuity verification.

[0065] In one exemplary embodiment, executing a fault drill plan includes: Based on user tag information, business requests are intercepted; user tag information is determined based on at least one of the following dimensions: user value dimension, user behavior dimension, user channel dimension, or user risk dimension.

[0066] User tagging information refers to structured data describing user characteristics obtained from a system that stores user profiles, such as high-value users or active users. Injection faults refer to interventions such as delays or error returns imposed on matching requests at the gateway processing stage.

[0067] For example, when the fault simulation system executes a fault simulation plan generated by the AI ​​model, it can extract the preset user tag filtering conditions from the plan and synchronize these conditions to the fault injection plugin at the gateway layer. When a business request arrives at the gateway, the gateway plugin extracts the user identifier from the request and obtains the user's tag set by calling the user profiling system or querying the local cache. Subsequently, the gateway plugin compares the user tags with the target tags in the fault simulation plan. If the tags match completely, a fault injection operation (such as adding a delay) is performed on the request; if they do not match, the request is allowed to proceed normally without any impact.

[0068] As an example, a fault simulation plan can be set with the target being "User XX". After receiving a business request, the gateway queries that User A's tag is "User XX", and then injects a 500ms delay into User A's request; User B's tag does not contain this information, so User B's business request will not be affected in any way.

[0069] In this embodiment, by introducing a traffic filtering and precise fault injection mechanism based on multi-dimensional user tags at the gateway layer, the fault drill system can achieve fine-grained control over the scope of fault impact. This enables drills to accurately simulate localized faults that only affect specific user groups in the real world, thereby verifying the system's resilience to high-risk user groups while minimizing interference with ordinary users and non-target traffic.

[0070] In one exemplary embodiment, executing the fault drill plan further includes: Verify whether the intercepted business request matches the user tag information; Inject faults into business requests that match user tag information.

[0071] In this context, the gateway layer refers to the system traffic entry point, responsible for request routing and filtering. User tagging information refers to structured data describing user characteristics, obtained from a system storing user profiles, such as high-value users or active users. Injection faults refer to interventions such as delays or error returns imposed on matching requests at the gateway processing point.

[0072] For example, the fault simulation system compiles the target user conditions set in the fault simulation scheme generated by the AI ​​model into filtering rules on the gateway side. User tag information can be determined based on multiple dimensions, such as: user value dimension (e.g., "high-value user: cumulative consumption greater than 1000 yuan in the past 30 days"), user behavior dimension (e.g., "daily active user: logged in on the current day"), user channel dimension (e.g., "App user"), and user risk dimension (e.g., "low-risk user: no recent fraudulent behavior"). When a user request arrives at the gateway, the gateway plugin extracts the user identifier and queries the tag system in real time. The gateway only performs fault injection on the request if the user tag completely matches the rule; otherwise, it allows the request to pass normally. By implementing precise traffic filtering and fault injection based on multi-dimensional user tags at the gateway layer, the impact of the fault can be strictly controlled within a preset, limited business subset (e.g., only affecting a small number of high-value customers for verification). This greatly reduces the "explosion radius" of fault drills and the risk of business disruption to non-target user groups, making it possible to simulate real localized faults that affect specific user groups in the production environment, thereby verifying system resilience while ensuring the stability of overall business operations.

[0073] As an example, a fault simulation plan can be set with the target being "User XX". After receiving a business request, the gateway queries that User A's tag is "User XX", and then injects a 500ms delay into User A's request; User B's tag does not contain this information, so User B's business request will not be affected in any way.

[0074] In this embodiment, by implementing precise traffic filtering and fault injection based on multi-dimensional user tags at the gateway layer, the fault simulation system can strictly control the impact of faults within a preset business subset (such as a small number of high-value customers). This allows fault simulations to be conducted safely in the production environment with a smaller blast radius, simulating real localized faults that affect specific user groups. This verifies system resilience while reducing the risk of interference with overall business operations.

[0075] In an exemplary embodiment, before intercepting service requests at the gateway layer based on user tag information carried in the fault simulation scheme, the method further includes: Based on at least one feature of the target business scenario, determine the user tag information and fault type that match at least one feature.

[0076] In this context, "features" refers to the description of an interface from both technical and business perspectives. Features can include at least one of the following: peak request rate, business link type, and dependency type. Peak request rate measures the traffic volume of the interface. Business link type identifies the importance of the interface (e.g., a core transaction link). Dependency type describes the interface's dependence on downstream components (e.g., a strong database dependency). Matching refers to the AI ​​model assigning appropriate training strategies (user tags and fault types) to interfaces with different features based on pre-defined decision rules.

[0077] For example, when generating solutions, the AI ​​model of the fault simulation system can analyze the characteristics of interfaces related to the business scenario. Based on these characteristics and preset protection principles (such as protecting core transactions and high-value users), it dynamically decides which user tags to use for screening and what fault type of severity to select.

[0078] As an example, for a payment interface with core transaction links and high traffic characteristics, AI decision rules may indicate that user tags should be limited to high-value users, fault types should be selected as reversible and controllable faults such as 100ms delay, and error injection that may cause transaction failure should be avoided.

[0079] In this embodiment, by using interface features as input for AI decision-making, the exercise strategy can be made intelligent and differentiated. This allows the generated fault scenarios to more accurately reflect the actual risk tolerance and protection priority of different business interfaces, enabling more conservative and precise exercises on core links. Under the premise of controlling overall risk, this ensures that the exercises effectively address key risk points in real business operations.

[0080] In one exemplary embodiment, executing a fault drill plan includes: The fault simulation plan is parsed into executable fault injection instructions; Based on fault injection instructions, faults are injected into the technical components that the service running the target business scenario depends on.

[0081] Here, parsing refers to the process of converting high-level semantic descriptions into formatted, specific operational instructions that the underlying infrastructure can recognize and execute. Executable fault injection instructions refer to atomic operation commands targeting specific chaos engineering tools or underlying platforms (such as container orchestration systems or virtual machine management platforms), which can clearly specify which fault operation to perform on which component. Fault injection refers to artificially inflicting abnormal states on target technology components by executing low-level instructions, such as simulating network outages, resource exhaustion, or process suspension.

[0082] For example, after receiving the fault simulation plan generated by the AI ​​model, the fault simulation system's internal execution engine can parse the plan. The execution engine has built-in adapters for various chaos engineering tools and underlying platforms, enabling it to translate business logic such as "injecting 5 minutes of network packet loss into Redis cluster A" into a configuration format recognizable by specific chaos engineering tools. After parsing, the execution engine calls the control interface of the chaos engineering tool to send the generated instructions to the runtime environment of the target technical component, thereby activating and executing the fault injection operation in the actual system.

[0083] As an example, the failure scenario generated by the AI ​​model is described as: "Simulate a complete network outage for 300 seconds for the Redis cache cluster that the membership service depends on." The execution engine of the failure simulation system parses this scenario and converts it into a network isolation experiment configuration that can be recognized by the underlying chaos engineering platform. Subsequently, the execution engine calls the application programming interface of the chaos engineering platform to create and start the experiment in the container environment where the Redis cluster resides, thus completely isolating the Redis cluster from the external network for the next 300 seconds.

[0084] In this embodiment, by automatically parsing the fault drill plan from a business perspective into low-level executable fault injection instructions, the fault drill system can achieve a seamless transition from business risk intent to technical operation execution. This mechanism enables the reliable and accurate implementation of complex drill designs generated by AI and tailored to real business scenarios, avoiding deviations and delays that may occur during manual translation and operation. This ensures that the drill process can strictly reproduce the preset risk scenarios, thereby enabling the final verification results to truly reflect the system's actual performance when facing anticipated business risks.

[0085] In one exemplary embodiment, Figure 5 This is a flowchart illustrating the steps of implementing a fault drill scheme according to an embodiment of the present disclosure, as shown below. Figure 5 As shown, step S203 involves executing a fault drill plan, including: S501. Convert the fault simulation plan into fault injection rules for identification using chaos engineering tools; S502. Call the interface of the chaos engineering tool to execute the fault injection rules in the runtime environment where the technical component is located.

[0086] Chaos engineering tools can refer to software platforms used to conduct controlled fault experiments in complex systems. Fault injection rules can refer to declarative configurations defined by these tools that describe fault experiments. The runtime environment can refer to the infrastructure environment in which the technical components are actually deployed, such as a Kubernetes cluster. Kubernetes refers to a container orchestration platform, or K8s, which can be a platform used for automated deployment, scaling, and management of containerized applications.

[0087] For example, the execution engine within the fault simulation system can receive business-oriented fault scenario descriptions generated by AI models. This execution engine can have built-in adapters for various chaos engineering tools, translating general scenario descriptions into configuration formats required by specific tools. The execution engine can create and activate fault experiments in the target environment by calling the control interfaces provided by the chaos engineering tools. In this way, fault scenarios based on high-level business logic are automatically converted into standardized rules recognized by the underlying chaos engineering tools and executed by calling their mature interfaces, achieving seamless and secure integration with existing operational infrastructure and the chaos engineering ecosystem. This approach shields the technical complexity and risks of directly manipulating underlying resources, reuses proven and reliable fault injection capabilities, ensures the repeatability and security of simulation operations, and avoids additional system instability factors that might be introduced by a self-developed injection engine.

[0088] As an example, the execution engine will inject a fault simulation scenario of 5 minutes of network packet loss into Redis cluster A, convert it into a NetworkChaos resource configuration file in YAML format that can be recognized by chaos engineering tool A, and create this resource in the cluster through the Kubernetes Operator of tool A, thereby triggering the actual fault injection.

[0089] In this embodiment, through automated fault simulation scheme conversion and tool invocation, the fault simulation system achieves seamless integration with the underlying infrastructure and the chaos engineering ecosystem. This allows complex simulation schemes designed based on high-level business logic to be executed safely, reliably, and in a standardized manner, shielding the complexity of technical details and improving the efficiency and repeatability of the simulations.

[0090] In one exemplary embodiment, executing fault injection rules in the runtime environment where the technical component resides includes: When the technical component is a cache or database, execute fault injection rules to simulate network isolation or network packet loss.

[0091] Network isolation can refer to completely depriving a target technology component instance of its network communication capabilities. Network packet loss can refer to randomly or proportionally dropping data packets on the network to simulate an unstable network situation.

[0092] For example, when the goal of a fault drill is to verify the application's resilience to dependencies on caching or databases, the execution engine can prioritize network-level fault injection rules. This effectively verifies the application's fault tolerance and degradation mechanisms.

[0093] As an example, to verify the membership service's ability to degrade to Redis, the execution engine can call chaos engineering tools to execute network isolation rules on all nodes of the Redis cluster, creating a complete cache unavailability scenario for high-pressure testing.

[0094] In this embodiment, critical data layer dependencies are verified by focusing on network-level failures. The fault simulation system can capture the most common and impactful fault types in distributed systems. This allows fault simulations to directly simulate dependency failures caused by network jitter, partitions, etc., in a real production environment. This effectively verifies the system's ability to ensure business continuity when critical connectivity is lost, making the fault simulation scenarios highly realistic in terms of infrastructure risks.

[0095] In one exemplary embodiment, Figure 6 A flowchart illustrating the steps for determining the dependence of a service on technical components, as provided in this embodiment of the disclosure, is shown below. Figure 6 As shown, based on preset business rules, the operational data is analyzed to determine the service's dependence on technical components, including: S601. Obtain the business rules preset for the target business scenario through the scenario platform. The business rules are used to define the core business functions and the corresponding health thresholds. S602. Compare the business metrics in the collected operational data with the health thresholds; S603. Based on the comparison results, determine whether the service's dependence on the technical components is strong or weak.

[0096] Here, "business rules" can refer to quantifiable standards stored in the scenario platform and jointly defined by business and technology. "Core business functions" can refer to the business outputs that must be guaranteed in this scenario. "Health thresholds" can refer to the minimum acceptable standards set for core business metrics. "Comparison results" can refer to the objective conclusions drawn regarding whether business metrics fall below the thresholds during a failure period.

[0097] For example, after the fault simulation system completes the simulation, it can proactively query the judgment rules for the business scenario from the scenario platform. The judgment rules can explicitly specify core business indicators and their health thresholds. For instance, for a transaction scenario, the rule might be "order creation success rate ≥ 99.5%"; for a query scenario, the rule might be "P95 interface response time ≤ 200ms"; and for a risk control scenario, the rule might be "frequency of specific risk control error codes < 0.1%". The fault simulation system can then calculate the actual values ​​of the corresponding indicators within the fault window period and compare them with the thresholds, automatically outputting a "strong dependency" or "weak dependency" judgment based on the rules.

[0098] As an example, in the business scenario of "asset delivery," the rule is "delivery success rate ≥ 95%." After fault injection, analysis shows that the delivery success rate is 96%, which is higher than the threshold. Therefore, the fault simulation system determines that the current service has a weak dependency on the faulty technical component.

[0099] In this embodiment, by introducing and strictly enforcing pre-defined, quantified business rules as the final adjudication criteria, the fault simulation system can transform the determination of dependency intensity from subjective technical perception to objective business facts. This ensures that the simulation conclusions directly reflect the actual impact of the fault on real business objectives, guaranteeing that all subsequent optimization suggestions closely revolve around the fundamental purpose of safeguarding business outcomes, and driving architectural governance to align with real business risk tolerance.

[0100] In one exemplary embodiment, the collection of operational data for the target business scenario includes: Collect the interface success rate and interface response time of the target business scenario, as well as the health status indicators of technical components; Based on interface success rate, interface response time, and health status indicators of technical components, operational data for the target business scenario is obtained.

[0101] Among these, the interface success rate refers to the proportion of successful API calls in the relevant business scenario. Interface response time refers to the time spent processing a request. The health status metrics of technical components refer to measures reflecting the operational status of dependent components, such as the number of connections, memory usage, and cluster status.

[0102] For example, the fault simulation system can be integrated with the monitoring system to perform high-frequency data collection on pre-defined monitoring targets (i.e., business scenario interfaces and dependent technical components) during the simulation. These metrics are collected, stored, and correlated with the simulation experiment.

[0103] As an example, when rehearsing the impact of a Redis failure on member queries, the failure simulation system can collect data in parallel: at the interface level: the number of requests per second, average response time, and error rate of / api / getUserType; at the technical component level: the network input / output traffic of the target Redis cluster, the current number of connections, and whether it is in the "CLUSTERDOWN" state.

[0104] In this embodiment, by systematically collecting multi-dimensional indicators covering application layer performance and dependency layer status, the fault simulation system constructs a complete chain of evidence for assessing the impact of faults. This allows the analysis to not only determine business outcomes (success or failure), but also to understand performance changes (speed) and pinpoint the root cause of faults (technical component status), thereby enabling more accurate attribution analysis and dependency strength determination. This makes the simulation evaluation process as comprehensive and objective as real fault diagnosis.

[0105] In one exemplary embodiment, the method further includes: Based on the determined dependency strength, a verification report containing a dependency list and optimization suggestions is generated; The verification report is sent to the fault simulation platform and the scenario platform via a message queue.

[0106] The verification report can refer to a formatted document summarizing the entire process of a single exercise. The dependency list clearly lists the verified components and their dependency strength categories. Optimization suggestions refer to improvement guidelines based on dependency strength assessment and impact analysis. Message queues can refer to a technical component used for asynchronous, decoupled communication.

[0107] For example, after the drill analysis is completed, the report generation module of the fault drill system can automatically fill the scenario, solution, data, and judgment results into a preset template to form a structured report. After the report is generated, the fault drill system can serialize it into a message and publish it to a specified message queue topic. The fault drill platform and the scenario platform can act as consumers, subscribing to and consuming the message to update their respective databases and user interfaces. In this way, asynchronous distribution of verification reports through message queues decouples the fault drill system from downstream platforms. This mechanism ensures that reports are not lost even if downstream platforms are temporarily unavailable, guaranteeing reliable data transmission and eventual consistency. At the same time, the asynchronous processing method avoids blocking the main process of the fault drill system, improving the overall system throughput and responsiveness.

[0108] As an example, the report content could also include: Scenario: User member query. Dependency verification: Redis-Cluster-01. Strength assessment: Weak dependency. Evidence: Interface response time increased by 35%, order success rate remained stable at 99.8%. Recommendation: The degradation mechanism is effective; it is recommended to monitor the database load during degradation. This report can be synchronized to various platforms via message queue.

[0109] In this embodiment, by automatically generating standardized verification reports and utilizing message queues to achieve data synchronization between platforms, the fault drill system can ensure the visualization, traceability, and transferability of drill results. This enables the business risks and architectural weaknesses revealed by the drills to efficiently reach a consensus among different teams, driving the formation of an optimization loop based on shared facts, and ensuring that each fault drill effectively promotes the system's evolution towards a direction more resilient to real risks.

[0110] In one exemplary embodiment, the method further includes: Listen for code change events in the business system; In response to detected code change events, if the change time involves technical components that are determined to be strongly dependent, a new round of fault drills for the strongly dependent technical components will be automatically triggered.

[0111] Code change events refer to operations such as commits and merges that occur within a version control system. Listening operations refer to operations implemented through hook mechanisms integrated into continuous integration / continuous deployment pipelines. Automatic triggering refers to new drill tasks being automatically created and executed by the fault drill system according to a strategy, without manual intervention.

[0112] For example, a fault drill system can be integrated with a code repository or CI / CD platform to subscribe to code push events. When an event notification is received, the fault drill system can analyze the code changes to determine if they modify code related to technical components identified as "strong dependencies" in previous drills (such as connection logic or configuration). If so, the fault drill system can automatically launch a fault drill targeting that strong dependency to verify whether the change has introduced new vulnerabilities. In this way, by listening to code change events and automatically triggering verification of strongly dependent components, continuous and proactive protection against system architecture risks is achieved. This mechanism deeply integrates fault drills into the software development lifecycle, ensuring that any code changes that may affect high-risk dependencies can be verified for resilience in a timely manner. This allows for the early detection and repair of potential vulnerabilities introduced by changes before code merging or release, effectively preventing the system's fault tolerance from being unintentionally compromised during continuous iteration.

[0113] As an example, a developer submitted code that modified the database connection for the "payment service". The fault simulation system can detect that database A is strongly dependent on the payment scenario, and can then automatically orchestrate and execute a fault simulation exercise targeting network latency for that database, feeding the results back to the code review process.

[0114] In this embodiment, by deeply integrating with the R&D process and intelligently triggering based on historical dependency strength knowledge, the fault simulation system can achieve continuous and proactive protection against architectural risks. This ensures that any code changes that may affect high-risk dependencies can be timely and resiliently verified, guaranteeing that the fault tolerance of the fault simulation system is not unintentionally compromised during continuous iteration, and transforming verification capabilities that closely reflect real business risks into automated quality verification throughout the software development lifecycle.

[0115] In some exemplary embodiments, the fault simulation scheme generation process also involves more detailed risk fault analysis and scheme design logic. Specifically, the AI ​​model of the fault simulation system can act as an attack and defense simulation expert, receiving detailed input from the scenario platform. This input includes information such as the business scenario name, affected interfaces and their priorities, and the technical components involved in the interface links. The AI ​​model is required to design the core risk faults that best fit the current business scenario from three dimensions: node dimension, network dimension, and configuration dimension. Its output is a structured risk fault description, which clearly identifies the associated applications, interfaces, and technical components, and elaborates on the core risks (e.g., "Redis connection timeout"), their specific impact levels (e.g., P0), and expected fault impacts (e.g., "RT increases, but the business can be downgraded to the database, and the call will not fail"). This part of the analysis provides direct and accurate input for the subsequent generation of executable fault injection schemes, making the design of fault scenarios based on a deep analysis of business logic and dependent technologies, thereby ensuring that the generated fault schemes can accurately hit the real business vulnerabilities.

[0116] In some exemplary embodiments, the fault scenario generation process more clearly distinguishes and handles startup faults and runtime faults. The AI ​​model of the fault drill system generates two different verification targets for the same underlying technical fault (such as Redis unavailability). The startup fault scenario focuses on verifying whether the service can successfully complete the startup, deployment, or scaling process in the event of a failure in a dependent component. Its technical implementation may involve dynamically inserting fault logic into the application initialization code path. The runtime fault scenario focuses on verifying whether the business functions have the ability to degrade operation when the service is already running and faces a sudden failure in a dependent component. Its scenario parameters (such as latency) can be dynamically set based on historical monitoring data. Through this division, the system enables fault drills to cover two key stages of the service lifecycle, more completely mapping the real business risks that may be faced from the occurrence of a fault to emergency response.

[0117] In some exemplary embodiments, the implementation of precise fault injection includes a more complete interaction process and technical details. The execution process of the fault simulation system begins with the AI ​​engine generating fault rules containing the target user type and storing these rules in a cache (such as Redis). When a user request reaches the system gateway, the gateway intercepts the request and verifies whether the user identifier in the request matches the preset target user type in the fault rules. Fault injection (such as latency or error) is performed only on matching requests; requests that do not match are allowed to proceed normally. Simultaneously, the monitoring system specifically collects relevant metrics (such as failure rate) of the target user requests for subsequent verification of precise effects. This complete chain, from rule generation and dynamic matching to targeted monitoring, enables fault injection to precisely affect a preset business subset (such as high-net-worth clients), greatly reducing the blast radius of the simulation and interference with non-target users, thereby enabling the safe simulation of real localized business risks affecting specific groups in a production environment.

[0118] In some exemplary embodiments, the closed-loop optimization mechanism encompasses more specific three-party collaborative processes and technical implementations. First, the fault simulation platform systematically collects verification result data, including changes in business metrics, dependent component status, and user tag matching, forming a structured record. Second, this result data is used to optimize the AI ​​model, for example, by building a classification model, using verification result features as input to predict dependency levels, and triggering incremental training of the model when the prediction deviates significantly from the actual situation. Next, the optimized AI model output (such as updated dependency levels) and risk analysis conclusions are synchronized to the fault simulation platform and the scenario platform via a message queue, driving the platform to update fault parameters in the scenario library and risk tags for components. Finally, this closed loop can be automatically triggered, for example, by integrating with the CI / CD process, automatically initiating a new round of verification when code changes involve components marked as strongly dependent. This entire mechanism, from data collection and model iteration to platform synchronization and automatic triggering, makes the entire fault drill system an intelligent system capable of self-learning and self-evolving. It ensures that its understanding and verification strategies for business risks can be continuously optimized as the system and business develop, always keeping pace with the ever-changing real business risk situation.

[0119] In some exemplary embodiments, Figure 7 This is a schematic diagram of an interactive process for fault drills provided in an embodiment of this disclosure; such as... Figure 7 As shown, the interaction flow includes: S701, the AI ​​model of the fault simulation system queries the scenario platform for related technical components (such as MySQL, Redis) and interacts with them.

[0120] Among these, "related technical components" refers to technical components that have a calling dependency relationship with the target business scenario, such as MySQL databases and Redis caches. "Query interaction operations" refers to the AI ​​model requesting detailed technical dependency data for a specific business scenario by calling the application programming interface provided by the scenario platform.

[0121] For example, when the fault drill system starts a drill orchestration for a certain business scenario (such as "user membership identity query"), its built-in AI model will first send a request to the scenario platform to query the technical link information associated with the scenario, especially the inter-component instances involved.

[0122] As an example, the AI ​​model calls the API (Application Programming Interface) of the scenario platform, passes in the scenario ID (Identifier), and requests the complete call chain details of the business scenario of querying user membership identity.

[0123] S702, The scenario platform returns scenario data to the AI ​​model of the fault simulation system.

[0124] The returned scenario data may refer to the structured information stored by the scenario platform that corresponds to the request, including but not limited to the business scenario name, the list of associated core interfaces, the interface priority, and the list of dependent technology components and their network addresses obtained through historical link analysis.

[0125] For example, after receiving a query request, the scenario platform can retrieve relevant information from its metadata storage and respond to the AI ​​model in a structured data format, such as JSON (JavaScript Object Notation), to provide input for AI to perform risk analysis and solution design.

[0126] As an example, the data returned by the scenario platform clearly states that the business scenario is user membership identity query, the core interface is " / api / getUserType", and the technical components it depends on are "Redis@cluster-01" and "MySQL@10.2.3.4:3306".

[0127] After the AI ​​model of the S703 fault simulation system performs AI design, it returns the fault solution to the scenario platform.

[0128] AI design refers to the process of using scenario data obtained by an AI model, combined with its built-in failure mode knowledge base, to perform risk analysis, failure matching, and solution generation. The returned failure solution is a structured, executable action plan.

[0129] For example, the AI ​​model integrates business scenario information and technology dependency information, runs its analysis logic, generates one or more customized fault drill plans, and submits these plans to the core scheduling module of the fault drill platform.

[0130] As an example, the AI ​​model designs and returns a solution that could be: "Inject a network packet loss fault into 'Redis@cluster-01' for 300 seconds, effective only for requests with the user tag containing 'high value'." S704, The scenario platform can push a list of injectable faults (such as database connection failure / interface delay) to the fault drill system.

[0131] The injectable fault list refers to a menu of selectable fault options presented to operators (such as operations engineers) after being designed and generated by AI and approved by the platform. Push notifications refer to the platform dynamically updating the operation interface through web interfaces or message notifications.

[0132] For example, after receiving the solution returned by the AI ​​model, the fault simulation platform formats it and displays it in a clear list on the platform's console interface. It typically includes information such as fault description, target object, injection parameters, and estimated impact, waiting for user confirmation or selection to execute.

[0133] As an example, after the platform interface is refreshed, two optional failures are displayed: "Database connection failed (simulating MySQL connection timeout)" and "Interface delay (adding 500ms delay to the specified interface)", and an "Execute" button is provided.

[0134] S705, the fault simulation system performs corresponding internal processing operations.

[0135] The internal processing operations refer to a series of automated processing steps performed by the backend of the fault simulation platform after the user triggers the execution command through the interface, including: solution analysis, resource verification, interaction with chaos engineering tools, generation of monitoring tasks, etc.

[0136] For example, once a user selects a fault from the list and confirms execution, the execution engine of the fault simulation platform begins to work. It translates the user-selected solution into instructions that the underlying system can recognize, calls the relevant APIs to prepare and inject the fault in the target environment, and simultaneously starts the corresponding monitoring data collection task.

[0137] As an example, when a user selects and executes the "interface latency" fault, the platform backend immediately calls the interface of the chaos engineering tool ChaosMesh to create a corresponding NetworkChaos resource in the Kubernetes cluster and start dedicated monitoring of the target interface. Here, NetworkChaos can be a network chaos experiment or a network fault experiment, which can be used to simulate abnormal scenarios such as network latency, packet loss, and partitioning. It can be a declarative resource object defined in a chaos engineering tool (such as ChaosMesh) to describe network fault experiments. By configuring this type of resource, abnormal conditions such as network latency, network packet loss, network corruption, and network partitioning can be simulated on the target container or container instance, thereby verifying the resilience of the distributed system under network instability or interruption.

[0138] S706, The AI ​​model of the fault drill system periodically performs interactive operations with the fault drill platform.

[0139] Among them, regularly executing interactive operations can refer to the mechanism by which the AI ​​model proactively and periodically triggers the fault drill process based on preset strategies or changes in risk labels of the scenario platform.

[0140] For example, in addition to a single exercise being triggered manually, the AI ​​model can be configured to automatically send a new exercise orchestration request to the fault simulation platform at a certain period (such as weekly) or when the risk level of a component in the scenario platform is monitored to increase, thereby driving periodic or event-driven automated verification.

[0141] As an example, the AI ​​model runs automatically every weekend, analyzing system changes and risk labels from the past week, generating new fault drill plans for high-risk scenarios and submitting them to the platform, which then automatically executes the fault drill plans during off-peak hours.

[0142] This disclosure allows for the automatic generation of "startup + runtime" fault scenarios and dependency attributes based on AI large-scale model parsing technology component documentation. It also generates a fault scenario library based on this documentation and dynamically customizes business-specific fault solutions in conjunction with risk tags. Furthermore, it allows for the injection of user tags (such as high-net-worth clients) through the gateway to limit the scope of fault effectiveness, enabling targeted verification. Finally, it combines scenario platform rules, using core business interruption as the strong dependency criterion, to output a list of optimization suggestions.

[0143] In some exemplary embodiments, the business scenario fault simulation method provided in this disclosure is applied to the asset settlement business scenario of a financial transaction system. Specifically, the fault simulation system acquires scenario information of the asset settlement business scenario, which includes the core middleware that the scenario depends on, including risk control rule cache (Redis cluster), transaction log database (MySQL master database), and message queue (RocketMQ). Based on this scenario information, the AI ​​model analyzes the strong dependency characteristics of the settlement process on the risk control rule cache and generates a customized fault simulation plan: simulating the failure of the Redis cluster master node, verifying whether the settlement service can obtain rules from the risk control engine in real time and complete the settlement through degradation logic after the cache expires. After the fault injection is executed, the system collects business indicators such as settlement success rate and average settlement time, and compares them with preset business rules (such as settlement success rate ≥ 99.99%) to determine the dependency strength. If the settlement success rate falls below the threshold, Redis is determined to be a strong dependency, and optimization suggestions can be generated: increase local caching of risk control rules and multi-level degradation strategies. In this embodiment, by applying the method to the transaction business scenario of asset delivery, the fault simulation can accurately verify the resilience of fund transfer business to key middleware, ensuring that the core fund link is not affected when a real fault occurs, thereby ensuring the business continuity and fund security of the transaction system.

[0144] In some exemplary embodiments, the business scenario fault simulation method provided in this disclosure is applied to the personalized recommendation business scenario of a content delivery network. Specifically, the fault simulation system can obtain scenario information for the homepage information stream recommendation business scenario. This information includes the technical components that the scenario relies on, including a user profile database (MySQL), a recommendation model feature cache (Redis), and content material storage (Object Storage Service OSS). Based on this personalized recommendation business scenario information, the AI ​​model analyzes the high-frequency read characteristics of the feature cache in the recommendation chain and generates a differentiated fault simulation plan: simulating a 500-millisecond increase in Redis cache service response latency to verify whether the recommendation service can ensure the rapid return of homepage content through asynchronous preloading or a circuit breaker mechanism when cache performance degrades. After the fault injection is executed, the system can collect business metrics such as the response time of the recommendation interface and the exposure rate of recommended content, and compare them with preset business rules (such as P99 response time ≤ 800 milliseconds) to determine the dependency strength. If the response time exceeds the threshold but the exposure rate does not decrease, Redis is determined to be a weak dependency, and optimization suggestions are generated, such as optimizing the cache read timeout configuration and adding a hot data preloading mechanism. In this embodiment, by applying the method to the high-concurrency business scenario of personalized recommendation, the fault drill can effectively verify the user experience guarantee capability of the content distribution system when facing the performance degradation of dependent components, ensuring that users can still obtain a smooth browsing experience when a real fault occurs, thereby maintaining the core user value of the content platform.

[0145] In some exemplary embodiments, the business scenario fault simulation method provided in this disclosure is applied to the business scenario of a flash sale on an e-commerce platform. Specifically, the fault simulation system obtains scenario information for the "flash sale" business scenario. This scenario information may include the technical components that the scenario depends on, including the inventory deduction database (MySQL master database), the activity status cache (Redis cluster), and the rate limiting gateway (Sentinel). Based on this scenario information, the AI ​​model analyzes the extremely high concurrent write characteristics of the flash sale scenario to the inventory database and generates a targeted fault simulation plan: simulating the master-slave synchronization delay of the inventory database, verifying whether the flash sale service can ensure that overselling does not occur when data consistency is compromised through optimistic locking or asynchronous compensation mechanisms. After the fault injection is executed, the system can collect business indicators such as the flash sale success rate and the number of oversold orders, and compare them with preset business rules (such as oversold rate = 0%) to determine the dependency strength. If the master-slave delay causes overselling to occur, the database synchronization mechanism is determined to be a strong dependency, and optimization suggestions are generated, such as upgrading to a distributed transaction scheme and adding a secondary verification before inventory deduction. In this embodiment, by applying the method to the high-pressure, high-consistency business scenario of flash sales, the fault simulation can accurately verify the e-commerce system's ability to ensure data consistency under extreme traffic conditions, ensuring the accuracy of core transaction data and the seriousness of business rules when real faults occur, thereby protecting the platform's reputation and consumer rights.

[0146] In some exemplary embodiments, the business scenario fault simulation method provided in this disclosure is applied to the live teaching business scenario of an online education platform. Specifically, the fault simulation system acquires scenario information of the real-time interactive classroom business scenario. This scenario information may include the technical components that the scenario depends on, including a signaling server (WebSocket cluster), an audio and video stream forwarding service (media server), and a classroom state cache (Redis). Based on this scenario information, the AI ​​model analyzes the low-latency dependency characteristics of the live scenario on the signaling service and generates a targeted fault simulation plan: simulating a WebSocket signaling server network partition to verify whether the teaching service can ensure the continuity of basic classroom functions (such as courseware synchronization and text chat) through client reconnection mechanisms and local state cache when the signaling channel is interrupted. After the fault injection is executed, the system can collect business indicators such as signaling delivery success rate and classroom interruption rate, and compare them with preset business rules (such as signaling delivery rate ≥ 99.5%) to determine the dependency strength. If the signaling delivery rate falls below a threshold but the class interruption rate does not increase significantly, the signaling service is determined to be weakly dependent, and optimization suggestions are generated, such as increasing client-side local state persistence and optimizing reconnection backoff strategies. In this embodiment, by applying this method to the highly real-time business scenario of online live streaming, fault drills can accurately verify the business continuity assurance capabilities of the real-time communication system in the face of network fluctuations, ensuring that the core teaching scenario is not interrupted when a real fault occurs, thereby maintaining the service quality and user retention of the online education platform.

[0147] In some exemplary embodiments, the business scenario fault simulation method provided in this disclosure is applied to the device command issuance business scenario of an IoT platform. Specifically, the fault simulation system acquires scenario information of the remote device control business scenario. This scenario information may include the technical components that the scenario depends on, including the device shadow database (MongoDB), the command issuance message queue (Kafka), and the device connection gateway (MQTT cluster). Based on this scenario information, the AI ​​model analyzes the reliable delivery dependency characteristics of the device control scenario on the message queue and generates a targeted fault simulation plan: simulating Kafka message queue backlog, verifying whether the command issuance service can ensure the timely delivery of critical control commands through command priority scheduling and timeout retry mechanisms when messages are piling up. After the fault injection is executed, the system can collect business indicators such as command issuance success rate and average command delivery delay, and compare them with preset business rules (critical command delay ≤ 1 second) to determine the dependency strength. If the critical command delay exceeds the threshold but the final success rate remains high, the message queue is determined to be a weak dependency, and optimization suggestions are generated, such as: adding a dedicated Topic for critical commands and configuring differentiated consumer groups. In this embodiment, by applying the method to the high-reliability business scenario of IoT device control, the fault drill can accurately verify the message middleware's ability to deliver instructions under abnormal loads, ensuring that critical control instructions can still be reliably executed when a real fault occurs, thereby ensuring the IoT platform's effective control over terminal devices and user safety.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] The following describes the business scenario fault simulation device provided in the embodiments of this disclosure. The business scenario fault simulation device has the same inventive concept as the business scenario fault simulation method described above. The solution to the problem provided by the device is similar to the solution described in the above method. Therefore, the specific limitations of one or more business scenario fault simulation device embodiments provided below can be referred to the limitations of the business scenario fault simulation method above. The business scenario fault simulation device described below and the business scenario fault simulation method described above can be referred to each other, and will not be repeated here.

[0150] In one exemplary embodiment, Figure 8 This is a schematic diagram of the structure of a business scenario fault drill device provided in an embodiment of this disclosure, as shown below. Figure 8 As shown, the business scenario fault drill device 80 includes: a business scenario information acquisition module 810, a fault drill plan acquisition module 820, a fault injection execution module 830, and a dependency strength analysis and judgment module 840, wherein: The business scenario information acquisition module 810 is used to acquire business scenario information of the target business scenario; wherein, the business scenario information includes the call relationship of at least one technical component on which the target business scenario depends.

[0151] The fault drill plan acquisition module 820 is used to input business scenario information into the AI ​​model to generate a fault drill plan for the target business scenario.

[0152] The fault injection execution module 830 is used to execute the fault drill plan and inject faults into the technical components on which the service of the target business scenario depends.

[0153] The dependency strength analysis and judgment module 840 is used to collect the operation data of the target business scenario after the injection of fault, and analyze the operation data based on the preset business rules to determine the dependency strength of the service on the technical components. The dependency strength is used to characterize the degree of impact of the failure of the technical components on the core functions of the target business scenario.

[0154] In an exemplary embodiment, the fault simulation scheme acquisition module 820 is used to feed back the determined dependency strength to the AI ​​model, and the AI ​​model is used to optimize the generation of fault simulation schemes based on the dependency strength.

[0155] In an exemplary embodiment, the fault simulation scheme acquisition module 820 is used to use the verification results containing dependency strength as training samples to incrementally train the AI ​​model; and to update the adjustment strategy parameters of the generated fault simulation scheme according to the trained AI model.

[0156] In an exemplary embodiment, the business scenario information acquisition module 810 is used to query the list of interfaces and call chains associated with the target business scenario from the scenario platform; and extract the identification information of at least one technical component on which the target business scenario depends from the call chain.

[0157] In one exemplary embodiment, generating a fault drill plan for a target business scenario includes: an AI model analyzing the dependency of the target business scenario on at least one technical component based on business scenario information, and generating a fault drill plan for the target business scenario.

[0158] In an exemplary embodiment, the AI ​​model analyzes the dependency of a target business scenario on at least one technical component based on business scenario information, including: the AI ​​model obtains the potential failure modes of the technical component by parsing the official documentation of the at least one technical component; and generates a failure drill plan for the target business scenario based on the business scenario information and the potential failure modes; the failure types described in the official documentation include at least one of the node dimension, network dimension and configuration dimension.

[0159] In an exemplary embodiment, the AI ​​model generates a fault simulation plan for a target business scenario, including: generating a startup fault plan to verify whether the service can be successfully started or scaled up when a technical component fails; and / or generating a runtime fault plan to verify whether the business functions of the service can be degraded when a technical component fails.

[0160] In an exemplary embodiment, the fault injection execution module 830 is used to intercept business requests based on user tag information; the user tag information is determined based on at least one of user value dimension, user behavior dimension, user channel dimension, or user risk dimension.

[0161] In an exemplary embodiment, the fault injection execution module 830 is further configured to verify whether the intercepted service request matches the user tag information; and to inject a fault into the service request that matches the user tag information.

[0162] In an exemplary embodiment, the fault injection execution module 830 is used to determine user tag information and fault type that match at least one feature of the target business scenario.

[0163] In an exemplary embodiment, the fault injection execution module 830 is used to parse the fault drill plan into an executable fault injection instruction; based on the fault injection instruction, faults are injected into the technical components on which the service running the target business scenario depends.

[0164] In an exemplary embodiment, the fault injection execution module 830 is used to convert the fault simulation scheme into fault injection rules for identification by the chaos engineering tool; and to call the interface of the chaos engineering tool to execute the fault injection rules in the runtime environment where the technical component is located.

[0165] In an exemplary embodiment, the fault injection execution module 830 is used to execute fault injection rules that simulate network isolation or network packet loss when the technical component is a cache or database.

[0166] In an exemplary embodiment, the dependency strength analysis and determination module 840 is used to obtain, through the scenario platform, preset business rules for the target business scenario, the business rules are used to define core business functions and corresponding health thresholds; compare the business indicators in the collected running data with the health thresholds; and determine whether the service's dependency on the technical components is strong or weak based on the comparison result.

[0167] In an exemplary embodiment, the dependency strength analysis and determination module 840 is used to collect the interface success rate and interface response time of the target business scenario, as well as the health status indicators of the technical components; based on the interface success rate, interface response time, and health status indicators of the technical components, the operating data of the target business scenario is obtained.

[0168] In an exemplary embodiment, the dependency strength analysis and determination module 840 is used to generate a verification report containing a dependency list and optimization suggestions based on the determined dependency strength; and send the verification report to the fault drill platform and the scenario platform through a message queue.

[0169] In an exemplary embodiment, the fault injection execution module 830 is used to listen for code change events in the business system; in response to the listened code change event, if the change time involves a technical component that is determined to be strongly dependent, a new round of fault drill process for the strongly dependent technical component is automatically triggered.

[0170] Each module in the aforementioned business scenario fault simulation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0171] In one exemplary embodiment, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the business scenario fault simulation methods described in the above embodiments.

[0172] In one exemplary embodiment, this disclosure also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the business scenario fault simulation methods described in the above embodiments.

[0173] In one exemplary embodiment, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the business scenario fault simulation methods described in the above embodiments.

[0174] Indicatively, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the internal structure of a computer device 900 provided in an embodiment of the present disclosure. The computer device 900 can be provided as a server. (Refer to...) Figure 9 The computer device 900 includes a processor 902, which further includes one or more processors, and memory resources represented by memory 901 for storing instructions executable by the processor 902, such as computer programs. The computer program stored in memory 901 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 902 is configured to execute instructions to perform the business scenario fault simulation method of any of the above embodiments. The computer device 900 can operate on an operating system stored in memory 901, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0175] The computer device 900 may also include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. Wireless operation can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a business scenario fault simulation method. The display unit 907 of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink display screen. The input device 906 of the computer device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad located on the computer device casing, or an external keyboard, touchpad, or mouse, etc.

[0176] Those skilled in the art will understand that Figure 9The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0178] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0179] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this disclosure.

[0180] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A method for fault simulation in a business scenario, characterized in that, The method includes: Obtain business scenario information of the target business scenario; wherein, the business scenario information includes the call relationship of at least one technical component on which the target business scenario depends; The business scenario information is input into the AI ​​model to generate a fault drill plan for the target business scenario; The fault simulation plan is executed to inject faults into the technical components on which the service running the target business scenario depends; After the fault is injected, the operation data of the target business scenario is collected, and the operation data is analyzed based on business rules to determine the dependence strength of the service on the technical component. The dependence strength is used to characterize the degree of impact of the fault of the technical component on the core function of the target business scenario.

2. The method according to claim 1, characterized in that, The method further includes: The determined dependency strength is fed back to the AI ​​model, which is used to optimize the generation of fault simulation schemes based on the dependency strength.

3. The method according to claim 2, characterized in that, The AI ​​model is generated based on the dependency strength optimization fault simulation scheme, including: The verification results containing the dependency strength are used as training samples to incrementally train the AI ​​model; Based on the trained AI model, update the adjustment strategy parameters for generating fault simulation plans.

4. The method according to claim 1, characterized in that, The acquisition of business scenario information for the target business scenario includes: Query the list of interfaces and call chains associated with the target business scenario from the scenario platform; Extract the identification information of at least one technical component that the target business scenario depends on from the call chain.

5. The method according to claim 1, characterized in that, The step of inputting the business scenario information into the AI ​​model to generate a fault drill plan for the target business scenario includes: Based on the business scenario information, the AI ​​model analyzes the dependency relationship between the target business scenario and the at least one technical component, and generates a fault drill plan for the target business scenario.

6. The method according to claim 5, characterized in that, The AI ​​model analyzes the dependency relationship between the target business scenario and the at least one technical component based on the business scenario information, including: The AI ​​model obtains the potential failure modes of the technical components by parsing the official documentation of at least one of the technical components; and generates a failure drill plan for the target business scenario based on the business scenario information and the potential failure modes; the failure types described in the official documentation include at least one of the node dimension, network dimension and configuration dimension.

7. The method according to claim 6, characterized in that, The generation of a fault simulation plan for the target business scenario includes: Generate a startup failure scenario to verify whether the service can successfully start or scale up in the event of a failure of the aforementioned technical components; and / or, Generate runtime fault scenarios to verify whether business functions can be degraded to operate when the technical components fail.

8. A business scenario fault simulation device, characterized in that, The device includes: A business scenario information acquisition module is used to acquire business scenario information of a target business scenario; wherein, the business scenario information includes the call relationship of at least one technical component on which the target business scenario depends; The fault drill plan acquisition module is used to input the business scenario information into the AI ​​model to generate a fault drill plan for the target business scenario. The fault injection execution module is used to execute the fault drill plan and inject faults into the technical components on which the service running the target business scenario depends. The dependency strength analysis and determination module is used to collect the operation data of the target business scenario after the fault is injected, and analyze the operation data based on business rules to determine the dependency strength of the service on the technical component. The dependency strength is used to characterize the degree of impact of the fault of the technical component on the core function of the target business scenario.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images