Chaotic experiment case generation method and device, computer device and storage medium

Abstract

Embodiments of the present application relate to a kind of chaos experiment use case generation method, device, computer equipment and storage medium, by obtaining mirror image flow in production environment;The mirror image flow is analyzed, and flow characteristic data is obtained;Service topology is constructed based on flow characteristic data, and the service topology is analyzed, and target call mode is identified, and target call mode is formed by multiple service calls according to the call behavior characteristics of association relationship based on flow characteristic data;Chaos experiment use case is generated based on target call mode;Chaos experiment use case is executed in isolated environment, and chaos experiment use case is evaluated and screened according to execution result.Therefore, it realizes the automation conversion from flow to fault injection experiment, effectively improves the pertinence, accuracy of chaos experiment, reduces the complexity of artificial construction experiment use case and improves system stability verification efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of cloud computing technology, and in particular to a method, apparatus, computer device and storage medium for generating chaotic experimental cases. Background Technology

[0002] In a microservices architecture, the system is broken down into multiple independently deployed services that interact with each other via lightweight communication protocols. While this architecture enhances system flexibility and scalability, it also introduces complex service dependencies and potential risks of fault propagation. To ensure the stability and reliability of the system in the face of unexpected failures, chaos engineering has emerged. It proactively injects faults (such as service latency, error returns, node failures, etc.) into the system to verify its fault tolerance and resilience.

[0003] Currently, chaos engineering implementation primarily relies on two models: one is fault injection based on chaos engineering platforms (such as Chaos Mesh), which injects pre-defined faults into the infrastructure layers of Kubernetes clusters, including Pods (container groups), networks, and kernels; the other is fault injection based on service meshes (such as Istio), which leverages its built-in traffic management capabilities to inject faults at the application layer. However, both models share a common drawback: the design of experimental scenarios heavily depends on engineers' experience and judgment, typically based on theoretical fault modes or historical incident replays, which may differ significantly from the dynamic and ever-changing business traffic patterns currently being handled by the system. This can lead to experiments failing to reach the core call chains, resulting in low testing efficiency, and even missing deep-seated defects that only trigger under specific traffic combinations.

[0004] Current technologies only address the issue of how to safely conduct experiments using real traffic, but fail to solve the fundamental problem of designing the right chaos experiments for real traffic. Generating test cases remains a manual, offline process highly dependent on expert experience, lacking the intelligent capability to automatically analyze and derive high-value, highly relevant chaos test cases from real traffic. Therefore, how to automatically generate targeted chaos engineering test cases based on real production traffic has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, in order to solve the above-mentioned technical problems or some of the technical problems, the present invention provides a method, apparatus, computer equipment and storage medium for generating chaotic experimental cases.

[0006] In a first aspect, embodiments of the present invention provide a method for generating chaotic experimental cases, comprising: Obtain mirrored traffic from the production environment; The mirrored traffic is parsed to obtain traffic feature data, which is used to characterize the call relationship and call behavior between services; A service topology is constructed based on the traffic feature data, and the service topology is analyzed to identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their associations, based on the traffic feature data. Chaos experiment test cases are generated based on the target call pattern. The chaos experiment test cases are experimental configurations used to describe fault injection for the target service. The chaotic experimental cases are executed in an isolated environment, and the chaotic experimental cases are evaluated and screened based on the execution results.

[0007] In one possible implementation, acquiring mirrored traffic in the production environment includes: Through the service mesh's traffic mirroring mechanism, a complete copy of the real business traffic is replicated from the production environment according to a preset ratio as the initial mirrored traffic. The preset ratio is dynamically adjusted according to the load of the production environment and analysis needs. Perform a filtering operation on the initial mirror traffic to obtain the mirror traffic. The filtering operation includes at least one of the following: filtering, sampling, or removing invalid or duplicate requests from the initial mirror traffic.

[0008] In one possible implementation, parsing the mirrored traffic to obtain traffic characteristic data includes: Extract request and response data from the service calls within the mirrored traffic; Based on the request data and the response data, identify the service identity information and the corresponding interface identification information of the service participating in the service call; Request parameter information for characterizing the called content is extracted from the request data, and response result information and call performance information for characterizing the call result are extracted from the response data. The traffic feature data is generated based on the service identity information, interface identification information, request parameter information, response result information, and call performance information.

[0009] In one possible implementation, the step of constructing a service topology based on the traffic characteristic data and analyzing the service topology to identify target call patterns includes: Based on the service identity information and the interface identification information, a service topology representing the call relationship between services is constructed; Based on the request parameter information, identify service call sequences with the same business entity identifier, and determine service call combinations with data dependencies from them; Based on the time information in the call performance information, identify the service call sequence that occurs continuously within a preset time window to obtain the time-series associated service call sequence. Based on the call chain length in the service topology, call chains that span a preset number of services are identified to obtain long call chains; The target call pattern is determined by at least one of the service call combination with data dependencies, the service call sequence with time correlation, and the long call chain.

[0010] In one possible implementation, generating chaotic experimental use cases based on the target invocation pattern includes: Based on the target invocation pattern, determine the corresponding target service and the invocation relationship between the services; Based on the call relationship and the service topology, the target location for fault injection is determined, wherein the target location includes: service nodes in the service topology or call links between services; Based on the service call sequence associated with the time sequence, the time location of fault injection is determined; Based on the target invocation pattern, a corresponding fault injection method is matched from the preset fault injection strategy; Based on the target location, the time location, and the fault injection method, a corresponding chaotic experimental case is generated.

[0011] In one possible implementation, generating chaotic experimental use cases based on the target invocation pattern includes: Determine the fault injection method that matches the target invocation pattern from the preset fault injection strategies; For the service call combination with data dependency, based on the fault injection method, experimental cases are generated to inject faults into the target service or call relationship in the call chain of the service call combination, and the scope of fault injection is limited to requests with the same business entity identifier. For the time-series associated service call sequence, based on the fault injection method, generate experimental test cases to inject delays or errors into the target service in the sequence within the preset time window; For the long call chain, based on the fault injection method, experimental test cases are generated to inject faults into the target service in the long call chain; Generate the chaos test cases based on the generated test cases.

[0012] In one possible implementation, the step of executing the chaos experiment cases in an isolated environment and evaluating and filtering the chaos experiment cases based on the execution results includes: The chaos experiment case is executed in the isolated environment, and the call performance information of the target service is collected during the execution. The call performance information includes at least response latency, error rate and call success rate. Based on the call performance information, determine the performance changes of the target service before and after fault injection; Based on the comparison between the performance changes and the preset performance threshold, the evaluation result of the chaos experiment case is determined. The chaotic experimental cases are screened based on the evaluation results, and the preset fault injection strategy is adjusted or updated based on the screening results for use in the generation of subsequent chaotic experimental cases.

[0013] Secondly, embodiments of the present invention provide a chaotic experimental case generation device, comprising: The acquisition module is used to acquire mirror traffic in the production environment; The parsing module is used to parse the mirrored traffic to obtain traffic feature data, which is data used to characterize the call relationship and call behavior between services; The construction module is used to construct a service topology based on the traffic feature data, analyze the service topology, and identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their associations, based on the traffic feature data. A generation module is used to generate chaotic experimental cases based on the target call pattern. The chaotic experimental cases are experimental configurations used to describe fault injection for the target service. The processing module is used to execute the chaos experiment cases in an isolated environment and evaluate and filter the chaos experiment cases based on the execution results.

[0014] Thirdly, embodiments of the present invention provide a computer device, including: a processor and a memory, wherein the processor is configured to execute a chaos experiment case generation program stored in the memory to implement the chaos experiment case generation method described in any one of the first aspects above.

[0015] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the chaotic experimental case generation method described in any one of the first aspects.

[0016] The chaos experiment test case generation scheme provided in this invention obtains mirrored traffic from the production environment; parses the mirrored traffic to obtain traffic feature data, which characterizes the call relationships and behaviors between services; constructs a service topology based on the traffic feature data and analyzes the service topology to identify the target call pattern, which is a call behavior feature formed by multiple service calls according to their relationships based on the traffic feature data; generates chaos experiment test cases based on the target call pattern, which describe the experimental configuration for fault injection against the target service; executes the chaos experiment test cases in an isolated environment, and evaluates and filters the chaos experiment test cases based on the execution results. Therefore, by parsing mirrored traffic from the production environment, constructing a service topology, and identifying the target call pattern, and then automatically generating chaos experiment test cases and executing and evaluating them in an isolated environment, the scheme achieves automated transformation from real call behavior to fault injection experiments. This effectively improves the relevance, accuracy, and security of chaos experiments, reduces the complexity of manually constructing test cases, and improves the efficiency of system stability verification. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for generating chaotic experimental cases according to an embodiment of the present invention; Figure 2 A flowchart illustrating another method for generating chaotic experimental cases provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a chaotic experimental test case generation device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0020] Figure 1 This is a flowchart illustrating a method for generating chaotic experimental cases according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method specifically includes: S11. Obtain mirrored traffic in the production environment.

[0021] The chaos test case generation method provided in this invention is mainly applied to production or near-production environments of microservice architectures, and is particularly suitable for scenarios that require continuous assurance of system stability, such as pre-sales stress testing, post-architectural verification, pre-launch evaluation of new services, and routine chaos engineering practices. Through automated chaos test generation and verification, it helps operations and development teams identify potential system vulnerabilities in advance, improving the resilience and reliability of microservice systems.

[0022] The subject of this invention is an independently deployed intelligent derivative system within a computer device. This system collaborates with existing service meshes (such as Istio) and chaos engineering platforms (such as Chaos Mesh). The intelligent derivative system is responsible for acquiring mirrored traffic from the service mesh, performing deep analysis and pattern recognition, automatically generating chaos experimental cases, completing pre-validation in an isolated sandbox, and finally outputting the validated experimental cases to the chaos engineering platform for execution or storing them in the experimental asset repository.

[0023] In this embodiment, the traffic mirroring function of the service mesh is used to replicate a portion of the real business traffic in the production environment in real time and send it to the intelligent derivative system as the input data source for subsequent analysis. The service mesh (such as Istio), as the infrastructure layer for handling inter-service communication in a microservice architecture, provides this advanced traffic governance function of traffic mirroring. It can completely replicate a specified proportion of real-time request traffic and send it to a specified target, while the original traffic is still forwarded normally to its original destination, without affecting the processing and return results of the original request.

[0024] In the implementation, traffic mirroring rules are configured in the service mesh. Taking Istio as an example, by defining a VirtualService resource, the mirror field in the routing rules specifies the target address of the mirrored traffic as the intelligent derivative system, and the mirrorPercentage field controls the proportion of mirrored traffic, typically set to 1%-5%. After configuration, the service mesh will automatically copy the request traffic that meets the conditions and send it to the traffic receiving endpoint deployed on the intelligent derivative system.

[0025] In one possible implementation, a complete copy of the real business traffic is copied from the production environment according to a preset ratio through the traffic mirroring mechanism of the service mesh, which serves as the initial mirrored traffic. The preset ratio is dynamically adjusted according to the load of the production environment and analysis needs. A filtering operation is performed on the initial mirrored traffic to obtain the mirrored traffic. The filtering operation includes at least one of the following: filtering, sampling, and removing invalid or duplicate requests from the initial mirrored traffic.

[0026] In this embodiment, the traffic mirroring mechanism provided by the service mesh is first used to replicate real business requests from the production environment at a preset ratio when requests pass through the data plane proxy, generating initial mirrored traffic. The preset ratio can be dynamically adjusted based on the real-time load of the production environment. For example, when the system load is lower than a first preset load or the analysis demand is higher than a first preset demand, the replication ratio of the mirrored traffic is increased to obtain more comprehensive call data. Conversely, when the system load is higher than a second preset load, the replication ratio of the mirrored traffic is decreased to reduce the impact on the production environment. The load can be determined using metrics such as CPU utilization, request throughput, or response latency. After obtaining the initial mirrored traffic, a filtering process is performed, including filtering requests based on preset rules to remove irrelevant traffic (e.g., retaining only traffic from specified services or interfaces), sampling requests according to a set sampling strategy (further downsampling based on the mirror ratio), and deduplicating and cleaning up duplicate or invalid requests based on request identifiers or time windows (e.g., health check requests, repeatedly sent identical requests), thereby obtaining the mirrored traffic for subsequent analysis. The filtered mirrored traffic can contain complete request lines, request headers, request bodies, response status codes, response latency, and metadata information of the caller and the target service, providing a high-quality data foundation for subsequent traffic parsing and feature extraction.

[0027] S12. Parse the mirrored traffic to obtain traffic feature data, which is used to characterize the call relationship and call behavior between services.

[0028] In this embodiment, the mirrored traffic is parsed message by message. First, the requests are decoded based on the network protocol and message structure in the traffic to extract basic call information such as the source service identifier, target service identifier, interface identifier, and request method. The source service identifier is the unique identifier of the service initiating the request, the target service identifier is the unique identifier of the service receiving the request, and the interface identifier is used to distinguish different service interfaces. Further, the source service and target service are mapped using metadata or service registration information in the service mesh to determine the call relationship between services. The metadata or service registration information is configuration information used to record service instances and their attributes. Simultaneously, behavioral feature data during the call process is extracted from the request and response messages. This may include request parameter information, response status code, response latency, and timestamps, used to characterize the call behavior and its execution result. The request parameter information is a set of parameters characterizing the request content, the response status code is a status identifier characterizing the request processing result, the response latency is the time elapsed from sending the request to receiving the response, and the timestamp is a timestamp recording the time the request occurred.

[0029] For multiple call records in the same request chain, they can be associated and reassembled based on the request identifier or link tracing information to restore the complete service call path. Link tracing information is an association identifier used to identify the process of the same request being passed between different services. Finally, the call relationship information and call behavior information are structured and integrated to form traffic feature data, which is used for subsequent service topology construction and call pattern identification.

[0030] In one possible implementation, request and response data are extracted from the service call in the mirrored traffic; based on the request and response data, the service identity information and corresponding interface identification information of the service call are identified; request parameter information to characterize the call content is extracted from the request data, and response result information to characterize the call result and call performance information are extracted from the response data; traffic feature data is generated based on the service identity information, interface identification information, request parameter information, response result information and call performance information.

[0031] In this embodiment, deep analysis of the collected mirrored traffic is performed to extract structured feature data from the original requests and responses, providing a foundation for subsequent service topology construction and call pattern recognition. First, mirrored traffic is read from the message queue, and each service call is parsed at the protocol level to separate request and response data. Request data consists of the request message data carried when the service initiates the call, which may include: HTTP method, URL path, request headers, request body, etc. Response data consists of the response message data returned by the target service, which may include: response status code, response headers, response body, and response latency information obtained from service mesh metrics.

[0032] Based on the extracted request and response data, the system further identifies the service identity information and corresponding interface identification information involved in the service call. The service identity information is used to identify the service instance or service name involved in the call, and may include: the caller service identifier and the target service identifier, which can be obtained from the service mesh metadata or Kubernetes tag in the request header; the interface identification information is the path or method identifier used to distinguish different service interfaces, which is obtained from the combination of the URL path and HTTP method in the request line.

[0033] Subsequently, the request data undergoes deep analysis to extract request parameter information that characterizes the invoked content. Parameter request information is the set of parameters carried in the request, reflecting the business input content. For HTTP requests, the system deserializes the request body according to Content-Type, extracting business parameters from JSON, XML, or form data, particularly user identity identifiers (e.g., user_id), business entity identifiers (e.g., order_id, product_id), and operation types (e.g., create, pay, query). For response data, the system extracts response result information (e.g., business status codes and error messages in the response body) and call performance information (e.g., response latency, HTTP status codes, and time information representing request sending and completion times) for subsequent evaluation of the service call's health and performance. The response result indicates whether the call was successful, while the call performance information represents metrics characterizing the call process's performance.

[0034] Finally, service identity information, interface identification information, request parameter information, response result information, and call performance information are correlated and integrated to construct structured data records and generate traffic feature data for subsequent service topology construction and call pattern recognition. Optionally, each traffic feature data record may include fields such as timestamp, caller service identifier, target service identifier, interface identifier, user identity identifier, business entity identifier, operation type, response latency, and response status code, stored in key-value pairs or vectors for use by the subsequent service topology construction and call pattern recognition modules. The entire parsing process adopts an asynchronous batch processing method to ensure that the system can keep up with the traffic peaks of the production environment.

[0035] S13. Construct a service topology based on traffic feature data, analyze the service topology, and identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their relationship based on traffic feature data.

[0036] In this embodiment, based on service identity information and interface identifier information in traffic feature data, the call relationship between services is constructed, and a service topology is generated accordingly. The service topology is a graph structure model representing the call relationship between multiple services, where nodes represent services and edges represent the call relationship between services. After obtaining the service topology, it is analyzed based on request parameter information, call performance information, and time information in the traffic feature data. Request parameter information is used to identify service call sequences with the same association identifier to determine service call combinations with data dependencies. Call performance information and time information are used to identify service call sequences occurring consecutively within a preset time window to determine service call sequences with temporal correlation. Simultaneously, based on the call path length in the service topology, call paths spanning a preset number of service nodes are identified to determine long call chains. Finally, the service call combinations, temporally correlated service call sequences, and long call chains are summarized and integrated to obtain the target call pattern. The target call pattern is a call behavior feature composed of multiple service calls according to a preset association relationship, used to characterize the fault-sensitive critical call structure in the system.

[0037] Optionally, machine learning models can be applied to identify potentially fault-sensitive call patterns as target call patterns. During model training, five types of features are extracted from traffic feature data for model training. Temporal features include call timestamps, call intervals, and call frequencies, reflecting the temporal distribution of service calls; topology and dependency features include caller service identifiers, callee service identifiers, call depth, and fan-in / fan-out degrees, characterizing the dependencies between services; traffic features include interface paths, request methods, response status codes, and response latency, describing the basic attributes of a single call; business semantic features include user identity identifiers, business entity identifiers, and operation types, reflecting the business context of the call; and historical fault correlation features include the deviation from historical traffic patterns and whether the call pattern has historically caused faults, used to assess potential risks.

[0038] An unsupervised learning algorithm (such as Isolation Forest, Autoencoder, or Single-Class Support Vector Machine) is used to train the recognition model. The training data consists of historical normal traffic features. The model establishes a baseline by learning the distribution characteristics of normal call patterns. The system continuously collects mirrored traffic from the production environment and extracts features in real time to form a call feature stream. Every preset time window (e.g., hourly), the accumulated feature stream is input into the trained model. The model outputs a sensitivity score or anomaly index for each call pattern. This score indicates the degree to which the current call pattern deviates from the normal baseline; a higher score indicates that the pattern is more sensitive to faults.

[0039] All call patterns are sorted based on sensitivity scores. Top-ranked call patterns (such as "synchronous call from the payment core service to the risk control service" or "call to the inventory service within 3 seconds of order creation") are automatically pushed to the chaos test case synthesizer as priority test candidates. The push results also include the service topology context, traffic characteristic snapshot, and detailed composition of the sensitivity score corresponding to the pattern, allowing the synthesizer to generate accurate chaos test cases.

[0040] In one possible implementation, a service topology representing the call relationship between services is constructed based on service identity information and interface identification information; service call sequences with the same business entity identifier are identified based on request parameter information, and service call combinations with data dependencies are determined from them; service call sequences occurring consecutively within a preset time window are identified based on time information in call performance information to obtain time-series associated service call sequences; call chains spanning a preset number of services are identified based on the call chain length in the service topology to obtain long call chains; at least one of the service call combinations with data dependencies, time-series associated service call sequences, and long call chains is determined as the target call pattern.

[0041] In this embodiment, a service topology representing the call relationships between services is constructed based on the parsed service identity information and interface identifier information. This service topology is a directed graph structure, where nodes represent individual services, edges represent the call relationships between services, and each edge includes statistical information such as call frequency, average response latency, and error rate. The system continuously updates the service topology in real time using continuously received traffic characteristic data, dynamically identifying core hub services (such as services with high in-degree or high out-degree) and high-frequency critical call paths, laying the foundation for subsequent analysis.

[0042] Secondly, the system identifies service call combinations with data dependencies based on the business entity identifiers in the request parameters. Business entity identifiers include unique identity information such as user ID, order ID, and product ID, used to uniquely identify the object of the business operation. The system aggregates multiple service calls operating on the same business entity identifier in chronological order to obtain a call sequence reflecting the complete business operation process, such as a call sequence for creating an order, paying for an order, and querying an order, occurring sequentially for the same order ID. From these call sequences, call combinations with strong data dependencies are identified—that is, sets of operations that must be completed consecutively on the same business entity. If a failure occurs in any part of these combinations, the business process will be directly interrupted.

[0043] Based on the timestamps in the call performance information, we identify time-sensitive call sequences that occur consecutively within a preset time window. Analyzing the time intervals between calls reveals closely related call patterns within specific time windows. For example, the recommendation service is inevitably called within 3 seconds of a user logging in, or the inventory service is called within 500 milliseconds after an order is placed. These time-sensitive call sequences reflect the real-time requirements of the business process; a delay or error in any service within the sequence will directly impact the user experience or cause the entire business process to fail.

[0044] Based on the call chain length in the service topology, call chains that span a preset number of microservices are identified as long call chain patterns. The service topology is traversed, and the number of service nodes traversed by each complete call chain is counted. Call chains exceeding a preset threshold (e.g., 5 services) are marked as long call chains. These long chains often correspond to complex business scenarios. For example, an e-commerce order process may sequentially call multiple services such as the gateway, order service, inventory service, payment service, and points service. Failure of any intermediate node may cause the entire request to fail.

[0045] At least one of the following identified service call combinations with data dependencies, time-series related service call sequences, and long call chains is determined as the target call pattern. This pattern is then associated with and stored in conjunction with the corresponding service topology context and traffic characteristic snapshots for use by the subsequent chaos experiment test case generation module. These target call patterns accurately reflect real, critical business processes in the production environment, providing precise test targets for chaos engineering experiments and avoiding the problem of experimental scenarios being out of sync with real traffic caused by traditional methods relying on guesswork based on human experience.

[0046] S14. Generate chaotic experimental cases based on the target call pattern. The chaotic experimental cases are experimental configurations used to describe fault injection for the target service.

[0047] In this embodiment, the target of fault injection is determined based on the type of the target call pattern. If the target call pattern is a combination of service calls with data dependencies, the key service node in the combination is used as the injection target; if it is a time-related service call sequence, the downstream service in the sequence is used as the injection target; if it is a long call chain, the intermediate service node in the chain is used as the injection target.

[0048] The fault injection method is matched based on the characteristics of the target call pattern. For combinations of data dependencies, error injection (such as returning an HTTP 503 error) is matched; for time-series related sequences, delay injection (such as increasing response latency) is matched; for long call chains, node fault injection (such as simulating service instance termination) is matched. At the same time, the fault injection strength is dynamically determined based on the frequency of service calls and response latency in the target call pattern, with lower injection strength for higher frequencies.

[0049] Finally, the determined target service, fault injection method, injection strength, and business context (such as business entity identifier and time window) in the target call pattern are encapsulated to generate structured chaos test cases. Each test case should at least include: target service identifier, fault type, injection parameters, effective conditions, and impact scope constraints, and can be directly executed by the chaos engineering platform or service mesh.

[0050] In one possible implementation, chaotic experimental use cases are generated based on the target invocation pattern, including: Based on the target call pattern, determine the corresponding target service and the call relationship between services; based on the call relationship and service topology, determine the target location for fault injection, where the target location includes: service nodes in the service topology or call chains between services; based on the time-series associated service call sequence, determine the time location for fault injection; based on the target call pattern, match the corresponding fault injection method from the preset fault injection strategy; based on the target location, time location and fault injection method, generate the corresponding chaos experiment test cases.

[0051] In this embodiment, executable chaos experiment test cases are automatically generated based on the target invocation pattern, including determining the target location and time location of fault injection and matching the corresponding fault injection method, and finally encapsulating them into a complete experiment configuration.

[0052] First, based on the target call pattern, the service nodes and call connections between services are extracted to determine the corresponding target services and call relationships. Combining the call relationships and the connection structure between service nodes in the service topology, the target location for fault injection is located. The target location is the position of a service node or service in the call chain within the service topology. Further, based on the time information in the time-series service call sequence, the time location for fault injection is determined. The time location is the point in time or time interval in which the corresponding service call occurs. Simultaneously, based on the call relationship characteristics reflected in the target call pattern, a corresponding fault injection method is matched from a pre-configured fault injection strategy. The fault injection strategy is a set of rules pre-configured based on service call relationships or call chain structures to determine the type of fault injection and its mechanism of action. Finally, the target location, time location, and fault injection method are combined to generate corresponding chaos experiment cases.

[0053] In one possible implementation, chaotic experimental use cases are generated based on the target invocation pattern, including: From the preset fault injection strategies, determine the fault injection method that matches the target call pattern; for service call combinations with data dependencies, based on the fault injection method, generate experimental test cases to inject faults into the target service or call relationship in the call chain of the service call combination, and limit the scope of fault injection to requests with the same business entity identifier; for time-series related service call sequences, based on the fault injection method, generate experimental test cases to inject delays or errors into the target service in the sequence within a preset time window; for long call chains, based on the fault injection method, generate experimental test cases to inject faults into the target service in the long call chain; generate chaotic experimental test cases based on the generated experimental test cases.

[0054] In this embodiment, based on the type of the target call mode, the corresponding fault injection method is matched from the preset fault injection strategy library, and targeted chaos test cases are generated.

[0055] The system has a pre-defined fault injection strategy library, which defines the mapping relationship between various fault injection methods and call mode types. When a target call mode is identified, the system matches the corresponding fault injection method from the strategy library according to the type of the mode: for service call combinations with data dependencies, the system matches error injection methods; for time-related service call sequences, the system matches delayed injection methods; and for long call chains, the system matches node fault injection methods.

[0056] For service call combinations with data dependencies, experimental use cases are generated to inject faults at critical points in the call chain. Specifically, core service nodes or call relationships in the middle of the call combination are identified as injection points. For example, in a combination of creating an order, paying for an order, and querying an order, the payment order is selected as the critical step for fault injection. Simultaneously, the scope of fault injection is limited to requests with the same business entity identifier; that is, faults are only injected into requests operating on specific business entities such as the same order ID or user ID, achieving precise targeting and avoiding impact on other normal requests.

[0057] For time-series service call sequences, experimental test cases are generated to inject delays or errors within a preset time window. The system extracts timing features from the call sequence to determine the time window for fault injection. For example, in a sequence of "calling the recommendation service within 3 seconds of user login," the fault activation time is set to between 2 and 3 seconds after the user completes login. Downstream services in the sequence are selected as injection targets, such as the recommendation service, and the injection strength is dynamically determined based on the call frequency and response latency in the sequence, such as setting a delay of 500 milliseconds or an error rate of 5%.

[0058] For long call chains, experimental test cases are generated to inject faults into the target service within the chain. The sequence of service nodes traversed by the long call chain is identified from the service topology, and intermediate nodes in the chain are selected as fault injection points to test the chain's fault tolerance to single points of failure. Random node selection or specifying specific nodes (such as high-in-degree nodes) is supported for injection. Fault types include Pod termination, network latency, and resource exhaustion, ensuring comprehensive verification of the stability of long chains under different fault scenarios.

[0059] Finally, the experimental cases generated for various target invocation patterns will be uniformly encapsulated and the experimental configuration will be organized in a standard format (such as YAML or JSON). This will include at least the fields of experimental name, target service identifier, fault type, injection parameters (such as delay time, error code), effective conditions (such as time window, business entity identifier limitation), and impact scope constraints, generating complete and executable chaos experimental cases for subsequent isolation environment verification or direct execution in the production environment.

[0060] S15. Execute chaotic experimental cases in an isolated environment, and evaluate and filter the chaotic experimental cases based on the execution results.

[0061] In this embodiment, the system deploys an isolated sandbox environment. This sandbox environment maintains the same service topology as the production environment and deploys mirror versions of all services being tested, ensuring the authenticity of the experimental verification environment. The system automatically creates the sandbox using Infrastructure as Code (IaaS), including network configuration, service discovery, and monitoring components, enabling one-click setup and destruction of the environment.

[0062] The system automatically executes chaos experiment test cases in a sandbox environment. Each test case injects faults according to the configured fault type, injection location, injection strength, and time window through the fault injection interface of a chaos engineering platform (such as Chaos Mesh) or service mesh. During execution, the system collects real-time call performance information of the target service, including response latency, error rate, call success rate, and correlation indicators of upstream and downstream services, forming a comprehensive monitoring data of the experiment execution.

[0063] The effectiveness and impact of experimental cases are evaluated based on the collected performance information. Effectiveness evaluation is based on the performance changes of the target service before and after fault injection, determining whether the experiment triggered the expected anomalies. For example, does delayed injection significantly increase response latency, or does fault injection significantly decrease call success rate? Impact evaluation is based on the scope of fault propagation within the service topology, analyzing whether the fault was isolated as expected or whether a cascading effect occurred. For example, does a downstream service failure lead to upstream service circuit breaking or cascading failure? The performance changes are compared with preset thresholds to generate evaluation results, including three types: effective, ineffective, or too high risk.

[0064] Experimental cases are screened based on the evaluation results. Cases deemed effective and with manageable impact are retained as valid test cases. Invalid test cases (those without significant impact) or those with excessively high risk (impact exceeding expectations) are recorded for their failure reasons and marked as requiring adjustment or being discarded. The screening results are also used to optimize the pre-set fault injection strategy. For example, the injection strength threshold is adjusted based on experimental effectiveness, and the explosion radius control rules are modified based on the impact area, forming a feedback loop and continuously improving the accuracy of subsequent test case generation.

[0065] In one possible implementation, chaotic experimental cases are executed in an isolated environment, and call performance information of the target service is collected during execution. The call performance information includes at least response latency, error rate, and call success rate. Based on the call performance information, the performance changes of the target service before and after fault injection are determined. Based on the comparison results of the performance changes with preset performance thresholds, the evaluation results of the chaotic experimental cases are determined. The chaotic experimental cases are screened according to the evaluation results, and the preset fault injection strategy is adjusted or updated based on the screening results for use in the generation of subsequent chaotic experimental cases.

[0066] In this embodiment, the chaotic experimental cases are loaded and executed in an isolated environment to inject faults into the target service. Simultaneously, the service call process is monitored during execution, and call performance information is collected. This performance information includes at least response latency, error rate, and call success rate, representing the time overhead of processing requests, the proportion of failed requests, and the proportion of successful request completions, respectively. After obtaining the call performance information, various performance indicators before and after fault injection are statistically analyzed according to a preset time range, and the performance changes of the target service before and after fault injection are determined through comparative analysis. The performance changes are compared with a preset performance threshold, which serves as a benchmark value for determining whether service performance is abnormal. The corresponding evaluation result is determined based on the comparison result. The chaotic experimental cases are then screened based on the evaluation result, retaining those that meet preset conditions. Finally, the preset fault injection strategy is adjusted or updated based on the screening results, optimizing the parameters or applicable conditions of relevant rules in the strategy for use in the generation of subsequent chaotic experimental cases.

[0067] As an example, a chaos experiment test case targeting a service is executed in an isolated environment, injecting a target proportion of latency faults into the service. During execution, the call performance information of the target service is continuously collected, including response latency, error rate, and call success rate. Assume that before fault injection, the average response latency of the target service is 100ms, the error rate is 1%, and the call success rate is 99%; after fault injection, the average response latency increases to 300ms, the error rate increases to 8%, and the call success rate decreases to 92%. The performance changes are compared with preset performance thresholds, such as a preset response latency threshold of 250ms, an error rate threshold of 5%, and a call success rate threshold of 95%. Since all indicators exceed or fall below the corresponding threshold range after fault injection, the evaluation result of this chaos experiment test case is determined to be significantly impacting service stability. Based on this evaluation result, this experiment test case can be identified as a valid test case and retained during the screening phase. Simultaneously, during the strategy adjustment phase, the injection intensity or impact scope of this type of fault injection method can be reduced to avoid excessive impact on the system, thus enabling its use in the generation and optimization of subsequent chaos experiment test cases.

[0068] This application analyzes mirrored traffic in the production environment, constructs service topology, and identifies target call patterns. Based on real call behavior, it automatically generates chaos experiment test cases, which are then executed, evaluated, and filtered in an isolated environment. This achieves an automated closed loop for chaos experiments, from test case generation to result feedback. Compared to manually constructing experiments, this significantly improves the relevance and accuracy of test cases. Furthermore, through precise fault injection scope control and strategy optimization mechanisms, it enhances experimental security, reduces the impact on the system, and effectively improves the verification efficiency of distributed system stability and fault tolerance.

[0069] Figure 2 This is a flowchart illustrating another method for generating chaotic experimental cases provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method specifically includes: The intelligent derivation system for chaos engineering test cases is deployed independently and works in collaboration with service meshes (such as Istio) and chaos engineering platforms (such as Chaos Mesh). It consists of three phases: traffic learning and analysis, intelligent derivation of test cases, and delivery and execution.

[0070] Phase 1: Traffic Learning and Analysis Real business traffic in the production environment first passes through the service mesh. The system, by configuring the VirtualService resource of the service mesh, mirrors a specified proportion (e.g., 1%) of the production traffic to the traffic receiving endpoint of the intelligent derivative system, achieving zero-intrusion traffic collection. The traffic collection and deep analysis module performs protocol-level deep analysis on the mirrored traffic, not only recording the complete content of the original requests and responses, but also extracting key information from the HTTP / gRPC protocol headers and payloads, including user identity (via JWT), API endpoints, call parameters (especially the ID identifying the business entity), response latency and status codes, service identity and topology (purpose: accurately locating injection points to ensure that chaotic faults can accurately hit the specific service instances through which the traffic passes), protocol and interface fingerprints (purpose: refining fault scenarios and allowing fault injection for specific interfaces), request and response content characteristics (purpose: constructing realistic test data for generating requests that conform to business logic, defining precise success assertions, implementing follower strategies, and achieving more complex and realistic business scenario testing), and timing and performance baselines (purpose: setting reasonable fault parameters, quantifying the impact of experiments, and serving as a benchmark for evaluating the effectiveness of experiments), generating structured traffic feature data.

[0071] Based on continuous traffic characteristic data, the intelligent analysis engine automatically draws and updates a dynamic service topology diagram between microservices in real time, calculating metrics such as call frequency, average latency, and error rate for each call edge, and identifying core hub services and high-frequency critical paths. Simultaneously, the engine applies machine learning models to identify fault-sensitive call patterns, including time-series sensitive patterns (such as order placement call chains that inevitably occur shortly after a user logs in), data-dependent patterns (such as continuous "create-query-pay" operations on the same order ID), and long-chain calls (complex business processes spanning multiple microservices), providing precise targets for subsequent experimental generation.

[0072] Phase Two: Intelligent Derivation of Experimental Cases The chaos experiment test case synthesizer, based on the output of the intelligent analysis engine, automatically generates specific fault injection experiment configurations using multiple built-in derivative strategies. The critical path attack strategy generates network latency or Pod termination experiments for critical services in the topology; the dependency brute-force strategy generates HTTP error injection or high-latency experiments for identified strong dependency calls; and the shadow-following strategy generates fault rules that only affect specific request flows containing that ID, targeting sensitive business entity IDs. The synthesizer automatically adds explosion radius constraints when generating experiments, using a strict label selector to limit the impact to the relevant services or versions identified in the analysis, ensuring the security of the experiments.

[0073] The generated candidate chaos test cases are deployed to the security sandbox verification module. This module maintains a sandbox cluster with the same topology as the production environment, deploys mirror versions of all services, automatically executes candidate test cases in the sandbox, monitors the success rate, latency, and other metrics of key services, and compares them with the steady-state baseline to automatically evaluate the effectiveness and potential impact of the experiments. Based on the pre-execution results, the system scores and filters the test cases. Experiments that can cause a significant drop in core business metrics but are successfully circuit-broken are rated as high-value test cases, while experiments that do not produce a significant impact are marked as low-risk or require adjustment.

[0074] The experimental cases are scored, with the following dimensions: effectiveness score (highest weight: whether the experiment triggered the expected, observable anomaly), and core metrics: increase in error rate, increase in latency, and decrease in throughput. The scoring formula is normalizable. The effectiveness score is calculated as follows: the effectiveness score is determined based on the increase in error rate, increase in latency, and decrease in throughput. Specifically, the changes in these three metrics before and after fault injection are normalized, mapped to the 0-1 interval, and then weighted and summed to obtain the effectiveness score. The higher the score, the more effectively the experiment triggers the expected system anomaly, and the higher the weight.

[0075] Impact Surface Score (a trade-off indicator: whether the damage caused by the experiment was controllable and met expectations), core indicators: contamination level and fault propagation depth, scoring formula: S = 1 - (contamination level * propagation depth factor). The impact surface score is calculated as follows: the impact surface score is determined based on contamination level and fault propagation depth, where contamination level is the proportion of requests affected by the fault out of all requests, and fault propagation depth is the number of levels the fault spreads along the service topology. The impact surface score is calculated using the formula S = 1 - (contamination level × propagation depth factor), where the propagation depth factor is a preset coefficient greater than 0 and less than 1. The lower the contamination level and propagation depth, the higher the impact surface score, indicating that the damage caused by the experiment was more controllable and met expectations.

[0076] The resource and efficiency score (based on the cost of experiment execution) has the following core metrics: experiment duration, sandbox resource cost, and automation execution success rate. The resource and efficiency score is calculated based on the experiment duration, sandbox resource cost, and automation execution success rate. Specifically, the experiment duration is calculated based on the difference between the start and end times. The system presets an optimal experiment duration range; the score decreases when the experiment duration exceeds this range. The durations of different experiments are normalized and mapped to the 0-1 range. The sandbox resource cost is calculated by combining the peak and average CPU utilization and memory usage during the experiment execution. The actual resource consumption is compared with a preset resource consumption threshold; lower resource consumption results in a higher resource score. The automation execution success rate is determined by the proportion of successful automation executions of this test case out of the total number of executions in historical records. A 100% success rate earns a full score of 1 point; the score decreases linearly with a lower success rate. The resource and efficiency score is obtained by normalizing each of these three metrics and then weighted and summing them.

[0077] The chaos experiment cases are screened based on the comprehensive score, which is a weighted sum of the effectiveness score, the impact score, and the resource and efficiency score. The comprehensive score is calculated as follows: Comprehensive Score = α × Effectiveness Score + β × Impact Score + γ × Resource and Efficiency Score, where α, β, and γ are preset weight coefficients and α > β > γ, indicating that the effectiveness score contributes the most to the comprehensive score, followed by the impact score, and the resource and efficiency score contributes the least. The higher the total score, the higher the value of the experiment case. Experiment cases with a total score exceeding a preset threshold are identified as approved chaos experiment cases.

[0078] Phase Three: Delivery and Execution Validated high-value chaos experiment use cases are stored in a chaos experiment asset repository. Each use case is associated with the original traffic characteristic snapshot that generated it, the service topology context, and a pre-execution validation report. The asset repository provides retrieval functionality, which can trigger a review and automatic update suggestion for associated experiment use cases when online traffic patterns or service architectures change. Finally, the system outputs a package of validated chaos experiment use cases, which is delivered to a chaos engine (such as Chaos Mesh) for execution in a production or near-production environment, forming a complete automated closed loop from traffic collection, analysis, derivation to validation and delivery.

[0079] This application fundamentally changes the design pattern of chaos experiments, shifting from relying on human experience and guesswork to being driven by real production traffic data. The generated test cases naturally match the most core and active business processes of the current system, significantly improving the relevance of the experiments and the probability of discovering deep-seated defects. Through a sandbox pre-validation mechanism, potentially ineffective (no impact) or poorly designed (uncontrollable impact) experimental schemes can be identified and filtered out before the experiments are deployed to a more production-like environment, reducing the blind spots and potential risks of the experiments. Automation replaces a large amount of repetitive and tedious experiment design and configuration work, enabling testing and operations teams to conduct chaos experiments continuously and efficiently. Simultaneously, the accumulation of experimental assets allows chaos engineering knowledge to be solidified, inherited, and evolved with system iterations, overcoming the pain point of easily lost experimental assets in traditional models. Deeply utilizing standard cloud-native components such as service mesh and chaos engineering platforms, without requiring any modification to business code, it achieves out-of-the-box intelligent enhancements, conforming to best practices of cloud-native architecture.

[0080] Figure 3 A schematic diagram of a chaos experiment test case generation device provided in an embodiment of the present invention includes: Module 31 is used to acquire mirror traffic in the production environment; The parsing module 32 is used to parse the mirrored traffic to obtain traffic feature data, which is data used to characterize the call relationship and call behavior between services; The construction module 33 is used to construct a service topology based on the traffic feature data, analyze the service topology, and identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their association relationship based on the traffic feature data. The generation module 34 is used to generate chaotic experimental cases based on the target call pattern. The chaotic experimental cases are experimental configurations used to describe fault injection for the target service. The processing module 35 is used to execute the chaos experiment cases in an isolated environment and evaluate and filter the chaos experiment cases based on the execution results.

[0081] In one possible implementation, the acquisition module is specifically used to copy a complete copy of real business traffic from the production environment according to a preset ratio through the traffic mirroring mechanism of the service mesh, as the initial mirrored traffic. The preset ratio is dynamically adjusted according to the load of the production environment and analysis needs. Perform a filtering operation on the initial mirror traffic to obtain the mirror traffic. The filtering operation includes at least one of the following: filtering, sampling, or removing invalid or duplicate requests from the initial mirror traffic.

[0082] In one possible implementation, the parsing module is specifically used to extract request and response data from service calls in the mirrored traffic; Based on the request data and the response data, identify the service identity information and the corresponding interface identification information of the service participating in the service call; Request parameter information for characterizing the called content is extracted from the request data, and response result information and call performance information for characterizing the call result are extracted from the response data. The traffic feature data is generated based on the service identity information, interface identification information, request parameter information, response result information, and call performance information.

[0083] In one possible implementation, the construction module is specifically used to construct a service topology representing the call relationship between services based on the service identity information and the interface identification information; Based on the request parameter information, identify service call sequences with the same business entity identifier, and determine service call combinations with data dependencies from them; Based on the time information in the call performance information, identify the service call sequence that occurs continuously within a preset time window to obtain the time-series associated service call sequence. Based on the call chain length in the service topology, call chains that span a preset number of services are identified to obtain long call chains; The target call pattern is determined by at least one of the service call combination with data dependencies, the service call sequence with time correlation, and the long call chain.

[0084] In one possible implementation, the generation module is specifically used to determine the corresponding target service and the calling relationship between services based on the target calling pattern; Based on the call relationship and the service topology, the target location for fault injection is determined, wherein the target location includes: service nodes in the service topology or call links between services; Based on the service call sequence associated with the time sequence, the time location of fault injection is determined; Based on the target invocation pattern, a corresponding fault injection method is matched from the preset fault injection strategy; Based on the target location, the time location, and the fault injection method, a corresponding chaotic experimental case is generated.

[0085] In one possible implementation, the generation module is specifically used to determine a fault injection method that matches the target call pattern from a preset fault injection strategy; For the service call combination with data dependency, based on the fault injection method, experimental cases are generated to inject faults into the target service or call relationship in the call chain of the service call combination, and the scope of fault injection is limited to requests with the same business entity identifier. For the time-series associated service call sequence, based on the fault injection method, generate experimental test cases to inject delays or errors into the target service in the sequence within the preset time window; For the long call chain, based on the fault injection method, experimental test cases are generated to inject faults into the target service in the long call chain; Generate the chaos test cases based on the generated test cases.

[0086] In one possible implementation, the processing module is specifically used to execute the chaos experiment case in the isolated environment and collect the call performance information of the target service during the execution process. The call performance information includes at least response latency, error rate and call success rate. Based on the call performance information, determine the performance changes of the target service before and after fault injection; Based on the comparison between the performance changes and the preset performance threshold, the evaluation result of the chaos experiment case is determined. The chaotic experimental cases are screened based on the evaluation results, and the preset fault injection strategy is adjusted or updated based on the screening results for use in the generation of subsequent chaotic experimental cases.

[0087] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 4 The computer device 400 shown includes at least one processor 401, a memory 402, at least one network interface 404, and other user interfaces 403. The various components in the computer device 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to implement communication between these components. In addition to a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general designated all buses as Bus System 405.

[0088] The user interface 403 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0089] It is understood that the memory 402 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0090] In some implementations, memory 402 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 4021 and application program 4022.

[0091] The operating system 4021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 4022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 4022.

[0092] In this embodiment of the invention, by calling the program or instructions stored in the memory 402, specifically the program or instructions stored in the application program 4022, the processor 401 executes the method steps provided in each method embodiment, including, for example: Obtain mirrored traffic from the production environment; The mirrored traffic is parsed to obtain traffic feature data, which is used to characterize the call relationship and call behavior between services; A service topology is constructed based on the traffic feature data, and the service topology is analyzed to identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their associations, based on the traffic feature data. Chaos experiment test cases are generated based on the target call pattern. The chaos experiment test cases are experimental configurations used to describe fault injection for the target service. The chaotic experimental cases are executed in an isolated environment, and the chaotic experimental cases are evaluated and screened based on the execution results.

[0093] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 402. Processor 401 reads the information in memory 402 and, in conjunction with its hardware, completes the steps of the above method.

[0094] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0095] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0096] The computer device provided in this embodiment may be as follows: Figure 4 The device shown can perform, for example Figure 1 All steps of the method, thus achieving Figure 1 For details on the technical effects of the method shown, please refer to [link / reference]. Figure 1 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0097] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0098] When one or more programs in the storage medium can be executed by one or more processors to achieve the above-described method of execution on the device side.

[0099] The processor is used to execute a program stored in memory to implement the steps of the method performed on the device side: Obtain mirrored traffic from the production environment; The mirrored traffic is parsed to obtain traffic feature data, which is used to characterize the call relationship and call behavior between services; A service topology is constructed based on the traffic feature data, and the service topology is analyzed to identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their associations, based on the traffic feature data. Chaos experiment test cases are generated based on the target call pattern. The chaos experiment test cases are experimental configurations used to describe fault injection for the target service. The chaotic experimental cases are executed in an isolated environment, and the chaotic experimental cases are evaluated and screened based on the execution results.

[0100] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0101] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0102] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating chaotic experimental cases, characterized in that, include: Obtain mirrored traffic from the production environment; The mirrored traffic is parsed to obtain traffic feature data, which is used to characterize the call relationship and call behavior between services; A service topology is constructed based on the traffic feature data, and the service topology is analyzed to identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their associations, based on the traffic feature data. Chaos experiment test cases are generated based on the target call pattern. The chaos experiment test cases are experimental configurations used to describe fault injection for the target service. The chaotic experimental cases are executed in an isolated environment, and the chaotic experimental cases are evaluated and screened based on the execution results.

2. The method according to claim 1, characterized in that, The acquisition of mirrored traffic in the production environment includes: Through the service mesh's traffic mirroring mechanism, a complete copy of the real business traffic is replicated from the production environment according to a preset ratio as the initial mirrored traffic. The preset ratio is dynamically adjusted according to the load of the production environment and analysis needs. Perform a filtering operation on the initial mirror traffic to obtain the mirror traffic. The filtering operation includes at least one of the following: filtering, sampling, or removing invalid or duplicate requests from the initial mirror traffic.

3. The method according to claim 1, characterized in that, The step of parsing the mirrored traffic to obtain traffic feature data includes: Extract request and response data from the service calls within the mirrored traffic; Based on the request data and the response data, identify the service identity information and the corresponding interface identification information of the service participating in the service call; Request parameter information for characterizing the called content is extracted from the request data, and response result information and call performance information for characterizing the call result are extracted from the response data. The traffic feature data is generated based on the service identity information, interface identification information, request parameter information, response result information, and call performance information.

4. The method according to claim 3, characterized in that, The process of constructing a service topology based on the traffic characteristic data and analyzing the service topology to identify target call patterns includes: Based on the service identity information and the interface identification information, a service topology representing the call relationship between services is constructed; Based on the request parameter information, identify service call sequences with the same business entity identifier, and determine service call combinations with data dependencies from them; Based on the time information in the call performance information, identify the service call sequence that occurs continuously within a preset time window to obtain the time-series associated service call sequence. Based on the call chain length in the service topology, call chains that span a preset number of services are identified to obtain long call chains; The target call pattern is determined by at least one of the service call combination with data dependencies, the service call sequence with time correlation, and the long call chain.

5. The method according to claim 4, characterized in that, The generation of chaotic experimental cases based on the target invocation pattern includes: Based on the target invocation pattern, determine the corresponding target service and the invocation relationship between the services; Based on the call relationship and the service topology, the target location for fault injection is determined, wherein the target location includes: service nodes in the service topology or call links between services; Based on the service call sequence associated with the time sequence, the time location of fault injection is determined; Based on the target invocation pattern, a corresponding fault injection method is matched from the preset fault injection strategy; Based on the target location, the time location, and the fault injection method, a corresponding chaotic experimental case is generated.

6. The method according to claim 5, characterized in that, The generation of chaotic experimental cases based on the target invocation pattern includes: Determine the fault injection method that matches the target invocation pattern from the preset fault injection strategies; For the service call combination with data dependency, based on the fault injection method, experimental cases are generated to inject faults into the target service or call relationship in the call chain of the service call combination, and the scope of fault injection is limited to requests with the same business entity identifier. For the time-series associated service call sequence, based on the fault injection method, generate experimental test cases to inject delays or errors into the target service in the sequence within the preset time window; For the long call chain, based on the fault injection method, experimental test cases are generated to inject faults into the target service in the long call chain; Generate the chaos test cases based on the generated test cases.

7. The method according to claim 1, characterized in that, The process of executing the chaos experiment cases in an isolated environment, and evaluating and filtering the chaos experiment cases based on the execution results, includes: The chaos experiment case is executed in the isolated environment, and the call performance information of the target service is collected during the execution. The call performance information includes at least response latency, error rate and call success rate. Based on the call performance information, determine the performance changes of the target service before and after fault injection; Based on the comparison between the performance changes and the preset performance threshold, the evaluation result of the chaos experiment case is determined. The chaotic experimental cases are screened based on the evaluation results, and the preset fault injection strategy is adjusted or updated based on the screening results for use in the generation of subsequent chaotic experimental cases.

8. A chaotic experimental case generation device, characterized in that, include: The acquisition module is used to acquire mirror traffic in the production environment; The parsing module is used to parse the mirrored traffic to obtain traffic feature data, which is data used to characterize the call relationship and call behavior between services; The construction module is used to construct a service topology based on the traffic feature data, analyze the service topology, and identify the target call pattern. The target call pattern is a call behavior feature formed by multiple service calls according to their associations, based on the traffic feature data. A generation module is used to generate chaotic experimental cases based on the target call pattern. The chaotic experimental cases are experimental configurations used to describe fault injection for the target service. The processing module is used to execute the chaos experiment cases in an isolated environment and evaluate and filter the chaos experiment cases based on the execution results.

9. A computer device, characterized in that, include: A processor and a memory, the processor being configured to execute a chaos experiment test case generation program stored in the memory to implement the chaos experiment test case generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the chaotic experimental case generation method according to any one of claims 1 to 7.

Images