Istio and link tracking-based microservice chaos testing method and device

Abstract

This invention discloses a microservice chaos testing method and apparatus based on Istio and distributed tracing, relating to the field of cloud-native technology. The method includes: Step S1, submitting a test task; Step S2, initializing the test environment; Step S3, tracing the microservice-level call chain of this request using the distributed tracing function, and generating a request tracing tree by combining the request definition and the microservice-level call chain; Step S4, implementing a bottom-up testing strategy: starting from the leaf node of the request tracing tree, implementing test cases from bottom to top. The execution of a single test case includes injecting candidate faults, initiating a specified request, and triggering the Envoy filter to obtain a preset response; Step S5, generating the final test result. The generation and execution of test cases in this invention are both automatically completed by the system, greatly reducing the complexity and labor intensity of manual operations, and achieving efficient tracing data collection and processing.

Description

Technical Field

[0001] This invention belongs to the field of cloud-native technology, specifically relating to a microservice chaos testing method and apparatus based on Istio and tracing. Background Technology

[0002] With the development of cloud computing and container technologies, microservice architecture has gradually become the mainstream design for modern distributed systems. It breaks down a single application into multiple independent services, each responsible for specific business functions, and can be developed, deployed, and scaled independently. Compared to traditional monolithic architectures, microservice architectures offer advantages in scalability, development and deployment efficiency, maintainability, and manageability. However, systems based on microservice architectures are typically more complex, and a single failure can affect multiple aspects of the system. Therefore, for testers, identifying and resolving these complex failures is crucial to improving the resilience of microservice architecture systems.

[0003] Kubernetes (K8s) is an open-source container orchestration platform designed to automate the deployment, scaling, and management of applications. It makes running containerized applications in large-scale cluster environments more efficient and reliable by providing powerful features such as automated deployment, elastic scaling, service discovery, load balancing, and self-healing capabilities. Kubernetes is widely used in modern cloud computing and microservice architectures, becoming the standard solution for managing containerized applications.

[0004] Istio is an open-source service mesh solution designed for microservice architectures, providing features such as traffic management, security, observability, and policy control. By deploying an agent (Envoy) alongside each service instance, Istio enables inter-service communication, load balancing, authentication and authorization, monitoring, and fault injection, thereby enhancing system resilience, reliability, and security, and making the management and operation of distributed microservices more efficient and controllable.

[0005] Link tracing is a method for monitoring and debugging distributed systems. It generates a complete request path by recording and tracing each request through various services and components in the system. It helps developers and operations personnel understand the internal workings of the system, identify performance bottlenecks, and locate problems, thereby improving the system's observability and reliability.

[0006] Chaos engineering is used to simulate various failure scenarios that may occur in a real-world production environment, triggering potential problems the system might encounter when facing these failures. Based on the triggered problems, relevant personnel propose solutions, thereby improving the system's overall ability to cope with risks. However, manually conducting chaos engineering experiments is inefficient and requires significant manpower; random fault injection suffers from duplication and omissions, resulting in low execution efficiency and insufficient coverage of the test space. Testing in isolated environments requires substantial resources, and for organizations and enterprises with high information security requirements (such as those in finance, military, technology, and power industries), deploying a dedicated system for testing in other environments is virtually impossible. Existing microservice chaos testing tools include:

[0007] Chaos Mesh is a typical chaos engineering platform suitable for fault injection in Kubernetes environments. However, each fault injection requires manual specification of the injection target and configuration of fault parameters. After injection, a test request must be manually issued, and the effectiveness of the fault injection must be manually checked to ensure it meets expectations.

[0008] Filibuster, a representative HTTP layer fault injection tool, can inject pre-configured faults into downstream requests and responses at the network layer after issuing a test request. It automatically detects dependencies between microservices through request tracing and generates the necessary fault combinations, avoiding unnecessary fault injection and automatically checking whether the test results meet expectations. However, this method requires modification of the source code of the microservice under test to implement request tracing and fault injection.

[0009] Random fault injection tool: Chaos Monkey is a typical example of a random fault injection tool, supporting the periodic random injection of crash faults in production environments. While it can effectively discover system problems, it also impacts real users and makes it difficult to quantify test progress.

[0010] Microservice chaos testing tools based on automatic dependency discovery and proxy: These tools are highly dependent on resources, making them unsuitable for experimentation in production environments and difficult to apply and promote in practice.

[0011] Therefore, research on microservice chaos testing methods based on Istio and link tracing is of great significance. Summary of the Invention

[0012] To address the aforementioned technical issues, this invention provides a microservice chaos testing method and apparatus based on Istio and distributed tracing. By utilizing Envoy to intercept requests, request definitions are obtained and preset responses are returned. Dependent services do not need to actually process these requests; by directly returning preset responses, various service response scenarios can be quickly simulated, accelerating the testing process. Distributed tracing is used to obtain the relevant microservice-level links, and a request tracing tree corresponding to microservices and requests is generated by combining the request definitions intercepted by Envoy. Subsequently, a bottom-up testing strategy is implemented to comprehensively cover the test space.

[0013] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0014] On one hand, this invention provides a microservice chaos testing method based on Istio and tracing, the method comprising:

[0015] Step S1, Submit the test task: The test task includes the namespace where the system under test is located, the request definition to trigger the test link, and the fault candidate list for each microservice.

[0016] Step S2: Initialize the test environment: Ensure that Istio is installed and Sidecar mode is enabled on the system under test, and that the link tracing function has been integrated;

[0017] Step S3: Generate a request tracing tree: Initiate a request by triggering the request definition of the link under test after processing, use Envoy to filter out the request definition that passes through each microservice, and use the link tracing function to trace the microservice-level call link of this request, and generate a request tracing tree by combining the request definition and the microservice-level call link.

[0018] Step S4: Implement a bottom-up testing strategy: Start from the leaf node of the request tracing tree and implement test cases from the bottom up. The execution of a single test case includes injecting candidate faults, initiating a specified request, and triggering the Envoy filter to obtain a preset response.

[0019] Step S5: Generate final test results: By collecting information related to requests, preset responses, microservice calls, and injected fault types, generate test results in a specified format.

[0020] On the other hand, the present invention provides a microservice chaos testing device based on Istio and link tracing, the device comprising:

[0021] The Trace processor is used to mark and initiate requests to the link under test. It generates a request trace tree based on the link trace data of this request and the request definitions of each request filtered by the Envoy filter, and triggers the test case generator.

[0022] The test case generator is used to generate test cases based on the leaf nodes of the request tracing tree generated by the Trace processor and the list of fault candidates corresponding to each service submitted by the front end.

[0023] The Envoy filter is used to process requests to downstream services triggered by requests marked by the Trace processor into a specified format of request definition, and to trigger the generation of response filters or return preset responses based on existing response filters for specified requests initiated by the test case executor.

[0024] The test case executor is used to inject the fault specified in the test case, initiate the specified request, and trigger the Envoy filter. If the specified request has a corresponding response, a preset response will be obtained. Otherwise, the result returned by the service under test and the information of this test case will be submitted to the Envoy filter to generate a response filter.

[0025] The test result processor is used to generate test results in a specified format based on the test process data generated by the test case executor.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] 1. Low labor costs: This invention significantly reduces the need for manual intervention through an automated testing process. Utilizing Istio's automatic sidecar injection and link tracing tools, it achieves automatic identification and tracing of test links. The generation and execution of test cases are all completed automatically by the system, greatly reducing the complexity and labor intensity of manual operations.

[0028] 2. High Execution Efficiency: A bottom-up testing strategy is adopted, improving test execution efficiency through parallel execution of test cases. The Envoy filter effectively intercepts and processes requests and responses, reducing unnecessary network overhead. Efficient tracing data collection and processing are achieved through dynamic injection of the OpenTelemetry Agent and configuration of the OpenTelemetry Collector.

[0029] 3. Wide range of applications: This invention supports testing in production environments and is suitable for microservice systems of various sizes. Whether it's a simple, small-scale microservice architecture or a complex, large-scale distributed system, the method and apparatus of this invention can be effectively applied.

[0030] 4. Low resource consumption: Using Envoy to intercept and process requests and responses reduces the impact on system performance. The application of sampling strategies and preset response mechanisms further reduces the demand for computing and storage resources during testing. Attached Figure Description

[0031] Figure 1 This is a flowchart of a microservice chaos testing method based on Istio and tracing according to the present invention.

[0032] Figure 2 A flowchart of the request tracing tree stages is generated for this invention;

[0033] Figure 3 This is a flowchart illustrating the bottom-up testing strategy implemented in this invention.

[0034] Figure 4 This is a structural block diagram of a microservice chaos testing device based on Istio and link tracing according to the present invention. Detailed Implementation

[0035] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely specific embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention provides a microservice chaos testing method based on Istio and distributed tracing. The principle is as follows: by identifying and executing the initial request of the user-selected test path, the `request_handler` hook function (referred to as the request filter in this invention) in the Lua filter of Envoy, deployed by Istio's Sidecar mechanism, is triggered. The request filter extracts the request definitions (including request path, request method, request header, and request body) triggered by this request for each microservice. Simultaneously, the application service, dynamically injected with an OpenTelemetry Agent, is monitored by the tracing tool, which tracks the call chain of the target request. Based on the request call chain information and the request definitions of each microservice, a request tracing tree in a specified format is generated. The data format of this tracing tree is a tree structure, and each node represents a request triggered on this chain. Next, test case generation is performed through the request tracing tree, creating a coroutine for all nodes in the request tracing tree. In each coroutine, it is determined whether the microservice under test has downstream dependent microservices. If there are downstream dependent microservices, it's necessary to determine if the test cases for those microservices have been completed. If so, test case generation can be performed on the current microservice. Test case generation involves three steps: selecting the request for the microservice under test, selecting the fault to be injected, and selecting test cases for the downstream dependent microservices. Each generated test case is immediately executed by the test case executor. If the microservice under test has no downstream dependent microservices, test case generation is performed directly on the current microservice. During test case execution, fault objects that can be injected by the Chaos Mesh client are generated using the specified candidate fault definitions, and the fault injection operation is performed through the Chaos Mesh client. After the request receives a response, the fault is restored. Then, the selected request is appended with an execution status-specific identifier and executed again. The response filter intercepts specific requests and returns a preset response to the target service. Through a bottom-up execution strategy, the test task is completed until all test cases at the root node are executed. Finally, combining the test cases and the obtained responses, the test result processor processes the data into a specified format and returns it to the front end for display.

[0037] Example 1

[0038] Combination Figure 1 The method described in this invention will be explained in detail.

[0039] Step S1: Submit the test task. The test task includes the namespace of the system under test, the request definition to trigger the test link, and the list of fault candidates for each microservice.

[0040] Step S2: Initialize the test environment: Ensure that Istio is installed and Sidecar mode is enabled on the system under test, and that the link tracing function has been integrated.

[0041] Step S3: Generate a request tracing tree: Initiate a request using the processed request definition that triggers the tested link. Use Envoy to filter out the request definitions that pass through each microservice, and use a tracing tool to trace the microservice-level call chain of this request. Combine the request definitions and the microservice-level call chain to generate a request tracing tree.

[0042] Step S4: Implement a bottom-up testing strategy: Start testing from the leaf nodes of the request tracing tree and proceed from the bottom up. Initiate a request stored on the service node. If the service node does not initiate a request to a downstream service, its response is used as a preset response for the next identical request to be intercepted and returned by Envoy. If the service node initiates a request to a downstream service, the request will be intercepted by Envoy and a preset response will be returned. The execution of a single test case includes four steps: injecting a candidate fault, initiating a specified request, obtaining a preset response, and setting a response filter for this test case.

[0043] Step S5: Generate final test results: By collecting requests, responses, microservices, injected fault types, and downstream dependency fault types, generate the final test results and provide them to the front end for display.

[0044] In one embodiment, step S1 includes:

[0045] Step S11: Select the namespace where the system under test is located, where the namespace refers to the namespace in Kubernetes;

[0046] Step S12: Select the request definition that triggers the test link. The test link is a series of request links triggered by the requests corresponding to the functional points displayed by the front end. For example, when the functional point displayed by the front end is the order placement function, the series of business processes it can trigger can be simply described as follows: the front end service first requests the user service to obtain user verification information, then calls the payment service based on the verification information. If the payment is successful, the shopping cart service will be triggered to clear the shopping cart, and the inventory service will also reduce the corresponding item quantity, and the transportation service will start generating transportation information. This request link can be described as: Front end service -> User service -> Payment service, Front end service -> Shopping cart service, Front end service -> Transportation service, Front end service -> Inventory service, where "A -> B" indicates that A initiates a request to service B. In this invention, the request definition is considered to be the definition of an HTTP request, including the request path, request method, request header, and request body.

[0047] Step S13: Configure the fault candidate list for each microservice and configure the chaotic faults to be injected into each microservice. The chaotic faults are applied to the container of the application service, such as network faults, CPU pressure faults, memory pressure faults, I / O faults, etc.

[0048] In one embodiment, step S2 above, initializing the test environment, involves ensuring that Istio is installed and Sidecar mode is enabled on the system under test. Simultaneously, the OpenTelemetry Agent is dynamically injected into all applications on the system, and the corresponding OpenTelemetry Collector is configured. Specifically, this includes:

[0049] Step S21: Check whether Istio is installed and whether the system under test has enabled Sidecar mode. Istio and Sidecar mode are prerequisites for ensuring that each application service is equipped with Envoy, and Envoy is a prerequisite for ensuring that the request filtering module and response filtering module are effective. If Istio is not installed, you need to exit the current process and install Istio.

[0050] Step S22: Check if the system has integrated the tracing function. If the tracing function is not integrated, the following steps can be performed to implement the tracing function in a non-intrusive way. First, the OpenTelemetry Agent needs to be dynamically injected into the runtime of all application containers in the system. The method used is to inject the prepared OpenTelemetry Agent file into the runtime of the application container using the jattach tool (this invention currently only considers the JVM runtime). Then, the OpenTelemetry Collector is configured, and the tracing service is configured in the Collector. In this invention, jaeger is used as an example.

[0051] In one embodiment, combined Figure 2 Step S3 is described below, and specifically includes:

[0052] Step S31: Process the request definition and initiate the request: Add a unique identifier to the request header of the request definition to distinguish the Trace triggered by this request from a large number of Traces, and then initiate this request. Record the request initiation time and response time as the time interval for obtaining the Trace;

[0053] Step S32: Filtering request definitions for microservices: All request definitions for each microservice can be filtered out using the configured Envoy Lua Filter script and a custom log handler.

[0054] Step S33: Obtain the request call chain and generate the request call tree: Trace the microservice call chain of this request using a tracing tool and generate the request call tree by combining the request definition data with the request call chain;

[0055] In one embodiment, step S32 specifically includes:

[0056] Step S321, Configure the request filter: This mainly configures the namespace where the test system resides, the `envoy_on_request` hook function, and the byte stream parsing function. Detailed explanations are as follows:

[0057] The envoy_on_request hook function is defined to handle related operations when a request is received; obtain the request method and request path; collect all request headers; and output basic information such as the request URL, method, path, and request headers.

[0058] Retrieve Request Body: Ensure the request body is retrieved, even if it is empty; if the request method is POST and the request body is not empty, read and convert the request body into a string; parse the JSON data and record the parsing result or error log.

[0059] In this configuration, key components include using the `envoy_on_request` hook to extract and log basic request information (such as request method, path, headers, and body), and using a byte stream parsing function (pcall) to convert the request body from a byte stream into a readable string and attempt to parse the JSON data. If parsing succeeds, the parsed JSON data is logged; if parsing fails, an error log is logged.

[0060] The above configuration ensures that the basic information of the request is effectively extracted and processed in the envoy_on_request hook function, thereby providing the necessary data support for subsequent chaos testing and tracing.

[0061] Step S322: Parse Envoy logs to obtain request definition data for each microservice: Based on the service name, retrieve the service logs and filter out the target logs. Specifically:

[0062] Retrieve service logs based on service name and filter out target logs. Specifically:

[0063] If this request triggers a request to the user service, the user service's Envoy log will contain output information about the request definition. First, you need to raise the service's Envoy log level to view Info level logs. For example, you can use the following command to raise the log output level:

[0064] kubectl exec user-5d7b595bcd-w85vh -c istio-proxy -n sock-shop --curl -X POST http: / / localhost:15000 / logging?level=debug

[0065] This command can be used to increase the log output level. Then, use the following command to filter the output for each part of the request definition belonging to this call chain:

[0066] kubectl logs -f user-5d7b595bcd-w85vh -c istio-proxy -n sock-shop | grep "info" | grep ${traceId}

[0067] Finally, the corresponding request definition data is parsed out using custom regular expression logic.

[0068] In one embodiment, step S33 specifically includes:

[0069] Step S331: Obtain all trace data within a specified time interval by recording the request initiation time and response time, as well as the names of the main services involved. Then, filter out the target trace data based on a specific identifier. The specific steps are as follows:

[0070] In this invention, all triggered requests are issued by the frontend; therefore, the frontend service name can be used as a parameter for querying the trace. The trace is obtained using the API provided by the tracing tool. For example, all traces within the last 5 minutes can be obtained via the following URL:

[0071] http: / / localhost:8083 / jaeger / api / traces?end=1686837600000000&limit=10&lookback=5m&service=front-end&start=1686837300000000

[0072] Then, the target trace, i.e. the request call chain, is filtered out from the spans of all front-end services based on a specific identifier.

[0073] S332. Generate a request call tree by combining request definition data and trace: Generate a request call tree structure based on the obtained trace. Specifically: Obtain the service names in all spans and remove duplicates to determine the nodes of the request call tree. Organize the relationships between the nodes using the parentId in the span to construct the edges of the request call tree. Combine the request definition data with each node to complete the generation of the request call tree.

[0074] In one embodiment, combined Figure 3 Step S4 will be described in detail below. Specifically, step S4 includes:

[0075] Step S41: Traverse all nodes in the request call tree and generate a coroutine for each node: Traverse all nodes in the request call tree and generate a coroutine for each node. Each coroutine is responsible for handling the microservice corresponding to a node, ensuring parallel execution of test cases to improve testing efficiency. This process utilizes the concurrency characteristics of coroutines, allowing multiple nodes to process simultaneously, thereby speeding up the testing process.

[0076] Step S42: Monitor each node and determine whether the test can be executed based on the node's status. After generating the coroutine, determine whether the test can be executed based on the status of each node. Specific judgment criteria include: determining whether it is a leaf node; if it is a leaf node, immediately generate test cases; if it is not a leaf node, it is necessary to determine whether the test tasks of its downstream dependent services have been completed; if the test tasks of all downstream services are completed, then test cases can be generated for the currently tested service.

[0077] Step S43: Generate test cases for the microservices corresponding to the nodes: Each test case should include the ID of the service under test, the request definition, a combination of test cases for downstream dependent services (select one test case for each downstream dependent service), and the fault injection type selected from the submitted test tasks. At the same time, each microservice's test case should include a case where no fault is injected.

[0078] Step S44: Sequentially execute test cases: Based on the selected fault definition, inject faults using the Chaos Mesh client, initiate generated test requests to the microservice, simulate real request scenarios, return preset responses using response filters, record test results, and set corresponding response filter configurations to set the preset response for the current request.

[0079] In one embodiment, step S43 specifically includes:

[0080] Step S431: Select Request Definition: Based on the request definition list of the node, determine the specific request in the test case, including the request path, method, header information and request body, to ensure that the test case can accurately reflect the real request behavior of the microservice.

[0081] Step S432: Select a test case combination for downstream dependent services: Select a corresponding test case combination for each downstream dependent service to ensure that the current service's calls to downstream dependent services are fully considered during testing, thereby guaranteeing the completeness and consistency of the test. For example, when testing the Order Service, if the Order Service depends on the User Service and the Inventory Service, test cases need to be selected separately for the User Service and the Inventory Service: User Service Test Case: Select a test case that simulates the response of the User Service under normal and network interruption failure conditions, such as scenarios where user information retrieval is successful or unsuccessful. Normal situation: The User Service returns user information normally. Network interruption failure situation: The User Service is unavailable and returns an error message. Inventory Service Test Case: Select a test case that simulates the response of the Inventory Service under normal and service crash failure conditions, such as scenarios where inventory query is successful or unsuccessful. Normal situation: The Inventory Service returns inventory information normally. Service crash failure situation: The Inventory Service is unavailable and returns an error message. In this way, when testing the Order Service, its dependencies on the User Service and the Inventory Service can be fully covered.

[0082] Step S433: Select the fault injection type: Select the fault type to be injected from the submitted test task, clarify the fault scenarios in the test cases, and ensure that each test case includes a case without injected faults, because a case without faults is also a scenario.

[0083] In one embodiment, step S44 above includes:

[0084] Step S441: Inject faults using the Chaos Mesh client based on the selected fault definition. The fault definition is a type of fault data that the Chaos Mesh client can recognize, such as: Network latency: Simulate network latency faults by introducing latency on a specified network connection to test the performance and response of microservices under unstable network conditions. For example, introduce a 500ms latency into the network connection between the Order Service and the User Service.

[0085] Step S442: Send the generated test request to the microservice according to the request definition selected by the test case. Before sending the request, you need to add a specific identifier for the execution status generated based on the microservice ID, fault injection type, and request path to the request header, such as "001-user-service-cpu-overload- / api / v1 / users".

[0086] Step S443: A request to the service under test triggers a request to the downstream service, and the response filter returns a preset response: If the service under test has downstream dependent services, then the request will inevitably trigger the response filter. The response filter will return a specified response content, which is a preset response under a specific request, a specific combination of downstream dependent service test cases, and a specific failure condition. The specific operation is as follows:

[0087] The response filter is implemented based on the `envoy_on_response` hook function of Envoy's Lua filter. Each response filter corresponds one-to-one with a request. In the `envoy_on_response` hook function, it first checks whether the request path corresponding to the current response matches the specified target request path and test case ID. If the request path matches the target path and the test case ID matches the target ID, the corresponding response data is retrieved from the preset response content and returned to the client. The preset response content is generated based on a combination of specified requests, specified downstream dependent service test cases, and specified fault conditions, ensuring the accuracy and consistency of the tests.

[0088] Step S444: Recover from the fault after receiving the response: After each test case is completed, it is necessary to ensure that the fault is recovered immediately. This is to prevent the fault state of the current test case from affecting the execution of the next test case. The steps to recover from the fault include canceling the injected fault, resetting the system state, and ensuring that all services are restored to normal operating conditions.

[0089] Step S445: After receiving the response, a response filter for this test case needs to be generated: Based on a predefined response filter template, the filter is customized using string replacement. Replace ${namespace} with the current system namespace. Replace ${req_url} with the current request path. Replace ${testCase_ID} with the ID of the current test case. Replace ${respond_body} with the specific response content to be returned. Apply the customized response filter to the system to ensure that the preset content is returned in the response to the corresponding request. Finally, deploy it to the current Kubernetes environment using a Kubernetes client.

[0090] Example 2

[0091] like Figure 4 As shown, this embodiment of the invention provides a microservice chaos testing device based on Istio and link tracing, the device comprising:

[0092] A microservice chaos testing device based on Istio and distributed tracing, the device comprising:

[0093] The Trace processor is used to mark and initiate requests to the link under test. It generates a request trace tree based on the link trace data of this request and the request definitions of each request filtered by the Envoy filter, and triggers the test case generator.

[0094] The test case generator is used to generate test cases based on the leaf nodes of the request tracing tree generated by the Trace processor and the list of fault candidates corresponding to each service submitted by the front end.

[0095] The Envoy filter is used to process requests to downstream services triggered by requests marked by the Trace processor into a specified format of request definition, and to trigger the generation of response filters or return preset responses based on existing response filters for specified requests initiated by the test case executor.

[0096] The test case executor is used to inject the fault specified in the test case, initiate the specified request, and trigger the Envoy filter. If the specified request has a corresponding response, a preset response will be obtained. Otherwise, the result returned by the service under test and the information of this test case will be submitted to the Envoy filter to generate a response filter.

[0097] The test result processor is used to generate test results in a specified format based on the test process data generated by the test case executor.

[0098] The Trace processor includes a tested link processing module and a tracing data processing module.

[0099] Test case generator, which includes a test case generation module based on request tracing tree;

[0100] Envoy filters, which include request filtering modules and response filtering modules;

[0101] The test case executor includes a request processing module, a response filter processing module, and a fault injection module.

[0102] Specifically, the tested link processing module attaches a tracking status-specific identifier to the request definition selected by the front end, then initiates a specific network request according to the request category, and records the request initiation time and response time.

[0103] Specifically, the data processing module uses the API provided by the tracing tool to query all traces within the time interval recorded by the tested tracing module based on the request initiation time and response time. Since this invention supports implementation in a production environment, it is necessary to distinguish the target trace. This module distinguishes the target trace using a specific identifier specified by the tested tracing module, and generates a request tracing tree by combining the obtained trace with the request definitions of each microservice filtered by the request filter.

[0104] Specifically, the test case generation module based on the request tracing tree determines whether the microservice under test has downstream dependent microservices for operation. If there are downstream dependent microservices, it needs to determine whether the test cases for those downstream dependent microservices have been executed successfully; if so, test case generation can be performed on that microservice. If the microservice under test does not have downstream dependent microservices, test case generation can be performed directly on the microservice under test. Test case generation consists of three steps: selecting the request for the microservice under test, selecting the fault to be injected, and selecting the test cases for the downstream dependent microservices.

[0105] Specifically, the request filtering module uses the TraceId obtained by the Trace processor to filter out the various request definitions in the current request call chain, and parses out the request path, request method, request parameters and request headers.

[0106] Specifically, the response filtering module intercepts specified requests and returns a preset response to the target service.

[0107] Specifically, the request processing module retrieves the request definition of the currently tested microservice from the request tracing tree, appends an execution status-specific identifier, and then initiates the request.

[0108] Specifically, the response filter processing module, after receiving a response to a request, generates an execution status-specific identifier based on the microservice ID, fault injection type, and request path. It then generates a preset response for this test case based on the response, the specific identifier, and the response filter template resource. The preset response is replayed using the specified execution status identifier of the request; if no preset response exists, the microservice itself returns the response.

[0109] Specifically, the fault injection module generates fault objects that can be injected via the Chaos Mesh client by specifying candidate fault definitions, then performs the fault injection operation via the Chaos Mesh client, and recovers the fault after receiving a response to the request.

[0110] Specifically, the test result processor generates test results in a specified format based on the test process data generated by the test case executor.

[0111] Through the collaborative work of the above components, this invention provides an efficient and automated microservice chaos testing device that can effectively test the stability and robustness of a system in a production environment and provide detailed test results for analysis and optimization.

[0112] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A microservice chaos testing method based on Istio and distributed tracing, characterized in that, The method includes: Step S1, Submit the test task: The test task includes the namespace where the system under test is located, the request definition to trigger the test link, and the fault candidate list for each microservice. Step S2: Initialize the test environment: Ensure that Istio is installed and Sidecar mode is enabled on the system under test, and that the link tracing function has been integrated; Step S3: Generate a request tracing tree: Initiate a request by triggering the request definition of the link under test after processing, use Envoy to filter out the request definition that passes through each microservice, and use the link tracing function to trace the microservice-level call link of this request, and generate a request tracing tree by combining the request definition and the microservice-level call link. Step S4: Implement a bottom-up testing strategy: Start from the leaf nodes of the request tracing tree and implement test cases from the bottom up. The execution of a single test case includes injecting candidate faults, initiating a specified request, and triggering the Envoy filter to obtain a preset response; including: Step S41: Traverse all nodes of the request call tree, generate a coroutine for each node, and each coroutine is responsible for handling the microservice corresponding to a node. Step S42: Monitor each node and determine whether the test can be executed based on the node's status. If it is a leaf node, test cases can be generated. If it is not a leaf node, determine whether the test tasks of its downstream dependent services have been completed. If they have been completed, test cases can be generated. Step S43: Generate test cases for the microservices corresponding to the nodes. Each test case includes the ID of the service under test, the request definition, the test case combination of downstream dependent services, and the fault injection type selected from the submitted test tasks. Step S44: Based on the selected fault definition, inject faults using the Chaos Mesh client, initiate generated test cases to the microservice, simulate real request scenarios, and return preset responses using the response filter; Step S5: Generate final test results: By collecting information related to requests, preset responses, microservice calls, and injected fault types, generate test results in a specified format.

2. The microservice chaos testing method based on Istio and link tracing according to claim 1, characterized in that, Step S1 includes: Step S11: Select the namespace where the system under test is located; Step S12: Select the request definition that triggers the tested link. The tested link refers to a series of request links triggered by the request corresponding to the function point displayed by the front end. Step S13: Configure the fault candidate list for each microservice, including configuring the chaotic faults that need to be injected into each microservice.

3. The microservice chaos testing method based on Istio and link tracing according to claim 1, characterized in that, Step S2 includes: Step S21: Check whether Istio is installed on the system under test and whether Sidecar mode is enabled; Step S22: Check whether the system under test has integrated the link tracing function. If not, dynamically inject the OpenTelemetry Agent into the containers of all applications in the system under test, and configure the link tracing service in the OpenTelemetry Collector.

4. The microservice chaos testing method based on Istio and link tracing according to claim 1, characterized in that, Step S3 includes: Step S31: Process the request definition and initiate the request: Add a unique identifier to the request header of the request definition, initiate the request, and record the request initiation time and response time as the time interval for obtaining the Trace; Step S32: Filter the request definitions of microservices: Filter all request definitions for each microservice using the configured Envoy Lua Filter script and a custom log handler; Step S33: Obtain the request call chain and generate the request call tree: Trace the microservice call chain of the initiated request using a link tracing tool, and generate the request call tree by combining the request definition data with the microservice call chain of the request.

5. A microservice chaos testing device based on Istio and link tracing, applied to the method described in any one of claims 1-4, characterized in that, The device includes: The Trace processor is used to mark and initiate requests to the link under test. It generates a request trace tree based on the link trace data of this request and the request definitions of each request filtered by the Envoy filter, and triggers the test case generator. The test case generator is used to generate test cases based on the leaf nodes of the request tracing tree generated by the Trace processor and the list of fault candidates corresponding to each service submitted by the front end. The Envoy filter is used to process requests to downstream services triggered by requests marked by the Trace processor into a specified format of request definition, and to trigger the generation of response filters or return preset responses based on existing response filters for specified requests initiated by the test case executor. The test case executor is used to inject the fault specified in the test case, initiate the specified request, and trigger the Envoy filter. If the specified request has a corresponding response, a preset response will be obtained. Otherwise, the result returned by the service under test and the information of this test case will be submitted to the Envoy filter to generate a response filter. The test result processor is used to generate test results in a specified format based on the test process data generated by the test case executor.

6. A microservice chaos testing device based on Istio and link tracing according to claim 5, characterized in that, The Trace processor includes a tested link processing module and a trace data processing module; The test case generator includes a test case generation module based on a request tracing tree; The Envoy filter includes a request filtering module and a response filtering module; The test case executor includes a request processing module, a response filter processing module, and a fault injection module.

7. A microservice chaos testing device based on Istio and link tracing according to claim 6, characterized in that, The tested link processing module appends a specific identifier for the tracing status to the request definition selected by the front end, then initiates a network request according to the request category, and records the request initiation time and response time; the tracing data processing module, through the API provided by the link tracing tool, queries all traces within this time interval based on the request initiation time and response time recorded by the tested link processing module, and generates a request tracing tree.

8. A microservice chaos testing device based on Istio and link tracing according to claim 6, characterized in that, The test case generation module based on the request tracing tree determines whether the microservice under test has downstream dependent microservices for operation. If there are downstream dependent microservices, it needs to determine whether the test cases of the downstream dependent microservices have been executed. If the execution is complete, then perform test case generation for this microservice; If the microservice under test has no downstream dependent microservices, then the test case generation operation is performed directly on the microservice under test.

9. A microservice chaos testing device based on Istio and link tracing according to claim 6, characterized in that, The request filtering module uses the Trace processor to filter out each request definition in the request call chain and parses out the request path, request method, request parameters and request headers; The response filtering module intercepts specified requests and returns a preset response to the target service.

10. A microservice chaos testing device based on Istio and link tracing according to claim 6, characterized in that, The request processing module will obtain the request definition of the currently tested microservice from the request tracing tree, append an execution status-specific identifier, and then initiate the request. The response filter processing module generates a specific identifier for the execution status based on the microservice ID, fault injection type, and request path after receiving a response from the request. It then generates a preset response for this test case based on the response, the specific identifier, and the response filter template resource. The fault injection module generates a fault object for fault injection based on a specified candidate fault definition, and then performs the injection operation of the fault object through the Chaos Mesh client. After the request is responded to, the fault is restored.

Images