Distributed testing system and method based on micro-service architecture

Abstract

The invention discloses a distributed testing method and system based on micro-service architecture. The method comprises the steps of generating a test task and a fault injection task according to a test plan; distributing each test case to a plurality of test agents according to the test task, so that the plurality of test agents respectively execute the corresponding test cases, and the plurality of test agents and the plurality of micro-service instances of the tested system run on a plurality of service nodes in a one-to-one manner; injecting a corresponding fault in an execution environment of the tested system according to the fault injection task; and collecting and summarizing log data related to the test, and outputting a test result and the influence of the fault on the test result according to the log data. According to the embodiment of the invention, a complete test process from test plan generation to test result analysis of the tested system is supported, high cooperation of fault injection and test execution is realized through accurate control of fault injection, and meanwhile, the distributed test system constructed based on the micro-service architecture can be adaptively expanded according to the scale of the tested system.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a distributed testing system and method based on a microservices architecture. Background Art

[0002] With the wide application of the microservices architecture, software systems are becoming increasingly complex, and traditional testing systems face the following challenges: First, there is a lack of organization in each testing link. Software testing is a systematic process that involves many links such as requirements analysis, test plan writing, test case design, test environment setup, test execution, defect tracking and management. Currently, the testing system only supports a few of these links. Second, the ability to simulate faults is insufficient, mainly manifested in that the current testing system lacks precise control over faults and cannot simulate complex fault combination scenarios. The scalability is limited, mainly manifested in that it is difficult for the current testing system to scale as the system under test expands. Therefore, it is necessary to design a new type of distributed testing system to better support the testing requirements of software systems based on the microservices architecture. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present disclosure is to provide a distributed testing system and method based on a microservices architecture to solve the above problems.

[0004] In a first aspect, embodiments of the present disclosure provide a distributed testing method based on a microservices architecture, including:

[0005] Generating test tasks and fault injection tasks according to a test plan;

[0006] Allocating each test case to a plurality of test agents according to the test tasks, so that the plurality of test agents respectively execute the corresponding test cases, and the plurality of test agents and a plurality of microservice instances of the system under test run one-to-one on a plurality of service nodes;

[0007] Injecting corresponding faults into the execution environment of the system under test according to the fault injection tasks;

[0008] Collecting, summarizing and testing relevant log data, and outputting test results and the impact of faults on test results according to the log data.

[0009] In some embodiments, the test agent is further configured to: collect status data from the service node, adjust relevant service processes and microservice instances according to the status data, and recover relevant service processes and microservice instances when a fault occurs.

[0010] In some embodiments, the injection of corresponding faults includes: injecting network faults, injecting resource faults, and / or injecting application faults. The injected network faults include one or more of simulating network latency, packet loss, and disconnection. The injected resource faults include one or more of simulating high CPU occupancy, memory leak, and disk fullness. The injected application faults include one or more of simulating service crash and abnormal response.

[0011] In some embodiments, for the network faults, the TC command is used to simulate network latency, iptable is used to implement packet loss simulation, and a network proxy is used to simulate disconnection. For resource faults, the stress tool is used to simulate CPU stress, memory filling is used to simulate memory leak, and file I / O is used to simulate disk occupancy. For application fault simulation, at least one of process control, request interception, request response interception, and abnormal response is used to implement.

[0012] In some embodiments, the fault injection task is configured to adopt one of the following triggering methods: time trigger, condition trigger, and progressive trigger. The time trigger is to inject faults at a specified time point. The condition trigger is to inject faults when specific conditions are met. The progressive trigger is to gradually inject multiple faults according to a predetermined rule.

[0013] In some embodiments, it further includes: adjusting the test progress and test intensity according to the current state of the system under test, identifying and skipping invalid tests, and retrying failed tests.

[0014] In a second aspect, an embodiment of the present disclosure provides a distributed test system based on a microservice architecture, including:

[0015] Multiple test agents, running one-to-one with multiple microservice instances of the system under test on multiple service nodes;

[0016] A central control unit, configured to generate a test task and a fault injection task according to a test plan input by a user;

[0017] An automated execution engine, configured to allocate each test case to multiple test agents according to the test task, so that the multiple test agents respectively execute the corresponding test cases;

[0018] A log management module, configured to collect and manage log data fed back by the multiple test agents;

[0019] A fault simulator, configured to inject corresponding faults into the execution environment of the system under test according to the fault injection task;

[0020] The central control unit is further configured to analyze the test results and the impact of faults on the test results according to the log data.

[0021] In some embodiments, the test agent includes:

[0022] A data collection module for collecting status data from the service node where it is located;

[0023] A local analysis module for integrating and analyzing the collected status data to obtain analysis log data;

[0024] An adaptive control module for adjusting the service process and microservice instances of the service node where it is located according to the analysis log data, detecting faults, and recovering the service process and the microservice instances from the faults.

[0025] In some embodiments, the log management system also provides functions of real-time log search and filtering, log correlation analysis, cross-service log tracing, specific exception detection, and visualization display.

[0026] In some embodiments, the central control unit includes:

[0027] A test plan management module for managing the test plan and generating the test tasks and the fault injection tasks according to the test plan;

[0028] A task scheduler for allocating and scheduling the test tasks and the fault injection tasks;

[0029] A result aggregator for collecting and integrating the log data fed back by the multiple test agents.

[0030] In some embodiments, the fault simulator includes:

[0031] A fault management unit for designing various fault modes;

[0032] A scenario orchestration module for supporting visual fault scenario design;

[0033] An injection controller for controlling the timing and manner of fault injection;

[0034] An effect analyzer for analyzing the execution results of fault injection.

[0035] In a third aspect, an embodiment of the present disclosure provides a readable storage medium, on which a processing program is stored, and when the processing program is executed by a processor, the distributed test method described above is implemented.

[0036] In summary, the distributed test system and test method provided in this embodiment support the complete test process of the system under test from test plan generation to test result analysis, achieve a high degree of coordination between fault injection and test execution through precise control of fault injection, can simulate the scenarios of combined faults, and the distributed test system based on the microservices architecture can adaptively scale according to the scale of the system under test. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, the above and other objects, features, and advantages of the embodiments of the present disclosure will become clearer. In the drawings:

[0038] Figure 1 is a schematic structural diagram of a distributed test system based on the microservices architecture provided by an embodiment of the present disclosure;

[0039] Figure 2 is a flowchart of a distributed test method based on the microservices architecture provided by an embodiment of the present disclosure;

[0040] Figure 3 is a schematic structural diagram of a data center for deploying the distributed system of an embodiment of the present disclosure;

[0041] Figure 4 is Figure 1 a schematic diagram of the modules of the test agent in

[0042] Figure 5 is a schematic diagram of the modules of the fault injection system provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. In the respective drawings, like elements are denoted by like reference numerals. For clarity, the various parts in the drawings are not drawn to scale. In addition, some well-known parts may not be shown.

[0044] The following describes the embodiments of the present disclosure based on embodiments, but the embodiments of the present disclosure are not limited to these embodiments. In the following detailed description of the embodiments of the present disclosure, some specific details are described in detail. Those skilled in the art can fully understand the embodiments of the present disclosure without the description of these details. To avoid obscuring the essence of the embodiments of the present disclosure, well-known methods, processes, procedures, elements, and circuits are not described in detail.

[0045] Unless the context clearly requires otherwise, the terms "including", "comprising" and similar words throughout the specification and claims shall be construed in an inclusive sense rather than an exclusive or exhaustive sense; that is, the meaning of "including but not limited to". In addition, in the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0046] The microservices architecture is a software development architecture that decomposes an application into a set of small services, each running in its own independent process and typically built around a specific business capability. These services can communicate through well-defined application programming interfaces (APIs) or lightweight communication protocols. The microservices architecture has the following key characteristics: 1) modularity, the microservices architecture decomposes the application into a set of small, loosely coupled services, each implementing a specific business function; independent deployment: each microservice can be deployed, updated, and scaled independently of other services; technology diversity: different microservices can be developed using different programming languages, databases, or other technology stacks; business capability alignment: each service is typically built around a specific business capability, which helps the team better understand the business value of the service.

[0047] Embodiments of the present disclosure build a distributed test system for a system under test based on the microservices architecture. The schematic structural diagram of the distributed test system is as Figure 1 shown.

[0048] In Figure 1 it, the distributed test system includes a plurality of test agents 1 to 3, an automated execution engine 101, a log management system 102, a fault simulator 103, and a central control unit 104.

[0049] The system under test is based on a microservices architecture. During operation, multiple service instances are generated. Only microservice instances 1 to 3 of the system under test are exemplarily shown in the figure. Microservice instances 1 to 3 respectively perform specific business functions, and each service instance can also generate multiple replica instances. To test microservice instances 1 to 3, test agents 1 to 3 are deployed and run one-to-one on the service nodes where microservice instances 1 to 3 are located. Each test agent can not only execute commands and test cases from the automated execution engine 101 and the central control unit 104, but also regularly collect status data of entities such as CPU, memory, network, various system and application processes, detect the status of the corresponding microservice instance, capture the real-time logs generated by the corresponding microservice instance in real time, and actively or passively provide this data to the log management system 102, which stores this data as log data. Each test agent also has the ability of adaptive load, such as pausing certain service processes, reducing or increasing replica instances according to the status of CPU, memory, and network. Each test agent also has the ability of fault recovery, such as restarting or restoring service processes and / or microservice instances when a fault occurs. Each test agent can also perform intelligent analysis on various data collected or captured by itself and provide the analysis results to the log management system 102. In some embodiments, to reduce the negative impact of the test agent on the service under test, each test agent is implemented as a lightweight test agent and developed using a high-performance language, and containers (such as Docker) can be used to deploy each test agent.

[0050] In some embodiments, the test agents such as test agents 1 to 3 mainly include, for example Figure 4The three major functional modules shown: data acquisition module 401, local analysis module 402, and adaptive control module 403. The data acquisition module 401 configures an acquisition mechanism for each metric. For example, it samples the CPU usage rate, memory occupancy (including heap memory and non-heap memory), network traffic (including inbound and outbound traffic), and service status (detecting whether a specified service is alive and its response time) every 5 seconds, and stores the acquired data locally and reports it to the log management system 102. The local analysis module 402 analyzes based on the acquisition data stored locally, including calculating the change trend of key metrics, identifying outliers in the data using statistical methods, and predicting the load trend based on historical data. The adaptive control module 403 performs adaptive control according to the analysis results provided by the local analysis module 402, including adjusting the number of service processes and microservice instances, adjusting the sampling frequency, and adjusting the data reporting strategy. For example, when the local analysis module 402 outputs multiple outliers of CPU and memory occupancy, or indicates that the CPU and memory occupancy have been increasing over a certain period of time, the sampling frequency is reduced and the reporting frequency is increased. Another example is that when the local analysis module 402 outputs the change trend of network traffic over a certain period of time and predicts that the future network traffic will reach a critical point, the number of microservice instances is reduced. The adaptive control module 403 also includes detecting and recovering service processes and microservice instances from failures.

[0051] The central control unit 104 provides a user interface to testers so that testers can create test plans through the user interface. A test plan usually includes information such as test objective and scope definition, list of services under test and their configurations, test scenarios and test case sets, fault injection configuration, execution parameters and conditions, and expected result definition. The central control unit 104 constructs test tasks and fault injection tasks according to the test plan, provides the test tasks to the automated execution engine 101, and provides the fault injection tasks to the fault simulator 103. The central control unit 104 also collects test-related data and determines the impact of faults on test results based on the data. This data can come directly from the log management system 102 and / or microservice instances 1 to 3. In some cases, the central control unit 104 can also directly issue commands to test agents 1 to 3.

[0052] In some embodiments, the main functional modules of the central control unit 104 include: a test plan management module 1041, a task scheduler 1042, a result aggregator 1043, and an interface module 1044. The test plan management module 1041 is used to create, modify, and delete test plans, and generate test tasks and fault injection tasks according to the test plans. The task scheduler 1042 is used to allocate and schedule test tasks and fault injection tasks. The task scheduler 1042 assigns test tasks to the automated execution engine 101 according to the test task types. The result aggregator 1043 is used to collect and integrate the log data from the log management system 102 and test agents 1 to 3. The interface module 1044 is used to provide various communication interfaces. The central control unit 104 can also monitor the status of all test agents and microservice instances in real time through the log management system 102 and display it on the graphical interface.

[0053] The automated execution engine 101 assigns the test cases included in the test task to the corresponding test agent among test agents 1 to 3 according to the received test task, so that the corresponding test agent executes the test cases. The test agent is a distributed execution unit of the automated execution engine 101, and the automated execution engine 101 is responsible for the overall test process control and coordination.

[0054] The fault simulator 103 injects corresponding faults into the execution environment of the system under test according to the received fault injection tasks. There are mainly three categories of fault types: network faults such as network latency, packet loss, and disconnection; resource faults such as high CPU occupancy, memory leakage, and disk full; and application layer faults such as service crash and abnormal response. The fault simulator 103 supports multiple fault simulation methods. For example, network latency, packet loss, and disconnection are implemented through network proxies; high CPU occupancy and memory leakage are implemented through resource control; and service exceptions are implemented through request interception. Moreover, the fault simulator 103 supports network faults between microservice instances, network faults between the service nodes where the microservice instances are located and external systems, and network faults in the network where the distributed test system is located. For this reason, the fault simulator 103 also has a fault configuration function, which can not only configure the fault type and the scope of fault occurrence to be injected, but also configure its triggering method. The triggering methods of faults include, but are not limited to: time trigger, condition trigger, and progressive trigger. The time trigger means injecting faults at a specified time point. The condition trigger means injecting faults when specific conditions are met. The progressive trigger means injecting multiple faults step by step according to a predetermined rule. For example, the progressive trigger is injecting faults at a certain time interval. Another example is that the first fault is triggered at a specific time or when a certain condition is met, and subsequent faults can only be triggered when the execution results of the adjacent previous faults meet specific conditions. In some embodiments, the fault simulator 103 provides a fault scenario orchestration tool, which can automatically discover and display the status of registered microservice instances, and at the same time provides a graphical interface so that testers can design fault injection through the graphical interface.

[0055] The main functions of the log management system 102 include: a log collector 112, a log transmission pipeline 113, a central log storage 114, a log analysis engine 115, and a visualization tool 116. The log collector 112 collects various log formats of each service node. The log collector 112 can be designed to be deployed at the collection end of each service node and the service end deployed on the log server, and the log messages are transmitted between the two through a log transmission pipeline 113 (such as a message queue). The central log storage 114 can use a distributed search engine (such as Elasticsearch) to store logs to achieve index optimization and log lifecycle management. The log analysis engine 115 implements real-time log search and filtering functions, log correlation analysis, cross-service log tracing, and a specific anomaly detection module. The visualization tool 116 supports multi-dimensional data display and interactive query of log data.

[0056] In some embodiments, the central control unit 104 further includes: adjusting the test progress and test intensity according to the current state of the system under test, identifying and skipping invalid tests and retrying failed tests, for example, adjusting the test progress and test intensity through the automated execution engine 101. The central control unit 104 can also manage task prioritization, system load balancing, etc. across the entire distributed system.

[0057] In some embodiments, the distributed system 100 is built based on a microservices architecture, and the microservices architecture enables advantages such as modularization, pluggability, custom extensibility, and adaptability to systems under test of different scales for the distributed system 100.

[0058] Corresponding to the above distributed test system, the embodiments of the present disclosure provide a distributed test method based on a microservices architecture. Figure 2 The main steps of the distributed test method are given.

[0059] In step S201, test tasks and fault injection tasks are generated according to the test plan.

[0060] In step S202, each test case is assigned to multiple test agents according to the test tasks, so that the multiple test agents execute the corresponding test cases respectively.

[0061] In step S203, corresponding faults are injected into the execution environment of the system under test according to the fault injection tasks.

[0062] In step S204, log data related to the test is collected and summarized, and the test results and the impact of the faults on the test results are output according to the log data.

[0063] This embodiment provides a complete fault test process for the system under test, including key links from test plan generation to test result analysis. Specifically: receiving the test plan input by the tester, extracting key information from the test plan, constructing test tasks and fault injection tasks according to this key information, the test tasks include test cases to be executed by the test agents, then the test tasks are assigned to the test agents, and the test agents execute the corresponding test cases. During this process, faults are injected into the execution environment of the system under test, and the log data generated when the multiple test agents execute the test cases under the faults is collected and summarized, and the test results and the impact of the faults on the test results are analyzed according to the log data.

[0064] In a further embodiment, the fault injection task is configured to be triggered when certain conditions are met, triggered at a set time point, or perform progressive fault injection. Triggering when certain conditions are met means, for example, performing fault injection based on the execution status of the test task. For example, when certain specific test cases included in the test task all feedback successful execution, the fault injection task is executed. Another example is that when certain conditions are met in aspects such as the network environment where the system under test is located, resource occupancy, the status and quantity of microservice instances, etc., the fault injection task is executed. Progressive fault injection refers to combined fault injection. For example, injecting more than two faults into the execution environment of the system under test at certain time intervals, or after the first fault injection, determining whether to inject the second fault based on the execution result of the first fault. At the same time, different faults adopt different simulation methods. For example, for network faults, using the TC command to control network latency, using iptables to simulate packet loss, and using a network proxy to simulate disconnection; for resource faults, using the stress tool to simulate CPU stress, using memory filling to simulate memory leakage, and using file I / O to simulate disk occupancy; for application faults, using process control, request interception, request response interception, abnormal response injection, etc. to achieve simulation. Thus, through the configurability of faults, this embodiment realizes precise control over the timing and effect of fault injection.

[0065] In a further embodiment, the test task generated by the test plan indicates the name of the microservice or microservice use case to be tested, and then step S202 can allocate the test task to the test agent according to the name of the microservice or microservice use case to be tested.

[0066] In a further embodiment, the log data in step S204 is collected and managed in a dedicated log management system, and the log management system also provides functions such as real-time log search and filtering, log correlation analysis, cross-service log tracing, specific exception detection, and visual display.

[0067] In a further embodiment, the above method further includes the following steps: adjusting the test progress and test intensity according to the current state of the system under test, identifying and skipping invalid tests, and retrying failed tests. Figure 3 It is a schematic diagram of the structure of the data center for deploying the distributed system of the embodiments of the present disclosure. For the convenience of description, Figure 3 only the management node 310 and n service nodes 320-1 to 320-n are shown, where n is a positive integer.

[0068] The data center 300 is applied to various application scenarios such as Content Delivery Network (CDN), e-commerce, gaming, audio and video, Internet of Things, logistics, industrial brain, and urban brain, and provides business services for end users in various scenarios. Among them, the management node 310 is a computer device in the data center that manages and monitors n service nodes, and the n service nodes 320-1 to 320-n are computer devices in the data center that perform business processing. The management node 310l can perform various management operations in the data center to which each service node belongs. For example, it can monitor and manage operations and maintenance, logs, network status, etc. Through the management node 310, maintenance personnel can send commands to each service node, and the service node executes the commands received from the management node 310.

[0069] The management node 310 and the service nodes 320-1 to 320-n can form a computer cluster. The management node 310 includes multiple servers, and a computer cluster management software is installed on one of the servers. Each service node corresponds to one server, but the embodiments of the present disclosure are not limited thereto. Generally, on the basis of hardware devices including a processor, memory, storage, network device, audio device, input / output device, etc., various software is also installed on the server. For example, an operating system and application software are set on each server above the underlying hardware. The operating system is, for example, an operating system such as UNIX operating system, Linux operating system, etc. that can be used on the server. The application software may include and is not limited to programs for controlling or responding to external devices (such as biometric sensors, printers, microphones, speakers, flow valves, or other I / O components, sensors, actuators, or devices), programs for various I / O tasks, security programs, authentication programs, various computing modules, communication programs, communication support protocols, or other programs, or combinations thereof.

[0070] To adapt to this embodiment, the management node 310 deploys each module in the distributed test system provided by the embodiments of the present disclosure: the central control unit 104, the automated execution engine 101, the log management system 102, and the fault simulator 103. Moreover, microservice instances 1 to n of the system under test are deployed one-to-one on service nodes 320-1 to 320-n, and at the same time, test agents 1 to n are also deployed one-to-one on service nodes 320-1 to 320-n. It should be understood that for simplicity, the above modules in the distributed test system are drawn in the same box in the figure. However, in fact, the above modules in the distributed test system are usually independently deployed on multiple servers, and the communication between the above modules in the distributed test system can be realized through the communication mechanism provided by the microservice architecture. In addition, it should be noted that for convenient deployment and management, container technology can be used to deploy the system under test and the distributed test system provided by the embodiments of the present disclosure. Container technology creates an independent running environment for different application programs, can achieve resource isolation, configuration, and security guarantee between application programs and modules, can meet the resource requirements for on-demand allocation of applications, and ensure the isolation and availability of applications. Correspondingly, the embodiments of the present disclosure also provide a fault injection system, Figure 5 which shows the main modules of the system. As Figure 5 shown, the system includes a fault management unit 501, a scenario orchestration module 502, an injection controller 503, and an effect analyzer 504. The fault management unit 501 is used to manage various fault modes, including providing a mode design interface for the user to input information such as fault name, fault type, and fault simulation method to generate a new fault mode and store it in the fault mode library. The scenario orchestration module 502 is used to design fault scenarios, including providing a visual fault scenario design interface for the user to select fault modes and fault triggering methods. The injection controller 503 is used to execute fault injection according to the fault scenarios (including fault modes and fault triggering methods) designed by the scenario orchestration module 502. The effect analyzer 504 is used to analyze the execution results of fault injection. It should be noted that each module included in the fault injection system of this embodiment and its functions are applicable to Figure 1 the fault simulator 103 in

[0071] In summary, the distributed test system and method provided in this embodiment have the following innovative points: 1) Test agent collaboration mechanism, including supporting adaptive collaboration between test agents, supporting dynamic load balancing and task reallocation for each service node, and supporting local decision-making for each service node; 2) Multi-dimensional fault injection, including supporting network, resource, and application layer fault simulation, achieving precise triggering, control, and providing visual fault scenario orchestration of faults; 3) Distributed log analysis, including log correlation analysis, cross-service call chain tracing, log anomaly detection and location; 4) Deep adaptation to the microservices architecture, including native service discovery and registration, automatically handling service dependencies, and supporting dynamic scaling of services. Moreover, through experimental verification, it is found that compared with the traditional solution, the distributed test system of this embodiment reduces the test execution time by 40%, increases the fault scenario coverage rate by 60%, improves the system resource utilization rate by 50%, and shortens the fault location time by 70%.

[0072] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by computer instructions, or by controlling related hardware through computer instructions. These computer instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, the embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program or instruction is stored. When the computer program or instruction is executed by the processor, the various processes of the above embodiments can be implemented.

[0073] Since the computer instructions stored in this storage medium can execute the steps in the distributed test method provided by the embodiments of the present disclosure, the beneficial effects achievable by the distributed test method provided by the embodiments of the present disclosure can be realized. For details, see the previous embodiments and will not be elaborated here. The specific implementation of each of the above operations can be seen in the previous embodiments and will not be elaborated here.

[0074] As described above in the embodiments in accordance with the present disclosure, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can make good use of the embodiments of the present disclosure and the modifications based on the embodiments of the present disclosure. The embodiments of the present disclosure are only limited by the claims and their full scope and equivalents.

Claims

1. A distributed testing method based on microservice architecture, comprising: Generate test tasks and fault injection tasks according to the test plan; Allocate each test case to a plurality of test agents according to the test task, so that the plurality of test agents respectively execute the corresponding test cases, and the plurality of test agents and the plurality of microservice instances of the system under test run one-to-one on a plurality of service nodes; Injecting corresponding faults into the execution environment of the system under test according to the fault injection task; Collect and summarize the log data related to the test, and output the test results and the impact of the failure on the test results based on the log data.

2. The distributed testing method according to claim 1, wherein: The test agent is also used to collect status data from the service node, adjust related service processes and microservice instances according to the status data, and restore related service processes and microservice instances when a failure occurs.

3. The distributed testing method according to claim 1, wherein: The injected corresponding faults are: injected network faults, injected resource faults and / or injected application faults. The injected network faults are one or more simulated network delays, packet loss and disconnections. The injected resource faults are one or more simulated CPU occupancy, memory leaks and disk full. The injected application faults are one or more simulated service crashes and abnormal responses.

4. The distributed testing method according to claim 3, wherein: For the network failure, use the TC command to simulate network delay, use iptable to achieve packet loss simulation, and use the network proxy to simulate disconnection; for resource failure, use the stress tool to simulate CPU pressure, use memory filling to simulate memory leakage, and use file IO to simulate disk occupancy; for application failure simulation, use at least one of process control, request interception, request response interception, and exception response to achieve it.

5. The distributed testing method according to claim 1, wherein: The fault injection task is configured to adopt one of the following triggering methods: time triggering, condition triggering and progressive triggering. The time triggering is to inject a fault at a specified time point, the condition triggering is to inject a fault when a specific condition is met, and the progressive triggering is to gradually inject multiple faults according to a predetermined rule.

6. The distributed testing method according to claim 1, further comprising: Adjust test progress and test intensity according to the current state of the system under test, identify and skip invalid tests, and retry failed tests.

7. A distributed testing system based on microservice architecture, comprising: Multiple test agents, running one-to-one with multiple microservice instances of the system under test on multiple service nodes; A central control unit, used for generating test tasks and fault injection tasks according to a test plan input by a user; An automated execution engine, configured to assign each test case to a plurality of test agents according to the test task, so that the plurality of test agents respectively execute the corresponding test cases; A log management module, used to collect and manage log data fed back by the multiple test agents; A fault simulator, used for injecting corresponding faults into the execution environment of the system under test according to the fault injection task; The central control unit is also used to analyze the test results and the impact of the fault on the test results according to the log data.

8. The distributed testing system according to claim 7, wherein the testing agent comprises: A data collection module is used to collect status data from the service node; The local analysis module is used to integrate and analyze the collected status data to obtain analysis log data; The adaptive control module is used to adjust the service process and microservice instance of the service node according to the analysis log data, and detect faults and recover the service process and the microservice instance from the faults.

9. The distributed testing system according to claim 7, wherein: The log management system also provides real-time log search and filtering functions, log correlation analysis, cross-service log tracking, specific anomaly detection, and visual display functions.

10. The distributed testing system according to claim 7, wherein: The central control unit comprises: A test plan management module, used to manage the test plan and generate the test task and the fault injection task according to the test plan; A task scheduler, used for allocating and scheduling the test tasks and the fault injection tasks; The result aggregator is used to collect and integrate the log data fed back by the multiple test agents.

11. The distributed testing system according to claim 7, wherein: The fault simulator comprises: Fault management unit, used to design various fault modes; Scenario arrangement module, used to support visual fault scenario design; An injection controller, used to control the timing and method of fault injection; Effect Analyzer, used to analyze the execution results of fault injection. 12 . A readable storage medium having a processing program stored thereon, wherein the processing program, when executed by a processor, implements the distributed testing method according to claim 1 .

Images