A system-oriented fault injection and simulation testing method, platform, and equipment

Abstract

This application relates to the field of system testing technology, and in particular to a system-oriented fault injection and simulation testing method, platform, and device. The method includes: registering the topology information of the system under test after responding to all test task requests through a management service; deploying a fault injection probe within the target server of the system under test; determining the generation mode of a traffic strategy in response to orchestration operations on a visual interface; invoking a test service matching the generation mode; injecting background traffic into the system under test based on the test service; and having a reliability test service send instructions to the fault injection probe of the target server based on the background traffic and preset orchestration logic, and perform fault injection and simulation testing. This application achieves a high degree of simulation of real-production traffic models and business scenarios, improving the authenticity and effectiveness of test background pressure, and making the system bottlenecks exposed by the test have high production reference value.

Description

Technical Field

[0001] This application relates to the field of system testing technology, and in particular to a system-oriented fault injection and simulation testing method, platform and equipment. Background Technology

[0002] In today's rapidly evolving landscape of software systems and infrastructure, system reliability has become a core element in ensuring business continuity and user experience. Especially in critical sectors such as finance, telecommunications, and the Internet of Things (IoT), system failures can lead to severe economic losses and reputational risks. Therefore, conducting thorough and effective reliability testing before system deployment is essential to ensuring its long-term stable operation.

[0003] Given this broader technological context, the key challenges in reliability testing of large-scale enterprise-level business systems are as follows: Background traffic generation: The authenticity of test traffic is the cornerstone of evaluating system reliability. Effective testing must be able to simulate background traffic that is highly consistent with the production environment. This includes realistic user behavior patterns (such as the proportion and sequence of login, browsing, and transaction operations), time patterns that conform to business cycles (such as daytime peaks and nighttime batch tasks), and diverse combinations of business processes and data distribution. Simply simulating simple constant pressure or linearly increasing traffic cannot expose potential performance bottlenecks and stability issues of the system under complex real-world loads.

[0004] Fault Scenario Simulation: Production environment unavailability is rarely caused directly by a single, isolated failure. Especially in distributed systems, complex failure modes are more common and destructive. Multiple failure points (such as a microservice instance failure, localized network latency, database connection pool exhaustion, cache node failure, etc.) occur successively or simultaneously within a short period of time, creating a cumulative, cascading, or even amplified effect, ultimately leading to system avalanche or service unavailability. Reliability testing must be able to accurately and controllably simulate these complex failure scenarios to verify the system's fault tolerance, self-healing, and resilience.

[0005] System Monitoring and Analysis: During fault injection and stress testing, monitoring is the "eye" that reveals system behavior. However, traditional monitoring often focuses on technical metrics such as CPU, memory, and network I / O. Business monitoring (such as transaction success rate, response time distribution, throughput of key business processes, and business error code distribution) is more critical. It directly reflects the impact of faults on end-user experience and business objectives, and is the ultimate basis for judging whether a system is "truly reliable." Without in-depth monitoring and analysis of business-related metrics, testing will struggle to pinpoint root causes and assess business impact. Summary of the Invention

[0006] The main objective of this application is to propose a system-oriented fault injection and simulation testing method, platform, and device.

[0007] To achieve the above objectives, one aspect of this application proposes a system-oriented fault injection and simulation testing method, the method comprising: After responding to all test task requests through the management service, the topology information of the system under test is registered, and a fault injection probe is deployed in the target server of the system under test. Responding to the orchestration operations of the visual interface, determine the generation mode of the traffic strategy; Invoke a test service that matches the generation pattern, and inject background traffic into the system under test based on the test service; Based on the background traffic and preset orchestration logic, the reliability testing service sends instructions to the fault injection probe of the target server and performs fault injection and simulation testing.

[0008] In some embodiments, the method further includes: Deploy monitoring probes within the target server of the system under test; The monitoring service collects and displays feedback data from the monitoring probes in real time.

[0009] In some embodiments, the test service includes a pipeline replay service, wherein calling the test service that matches the generation mode and injecting background traffic into the system under test based on the test service specifically includes: In response to the generation mode determined through the visual interface, the backend of the stream playback service obtains the stream file uploaded by the user and stores it in the database; When the streaming playback service is triggered, the task configuration is sent to the streaming playback engine; The pipeline playback engine reads the pipeline file from the database based on the task configuration, parses the pipeline file, and generates a structured request sequence. The pipeline playback engine calls the interface adaptation layer, determines the protocol specification based on the environment information of the system under test, sends the structured request sequence to the corresponding entry point of the system under test according to the protocol specification, and injects background traffic into the system under test.

[0010] In some embodiments, the testing service includes a stress testing service, and the invocation of the testing service matching the generation pattern, and the injection of background traffic into the system under test based on the testing service, specifically includes: In response to the user's configuration operations in the visual interface, the stress test service backend creates a scenario configuration file, establishes an SSH connection with the execution machine, transmits the stress engine compressed package to the target execution machine and decompresses it, thus completing the deployment of the stress engine; The stress testing service backend sends a start command to the deployed stress engine via SSH connection to start the stress engine and continuously monitors the initialization status of the stress engine. Once the stress engine initialization is complete, the stress test service backend sends the scenario configuration file and execution commands to the stress engine. The pressure engine receives and parses the scenario configuration file, and sends parameterized pressure requests to the system under test; Based on the pressure request, background traffic is injected into the system under test.

[0011] In some embodiments, injecting background traffic into the system under test based on the pressure request specifically includes: Based on the scenario configuration file, construct the specific request content; Based on the generation mode, the request sending rate per unit time is dynamically adjusted. Based on the business proportion, allocate the number of requests for different business types according to their weights; During the injection process, the pressure engine adjusts the background flow based on pressure parameters and injects it into the system under test. The pressure parameters include the specific request content, request sending rate, and number of requests.

[0012] In some embodiments, the reliability testing service, based on the background traffic and preset orchestration logic, sends instructions to the fault injection probe of the target server and performs fault injection and simulation testing, specifically including: In response to fault test scenarios created by users in the visual interface, configure complex fault sequences and execution logic for the target server; In response to the user's trigger operation on the visual interface, a fault task instruction is generated, and the reliability test service backend distributes the fault task instruction to the fault injection probe. The fault injection probe maintains the fault task instructions in the probe's local memory; When the timer of the fault injection probe reaches a preset threshold, the corresponding fault action is executed.

[0013] In some embodiments, the monitoring service collects and displays feedback data from the monitoring probe in real time through the monitoring probe, specifically including: All target servers and application instance information to be monitored are uniformly registered to the service discovery middleware to form a dynamic monitoring target list; In response to the user's trigger operation on the visual interface, the monitoring service backend connects to the target server via SSH, pushes the monitoring probe executable file to the specified directory and starts it as a daemon process, and then registers the probe's IP and port information to the service discovery middleware. The data collector listens to the service discovery middleware, dynamically creates and manages collection task timers based on the dynamic monitoring target list, and periodically and proactively pulls data collected by monitoring probes deployed on each target server via the HTTP protocol. The collected data is stored in a time-series database; In response to a query request initiated by the visualization interface, monitoring data is retrieved from the time-series database and real-time charts are generated for display.

[0014] A system-oriented fault injection and simulation testing platform, characterized in that the platform comprises: The probe deployment module is used to register the topology information of the system under test after responding to all test task requests through the management service, and to deploy fault injection probes in the target server of the system under test. The data interaction module responds to the orchestration operations of the visual interface and determines the generation mode of the traffic strategy; A traffic injection module is used to inject background traffic into the system under test based on a test service that matches the generation pattern. The test execution module, based on the background traffic and preset orchestration logic, sends instructions to the fault injection probe of the target server to perform fault injection and simulation testing.

[0015] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0016] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the above-described method.

[0017] The embodiments of this application include at least the following beneficial effects: This application achieves a high degree of simulation of real production traffic models and business scenarios, improving the authenticity and effectiveness of test background pressure, making the system bottlenecks exposed by the test have high production reference value, and realizing full-link deep observability from infrastructure to business logic. The test method of this application is flexible to expand and has very low modification cost. Attached Figure Description

[0018] Figure 1 A flowchart of a system-oriented fault injection and simulation testing method provided in an embodiment of this application; Figure 2 A schematic diagram of a system-oriented fault injection and simulation testing method provided in an embodiment of this application; Figure 3 The flowchart of the pipeline playback service provided in the embodiments of this application is as follows; Figure 4 This is a flowchart illustrating the execution of a stress testing service provided in an embodiment of this application. Figure 5 A flowchart illustrating the execution of reliability services provided in this application's embodiments; Figure 6 The execution flowchart of the monitoring service provided in the embodiments of this application; Figure 7 A block diagram of the system-oriented fault injection and simulation test platform provided in the embodiments of this application; Figure 8 This is a block diagram of the fault injection and simulation test platform under a distributed architecture provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0020] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0021] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0024] SSH (Secure Shell) is an encrypted network transport protocol used to enable secure communication and data transmission between computers in insecure networks.

[0025] WebUI, a browser-based web user interface.

[0026] Grafana, as a visualization platform for monitoring data, uses PromQL to query monitoring data in Prometheus and generates real-time charts for display on the front end.

[0027] Prometheus, as a data collector, periodically queries the service discovery middleware to obtain monitoring targets, pulls probe data via HTTP protocol, and persists it.

[0028] In related technologies, common reliability testing processes often struggle to simultaneously guarantee high fidelity of background traffic, complex and controllable fault scenarios, and in-depth business focus in monitoring and analysis. Specifically: traffic generation tools struggle to simulate real-world business distribution and user behavior sequences; fault injection capabilities are often limited to simulating simple faults in single components or types, lacking the ability to orchestrate complex fault scenarios; and on the monitoring side, technical and business metrics are generally disconnected, making it difficult to quickly and intuitively establish the correlation between resource bottlenecks, infrastructure failures, and the final business outcome during testing. This weakens the persuasiveness and guiding value of problems discovered during testing at the business level.

[0029] In view of this, this application provides a system-oriented fault injection and simulation testing method, platform and equipment.

[0030] refer to Figures 1-2 As shown, to achieve the above objectives, this application proposes a system-oriented fault injection and simulation testing method, including: S101: After responding to all test task requests through the management service, register the topology information of the system under test and deploy a fault injection probe in the target server of the system under test; S102: In response to orchestration operations on the visual interface, determine the generation mode of the traffic strategy; S103: Invoke the test service that matches the generation mode, and inject background traffic into the system under test based on the test service; S104: Based on the background traffic and preset orchestration logic, the reliability service sends instructions to the fault injection probe of the target server to perform fault injection and simulation testing.

[0031] Specifically, step S101 includes: The system registers the topology information of the system under test, including key nodes such as traffic injection entry points, application servers under test, and databases, enabling the system to accurately identify and manage test targets.

[0032] The fault injection probe is deployed in the target server, receives instructions from the reliability testing service, and executes specific fault simulation actions (such as CPU full load, network packet loss, service anomaly, etc.).

[0033] The monitoring probes are deployed within the target server. The monitoring probes are divided into resource monitoring probes and business monitoring probes. The former collects infrastructure indicators such as server CPU, memory, and network I / O; the latter collects key business indicators such as business success rate, response time, and transaction volume through methods such as data entry or log analysis.

[0034] Specifically, in step S101, the system under test in this application includes a distributed microservice system, a cloud-native system, and a traditional distributed system.

[0035] Distributed microservice systems consist of multiple independently deployed services that communicate with each other; cloud-native systems are systems deployed on container cloud platforms; and traditional distributed systems are distributed architectures based on technologies such as RPC and message queues.

[0036] This application, in addition to infrastructure monitoring, introduces business monitoring probes. Through non-intrusive methods such as parsing application logs and capturing link data, it collects core business metrics such as success rate, response latency, and throughput. The entire monitoring system adopts a plug-in design; for new monitoring needs or different systems under test, only corresponding monitoring probe plugins need to be customized for quick integration. This solution achieves deep observability across the entire link from infrastructure to business logic, and is flexible in expansion with extremely low modification costs.

[0037] Specifically, the visual interface of this application adopts WebUI, which is a browser-based web page user interface.

[0038] Specifically, step S102 includes: Users can select one or more target resource groups in the system's visual interface and orchestrate a series of "atomic faults" (such as process termination, memory consumption, network latency, etc.) for each resource group. The system supports setting independent manual triggering or automatic execution strategies for each atomic fault and can orchestrate the concurrency and sequence dependencies between faults, thereby accurately simulating complex fault modes with multiple faults superimposed in the production environment.

[0039] Users need to select how background traffic is generated. The system provides two traffic generation modes: Standard stress mode: Driven by the stress testing service, it generates parameterizable linear, stepped, and other stress models.

[0040] Pipeline replay mode: Driven by the pipeline replay service, it reproduces user behavior and business links that are highly consistent with the production environment by replaying real production traffic data offline, thereby improving the authenticity of the test.

[0041] Users can customize the monitoring dashboard and flexibly add key technical and business metrics that need to be focused on during testing. The monitoring service will then periodically pull data from each probe based on this configuration and display and store it in real time during test execution.

[0042] Specifically, steps S103 and S104 include: S201: The system schedules either a stress test service or a pipeline playback service to inject highly realistic background traffic into the system under test, based on the selected generation mode.

[0043] S202: The reliability testing service sends instructions to the fault injection probes of the target resource group sequentially or concurrently according to the preset orchestration logic to simulate complex fault scenarios.

[0044] S203: The monitoring service gathers technical and business indicator data reported by all monitoring probes in real time and refreshes it every second through the front-end monitoring interface, providing users with a panoramic view of system resource consumption, application performance and business health, thereby clearly and intuitively exposing the true reliability status and bottlenecks of the system under the superposition of complex faults and real loads.

[0045] Specifically, in step S201, the pipeline replay service is the core module for achieving highly realistic background load and ensuring that the test scenario closely resembles the production environment. This service collects, analyzes, and replays real production traffic data to accurately simulate user behavior, thereby reproducing system stress highly consistent with the production environment during testing. This service adopts a modular design, consisting of two main parts: the replay service backend and the pipeline replay engine. Replay Service Backend: As the control center, it is responsible for the lifecycle management, scheduling, and state maintenance of the pipeline replay service. When a test task is triggered, the backend service is responsible for scheduling the pipeline replay engine instance to execute the specific replay task.

[0046] The pipeline replay engine, as the task execution unit, adopts a layered architecture design, comprising an engine core and an interface adaptation layer. The engine core is the data processing hub for the replay task; one of its core functions is pipeline file parsing. It cleans, denoises, converts formats, and extracts parameters from the raw pipeline data, filtering invalid requests and transforming it into a structured sequence of user requests, outputting it as an intermediate file for subsequent stages. The interface adaptation layer acts as a bridge connecting to different systems under test. It receives the structured request files produced by the engine core and accurately sends the requests to the corresponding entry points of the systems under test according to protocol specifications (such as HTTP, gRPC, custom RPC, etc.), achieving real traffic injection.

[0047] refer to Figure 3 As shown, the specific execution flow of the playback service includes: In response to the generation mode determined through the visual interface, the backend of the stream playback service obtains the stream file uploaded by the user and stores it in the database; When the pipeline replay service is triggered, the task configuration (including the path of the pipeline file in Minio and the information of the tested environment) is sent to the pipeline replay engine. The streaming playback engine reads the streaming file from the database based on the task configuration, parses the streaming file, and generates a structured request sequence.

[0048] The pipeline playback engine calls the interface adaptation layer, determines the protocol specification based on the environment information of the system under test, and sends the structured request sequence to the corresponding entry point of the system under test according to the protocol specification (such as HTTP, gRPC, custom RPC, etc.), injecting background traffic into the system under test.

[0049] During task execution, the pipeline playback engine will report the execution status in real time by calling the API provided by the pipeline playback service.

[0050] This application utilizes an independent pipeline replay service to deeply analyze, structure, and adapt raw production traffic for multiple protocols, transforming it into a sequence of user requests that can be accurately replayed to the system under test. This method achieves a high degree of simulation of real production traffic models and business scenarios, fundamentally improving the authenticity and effectiveness of test background stress, and making the system bottlenecks exposed by the test highly valuable for production reference.

[0051] Furthermore, in step S201, the stress testing service is the core module for implementing regular background traffic injection, including: Stress testing service backend: The stress testing backend is the control center for stress generation, responsible for the full lifecycle management of stress testing scenarios. Users can create test scenarios, select stress generation modes, configure stress levels and the proportion of various business operations, and control the stress engine status through the platform backend. When a task needs to be executed, the server assembles the scenario configuration file and distributes it to a fixed path on the execution machine, and sends instructions to the execution machine via SSH connection to control the stress engine status.

[0052] Stress testing engine: The stress testing engine is responsible for reading the test scenario files sent by the server and accepting the control commands sent by the server. The server first starts the stress engine and sends an initialization command. The server continuously checks whether the stress engine status has been initialized. Once it has been initialized, the server sends the start execution command, and the stress engine begins to send stress according to the scenario requirements.

[0053] refer to Figure 4 As shown, the execution flow of the stress testing service further includes: In response to the user's configuration operations in the visual interface, the stress test service backend creates a scenario configuration file, establishes an SSH connection with the execution machine, transmits the stress engine compressed package to the target execution machine and decompresses it, thus completing the deployment of the stress engine; The stress testing service backend sends a start command to the deployed stress engine via SSH connection to start the stress engine and continuously monitors the initialization status of the stress engine. Once the stress engine initialization is complete, the stress test service backend sends the scenario configuration file and execution commands to the stress engine. The pressure engine receives and parses the scenario configuration file, and sends parameterized pressure requests to the system under test; Based on the pressure request, background traffic is injected into the system under test, and the execution status is returned to the server in real time through the callback interface.

[0054] Furthermore, based on the pressure request, background traffic is injected into the system under test, specifically including: Based on the scenario configuration file, construct the specific request content; Based on the generation mode, the request sending rate per unit time is dynamically adjusted. Based on the business proportion, allocate the number of requests for different business types according to their weights; During the injection process, the pressure engine adjusts the background flow based on pressure parameters and injects it into the system under test. The pressure parameters include the specific request content, request sending rate, and number of requests.

[0055] Specifically, in step S202, the reliability testing service is the core module for the system to implement fault injection and scenario execution. It adopts a distributed architecture with server-probe collaboration, consisting of a reliability testing service backend and fault injection probes deployed on each target node. Reliability testing service backend: As the control center, it is responsible for the global orchestration, scheduling, and full lifecycle management of test tasks. When a test starts, the backend distributes the user-orchestrated fault scenarios and task instructions to the relevant fault injection probes in one go.

[0056] Fault Injection Probe: As a lightweight execution terminal, it receives fault tasks from the backend and is responsible for accurately and timely executing various fault actions in the target environment. To achieve efficient autonomy and precise control, the probe's internal design is as follows: Task storage: To meet the requirements of lightweight and high performance, the probe maintains the entire list of received fault tasks in memory, ensuring extremely low management overhead and fast response capabilities.

[0057] Task scheduling: Each fault injection probe runs a high-precision timer that continuously checks the task list in memory, autonomously judges and executes fault actions that have reached the preset trigger time, thereby achieving distributed, decentralized and precise scheduling.

[0058] This design distributes the centralized scenario scheduling pressure to each edge probe, ensuring the timing accuracy of fault injection while significantly improving the overall scalability and execution stability in large-scale, multi-node test scenarios.

[0059] Further reference Figure 5 As shown, in step S202, instructions are sent sequentially or concurrently to the fault injection probes of the target resource group according to the preset orchestration logic of the reliability service to simulate complex fault scenarios, specifically including: In response to fault test scenarios created by users in the visual interface, configure complex fault sequences and execution logic for the target server; In response to the user's trigger operation on the visual interface, a fault task instruction is generated, and the reliability test service backend distributes the fault task instruction to the fault injection probe. The fault injection probe maintains the fault task instructions in the probe's local memory; When the timer of the fault injection probe reaches a preset threshold, the corresponding fault injection and simulation test are performed.

[0060] After receiving the instruction, the fault injection probe executes the corresponding fault injection operation and reports the execution result to the service backend in real time through the callback interface, thereby completing the closed-loop task scheduling and state synchronization.

[0061] In some embodiments, fault task instructions can also be manually triggered at the front end.

[0062] Specifically, the monitoring service in step S203 is a key component for the system to achieve observability of the testing process and analyzability of the results. It is responsible for panoramic, real-time, and multi-dimensional data collection, storage, and display of the entire testing chain.

[0063] Its core design goal is not only to monitor the status of infrastructure resources, but also to directly gain insight into the health and performance of business operations. The service employs a loosely coupled, scalable architecture, primarily consisting of four collaborative components: a monitoring probe, a data collector, a data storage service, and a data query service. Service registration and discovery: When a reliability test task is started, the reliability test service will register all target servers and application instance information to be monitored with the service discovery middleware to form a dynamic monitoring target list.

[0064] Distributed data acquisition: The data collector acts as the scheduling hub, listening to the service discovery middleware and dynamically creating and managing collection task timers based on the registration list. The collector periodically and proactively pulls data collected by various monitoring probes deployed on target servers. The probe system is highly scalable, including: Infrastructure probe: Collects hardware resource metrics such as server CPU, memory, disk I / O, and network traffic, as well as the status of key processes.

[0065] Business Monitoring Probe: This is the core innovative monitoring capability of the system. This probe non-intrusively extracts metrics that directly reflect user experience and system business capabilities, such as business request success rate, request response latency, and system throughput, by real-time parsing of application logs and capturing business link tracing data.

[0066] Data persistence and query: The data storage service receives and processes massive amounts of monitoring data reported by the collectors, efficiently and persistently storing it in a time-series database. The data query service provides a unified query interface, providing data support for front-end monitoring dashboards, test report generation, and root cause analysis.

[0067] Further reference Figure 6 As shown, the execution flow of the monitoring service includes: Deploy monitoring probes within the target server of the system under test; The monitoring service backend registers all target server and application instance information to be monitored in a unified manner with the service discovery middleware, forming a dynamic monitoring target list; In response to the user's trigger operation on the visual interface, the monitoring service backend connects to the target server via SSH, pushes the monitoring probe executable file to the specified directory and starts it as a daemon process, and then registers the probe's IP and port information to the service discovery middleware. The data collector listens to the service discovery middleware, dynamically creates and manages collection task timers based on the dynamic monitoring target list, and periodically and proactively pulls data collected by monitoring probes deployed on each target server via the HTTP protocol. The collected data is stored in a time-series database; In response to a query request initiated by the visualization interface, monitoring data is retrieved from the time-series database and real-time charts are generated for display.

[0068] Users can configure PromQL queries in visualization platforms such as Grafana to retrieve monitoring data from Prometheus and generate real-time charts for display on the front end.

[0069] The dynamic monitoring target list is discovered through a periodic query service of the data collector (Prometheus), which dynamically obtains monitoring targets and adds them to the dynamic monitoring target list.

[0070] refer to Figure 7 As shown, in another aspect of the embodiments of this application, a system-oriented fault injection and simulation test platform is also proposed, including: The probe deployment module is used to register the topology information of the system under test after responding to all test task requests through the management service, and to deploy fault injection probes in the target server of the system under test. The data interaction module responds to the orchestration operations of the visual interface and determines the generation mode of the traffic strategy; A traffic injection module is used to inject background traffic into the system under test based on a test service that matches the generation pattern. The test execution module, based on the background traffic and preset orchestration logic, sends instructions to the fault injection probe of the target server to perform fault injection and simulation testing.

[0071] This application provides a visual interface through a data interaction module, supporting users to orchestrate complex fault scenarios involving multiple targets and types of atomic faults (such as resource, network, and service anomalies) with defined timing and dependencies. It adopts a centralized orchestration and edge execution architecture: a central service is responsible for task definition and instruction issuance; lightweight probes deployed on each target node maintain task queues in local memory and execute autonomously using built-in high-precision timers. This architecture distributes scheduling pressure to the edge, enabling accurate and scalable simulation of complex fault modes in large-scale distributed systems.

[0072] refer to Figure 8 As shown, further, in some embodiments, the system-oriented fault injection and simulation testing platform of this application adopts a distributed architecture with front-end and back-end separation. The back-end is divided into multiple collaborative sub-services according to function, and can be divided into three layers: application business layer, service layer, and data layer. The application business layer serves as the user interface, providing two access methods: Web UI and OpenAPI. It integrates all the core functions of the system and is a unified window for users to perform test orchestration, task execution, and monitoring and analysis. The service layer, as the core back-end of the system, is responsible for handling all business logic. Based on the key functional dimensions of reliability testing, this system subdivides it into five core services. The data layer provides persistent storage for each service of the system, used to save key information such as test configuration, scenario orchestration, monitoring data, and replay pipelines.

[0073] The methods provided in this application relate to the field of information technology. The methods provided in this application can be applied to terminals, servers, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the method, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0076] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0077] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0078] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0079] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0080] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0081] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0084] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0085] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0087] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A system-oriented fault injection and simulation testing method, characterized in that, The method includes: After responding to all test task requests through the management service, the topology information of the system under test is registered, and a fault injection probe is deployed in the target server of the system under test. Responding to the orchestration operations of the visual interface, determine the generation mode of the traffic strategy; Invoke a test service that matches the generation pattern, and inject background traffic into the system under test based on the test service; Based on the background traffic and preset orchestration logic, the reliability testing service sends instructions to the fault injection probe of the target server and performs fault injection and simulation testing.

2. The system-oriented fault injection and simulation testing method according to claim 1, characterized in that, The method further includes: Deploy monitoring probes within the target server of the system under test; The monitoring service collects and displays feedback data from the monitoring probes in real time.

3. The system-oriented fault injection and simulation testing method according to claim 1, characterized in that, The testing service includes a pipeline playback service. The invocation of the testing service, which matches the generation pattern, injects background traffic into the system under test based on the testing service. Specifically, this includes: In response to the generation mode determined through the visual interface, the backend of the stream playback service obtains the stream file uploaded by the user and stores it in the database; When the streaming playback service is triggered, the task configuration is sent to the streaming playback engine; The pipeline playback engine reads the pipeline file from the database based on the task configuration, parses the pipeline file, and generates a structured request sequence. The pipeline playback engine calls the interface adaptation layer, determines the protocol specification based on the environment information of the system under test, sends the structured request sequence to the corresponding entry point of the system under test according to the protocol specification, and injects background traffic into the system under test.

4. The system-oriented fault injection and simulation testing method according to claim 1, characterized in that, The testing service includes a stress testing service. The invocation of the testing service matching the generation pattern, and the injection of background traffic into the system under test based on the testing service, specifically includes: In response to the user's configuration operations in the visual interface, the stress test service backend creates a scenario configuration file, establishes an SSH connection with the execution machine, transmits the stress engine compressed package to the target execution machine and decompresses it, thus completing the deployment of the stress engine; The stress testing service backend sends a start command to the deployed stress engine via SSH connection to start the stress engine and continuously monitors the initialization status of the stress engine. Once the stress engine initialization is complete, the stress test service backend sends the scenario configuration file and execution commands to the stress engine. The pressure engine receives and parses the scenario configuration file, and sends parameterized pressure requests to the system under test; Based on the pressure request, background traffic is injected into the system under test.

5. The system-oriented fault injection and simulation testing method according to claim 4, characterized in that, The injection of background traffic into the system under test based on the pressure request specifically includes: Based on the scenario configuration file, construct the specific request content; Based on the generation mode, the request sending rate per unit time is dynamically adjusted. Based on the business proportion, allocate the number of requests for different business types according to their weights; During the injection process, the pressure engine adjusts the background flow based on pressure parameters and injects it into the system under test. The pressure parameters include the specific request content, request sending rate, and number of requests.

6. The system-oriented fault injection and simulation testing method according to claim 1, characterized in that, The reliability testing service, based on the background traffic and preset orchestration logic, sends instructions to the fault injection probe of the target server and performs fault injection and simulation testing, specifically including: In response to fault test scenarios created by users in the visual interface, configure complex fault sequences and execution logic for the target server; In response to the user's trigger operation on the visual interface, a fault task instruction is generated, and the reliability test service backend distributes the fault task instruction to the fault injection probe. The fault injection probe maintains the fault task instructions in the probe's local memory; When the timer of the fault injection probe reaches a preset threshold, the corresponding fault injection and simulation test are performed.

7. The system-oriented fault injection and simulation testing method according to claim 2, characterized in that, The monitoring service collects and displays feedback data from the monitoring probes in real time, specifically including: All target servers and application instance information to be monitored are uniformly registered to the service discovery middleware to form a dynamic monitoring target list; In response to the user's trigger operation on the visual interface, the monitoring service backend connects to the target server via SSH, pushes the monitoring probe executable file to the specified directory and starts it as a daemon process, and then registers the probe's IP and port information to the service discovery middleware. The data collector listens to the service discovery middleware, dynamically creates and manages collection task timers based on the dynamic monitoring target list, and periodically and proactively pulls data collected by monitoring probes deployed on each target server via the HTTP protocol. The collected data is stored in a time-series database; In response to a query request initiated by the visualization interface, monitoring data is retrieved from the time-series database and real-time charts are generated for display.

8. A system-oriented fault injection and simulation testing platform, characterized in that, The platform includes: The probe deployment module is used to register the topology information of the system under test after responding to all test task requests through the management service, and to deploy fault injection probes in the target server of the system under test. The data interaction module responds to the orchestration operations of the visual interface and determines the generation mode of the traffic strategy; A traffic injection module is used to inject background traffic into the system under test based on a test service that matches the generation pattern. The test execution module, based on the background traffic and preset orchestration logic, sends instructions to the fault injection probe of the target server to perform fault injection and simulation testing.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images