A service mesh-based adaptive fault injection method and system, electronic device and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SMALL BOATS SAILING EDUCATION TECH (BEIJING) CO LTD
- Filing Date
- 2026-03-12
- Publication Date
- 2026-06-30
Smart Images
Figure CN122316918A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of cloud computing, cloud-native architecture, and software resilience and reliability engineering technology. Specifically, it relates to an adaptive fault injection method, system, electronic device, and storage medium based on service mesh. Background Technology
[0002] In modern cloud-native microservice architectures, with rapid business iteration and dramatic expansion of system scale, the dependencies between services become increasingly complex, and the potential points of failure grow exponentially. To ensure the stability and reliability of complex distributed systems in production environments, chaos engineering proactively injects faults into the system to verify its resilience and recovery capabilities in the face of unpredictable production environment failures.
[0003] Current mainstream chaos engineering tools typically run as external systems, independent of the service under test, injecting faults by calling remote interfaces or executing scripts at the infrastructure layer. However, such existing technical solutions generally suffer from the following technical drawbacks:
[0004] The operation is highly complex and lacks automation. Most existing external chaos engineering tools require manual configuration by operations or development personnel based on experience, including setting the target, timing, intensity, and scope of fault injection. In a dynamically changing production environment, this model, which relies on manual judgment and intervention, is difficult to achieve continuous and refined automated resilience verification. It not only consumes a lot of manpower but is also prone to introducing risks due to improper configuration.
[0005] The lack of dynamic linkage and closed-loop feedback mechanisms with the real-time health status of services leads to uncontrollable security risks. Traditional chaos experiments, once initiated, typically execute according to a pre-set script, unable to dynamically adjust based on the real-time operational status of the target service (such as changes in service level agreement metrics). If the injected fault is too severe or the timing is inappropriate, injecting network latency into core services during peak business periods, the lack of automated security boundary protection mechanisms can easily cause localized faults to escalate into global cascading failures, posing a serious and uncontrollable threat to the stability of core business operations. While existing technologies offer solutions to terminate exercises by pre-setting steady-state indicators, this mechanism is usually a hard stop switch, lacking smooth, adaptive adjustment capabilities. Furthermore, its monitoring and injection execution are separated, resulting in significant response latency and failing to achieve tight coupling with data plane traffic.
[0006] Therefore, how to provide a chaos injection scheme that can achieve non-intrusive, dynamic adaptive decision-making capabilities in a service mesh environment and effectively prevent uncontrollable risks caused by chaos experiments has become a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0007] This invention provides an adaptive fault injection method, system, electronic device, and storage medium based on service mesh, which solves the technical problems of existing chaos engineering tools being complex to operate, lacking dynamic safety boundaries, and having difficulty in real-time risk management.
[0008] To achieve the above objectives, the first aspect of the present invention provides an adaptive fault injection method based on a service mesh, comprising:
[0009] Receive an injection method issued by the control plane, wherein the injection method defines the target service, fault type, and service health security boundary threshold;
[0010] By deploying the Sidecar agent within the target service, the service health indicators that characterize the running status of the target service are monitored in real time, and the service health indicators are reported to the real-time analysis engine.
[0011] The real-time analysis engine makes adaptive decisions based on the security boundary threshold defined in the injection method and the real-time reported service health indicators to generate fault injection instructions or safe exit instructions.
[0012] The fault injection command or the safe exit command is sent to the Sidecar agent;
[0013] The Sidecar agent executes the received instructions to perform fault injection or stop fault injection on network traffic flowing through the target service.
[0014] A second aspect of the present invention provides an adaptive fault injection system based on a service mesh, comprising:
[0015] The control plane is configured to define and issue injection logic, which defines the target service, fault type, and service health security boundary threshold.
[0016] At least one Sidecar agent is deployed along with the target service. The Sidecar agent is configured to monitor service health metrics that characterize the running status of the target service in real time and execute fault injection instructions or safe exit instructions.
[0017] The real-time analysis engine is communicatively connected to the control plane and the Sidecar agent. The real-time analysis engine is configured to receive the service health indicators reported by the Sidecar agent, and make adaptive decisions based on the security boundary threshold defined in the injection logic and the service health indicators to generate the fault injection instruction or the safe exit instruction, and send the generated instruction to the Sidecar agent.
[0018] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in the first aspect of the present invention.
[0019] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect of the present invention.
[0020] The fifth aspect of the present invention provides a computer program product including computer instructions that, when executed by a processor, implement the method as described in the first aspect of the present invention.
[0021] The beneficial effects of this invention are as follows:
[0022] This application embeds fault injection capabilities into the Sidecar agent of the service mesh data plane and combines it with a real-time analytics engine for automated decision-making. This architecture enables resilience verification to run continuously and automatically 24 / 7 in a production environment with low risk and a low blast radius, automatically discovering system weaknesses without continuous manual intervention from operations personnel, thus solving the technical challenges of complex operation of traditional tools.
[0023] This application constructs an adaptive closed-loop feedback mechanism based on real-time service health indicators. By pre-setting safety boundary thresholds, it ensures that chaotic experiments are always conducted within the system's tolerable safety range. Once the target service health indicators deteriorate and reach the safety threshold, the system can quickly and automatically pause or stop fault injection, thereby effectively avoiding cascading failures caused by uncontrolled experiments and solving the major drawback of uncontrollable risks in existing technologies.
[0024] Because the fault injection capability is embedded in the Sidecar proxy, the technical solution of this application is completely transparent to the business application code and does not intrude in any way. At the same time, it can fully utilize Sidecar's natural interception and fine-grained control capabilities over network traffic from Layer 4 to Layer 7 to achieve precise fault injection targeting specific API paths, specific request header fields, or specific percentages of traffic. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1This is a structural block diagram of an adaptive fault injection system provided in an embodiment of the present invention.
[0027] Figure 2 This is a flowchart of an adaptive fault injection method provided in an embodiment of the present invention.
[0028] Figure 3 This is a schematic diagram of system interaction in a specific implementation scenario provided by an embodiment of the present invention. Detailed Implementation
[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0031] Example 1
[0032] This embodiment provides an adaptive fault injection system and method based on a service mesh. The system and method are used to achieve secure, continuous, and automated system resilience verification by sinking fault injection capabilities to the data plane of the service mesh and constructing a closed-loop feedback control based on real-time service health.
[0033] Reference Figure 1 This embodiment provides a structural block diagram of the adaptive fault injection system 100. The system 100 is deployed on a service mesh infrastructure. The system 100 runs on one or more computing nodes, each including hardware resources such as processors and memory. The system 100 mainly includes a control plane 101, a real-time analytics engine 102, and at least one Sidecar agent 103.
[0034] Control plane 101 is the management and configuration center for the entire system. Its configuration is used to define and deploy injection logic. Specifically, operations personnel or developers can create and manage chaos experiments through the user interface or API of control plane 101. An injection logic typically contains the following key information:
[0035] • Define the application service that serves the purpose of this fault injection experiment.
[0036] • Define the specific fault mode to be injected. In this embodiment, the fault type may include network latency injection, network packet loss injection, or HTTP layer error response injection. It should be noted that although this embodiment uses the above types as examples, the fault types of this application are not limited to these. In other embodiments, it may also be CPU or memory resource saturation, disk I / O abnormalities, etc., as long as the fault can simulate potential abnormalities in the production environment.
[0037] • Service health safety boundary threshold. This is the core parameter for implementing adaptive safety control. This threshold defines the bottom line of performance degradation that the target service can tolerate. The safety boundary threshold for P99 latency can be set to 200ms, or the safety boundary threshold for error rate can be set to 1%. Once the real-time metrics of the service exceed this threshold, it indicates that the system health has reached the safety red line.
[0038] The control plane 101 typically contains a database (not shown in the figure) for storing configurations and experimental results, and sends the configured injection logic to the real-time analysis engine 102 via a network.
[0039] Sidecar Proxy 103 is the execution unit for fault injection. It is deployed in the same container pod as the target business service (not shown in the diagram) in a Sidecar pattern. In a service mesh architecture, Sidecar Proxy 103 naturally intercepts all network traffic entering and leaving its accompanying business service. In this application, Sidecar Proxy 103 is given two core functions:
[0040] 1. Through its built-in traffic monitoring capabilities, the system collects and calculates service health indicators that characterize the operational status of the target service in real time. These indicators serve as the basis for decision-making. In this embodiment, the service health indicators specifically include at least one of the following: P99 latency, error rate, or throughput. The collected indicator data is periodically reported to the real-time analysis engine 102. In a preferred implementation, this reporting channel can employ a high-throughput message queue system to ensure the real-time performance and reliability of the data.
[0041] 2. Receive and execute instructions issued by the real-time analysis engine 102. Instructions are of two types: fault injection instructions and safe exit instructions. A fault injection instruction is "Inject 50ms of network latency into 1% of inbound traffic." Upon receiving this instruction, the Sidecar agent 103 will utilize its built-in fault injection module to modify or process the intercepted network traffic to achieve fault injection. The safe exit instruction, on the other hand, commands the Sidecar agent 103 to immediately stop all ongoing fault injection activities and resume normal traffic forwarding.
[0042] The real-time analytics engine 102 is the system's decision-making brain. It communicates with the control plane 101 and the Sidecar agent 103, and is responsible for making real-time, adaptive decisions. Its workflow is as follows:
[0043] 1. Receive the injected logic from control plane 101 to clarify the experimental objectives and safety boundaries.
[0044] 2. Continuously receive real-time service health indicator data reported by one or more Sidecar agents 103.
[0045] 3. Make core decisions based on the security boundary thresholds defined in the injection logic and the real-time reported service health indicators. The decision logic is as follows: if the current service health indicators are within the security boundary thresholds, the service is considered healthy and fault injection can be safely performed, and a corresponding fault injection command is generated. Conversely, if the current service health indicators exceed or are about to exceed the security boundary thresholds, the service is considered to be in an unstable state, and injection must be stopped immediately to prevent the risk from escalating, and a safe exit command is generated.
[0046] 4. Send the generated instructions (fault injection or safe exit) to the corresponding Sidecar agent 103 for execution.
[0047] Through the collaborative work of the above three components, this application constructs a complete adaptive closed-loop control system that integrates "configuration distribution -> status monitoring -> intelligent decision-making -> command execution -> status feedback".
[0048] Combination Figure 2 The process of an adaptive fault injection method provided in this embodiment is described in detail.
[0049] After the system starts up, the control plane 101 first receives the defined injection method. This method defines the target service of the experiment, the type of fault to be injected, and the critical service health security boundary thresholds. This method is distributed and cached in the real-time analysis engine 102 as the basis for subsequent decisions.
[0050] The Sidecar proxy 103, deployed within the target service container, leverages its network traffic interception capabilities to continuously monitor key performance indicators of the service. Specifically, the Sidecar proxy 103 may include a traffic interception module configured to intercept Layer 4 to Layer 7 network traffic of the target service. By analyzing this traffic, service health metrics such as P99 latency, error rate, and throughput can be accurately calculated. This metric data is reported in real-time to the real-time analytics engine 102 via a message queue.
[0051] The real-time analytics engine 102 is the core of the decision-making process. It compares the real-time metric values received from the Sidecar agent 103 with the security boundary thresholds received from the control plane 101. If the real-time metric value is less than the security boundary threshold, it indicates that the service is healthy, and the real-time analytics engine 102 generates a fault injection command. If the real-time metric value is greater than or equal to the security boundary threshold, it indicates that the service is abnormal, and the real-time analytics engine 102 generates a safe exit command.
[0052] In a preferred embodiment, to achieve more refined and proactive risk control, the real-time analysis engine 102 may include a neural network model. This neural network model is trained offline based on historical service health metric data and fault injection records, enabling it to learn and predict the state change trends of the target service under different fault injection patterns. In this case, the adaptive decision-making process further includes: the neural network model predicts the service health metric within a future time window after executing the planned fault injection instruction, based on the current health metric and the planned fault injection instruction. If the predicted future metric exceeds the safety boundary threshold, even if the current metric remains safe, the engine will generate a safe exit instruction in advance, thereby achieving predictive risk avoidance and improving system security.
[0053] The real-time analysis engine 102 sends the decision-generated instructions to the Sidecar agent 103 corresponding to the target service. The Sidecar agent 103 executes these instructions. When executing an HTTP error injection instruction, the fault injection module of the Sidecar agent 103 returns a pre-defined HTTP 503 error response to Layer 7 network traffic that meets specific conditions (the requested API path or traffic containing specific fields in the request header), while other traffic is forwarded normally. This fine-grained Layer 7 injection capability ensures that the impact of fault injection is precisely controlled to a minimum.
[0054] After executing the command, the Sidecar agent 103 will also report the execution result status to the control plane 101, forming a complete control and feedback closed loop, which makes it easier for maintenance personnel to track the experimental status and audit it.
[0055] Furthermore, to further enhance the robustness of the system, this method can also add a resource verification step. Specifically, before generating fault injection instructions, the real-time analysis engine 102 verifies the resource utilization of the cluster nodes where the target service resides. If the resource utilization of the cluster nodes exceeds a preset resource safety threshold, the real-time analysis engine 102 will pause generating any fault injection instructions until the resource utilization returns to within the safety threshold. This effectively prevents the injection experiment itself from consuming excessive resources and affecting the normal operation of other services on the node.
[0056] Example 2
[0057] This embodiment, based on Embodiment 1, provides further optimizations or alternatives to the specific implementation of fault injection and the algorithm logic for adaptive decision-making.
[0058] A fault injection implementation based on eBPF
[0059] In Implementation Example 1, fault injection capability is embedded within the user-space process of the Sidecar agent. Alternatively, fault injection can be implemented at the operating system kernel level using eBPF technology.
[0060] In this scheme, the system also includes an eBPF program deployed in the operating system kernel. This eBPF program is dynamically loaded, updated, and unloaded by the Sidecar agent 103 according to instructions from the real-time analysis engine 102. When network fault injection is required, the Sidecar agent 103 mounts a compiled eBPF program onto the TC processing path of the network device associated with the target service network namespace. This eBPF program can directly intercept, modify, or drop network packets in kernel mode, thereby efficiently achieving fault injection.
[0061] Compared to implementing it in the Sidecar user space, the eBPF-based approach has a shorter path, lower performance overhead, and less intrusion into the Sidecar's own business logic, providing an alternative, efficient implementation path. This solution effectively supports the aforementioned technical features of using eBPF for fault injection.
[0062] An Adaptive Decision Logic Based on Proportional Control
[0063] The decision logic based on a hard "safety boundary threshold" described in Example 1 is a start / stop switch. To achieve smoother, more intelligent adaptive control, the decision logic can employ a proportional control algorithm.
[0064] In this scheme, the real-time analysis engine 102 includes a proportional controller. This proportional controller is configured to make the strength of fault injection inversely proportional to the service health level represented by the service health index, and to smoothly and dynamically adjust as the service health level changes.
[0065] The specific implementation is as follows:
[0066] A standardized comprehensive service health score is obtained by weighting multiple dimensions of service health indicators. The score The value range can be normalized to the interval [0, 1], and the higher the value, the healthier the service.
[0067] Based on the overall service health score To determine the strength of the fault injection. .strength This can refer to latency, packet loss rate, or percentage of injected traffic. Intensity With score It satisfies a pre-defined inverse proportional function relationship.
[0068] To better support the aforementioned technical feature of configuring the inverse proportional function relationship as a piecewise function, a preferred piecewise function implementation is presented below, which combines the smoothness of proportional control with the reliability of hard safety boundaries.
[0069] In control plane 101, two boundary values for health scores are preset: the lower boundary of the health interval. and the lower boundary of the warning zone .
[0070] The proportional controller is specifically configured as follows:
[0071] 1. When the overall service health score is... When in the first preset health range (i.e. ≥ This indicates that the service is very healthy, at which point fault injection strength can be applied. Set to the preset maximum strength value This is to conduct thorough toughness verification.
[0072] 2. When the overall service health score is... When in the second preset warning zone (i.e. ≤ < This indicates that the service health is starting to decline, requiring caution when injecting faults. At this point, the fault injection strength... It will decay linearly with the decrease in health. Specifically, this calculation logic is implemented through the following formula:
[0073]
[0074] in, This indicates the calculated current fault injection strength; Indicates the maximum injection intensity; This indicates the current overall health score of the service; and These represent the upper and lower boundaries of the warning zone, respectively. From this formula, it can be seen that when... from Towards During descent, from It decays smoothly to 0.
[0075] 3. When the overall service health score is... Below the lower boundary of the second preset warning range Time (i.e.) < This indicates that the service is in a dangerous state and must be stopped immediately. At this point, the fault injection strength... Setting it to zero is equivalent to executing a safe exit command.
[0076] By adopting this proportional control logic, the system can achieve dynamic and smooth adjustment of the fault injection intensity, avoiding sudden changes in injection intensity caused by hard thresholds, making the chaotic experiment process more stable and safe.
[0077] Example 3
[0078] Reference Figure 3 This embodiment describes a specific application scenario that combines the above-mentioned technical solutions.
[0079] Figure 3 This is a schematic diagram of system interaction in a specific implementation scenario. In the diagram, the "Management Platform" corresponds to the aforementioned control plane 101. The "AI Real-time Analysis" node corresponds to the real-time analysis engine 102. pod1 and pod2 are two container pods running in the cluster. Each pod contains a "Business Service" and an accompanying "Sidecar Agent" (corresponding to Sidecar Agent 103).
[0080] The entire workflow is as follows:
[0081] 1. Operations personnel set up logic on the "Management Platform," defining the "safe exit threshold" and configuring the weights and algorithms for "health score calculation." The platform can also configure "timed switches" to control the start and end times of experiments, as well as "alarm" rules.
[0082] 2. The Sidecar broker, running in pod1 and pod2, continuously monitors the performance metrics and service status of its associated business services. These key decision-making metrics are pushed to the "AI Real-Time Analytics" engine via message queues.
[0083] 3. The "AI Real-Time Analytics" engine receives logic from the management platform and real-time metrics from the Sidecar agent. Based on these inputs, it makes adaptive injection decisions.
[0084] 4. Based on the decision results, the "AI Real-Time Analysis" engine sends specific fault injection instructions to the downstream Sidecar agent. It sends an instruction of "1% traffic delay 50ms" to the Sidecar agent of pod1. At the same time, it may also execute more aggressive fault injection according to logic, sending an instruction of "pod destruction and reconstruction" to the node where pod2 is located to verify the self-healing capability of the service.
[0085] 5. Service status changes after the Sidecar agent executes instructions will be reported again through the message queue, forming a continuous and dynamically adjusted closed loop. If the overall service P99 latency of pod1 reaches the "safe exit threshold" after a 50ms delay, the "AI real-time analysis" engine will stop or reduce fault injection into pod1 in the next round of decision-making.
[0086] This scenario vividly demonstrates how the system proposed in this application works in a real cloud-native environment, achieving a secure, intelligent, and automated chaos engineering experiment through the collaboration of various components.
[0087] Finally, it should be noted that the above embodiments can be combined with each other to form more implementation methods. The eBPF injection method in Embodiment 2 can be used in conjunction with proportional control decision logic.
[0088] The solutions in this application embodiment can be implemented by an electronic device. This electronic device can be one or more servers. When there are multiple servers, it can be a distributed system. The electronic device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus. The memory is used to store computer programs. The processor is used to execute the computer programs stored in the memory to implement the steps in the foregoing method embodiments.
[0089] In summary, this invention effectively solves the technical challenges of complex operation and uncontrollable risks in existing chaos engineering tools by embedding chaos injection capabilities into the service mesh Sidecar agent and constructing an adaptive closed-loop feedback control system based on real-time service health indicators. This solution enables continuous, automated, and low-risk system resilience verification in a production environment, which has significant technical value and practical implications for ensuring the stability and reliability of complex distributed systems.
[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for adaptive fault injection based on service mesh, characterized in that, include: Receive an injection method issued by the control plane, wherein the injection method defines the target service, fault type, and service health security boundary threshold; By deploying the Sidecar agent within the target service, the service health indicators that characterize the running status of the target service are monitored in real time, and the service health indicators are reported to the real-time analysis engine. The real-time analysis engine makes adaptive decisions based on the security boundary threshold defined in the injection method and the real-time reported service health indicators to generate fault injection instructions or safe exit instructions. The fault injection command or the safe exit command is sent to the Sidecar agent; The Sidecar agent executes the received instructions to perform fault injection or stop fault injection on network traffic flowing through the target service.
2. The method according to claim 1, characterized in that, The service health metrics include at least one of the following: P99 latency, error rate, or throughput.
3. The method according to claim 1, characterized in that, The fault types defined in the injection method include at least one of the following: network delay injection, network packet loss injection, or HTTP layer error response injection; Optionally, the Sidecar proxy can monitor the service health indicators and implement the fault injection by intercepting the Layer 4 to Layer 7 network traffic of the target service; Optionally, the fault injection is precisely applied to Layer 7 network traffic that meets specific conditions, including a specific API path or a specific request header field.
4. The method according to claim 1, characterized in that, The real-time analysis engine includes a neural network model, which is trained based on historical service health index data to predict the state change trend of the target service under different fault injections. Optionally, the adaptive decision-making further includes: the neural network model predicts the future service health index after executing the fault injection instruction based on the current service health index and the fault injection instruction; if the predicted future service health index exceeds the security boundary threshold, then the safe exit instruction is generated to replace the fault injection instruction. Optionally, the fault injection is implemented by intercepting and modifying network packets using eBPF technology at the operating system kernel layer; Optionally, the adaptive decision-making adopts proportional control logic, wherein the intensity of the fault injection is inversely proportional to the service health level represented by the service health index, and is smoothly and dynamically adjusted as the service health level changes. Optionally, the smooth dynamic adjustment of the fault injection intensity includes: A standardized comprehensive service health score is calculated based on the service health indicators from multiple dimensions. ,in The value range is [0, 1], and the higher the value, the healthier the service; Based on the comprehensive service health score Determine the fault injection strength The strength therein With the score It satisfies an inverse proportional function relationship; Optionally, the inverse proportional function relationship is configured as a piecewise function, specifically including: When the overall health score of the service When the fault injection intensity is within the first preset health range, Set as maximum strength value ; When the overall health score of the service When the fault injection intensity is within the second preset warning range, According to the formula Perform linear decay, where and These are the upper and lower boundaries of the second preset warning zone, respectively; When the overall health score of the service Below the lower boundary of the second preset warning range At that time, the fault injection strength Setting it to zero means performing a safe exit.
5. The method according to claim 1, characterized in that, After executing the instruction, the Sidecar agent reports the execution result status, including success, failure, or abort, to the control plane, forming a control closed loop. Optionally, the security boundary threshold is dynamic, and the control plane periodically updates and distributes the security boundary threshold to the real-time analysis engine based on the periodic characteristics of external business traffic or the level of business importance. Optionally, the method further includes: Before the real-time analysis engine generates the fault injection instruction, the resource utilization of the cluster node where the target service is located is verified. If the resource utilization rate of the cluster node exceeds a preset resource safety threshold, the real-time analysis engine will suspend the generation of any fault injection instructions until the resource utilization rate recovers to within the resource safety threshold.
6. An adaptive fault injection system based on service mesh, characterized in that, include: The control plane is configured to define and issue injection logic, which defines the target service, fault type, and service health security boundary threshold. At least one Sidecar agent is deployed along with the target service. The Sidecar agent is configured to monitor service health metrics that characterize the running status of the target service in real time and execute fault injection instructions or safe exit instructions. The real-time analysis engine is communicatively connected to the control plane and the Sidecar agent. The real-time analysis engine is configured to receive the service health indicators reported by the Sidecar agent, and make adaptive decisions based on the security boundary threshold defined in the injection logic and the service health indicators to generate the fault injection instruction or the safe exit instruction, and send the generated instruction to the Sidecar agent.
7. The system according to claim 6, characterized in that, The service health metrics include at least one of the following: P99 latency, error rate, or throughput; Optionally, the fault types defined in the injection logic include at least one of the following: network latency injection, network packet loss injection, or HTTP layer error response injection; Optionally, the Sidecar proxy includes a traffic interception module configured to intercept Layer 4 to Layer 7 network traffic of the target service; Optionally, the Sidecar proxy further includes a fault injection module, which is configured to perform fault injection on Layer 7 network traffic that is intercepted by the traffic interception module and meets the conditions of a specific API path or a specific request header field.
8. The system according to claim 6, characterized in that, The real-time analysis engine includes a neural network model trained based on historical service health index data. The neural network model is used to predict the state change trend of the target service under different fault injections. Optionally, the system further includes an eBPF program deployed in the operating system kernel, which is loaded and controlled by the Sidecar agent to intercept and modify network packets to achieve fault injection. Optionally, the real-time analysis engine includes a proportional controller configured to calculate a fault injection intensity that is inversely proportional to the service health level based on the service health index, and generate the fault injection instruction accordingly. Optionally, the proportional controller is further configured to: A standardized comprehensive service health score is calculated based on the service health indicators from multiple dimensions. ; Based on the comprehensive service health score Determine the fault injection strength The strength therein With the score It satisfies the preset inverse proportional function relationship; Optionally, the proportional controller is characterized by having a preset inverse proportional function relationship as a piecewise function, and the proportional controller is specifically configured as follows: When the overall health score of the service When the body is in the first preset health range, generate a value with the maximum intensity. Fault injection instructions; When the overall health score of the service When the system is within the second preset warning range, a fault injection command with linearly decreasing intensity is generated, wherein the intensity... According to the formula The calculation shows that, among which and These are the upper and lower boundaries of the second preset warning zone, respectively; When the overall health score of the service Below the lower boundary At that time, a safe exit command is generated; Optionally, the Sidecar agent is also configured to report the status of its instruction execution results to the control plane; Optionally, the real-time analysis engine further includes a resource verification unit, which is configured to verify the resource utilization of the cluster node where the target service is located before the real-time analysis engine generates a fault injection instruction, and to suppress the generation of the fault injection instruction when the resource utilization exceeds a preset threshold.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method of any one of claims 1 to 5.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 5; Optionally, a computer program product includes computer instructions that, when executed by a processor, implement the method of any one of claims 1 to 5.