Path-aware fault handling method for cloud computing environment

Abstract

The application provides a path-aware fault processing method for a cloud computing environment, and relates to the technical fields of cloud computing, computer network security and system reliability. The method comprises the following steps: obtaining a real end-to-end network path of target service traffic in a network communication system for cloud computing, and representing the real end-to-end network path as an ordered path sequence; determining a fault type of a target fault to be injected and a target path node corresponding to the target fault in the ordered path sequence based on a preset fault model and the real end-to-end network path, generating a fault specification based on the fault type, the position of the target path node and the fault parameters of the target fault, and controlling the target path node to inject a fault injection primitive corresponding to the target fault; and in the case of injecting the fault injection primitive, performing correlation analysis on running data and the fault injection primitive to generate an influence analysis result of the target fault. The technical scheme of the application improves the accuracy of fault influence analysis.

Description

Technical Field

[0001] This application relates to the fields of cloud computing, computer network security and system reliability technology, and in particular to a path-aware fault handling method for cloud computing environments. Background Technology

[0002] With the popularization of cloud computing technology, business requests in various cloud platforms need to pass through complex virtual network environments built by software-defined networking. The forwarding path involves multiple components such as virtual switches, routers, tunnel encapsulation and load balancers, and is affected by mechanisms such as equal-cost multi-path (ECMP), network address translation (NAT), and dynamic scheduling, which affects the reliability of cloud computing systems.

[0003] To improve the reliability of cloud computing systems, various fault injection techniques have been proposed. One type is the fault injection framework represented by chaos engineering, which actively introduces anomalies such as node failures and network latency to test the system's fault tolerance. Another type uses network observability and diagnostic techniques, or static configuration analysis tools, to build network connectivity views or verify the correctness of configuration strategies.

[0004] However, the fault injection technique described above lacks path awareness and is difficult to accurately match the dynamic and ever-changing real business paths in the cloud computing environment. As a result, the injected faults cannot accurately correspond to the real business paths, making it impossible to conduct accurate fault impact analysis. Summary of the Invention

[0005] This application provides a path-aware fault handling method for cloud computing environments to address the shortcomings of existing fault injection methods that lack path awareness, resulting in the injected faults not accurately corresponding to the real business paths. This method enables the injected faults to accurately correspond to the real business paths, thereby improving the accuracy of fault impact analysis.

[0006] This application provides a path-aware fault handling method for cloud computing environments, including: Obtain the actual end-to-end network path of the target service traffic in the cloud-oriented network communication system, and represent the actual end-to-end network path as an ordered path sequence including multiple path nodes; Based on the preset fault model and the real end-to-end network path, the fault type of the target fault to be injected and the target path node corresponding to the target fault in the ordered path sequence are determined, and a corresponding fault specification is generated based on the fault type, the position of the target path node and the fault parameters of the target fault. Based on the fault specification, the target path node is controlled to inject the fault injection primitive corresponding to the target fault. The fault injection primitive is used to simulate the fault behavior at the target path node. When the fault injection primitive is injected, the operating data of the network communication system is obtained, and the operating data and the fault injection primitive are correlated to generate the impact analysis results of the target fault.

[0007] According to the path-aware fault handling method for cloud computing environments provided in this application, the ordered path sequence includes at least one of virtual machine network card interface, container network interface, virtual switch port, virtual router port, tunnel encapsulation endpoint, host physical network card interface, physical switch port, and physical router port.

[0008] According to the path-aware fault handling method for cloud computing environments provided in this application, the fault specification includes at least one of the following: The fault type field is used to indicate the fault type of the target fault; The fields of the fault parameters are used to describe the fault parameters of the target fault, including at least one of fault intensity, fault duration, or related fault configuration parameters; The fault injection constraint field is used to indicate the location constraints of the target path node and / or the network hierarchy constraints to which the target path node belongs; The recovery information field describes the recovery method and triggering conditions after the target fault ends.

[0009] According to the path-aware fault handling method for cloud computing environments provided in this application, the fault injection primitives include injecting erroneous forwarding table entries, modifying routing configurations, introducing packet loss or delay, modifying tunnel parameters, terminating network component processes, or simulating physical node and link failures.

[0010] According to the path-aware fault handling method for cloud computing environments provided in this application, the step of determining the fault type of the target fault to be injected, and the target path node corresponding to the target fault in the ordered path sequence, based on a preset fault model and the real end-to-end network path, includes: Dependency analysis is performed on the path nodes in the ordered path sequence to identify the key components in the multiple path nodes and the dependencies between the key components. Based on the dependencies between the key components and the preset priority strategy, the fault injection priority corresponding to each key component is determined. Based on the preset fault injection constraints, the fault injection constraint range corresponding to each of the key components is determined. Based on the fault injection priority and fault injection constraint range corresponding to each of the key components, the fault type of the target fault is selected from the preset fault model, and the target path node is selected from the ordered path sequence.

[0011] According to the path-aware fault handling method for cloud computing environments provided in this application, the step of determining the fault injection priority corresponding to each of the key components based on the dependencies between the key components and a preset priority strategy includes: Based on the dependencies between the key components, the structural importance weight of each key component is determined. Based on a preset priority strategy, the fault injection target corresponding to the target service traffic is determined; For each of the key components, obtain the historical fault injection results corresponding to the key components, and dynamically adjust the fault injection priority corresponding to the key components based on the historical fault injection results, the structural importance weight of the key components, and the fault injection target.

[0012] According to the path-aware fault handling method for cloud computing environments provided in this application, the step of dynamically adjusting the fault injection priority corresponding to the key components based on the historical fault injection results, the structural importance weights of the key components, and the fault injection target includes: If the number of historical verifications of a critical component exceeds a preset threshold and the historical performance stability score is greater than a preset stability score threshold, the structural importance weight of the critical component is lower than a first preset weight threshold, and the fault injection target is coverage priority or risk avoidance, the corresponding fault injection priority shall be reduced. If the historical failure impact score of the key component is greater than the preset impact threshold, the structural importance weight of the key component is higher than the second preset weight threshold, and the failure injection target is impact verification priority, the corresponding failure injection priority is increased. If the critical component is located in a path region where the historical fault injection coverage is less than the coverage threshold, the structural importance weight of the critical component is higher than the third preset weight threshold, and the fault injection target is coverage priority, the corresponding fault injection priority is increased.

[0013] According to the path-aware fault handling method for cloud computing environments provided in this application, the step of controlling the injection of fault injection primitives corresponding to the target fault into the target path node based on the fault specification includes: The fault specification is parsed to obtain a fault injection task, which includes at least one of the following: the identifier of the target path node, the location of the target path node, the identifier of the fault type, and the fault parameter or the fault duration. Based on the fault injection task, a fault injection instruction is sent to the target path node. The fault injection instruction includes the fault injection primitive, and the fault injection instruction is used to control the distributed proxy component in the target path node to inject the fault injection primitive.

[0014] According to the path-aware fault handling method for cloud computing environments provided in this application, the method further includes: Receive the state information before the fault injection primitive is injected, and the fault injection result after the fault injection primitive is injected, sent by the target path node; The status information includes at least one of the following: configuration file, forwarding table entry, or interface status.

[0015] According to the path-aware fault handling method for cloud computing environments provided in this application, the method further includes: Based on the preset recovery strategy and / or the recovery information in the fault specification, a recovery command is sent to the target path node; The recovery instruction is used to instruct the distributed proxy component to perform a recovery operation corresponding to the fault injection primitive based on the state information before the fault injection primitive was injected.

[0016] According to the path-aware fault handling method for cloud computing environments provided in this application, the step of performing correlation analysis between the runtime data and the fault injection primitives to generate the impact analysis results of the target fault includes: For each of the multiple path nodes, a correlation analysis is performed between the operational data and the fault injection primitives to determine the impact analysis parameters corresponding to the path node, and an impact score is determined based on the impact analysis parameters. The impact analysis parameters include: normalized performance degradation values ​​and durations of performance degradation for multiple key analysis indicators at the path node; numerical severity values ​​of abnormal events observed at the path node; and the propagation impact of the fault caused by the path node within the path. Based on the impact scores of each path node, the impact analysis results of the target fault are obtained.

[0017] According to the path-aware fault handling method for cloud computing environments provided in this application, the step of obtaining the actual end-to-end network path of the target service traffic in a cloud computing-oriented network communication system includes: Obtain a network path view pre-constructed by fusing information from multiple sources; Based on the target service traffic, the network path view lookup determines the actual end-to-end network path.

[0018] This application also provides a path-aware fault handling device for a cloud computing environment, comprising: The acquisition unit is used to acquire the actual end-to-end network path of the target service traffic in the cloud computing-oriented network communication system, and represent the actual end-to-end network path as an ordered path sequence including multiple path nodes; The processing unit is used to determine the fault type of the target fault to be injected and the target path node corresponding to the target fault in the ordered path sequence based on the preset fault model and the real end-to-end network path, and to generate a corresponding fault specification based on the fault type, the position of the target path node and the fault parameters of the target fault. The control unit is configured to, based on the fault specification, control the injection of fault injection primitives corresponding to the target fault into the target path node, wherein the fault injection primitives are used to simulate the fault behavior at the target path node. The analysis unit is used to obtain the operating data of the network communication system when the fault injection primitive is injected, and to perform correlation analysis between the operating data and the fault injection primitive to generate the impact analysis results of the target fault.

[0019] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the path-aware fault handling method for cloud computing environments as described in any of the preceding claims.

[0020] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the path-aware fault handling method for a cloud computing environment as described in any of the preceding claims.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the path-aware fault handling method for cloud computing environments as described in any of the preceding claims.

[0022] The path-aware fault handling method for cloud computing environments provided in this application, when performing fault injection and analysis, obtains the actual end-to-end network path of the target service traffic in the network communication system and represents it as an ordered path sequence containing multiple path nodes. It retains key network layer information and forwarding stage information in the ordered path sequence, achieving port-level and stage-level characterization of the cloud environment path. This enables fault injection to have path-aware capabilities and accurately match dynamically changing real service paths. Based on a preset fault model and the actual end-to-end network path, it determines the fault type and target path nodes, generates a fault specification, and binds the fault model to specific path nodes in the actual end-to-end network path. This ensures that the fault type precisely corresponds to a specific forwarding position in the path, solving the problem of traditional methods that "only specify the fault type without specifying the location of action," and achieving precise fault orchestration. Based on the fault specification... The specification explains that by injecting the target fault into the target path node using the corresponding fault injection primitive, the fault is ensured to act precisely on the specified forwarding position of the real business path, achieving precise fault type location and controllable injection. After fault injection, by acquiring operational data and performing correlation analysis with the fault injection primitive, impact analysis results are generated. The observed system behavior changes are directly correlated with specific fault injection events, restoring the fault propagation process within the virtual network, making the fault injection results interpretable. This achieves controllable, reproducible, and interpretable fault injection and verification, solving the problem that traditional methods can only observe the business layer manifestations and cannot trace the root cause of the fault. By introducing path awareness, precise orchestration, controllable fault injection, and correlation analysis into the entire fault injection process, the precise correspondence between faults and real business paths and the interpretable analysis of fault impacts are achieved, significantly improving the accuracy of fault impact analysis. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of a path-aware fault injection system architecture provided in an embodiment of this application.

[0025] Figure 2 This is a flowchart illustrating a path-aware fault handling method for a cloud computing environment, as provided in an embodiment of this application.

[0026] Figure 3 This is a flowchart illustrating a method for determining the fault type and target path node of a target fault to be injected, as provided in an embodiment of this application.

[0027] Figure 4 This is a flowchart illustrating a method for injecting fault injection primitives corresponding to target faults into target path nodes, as provided in an embodiment of this application.

[0028] Figure 5 This is a schematic flowchart illustrating a method for performing correlation analysis between runtime data and fault injection primitives to generate impact analysis results for a target fault, as provided in an embodiment of this application.

[0029] Figure 6 This is a schematic diagram of the structure of a path-aware fault handling device for a cloud computing environment, provided in an embodiment of this application.

[0030] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Network communication for cloud computing typically operates on a complex network architecture comprised of overlays of virtual and physical networks. Unlike traditional data centers or single physical network environments, cloud platforms, in order to achieve tenant isolation, elastic scaling, and unified operation and maintenance, generally employ virtual switching, virtual routing, overlay network tunnels, and various network policy mechanisms in the data forwarding process. Therefore, the actual forwarding path of business traffic from source to destination often spans multiple network domains and multiple forwarding stages. This path comprises not only the virtual machine or container itself, but also key forwarding locations such as virtual switch ports, virtual router ports, tunnel endpoints, host physical network interface cards (NICs), and physical switching and routing devices.

[0034] However, existing fault injection techniques only consider a single node, a single link, or a single interface in a cloud computing network communication system, without characterizing the path at the port and stage levels. This makes fault injection lack path awareness and difficult to accurately match the dynamic and ever-changing real business paths in the cloud computing environment. Consequently, the injected faults cannot accurately correspond to the real business paths, making accurate fault impact analysis impossible.

[0035] To improve the accuracy of fault impact analysis, this application provides a path-aware fault handling method for cloud computing environments, which can be implemented through a path-aware fault injection system. For example, see... Figure 1 As shown, Figure 1 This application provides a schematic diagram of a path-aware fault injection system architecture. The path-aware fault injection system may include a cloud network path discovery module, a fault orchestration module, a fault injection execution module, a fault feasibility and impact analysis module, and a cloud platform. The fault injection execution module can be completed collaboratively by a centralized control component and a distributed agent component. The cloud network path discovery module, fault orchestration module, centralized control component, and fault feasibility and impact analysis module can be deployed on network devices. The distributed agent component can be deployed in nodes of the cloud platform, including but not limited to computing hosts, network nodes, or virtual machine instances when necessary, to perform specific fault injection and recovery operations locally. The cloud platform includes multiple path nodes. Taking the real end-to-end network path of business traffic in the cloud platform as the basic modeling object, the target of fault injection is expanded from the traditional single node, single link, or single interface to fine-grained locations on the path that can be accurately located to specific device ports, tunnel endpoints, and virtual / physical or cross-domain boundaries. Thus, in a cloud computing environment containing complex mechanisms such as virtualized networks, overlay networks and underlying networks, multi-level forwarding, encapsulation and decapsulation, address translation, equivalent multipath forwarding, and load balancing, controllable, reproducible, and interpretable fault injection and verification can be achieved.

[0036] The cloud network path discovery module is used to pre-construct a network path view by fusing information from multiple sources, primarily telemetry systems, monitoring systems, and user-inputted path clues. It merges and organizes the network paths that service traffic in the cloud platform may traverse, forming a unified network path view to provide a path foundation for subsequent fault analysis and injection. The fault orchestration module, upon receiving target service traffic, obtains the actual end-to-end network path of the target service traffic in the cloud-oriented network communication system and represents the actual end-to-end network path as an ordered sequence of path nodes. Based on a pre-set fault model and the actual end-to-end network path, it determines... The system defines the type of the target fault to be injected and the target path node corresponding to the target fault in the ordered path sequence. Based on the fault type, the location of the target path node, and the fault parameters of the target fault, it generates a corresponding fault specification. The fault injection execution module controls the injection of the fault injection primitive corresponding to the target fault into the target path node based on the fault specification. The fault injection primitive is used to simulate the fault behavior at the target path node. The fault feasibility and impact analysis module is used to obtain the operating data of the network communication system after injecting the fault injection primitive, and to perform correlation analysis between the operating data and the fault injection primitive to generate the impact analysis results of the target fault.

[0037] As can be seen, in this embodiment, by obtaining the actual end-to-end network path of the target service traffic in the network communication system and representing it as an ordered path sequence containing multiple path nodes, and retaining key network layer information and forwarding stage information in the ordered path sequence, port-level and stage-level characterization of the cloud environment path is achieved, enabling fault injection to have path awareness capabilities and accurately match dynamically changing real service paths; and based on the preset fault model and the actual end-to-end network path, the fault type and target path node are determined, a fault specification is generated, and the fault model is bound to the specific path node in the actual end-to-end network path, so that the fault type corresponds precisely to the specific forwarding position in the path, solving the problem of the traditional method "only specifying the fault type without specifying the position of action", and realizing precise fault orchestration; based on the fault specification, the target path node is controlled. By injecting fault injection primitives corresponding to the target fault, the fault is ensured to act precisely on the specified forwarding position of the real business path, achieving precise fault type location and controllable injection. After fault injection, by acquiring operational data and performing correlation analysis with the fault injection primitives, impact analysis results are generated. The observed system behavior changes are directly correlated with specific fault injection events, restoring the fault propagation process within the virtual network. This makes the fault injection results interpretable, achieving controllable, reproducible, and interpretable fault injection and verification. This solves the problem that traditional methods can only observe the business layer manifestations and cannot trace the root cause of the fault. By introducing path awareness, precise orchestration, controllable fault injection, and correlation analysis into the entire fault injection process, the precise correspondence between faults and real business paths and the interpretable analysis of fault impacts are achieved, significantly improving the accuracy of fault impact analysis.

[0038] It should be noted that the technical solutions provided in this application can be applied to various typical scenarios such as communication within the same host machine, communication across hosts machine, communication across subnets, and communication in multi-layer overlay networks. These include communication scenarios within the same physical node and the same subnet, communication scenarios within the same physical node and across subnets, communication scenarios across physical nodes and the same subnet, communication scenarios across physical nodes and across subnets, scenarios for accessing external networks (public networks) or external access to cloud services, and communication scenarios in multi-layer overlay networks or "containers within virtual machines".

[0039] In communication scenarios within the same physical node and subnet, service traffic typically does not need to traverse virtual routers or overlay network tunnels. Traffic is forwarded within the same host machine by a virtual switch. A typical path is as follows: the source virtual machine port enters a port on the virtual switch, is forwarded internally by the virtual switch, exits through another port, and arrives at the destination virtual machine port. This scenario demonstrates that even if both communicating parties are located on the same host machine, the path may still require precise description down to the specific port on the virtual switch.

[0040] In communication scenarios spanning multiple subnets within the same physical node, service traffic needs to be forwarded across subnets via virtual routing devices. A typical path is as follows: the source virtual machine port enters the virtual switching device and is then directed to the port connected to the virtual routing device; after routing and forwarding through the ingress and egress ports on the virtual routing device, it returns to the virtual switching device and is forwarded to the destination virtual machine port via the corresponding port. This scenario demonstrates that the virtual routing device port is a critical path location; the path is not merely "through a router," but requires further identification of "which pair of ports / which forwarding stage on the router."

[0041] In communication scenarios spanning physical nodes within the same subnet, the communicating parties reside on different hosts but are logically within the same subnet. Cloud platforms typically utilize overlay network tunnels to achieve Layer 2 / Layer 3 connectivity across hosts. A typical path unfolds as follows: after the source virtual machine port enters the source host's virtual switch, encapsulation occurs at the tunnel endpoint. Subsequently, it passes through the source host's physical network interface card (NIC) into the physical switch and is forwarded within the physical network. Upon reaching the peer host, decapsulation occurs at the peer tunnel endpoint, and finally, the peer virtual switch forwards the message to the destination virtual machine port. This scenario clearly demonstrates the "encapsulation / decapsulation phase" and "tunnel endpoint location" within the path; these phased processes are a crucial component of the cloud environment's communication path.

[0042] In communication scenarios spanning physical nodes and subnets, a combination of cross-subnet forwarding and cross-host tunneling mechanisms is employed, typically resulting in longer paths and more forwarding stages. A typical path unfolds as follows: the source virtual switch hands over traffic to a virtual router for cross-subnet forwarding, then sends it into an overlay network tunnel for encapsulation, forwards it through the physical network to the peer host for decapsulation, and finally enters the peer virtual switch for forwarding to the destination virtual machine port. This scenario demonstrates that in cloud environments, paths may simultaneously traverse multiple layers, including virtual switches, virtual routers, overlay networks, and physical networks, and configuration or forwarding anomalies at any of these layers can lead to service failures.

[0043] In scenarios involving access to external networks (public networks) or external access to cloud services, when a virtual machine in the cloud accesses an external network, the traffic typically needs to pass through virtual routers / gateways and further enter the physical network and external routing domain. In this scenario, the path may involve processing stages such as gateway ports, address translation, or security policies, and routing devices on the external network side may also become part of the path. This scenario emphasizes that the boundary locations of the cloud platform often bear more policy and translation logic, making them typical critical path locations.

[0044] In multi-layered overlay networks or "containers within virtual machines" communication scenarios, container networks and virtual machine networks may overlap to form a multi-layered overlay network structure. In this case, business traffic may first undergo virtual switching and tunneling encapsulation on the container network side, then enter virtual switching and tunneling encapsulation on the virtual machine / host side, and finally be transmitted to the other end via the physical network, where it is decapsulated layer by layer. This scenario demonstrates that in cloud-native environments, path cross-domain characteristics are stronger, and path modeling needs to simultaneously adapt to container-side virtual ports, virtual machine-side ports, and host and physical network ports.

[0045] It is understood that the execution subject of the path-aware fault handling method for cloud computing environment provided in this application can be a computer, a server, or a specially set fault impact analysis device, or a path-aware fault handling device for cloud computing environment set in the electronic device. The path-aware fault handling device for cloud computing environment can be implemented by software, hardware, or a combination of both, and can be set according to actual needs.

[0046] The path-aware fault handling method for cloud computing environments provided in this application will be described in detail below through several specific embodiments. It is understood that these specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0047] Figure 2 A flowchart illustrating a path-aware fault handling method for a cloud computing environment provided in this application embodiment is shown below. For example, please refer to... Figure 2 As shown, this path-aware fault handling method for cloud computing environments may include: S201. Obtain the actual end-to-end network path of the target service traffic in the cloud-oriented network communication system, and represent the actual end-to-end network path as an ordered path sequence including multiple path nodes.

[0048] For example, in the embodiments of this application, the ordered path sequence includes at least one of the following: virtual machine network interface card interface, container network interface, virtual switch port, virtual router port, tunnel encapsulation endpoint, host physical network interface card interface, physical switch port, and physical router port.

[0049] For example, in this embodiment of the application, when the cloud network path discovery module can obtain the real end-to-end network path of the target service traffic in the cloud-oriented network communication system, it can first obtain a network path view constructed in advance by fusing multi-source information, mainly from telemetry systems, monitoring systems, and path clues input by users; and then, based on the target service traffic, the network path view searches and determines the real end-to-end network path. In this way, by pre-constructing the network path view, the path discovery calculation process can be completed offline or asynchronously, avoiding repeated full-network path detection and merging during each fault injection, thereby reducing the latency and system overhead of real-time path acquisition; at the same time, by searching and determining the real end-to-end network path in the view based on the target service traffic, the actual forwarding trajectory of the service traffic can be reconstructed using the multi-source information pre-fused in the view, so that the real end-to-end network path can be used as a constraint to accurately apply the fault to a specific path node, virtual port, or forwarding stage, which can significantly improve the consistency between fault injection and real service behavior, ensuring that the obtained real end-to-end network path can truly reflect the dynamic forwarding behavior in the cloud environment, such as ECMP, tunnel encapsulation, etc., providing an accurate path basis for subsequent fault injection.

[0050] Furthermore, in this embodiment, when representing a real end-to-end network path as an ordered sequence of multiple path nodes, it can be represented as a port-level ordered path sequence through hop-by-hop parsing and explicit labeling. Specifically, starting from the source virtual machine or container, each forwarding location traversed by the target service traffic can be tracked sequentially: first, it enters the specific port of the host virtual switch, and then cross-subnet routing is completed based on the specific port of the virtual router that the forwarding path may pass through; during cross-host communication, the target service traffic enters the tunnel endpoint for encapsulation, enters the physical network through the host physical network card, is forwarded to the peer host through the physical switch port, and after decapsulation at the peer tunnel endpoint, it finally reaches the destination virtual machine network card through the target virtual switch port. In this process, each virtual network card interface, virtual switch port, virtual router port, tunnel endpoint, physical network card, and physical switch port is treated as an independent path node, arranged sequentially according to the order in which the target service traffic passes through, and the corresponding nodes are explicitly labeled at key processing stages such as encapsulation, decapsulation, and address translation, ultimately forming a port-level ordered path sequence that can accurately characterize the network hierarchy and forwarding stages of the cloud environment. By representing the real end-to-end network path as an ordered sequence of multiple path nodes and retaining key network layer information and forwarding stage information in the ordered path sequence, port-level and stage-level characterization of cloud environment paths is achieved. This enables fault injection to have path awareness capabilities, accurately match dynamic and ever-changing real business paths, and provide a foundation for subsequent fault orchestration, fault injection execution, and impact analysis. It can realize a closed loop of path-aware fault injection and verification in complex cloud computing environments.

[0051] S202. Based on the preset fault model and the real end-to-end network path, determine the fault type of the target fault to be injected, as well as the target path node corresponding to the target fault in the ordered path sequence, and generate the corresponding fault specification based on the fault type, the position of the target path node and the fault parameters of the target fault.

[0052] The pre-defined fault model serves as a bridge between real-world failure experience and executable fault injection. It acts as both a container for fault knowledge and a guide for fault orchestration. The pre-defined fault model defines "what faults exist," such as controller errors and network packet loss. In specific experiments, based on this pre-defined fault model and the real end-to-end network path, the type of the target fault to be injected is selected. Different types of fault models are only applicable to specific node types within the network path. For example, tunnel configuration faults only affect tunnel endpoints, while packet loss and latency faults affect forwarding ports or network devices. By matching fault models with path node types, the rationality and interpretability of fault injection can be ensured. Therefore, the fault model set can characterize typical failure behaviors that may occur in a cloud computing environment at multiple levels, providing a clear fault type space for subsequent path-aware fault injection. Furthermore, by combining the location of the target path node and fault parameters, a fault specification is generated. This fault specification is a structured description of a specific injection behavior, transforming the abstract fault model into an executable instruction basis, providing a foundation for subsequent fault injection. It should be noted that, in this embodiment, the fault model is not a theoretical deduction, but rather derived from engineering practices in a real cloud environment. Its construction comprehensively considers public cloud incident reports, component defect databases, operational failure cases, and experimental verification results. Therefore, this model set can cover the most common and business-critical failure scenarios in cloud computing environments, ensuring the practical significance of fault injection.

[0053] For example, in the embodiments of this application, the fault type may include at least one of control plane fault, data plane fault, management plane fault, or physical layer fault, which can be specifically set according to actual needs. When the fault type includes any one of them, it can be understood as a single fault injection scenario; when the fault type includes two or more, it can be understood as a multi-fault injection scenario.

[0054] For example, see Table 1, which shows a variety of different faults, each with its own fault description and fault type.

[0055] Table 1

[0056] As shown in Table 1 above, control plane faults mainly involve abnormal situations in components such as controllers and gateways during policy issuance or rule maintenance, such as controllers issuing incorrect forwarding table entries or virtual gateway flow table configuration errors. These types of faults often affect multiple business paths and are characterized by wide propagation and high difficulty in localization.

[0057] Data plane failures primarily occur during actual data forwarding, including abnormal behavior of virtual switching or routing devices during forwarding, caching, or scheduling, such as random packet loss, abnormal latency, and incorrect configuration of overlay network tunnel parameters. These types of failures typically directly impact business performance metrics and are a key focus in cloud platform stability verification.

[0058] Management plane failures primarily stem from configuration or operational errors, such as incorrect security group rule configurations, incorrect service listening port configurations, or abnormal load balancer backend instance configurations. These types of failures are quite common in actual operation and are often characterized by their high degree of concealment, making them easily misdiagnosed as business-related issues.

[0059] Physical layer failures include physical server downtime, physical network link disconnection, or network interface hardware malfunctions. Although cloud platforms reduce the direct impact of physical failures through redundancy and migration mechanisms, such failures can still significantly impact business processes under certain conditions, thus necessitating their inclusion in a unified failure model system.

[0060] Fault models do not exist independently; they need to be combined with specific business network paths to be effective. Since different fault models are applicable to different network locations, the system must match the selected fault model with the port-level network path model before performing fault injection.

[0061] For example, for the fault model of incorrect configuration of the peer address of the overlay network tunnel, such as FM-05 in Table 2, the injection location should correspond to the path node in the network path that is identified as the tunnel endpoint; for faults such as random packet loss or abnormal delay of virtual switching devices, such as FM-07 and FM-08, the injection location should correspond to the specific forwarding port of the virtual switching device; and for faults such as physical link disconnection or physical interface abnormality, such as FM-14 and FM-15, the injection should be applied to the corresponding physical network interface or physical switching device port in the path.

[0062] By mapping the fault model to the path node one-to-one, the problem of "only specifying the fault type without specifying the location of action" in traditional fault injection can be avoided, thus ensuring that the injected fault is highly consistent with the actual business path.

[0063] To further achieve unified description and reuse management of the fault injection process, for example, in this embodiment, based on the fault model, the fault orchestration module can generate a corresponding fault specification based on the fault type, the location of the target path node, and the fault parameters of the target fault. This fault specification can be expressed in a structured configuration form, binding the fault model with specific path nodes in the real end-to-end network path, so that the fault type corresponds precisely to the specific forwarding position in the path, which is used to guide the selection, configuration, and execution of fault injection. This solves the problem of the traditional method "only specifying the fault type without specifying the location of action", realizes precise fault orchestration, and can be reused in different paths or different experimental rounds, thereby reducing the configuration complexity of fault injection experiments.

[0064] For example, in embodiments of this application, the fault specification may include at least one of the following: The fault type field indicates the type of the target fault; The fault parameter field describes the fault parameters of the target fault, including at least one of the fault intensity, fault duration, or related fault configuration parameters; The fault injection constraint field is used to indicate the location constraints of the target path node and / or the network hierarchy constraints to which the target path node belongs; The recovery information field describes the recovery method and triggering conditions after the target failure ends.

[0065] After generating the corresponding fault specification based on the fault type, the location of the target path node, and the fault parameters of the target fault, the following step S203 can be executed: S203. Based on the fault specification, control the injection of fault injection primitives corresponding to the target fault into the target path nodes. The fault injection primitives are used to simulate the fault behavior at the target path nodes.

[0066] For example, in embodiments of this application, fault injection primitives may include injecting error forwarding entries, modifying routing configurations, introducing packet loss or delay, modifying tunnel parameters, terminating network component processes, or simulating physical node and link failures.

[0067] In this way, during fault injection, the abstract fault model is mapped to directly executable fault injection primitives by using fault specifications. This ensures that each type of fault model only acts on the corresponding path node in the network path, achieving precise fault type location and controllable injection. Furthermore, the standardized fault injection primitives shield the differences between underlying cloud platforms, allowing the same fault logic to be reused in different environments. At the same time, since each fault injection primitive is associated with a predefined recovery operation, the environment can be reliably restored based on the recorded state information after injection. This enables precise, controllable, and reproducible fault injection verification in complex cloud network environments.

[0068] S204. With fault injection primitives injected, obtain the operating data of the network communication system, perform correlation analysis between the operating data and the fault injection primitives, and generate the impact analysis results of the target fault.

[0069] For example, the results analyzer in the fault feasibility and impact analysis module performs correlation analysis between observed operational data and specific fault injection events. This includes: acquiring operational data of the network communication system collected by the distributed agent component, such as telemetry data and monitoring data, and using the results analyzer to perform correlation analysis between the operational data and fault injection primitives. This directly correlates the observed changes in system behavior with specific fault injection events, reconstructs the propagation process of the fault within the virtual network, and makes the fault injection results interpretable. This achieves controllable, reproducible, and interpretable fault injection and verification, solving the problem that traditional methods can only observe the surface phenomena of the business layer and cannot trace the root cause of the fault.

[0070] As can be seen, in this embodiment of the application, when performing fault injection method and analysis, the actual end-to-end network path of the target service traffic in the network communication system is obtained and represented as an ordered path sequence containing multiple path nodes. Key network layer information and forwarding stage information are retained in the ordered path sequence, achieving port-level and stage-level characterization of the cloud environment path. This enables fault injection to have path-aware capabilities and accurately match dynamically changing real service paths. Based on a preset fault model and the actual end-to-end network path, the fault type and target path nodes are determined, and a fault specification is generated. The fault model is bound to specific path nodes in the actual end-to-end network path, ensuring that the fault type precisely corresponds to a specific forwarding position in the path. This solves the problem of traditional methods that "only specify the fault type without specifying the effective location," achieving precise fault orchestration. Based on the fault specification, control... The target path node is injected with the corresponding fault injection primitive, ensuring that the fault is accurately applied to the specified forwarding position of the real business path, thus achieving precise fault type location and controllable injection. After fault injection, the system obtains operational data and performs correlation analysis with the fault injection primitive to generate impact analysis results. This directly correlates the observed system behavior changes with the specific fault injection event, reconstructing the fault propagation process within the virtual network and making the fault injection results interpretable. This achieves controllable, reproducible, and interpretable fault injection and verification, solving the problem that traditional methods can only observe the business layer manifestations and cannot trace the root cause of the fault. By introducing path awareness, precise orchestration, controllable fault injection, and correlation analysis into the entire fault injection process, the system achieves precise correspondence between faults and real business paths and interpretable analysis of fault impacts, significantly improving the accuracy of fault impact analysis.

[0071] Based on any of the above embodiments, for example, in S202 above, the specific implementation of determining the fault type of the target fault to be injected based on the preset fault model and the real end-to-end network path, and the target path node corresponding to the target fault in the ordered path sequence, can be found below. Figure 3 The example shown.

[0072] Figure 3 This application provides a flowchart illustrating a method for determining the fault type and target path node of a target fault to be injected. For example, the method may include: S301. Perform dependency analysis on path nodes in an ordered path sequence to identify key components in multiple path nodes and the dependencies between key components.

[0073] For example, critical components can be path nodes located at network layer transition points, policy enforcement points, or traffic convergence points in an ordered path sequence; failure of such nodes will directly lead to interruption of service traffic forwarding, path switching, or cross-layer propagation of the fault range, and is a decisive factor affecting the reliability and stability of network paths.

[0074] For example, the fault orchestration module can rely on the analyzer to perform dependency analysis on path nodes in an ordered path sequence. First, it parses the type of each path node and its role in the network hierarchy. Then, based on the order of the path nodes in the ordered path sequence, it establishes direct dependencies between adjacent path nodes: the normal forwarding of a subsequent path node depends on the correct state of the preceding path node, thus forming a complete dependency chain from source to destination. Based on this, it identifies key components in the ordered path sequence, constructs a dependency graph between different path nodes, and marks the key components that have a decisive impact on path reliability and fault propagation. Simultaneously, it clarifies the upstream and downstream dependencies between these key components, providing a basis for determining the priority and constraint scope of subsequent fault injection.

[0075] S302. Based on the dependencies between key components and the preset priority strategy, determine the fault injection priority corresponding to each key component.

[0076] For example, the preset priority strategy may include at least one of the following: component criticality, historical failure rate, or business impact level, which can be set according to actual needs.

[0077] For example, in the embodiments of this application, determining the fault injection priority corresponding to each key component based on the dependencies between key components and a preset priority strategy may include: Based on the dependencies between key components, the structural importance weights of each key component are determined. Based on a preset priority strategy, the fault injection targets corresponding to the target business traffic are determined. For each key component, historical fault injection results are obtained, and the fault injection priority of the key components is dynamically adjusted based on these historical results, the structural importance weights of the key components, and the fault injection targets. By quantifying dependencies into structural importance weights and dynamically adjusting priorities in conjunction with preset fault injection targets and historical fault injection results, this approach focuses on key components in the path and achieves adaptive optimization based on verification targets and historical feedback. This maximizes the verification value of fault injection with limited resources, improving testing accuracy and efficiency.

[0078] The fault injection objective corresponding to the target business traffic refers to the testing purpose or verification direction to be achieved in this fault injection experiment for the current business traffic to be tested. For example, if the fault injection objective is "coverage priority", then the priority of components in path areas with historically low coverage will be increased; if the fault injection objective is "impact verification priority", then the priority of components with high structural importance weight and high historical impact score will be increased; if the fault injection objective is "risk avoidance", then the priority of some unnecessary components can be reduced, or more constraints can be added to test only the most critical redundant paths.

[0079] For example, the historical fault injection results for a critical component can include verification counts, stability scores, impact scores, and test coverage, which can be set according to actual needs. Verification counts refer to the number of times the critical component has been injected with faults in the past; stability scores refer to the stability exhibited by the critical component when it was injected with faults in the past. This is usually calculated based on a combination of indicators such as performance fluctuations during the fault period and whether anomalies were triggered; impact scores refer to the actual severity of the impact on the business when the critical component was injected with faults in the past; and test coverage refers to whether the critical component and its associated path area have been adequately tested.

[0080] For example, in this embodiment of the application, when determining the fault injection target corresponding to the target service traffic based on a preset priority strategy, if the priority strategy uses the service impact level as the main basis, then the traffic carrying high-level services is automatically assigned a "priority for impact verification" fault injection target to ensure that critical services are verified in a focused manner; if the priority strategy uses the criticality of components as a weighting factor, then the structural importance of each component in the path is identified, and a "priority for coverage" target is set for traffic containing high-criticality components to fill the testing gaps in important areas; if the priority strategy uses historical failure rate data, then a "risk focus" target is enabled for traffic containing components with frequent historical failures to increase the priority of targeted verification; if multiple strategy elements are enabled simultaneously, then the final fault injection target orientation is dynamically determined by weighted calculation of a comprehensive score. In this way, through the above mechanism, the static priority strategy can be dynamically mapped to a fault injection target that adapts to specific service traffic and operating scenarios, ensuring that each fault injection experiment has a clear verification orientation and is consistent with the overall testing strategy and operation and maintenance requirements.

[0081] For example, in the embodiments of this application, dynamically adjusting the fault injection priority corresponding to the key components based on historical fault injection results, the structural importance weight of key components, and the fault injection target may include: If the historical verification count of a critical component exceeds a preset threshold and its historical performance stability score is greater than a preset stability score threshold, and the structural importance weight of the critical component is lower than a first preset weight threshold, and the fault injection target is coverage priority or risk avoidance, then the corresponding fault injection priority is reduced. This appropriately reduces the testing frequency of low-importance components that have been sufficiently verified and exhibit stable performance, avoiding redundant testing and wasting resources. It allows limited fault injection resources to be focused on more critical or higher-risk components, improving overall testing efficiency. The preset stability score threshold can be set according to actual needs.

[0082] If the historical failure impact score of a critical component exceeds a preset impact threshold, the structural importance weight of the critical component is higher than a second preset weight threshold, and the failure injection target prioritizes impact verification, the corresponding failure injection priority is increased. This ensures that critical components that have historically caused serious consequences and are located at core hubs receive higher testing priority in scenarios requiring failure impact verification. This guarantees that high-risk points are thoroughly verified, potential risks are identified promptly, and the stability of core business operations is guaranteed. The preset impact threshold can be set according to actual needs.

[0083] When a critical component is located in a path region where historical fault injection coverage is less than the coverage threshold, the structural importance weight of the critical component is higher than the third preset weight threshold, and the fault injection target prioritizes coverage, the corresponding fault injection priority is increased. This increases the testing priority of core components that are structurally important but not yet fully tested, prioritizing the filling of testing gaps in important areas and ensuring that the reliability of the critical path is fully verified, avoiding undiscovered major risks due to testing blind spots. The coverage threshold value can be set according to actual needs.

[0084] Normally, the first preset weight can be less than the second and third preset weights. For example, the values ​​of the second and third preset weights can be equal or unequal, and can be set according to actual needs.

[0085] S303. Based on the preset fault injection constraints, determine the fault injection constraint range corresponding to each key component.

[0086] Among them, fault injection constraints are preset rules or restrictions used to define the boundaries that must be followed during the fault injection process. For example, fault injection constraints may include time window limits, spatial constraints, influence range limits, fault propagation depth limits, etc.

[0087] For example, the fault constraint scope is the actually executable fault injection boundary determined for a specific critical component in the network path.

[0088] For example, when determining the fault injection constraint scope for each critical component based on preset fault injection constraints, the global rules can be parsed first, including time window restrictions (e.g., prohibiting peak business periods), spatial constraints (e.g., limited to specific network layers), impact scope restrictions (e.g., not interrupting core business), and fault propagation depth restrictions (e.g., no more than three hops). For each critical component's type, hierarchical position, and business importance, the above rules are matched one by one: the injectable time window is determined based on the peak traffic period of the component's business; the maximum allowed propagation depth is calculated based on the component's position in the dependency graph; and the applicable fault types and impact intensity thresholds are selected by combining the component's importance weight. Finally, the multi-dimensional rules are instantiated into a structured boundary specific to that component, forming an executable fault injection constraint scope.

[0089] S304. Based on the fault injection priority and fault injection constraint range corresponding to each key component, select the fault type of the target fault from the preset fault model, and select the target path node from the ordered path sequence.

[0090] For example, based on the fault injection priority and fault injection constraint range corresponding to each key component, when selecting the target fault type from the preset fault models and the target path node from the ordered path sequence, the key components can be sorted from high to low fault injection priority, with the highest priority component being processed first. For the currently processed component, fault models matching the component type are selected from the preset fault model library. Subsequently, each selected fault model is matched with the fault injection constraint range, eliminating fault types that do not meet constraints such as time window limits, spatial constraints, impact range limits, and fault propagation depth limits. If multiple fault models meet the conditions, the target fault type is selected according to a preset strategy, such as random selection, fault strength priority, or minimum impact priority, and the current component is determined as the target path node. If no fault type meets the conditions for the current component, the next component is processed in priority order until the selection is completed.

[0091] As can be seen, by performing dependency analysis on ordered path sequences to identify key components and their dependencies, we can focus on the pivotal nodes in the path that play a decisive role in reliability, avoiding blind injection on non-critical components and improving the targeting of fault injection. Determining fault injection priority based on dependency relationships and priority strategies can prioritize the verification of components with higher risks or greater importance, achieving reasonable resource allocation. Determining the constraint range of each key component based on preset fault injection constraints ensures that fault injection is executed within controllable boundaries, avoiding the expansion of business impact due to uncontrolled injection. Finally, by combining priority and constraint range to select fault types and target path nodes, fault injection satisfies both the test objective orientation and strictly follows the security boundary, realizing a complete decision-making closed loop from network path analysis to precise selection, significantly improving the accuracy and controllability of fault orchestration.

[0092] Based on any of the above embodiments, for example, in S203 above, the specific implementation of the fault injection primitive corresponding to the target fault injected into the target path node based on the fault specification can be found below. Figure 4 The example shown.

[0093] Figure 4 This application provides a flowchart illustrating a method for injecting fault injection primitives corresponding to target faults into target path nodes. For example, the method may include: S401. Parse the fault specification to obtain the fault injection task. The fault injection task includes at least one of the following: the identifier of the target path node, the location of the target path node, the identifier of the fault type, fault parameters, or fault duration.

[0094] For example, when the centralized control component parses the fault specification, it can read its structured fields, extract the fault type identifier to determine the type of fault to be injected, extract the location information of the target path node to determine the specific injection location, extract configuration items such as fault intensity and duration from the fault parameter field, and obtain the unique identifier of the target path node. The above information is combined and encapsulated to form a fault injection task containing the target path node identifier, target path node location, fault type identifier, fault parameters, and fault duration.

[0095] The centralized control component parses the fault specification and obtains the fault injection task. It then performs unified scheduling and management of the fault injection process, determines the distributed agent component responsible for executing the injection operation on that node, and sends a fault injection command to the distributed agent component in the target path node, i.e., executes the following S402: S402. Based on the fault injection task, send a fault injection instruction to the target path node. The fault injection instruction includes a fault injection primitive. The fault injection instruction is used to control the distributed proxy component in the target path node to inject the fault injection primitive.

[0096] The distributed proxy component preferably runs as a lightweight process or service, possessing the ability to receive injection instructions, execute injection primitives, return execution status, and perform recovery operations. The deployment location of the distributed proxy component corresponds one-to-one with the path nodes, thereby ensuring that fault injection operations can operate on the specified forwarding location in the actual business path.

[0097] For example, after receiving a fault injection command, the distributed proxy component can first parse the fault injection primitives and parameter configurations in the command, and then use the fault executor to execute the corresponding low-level command or API call on the operating system or virtualization environment where the target path node is located, as shown in Table 2 below. For example, it can inject packet loss or latency into a specified virtual network card using the tc command, inject error flow rules into a virtual switch using ovs-ofctl add-flow, or inject the physical network card's disabled state using the ip link set command, so as to trigger the corresponding fault behavior on the specified network path or component. Before execution, the proxy component automatically records the current state of the target node, and after execution, it sends the result back to the centralized control component to ensure that the fault operation is accurately applied to the real path node, realizing the accurate correspondence between the fault and the real business path and the interpretable analysis of the fault impact, thereby significantly improving the accuracy of fault impact analysis.

[0098] Table 2

[0099] As shown in Table 2, by establishing a mapping mechanism between fault models and specific operation commands, abstract fault models are transformed into executable fault injection methods and corresponding recovery operations. Specifically, each fault model (such as FM-07OVS bridge packet loss injection) is pre-bound with specific fault injection primitives (such as injecting packet loss via tc netem loss) and their corresponding recovery operations (such as clearing tc injection rules). During the fault injection execution phase, the corresponding fault injection primitives can be automatically matched according to the selected fault type (such as FM-07 in Table 2). Specific commands are executed on the target path node through a distributed proxy component to trigger the expected fault behavior. Simultaneously, the recovery operations associated with the fault model are recorded. After fault injection is completed or in abnormal situations, corresponding recovery commands are executed (such as clearing tc rules, restoring configuration files, restarting services, etc.) to ensure that the target path node can reliably recover to its pre-injection state. Through the above mapping mechanism, a complete transformation from "selecting a fault type" to "executing fault injection operations" and "automatic recovery" is realized, enabling abstract fault models to be implemented and executed on real cloud network paths, and ensuring the controllability and reproducibility of fault injection experiments.

[0100] Based on any of the above embodiments, for example, in this embodiment of the application, the distributed proxy component records the state information of the target path node before injecting the fault injection primitive; and after injecting the fault injection primitive, it collects the fault injection result after the fault injection primitive is injected; and sends the state information of the target path node before the fault injection primitive is injected, as well as the fault injection result after the fault injection primitive is injected, to the centralized control component so as to monitor and record the fault injection process, which can be used for subsequent recovery operations.

[0101] The status information includes at least one of the following: configuration file, forwarding table entry, or interface status, which can be set according to actual needs.

[0102] For example, in this embodiment, a recovery command is sent to the target path node based on a preset recovery strategy or recovery information in the fault specification. The recovery command instructs the distributed proxy component to perform a recovery operation corresponding to the fault injection primitive, based on the state information before the fault injection primitive. This ensures that the system reliably returns to its normal state after fault injection, avoiding the continuous impact of fault residue on business operations. Simultaneously, recovery based on the precise state before fault injection ensures that each experiment is performed under the same initial conditions, guaranteeing the comparability and reproducibility of test results. Furthermore, incorporating the recovery operation into an automated process allows for fault cleanup and environment restoration without manual intervention, significantly improving the execution efficiency, security, and overall controllability of fault injection experiments.

[0103] For example, recovery operations may include at least one of deleting faulty configurations, restoring the original configuration, restarting relevant components, or enabling backup links, which can be configured according to actual needs.

[0104] For example, the preset recovery strategy can be found in the third column of Table 2 above. Specifically, it may include at least one of the following: deleting faulty configurations, restoring the original configuration, restarting relevant components, or enabling backup links. In actual applications, it can be flexibly configured as needed. For each type of fault injection primitive, a corresponding recovery operation is predefined. During the recovery process, the centralized control component issues a recovery command to the corresponding distributed agent component based on the preset recovery strategy and / or recovery information in the fault specification. Upon receiving the recovery command, the distributed agent component executes the recovery operation corresponding to the fault injection primitive based on the state information of the target path node recorded before the execution of the fault injection primitive. After the recovery operation is completed, the agent component verifies the state of the target path node to confirm that it has recovered to the normal state before injection or as expected. If the recovery operation fails or the state verification is inconsistent, the abnormal information will be recorded and included in the subsequent fault analysis and verification results to help locate potential system vulnerabilities.

[0105] Throughout the entire fault injection and recovery process, the execution process can be fully recorded. The recorded information can include: fault model identifier, fault injection primitive type, injection parameters, injection start and end times, execution node identifier, execution result, and recovery status. These fault injection results will serve as important inputs to the subsequent fault observability and impact analysis module, and will be further used to update the fault scheduling strategy. This allows for continuous adjustment of the fault injection location and strategy based on historical injection effects, achieving an adaptive fault verification process.

[0106] Based on any of the above embodiments, for example, in S204 above, the specific implementation of performing correlation analysis between operating data and fault injection primitives to generate the impact analysis result of the target fault can be found below. Figure 5 The example shown.

[0107] Figure 5 This application provides a flowchart illustrating a method for performing correlation analysis between runtime data and fault injection primitives to generate impact analysis results for a target fault. For example, the method may include: S501. For each path node in multiple path nodes, perform correlation analysis on the running data and fault injection primitives to determine the impact analysis parameters corresponding to the path nodes, and determine the impact score corresponding to the path nodes based on the impact analysis parameters.

[0108] The impact analysis parameters include: the normalized performance degradation degree and duration of performance degradation of multiple key analysis indicators at path nodes, the numerical severity of abnormal events observed at path nodes, and the propagation impact of faults caused by path nodes in the path.

[0109] With the first Taking the first path node as an example, the fault observability and impact analysis module performs correlation analysis between operational data and fault injection primitives to determine the first path node. When analyzing the impact parameters for each path node, the start and end times of the fault injection primitive can be aligned with the timestamps of the runtime data. Key performance indicators (KPIs) for each path node during the fault injection period, such as latency, throughput, and error rate, can be extracted. These KPIs can then be compared to the state before fault injection to calculate the KPIs at the specified timestamps. Normalized performance degradation value at each path node and the duration of performance degradation Simultaneously, abnormal events are identified from system logs, and their severity is quantified numerically. And by combining the dependency graph, we can analyze the degree of propagation impact of the faults it causes in the path. Thus, the first The impact analysis parameters corresponding to each path node.

[0110] Similarly, with the first Taking a path node as an example, when determining the impact score corresponding to a path node based on impact analysis parameters, the following formula can be used: in, Indicates the first The impact score corresponding to each path node. It represents several key performance indicators, including service latency, throughput, error rate, and resource utilization. Indicators representing multiple key performance indicators The corresponding preset weights are used to reflect the importance of this indicator. Indicators At the node The degree of normalization performance degradation, and its range is: This value is used to quantify the degree of deviation of the indicator from the baseline state. It is calculated by normalizing the deviation between the observed value and the baseline value. Indicators At the node The duration of performance degradation, and its range is: This is used to quantify the proportion of performance degradation that persists throughout the entire experimental period. Indicates at node The set of anomalous events observed above; Indicates the exception type Preset weights; A numerical severity value representing the anomaly, used to reflect the level of impact of the anomaly on system operation. The weighting parameters represent the impact of fault propagation. Represents a node The extent to which the fault propagates along the path.

[0111] This approach unifies the normalized performance degradation level, the duration of performance degradation, the numerical severity level, and the propagation impact level into a unified impact score. This makes the abstract impact of the fault measurable, comprehensively reflects the overall vulnerability of path nodes in the fault, and supports decomposition and tracing, providing a quantitative basis for the optimization of subsequent fault scheduling strategies.

[0112] S502. Based on the impact scores of each path node, the impact analysis results of the target failure are obtained.

[0113] For example, when obtaining the impact analysis results of the target fault based on the impact scores of each path node, the impact scores of all path nodes in the ordered path sequence can be summarized first, and the affected path node with the highest score can be identified as the key impact point. At the same time, the number and distribution of path nodes with scores exceeding a preset threshold can be counted to quantify the impact range and propagation depth of the fault. On this basis, combined with the scores of each path node, a structured impact analysis report can be generated, which includes the key nodes affected by the fault, the overall impact score, the affected path area, and a summary of abnormal events, thus forming a complete impact analysis result of the target fault.

[0114] The aforementioned impact analysis results, along with performance index data, anomaly event timelines, system logs and tracing information, and test coverage metrics, are uniformly stored in the results database according to the data structure shown in Table 3 below, using data categories and data content descriptions. By adopting a unified results database structure, the entire process of a fault injection experiment, from execution to impact analysis, can be completely recorded, providing comprehensive data support for subsequent comparative analysis of the impact of different fault models at different path nodes, optimization of fault scheduling strategies, and system reliability assessment.

[0115] Table 3

[0116] As can be seen from the above description, the path-aware fault injection method for cloud computing environments provided in this application has low-overhead characteristics that are friendly to production environments. It can adopt an on-demand triggering mechanism, injecting faults into target path nodes and collecting observation data only when performing fault injection verification tasks. There is no need to continuously inject or monitor the entire network, which effectively avoids unnecessary resource consumption and is suitable for routine verification in production environments.

[0117] Furthermore, the path-aware fault injection method for cloud computing environments provided in this application has good environmental adaptability and scalability. Since path modeling and fault injection are directly based on the runtime behavior of real business traffic, they can naturally adapt to complex mechanisms commonly found in cloud environments, such as ECMP multipath forwarding, NAT address translation, dynamic load balancing scheduling, and the overlay and underlay network superposition. This makes the verification results no longer dependent on static topology assumptions, giving them stronger practical applicability and providing systematic technical support for the reliability assessment and defense mechanism design of cloud platforms. By repeatedly injecting different types of fault models and quantifying node-level impact scores under the constraints of real end-to-end network paths, platform operators can identify potential architectural weaknesses in advance, assess the effectiveness of existing protection mechanisms, and thus reduce system risks and improve overall service reliability before actual failures occur.

[0118] The path-aware fault handling device for cloud computing environments provided in this application is described below. The path-aware fault handling device for cloud computing environments described below can be referred to in correspondence with the path-aware fault handling method for cloud computing environments described above.

[0119] Figure 6 A schematic diagram of a path-aware fault handling device for a cloud computing environment provided in this application embodiment is shown below. For example, please refer to... Figure 6 As shown, the path-aware fault handling device 60 for cloud computing environments may include: The acquisition unit 601 is used to acquire the real end-to-end network path of the target service traffic in the cloud computing-oriented network communication system, and represent the real end-to-end network path as an ordered path sequence including multiple path nodes; The processing unit 602 is used to determine the fault type of the target fault to be injected and the target path node corresponding to the target fault in the ordered path sequence based on the preset fault model and the real end-to-end network path, and to generate a corresponding fault specification description based on the fault type, the position of the target path node and the fault parameters of the target fault. The control unit 603 is used to control the injection of fault injection primitives corresponding to the target fault into the target path node based on the fault specification, wherein the fault injection primitives are used to simulate the fault behavior at the target path node. The analysis unit 604 is used to obtain the operating data of the network communication system when the fault injection primitive is injected, and to perform correlation analysis between the operating data and the fault injection primitive to generate the impact analysis result of the target fault.

[0120] For example, in the embodiments of this application, the ordered path sequence includes at least one of the following: virtual machine network interface card interface, container network interface, virtual switch port, virtual router port, tunnel encapsulation endpoint, host physical network interface card interface, physical switch port, and physical router port.

[0121] For example, in embodiments of this application, the fault specification includes at least one of the following: The fault type field is used to indicate the fault type of the target fault; The fields of the fault parameters are used to describe the fault parameters of the target fault, including at least one of fault intensity, fault duration, or related fault configuration parameters; The fault injection constraint field is used to indicate the location constraints of the target path node and / or the network hierarchy constraints to which the target path node belongs; The recovery information field describes the recovery method and triggering conditions after the target fault ends.

[0122] For example, in the embodiments of this application, the fault injection primitives include injecting erroneous forwarding table entries, modifying routing configurations, introducing packet loss or delay, modifying tunnel parameters, terminating network component processes, or simulating physical node and link failures.

[0123] For example, in an embodiment of this application, the processing unit 602 is used to determine the fault type of the target fault to be injected, and the target path node corresponding to the target fault in the ordered path sequence, based on a preset fault model and the actual end-to-end network path, including: Dependency analysis is performed on the path nodes in the ordered path sequence to identify the key components in the multiple path nodes and the dependencies between the key components. Based on the dependencies between the key components and the preset priority strategy, the fault injection priority corresponding to each key component is determined. Based on the preset fault injection constraints, the fault injection constraint range corresponding to each of the key components is determined. Based on the fault injection priority and fault injection constraint range corresponding to each of the key components, the fault type of the target fault is selected from the preset fault model, and the target path node is selected from the ordered path sequence.

[0124] For example, in an embodiment of this application, the processing unit 602 is configured to determine the fault injection priority corresponding to each of the key components based on the dependencies between the key components and a preset priority strategy, including: Based on the dependencies between the key components, the structural importance weight of each key component is determined. Based on a preset priority strategy, the fault injection target corresponding to the target service traffic is determined; For each of the key components, obtain the historical fault injection results corresponding to the key components, and dynamically adjust the fault injection priority corresponding to the key components based on the historical fault injection results, the structural importance weight of the key components, and the fault injection target.

[0125] For example, in an embodiment of this application, the processing unit 602 is configured to dynamically adjust the fault injection priority corresponding to the key component based on the historical fault injection results, the structural importance weight of the key component, and the fault injection target, including: If the number of historical verifications of a critical component exceeds a preset threshold and the historical performance stability score is greater than a preset stability score threshold, the structural importance weight of the critical component is lower than a first preset weight threshold, and the fault injection target is coverage priority or risk avoidance, the corresponding fault injection priority shall be reduced. If the historical failure impact score of the key component is greater than the preset impact threshold, the structural importance weight of the key component is higher than the second preset weight threshold, and the failure injection target is impact verification priority, the corresponding failure injection priority is increased. If the critical component is located in a path region where the historical fault injection coverage is less than the coverage threshold, the structural importance weight of the critical component is higher than the third preset weight threshold, and the fault injection target is coverage priority, the corresponding fault injection priority is increased.

[0126] For example, in an embodiment of this application, the control unit 603 is used to control the injection of fault injection primitives corresponding to the target fault into the target path node based on the fault specification, including: The fault specification is parsed to obtain a fault injection task, which includes at least one of the following: the identifier of the target path node, the location of the target path node, the identifier of the fault type, and the fault parameter or the fault duration. Based on the fault injection task, a fault injection instruction is sent to the target path node. The fault injection instruction includes the fault injection primitive, and the fault injection instruction is used to control the distributed proxy component in the target path node to inject the fault injection primitive.

[0127] For example, in an embodiment of this application, the path-aware fault handling device 60 for a cloud computing environment further includes: The receiving unit is used to receive the state information before the fault injection primitive is injected and the fault injection result after the fault injection primitive is injected, sent by the target path node. The status information includes at least one of the following: configuration file, forwarding table entry, or interface status.

[0128] For example, in an embodiment of this application, the path-aware fault handling device 60 for a cloud computing environment further includes: The sending unit is used to send a recovery command to the target path node based on a preset recovery strategy and / or the recovery information in the fault specification. The recovery instruction is used to instruct the distributed proxy component to perform a recovery operation corresponding to the fault injection primitive based on the state information before the fault injection primitive was injected.

[0129] For example, in this embodiment of the application, the analysis unit 604 is used to perform correlation analysis on the operating data and the fault injection primitives to generate the impact analysis results of the target fault, including: For each of the multiple path nodes, a correlation analysis is performed between the operational data and the fault injection primitives to determine the impact analysis parameters corresponding to the path node, and an impact score is determined based on the impact analysis parameters. The impact analysis parameters include: normalized performance degradation values ​​and durations of performance degradation for multiple key analysis indicators at the path node; numerical severity values ​​of abnormal events observed at the path node; and the propagation impact of the fault caused by the path node within the path. Based on the impact scores of each path node, the impact analysis results of the target fault are obtained.

[0130] For example, in this embodiment of the application, the acquisition unit 601 is used to acquire the actual end-to-end network path of the target service traffic in a cloud-oriented network communication system, including: Obtain a network path view pre-constructed by fusing information from multiple sources; Based on the target service traffic, the network path view lookup determines the actual end-to-end network path.

[0131] The path-aware fault handling device 60 for cloud computing environments provided in this application embodiment can execute the technical solution of the path-aware fault handling method for cloud computing environments in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the path-aware fault handling method for cloud computing environments. Please refer to the implementation principle and beneficial effects of the path-aware fault handling method for cloud computing environments. It will not be repeated here.

[0132] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 730 to execute a path-aware fault handling method for a cloud computing environment. This method includes: acquiring the actual end-to-end network path of the target service traffic in a cloud-based network communication system, and representing the actual end-to-end network path as an ordered path sequence including multiple path nodes; determining the fault type of the target fault to be injected, and the target path node corresponding to the target fault in the ordered path sequence, based on a preset fault model and the actual end-to-end network path; generating a corresponding fault specification based on the fault type, the location of the target path node, and the fault parameters of the target fault; controlling the injection of a fault injection primitive corresponding to the target fault into the target path node based on the fault specification, the fault injection primitive being used to simulate fault behavior at the target path node; and acquiring the operating data of the network communication system when the fault injection primitive is injected, performing correlation analysis between the operating data and the fault injection primitive, and generating an impact analysis result for the target fault.

[0133] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the path-aware fault handling method for cloud computing environments provided by the above methods. The method includes: obtaining the real end-to-end network path of the target service traffic in a cloud computing network communication system, and representing the real end-to-end network path as an ordered path sequence including multiple path nodes; determining the fault type of the target fault to be injected and the target path node corresponding to the target fault in the ordered path sequence based on a preset fault model and the real end-to-end network path, and generating a corresponding fault specification based on the fault type, the position of the target path node, and the fault parameters of the target fault; controlling the injection of a fault injection primitive corresponding to the target fault into the target path node based on the fault specification, the fault injection primitive being used to simulate the fault behavior at the target path node; and obtaining the operating data of the network communication system when the fault injection primitive is injected, and performing correlation analysis on the operating data and the fault injection primitive to generate an impact analysis result of the target fault.

[0135] In another aspect, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the path-aware fault handling method for cloud computing environments provided by the methods described above. This method includes: acquiring the actual end-to-end network path of target service traffic in a cloud computing-oriented network communication system, and representing the actual end-to-end network path as an ordered path sequence including multiple path nodes; determining the fault type of the target fault to be injected, and the target path node corresponding to the target fault in the ordered path sequence, based on a preset fault model and the actual end-to-end network path, and generating a corresponding fault specification description based on the fault type, the location of the target path node, and the fault parameters of the target fault; controlling the injection of a fault injection primitive corresponding to the target fault into the target path node based on the fault specification description, the fault injection primitive being used to simulate fault behavior at the target path node; and acquiring the operating data of the network communication system when the fault injection primitive is injected, and performing correlation analysis on the operating data and the fault injection primitive to generate an impact analysis result of the target fault.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A path-aware fault handling method for cloud computing environments, characterized in that, include: Obtain the actual end-to-end network path of the target service traffic in the cloud-oriented network communication system, and represent the actual end-to-end network path as an ordered path sequence including multiple path nodes; Based on the preset fault model and the real end-to-end network path, the fault type of the target fault to be injected and the target path node corresponding to the target fault in the ordered path sequence are determined, and a corresponding fault specification is generated based on the fault type, the position of the target path node and the fault parameters of the target fault. Based on the fault specification, the target path node is controlled to inject the fault injection primitive corresponding to the target fault. The fault injection primitive is used to simulate the fault behavior at the target path node. When the fault injection primitive is injected, the operating data of the network communication system is obtained, and the operating data and the fault injection primitive are correlated to generate the impact analysis results of the target fault.

2. The path-aware fault handling method for cloud computing environments according to claim 1, characterized in that, The ordered path sequence includes at least one of the following: virtual machine network interface card interface, container network interface, virtual switch port, virtual router port, tunnel encapsulation endpoint, host physical network interface card interface, physical switch port, and physical router port.

3. The path-aware fault handling method for cloud computing environments according to claim 1, characterized in that, The fault specification includes at least one of the following: The fault type field is used to indicate the fault type of the target fault; The fields of the fault parameters are used to describe the fault parameters of the target fault, including at least one of fault intensity, fault duration, or related fault configuration parameters; The fault injection constraint field is used to indicate the location constraints of the target path node and / or the network hierarchy constraints to which the target path node belongs; The recovery information field describes the recovery method and triggering conditions after the target fault ends.

4. The path-aware fault handling method for cloud computing environments according to claim 1, characterized in that, The fault injection primitives include injecting error forwarding table entries, modifying routing configurations, introducing packet loss or delay, modifying tunnel parameters, terminating network component processes, or simulating physical node and link failures.

5. The path-aware fault handling method for cloud computing environments according to any one of claims 1-4, characterized in that, The process of determining the fault type of the target fault to be injected, and the target path node corresponding to the target fault in the ordered path sequence, based on the preset fault model and the real end-to-end network path, includes: Dependency analysis is performed on the path nodes in the ordered path sequence to identify the key components in the multiple path nodes and the dependencies between the key components. Based on the dependencies between the key components and the preset priority strategy, the fault injection priority corresponding to each key component is determined. Based on the preset fault injection constraints, the fault injection constraint range corresponding to each of the key components is determined. Based on the fault injection priority and fault injection constraint range corresponding to each of the key components, the fault type of the target fault is selected from the preset fault model, and the target path node is selected from the ordered path sequence.

6. The path-aware fault handling method for cloud computing environments according to claim 5, characterized in that, The determination of the fault injection priority for each key component based on the dependencies between the key components and a preset priority strategy includes: Based on the dependencies between the key components, the structural importance weight of each key component is determined. Based on a preset priority strategy, the fault injection target corresponding to the target service traffic is determined; For each of the key components, obtain the historical fault injection results corresponding to the key components, and dynamically adjust the fault injection priority corresponding to the key components based on the historical fault injection results, the structural importance weight of the key components, and the fault injection target.

7. The path-aware fault handling method for cloud computing environments according to claim 6, characterized in that, The step of dynamically adjusting the fault injection priority corresponding to the key components based on the historical fault injection results, the structural importance weight of the key components, and the fault injection target includes: If the number of historical verifications of a critical component exceeds a preset threshold and the historical performance stability score is greater than a preset stability score threshold, the structural importance weight of the critical component is lower than a first preset weight threshold, and the fault injection target is coverage priority or risk avoidance, the corresponding fault injection priority shall be reduced. If the historical failure impact score of the key component is greater than the preset impact threshold, the structural importance weight of the key component is higher than the second preset weight threshold, and the failure injection target is impact verification priority, the corresponding failure injection priority is increased. If the critical component is located in a path region where the historical fault injection coverage is less than the coverage threshold, the structural importance weight of the critical component is higher than the third preset weight threshold, and the fault injection target is coverage priority, the corresponding fault injection priority is increased.

8. The path-aware fault handling method for cloud computing environments according to any one of claims 1-4, characterized in that, The step of controlling the injection of fault injection primitives corresponding to the target fault into the target path nodes based on the fault specification includes: The fault specification is parsed to obtain a fault injection task, which includes at least one of the following: the identifier of the target path node, the location of the target path node, the identifier of the fault type, and the fault parameter or the fault duration. Based on the fault injection task, a fault injection instruction is sent to the target path node. The fault injection instruction includes the fault injection primitive, and the fault injection instruction is used to control the distributed proxy component in the target path node to inject the fault injection primitive.

9. The path-aware fault handling method for cloud computing environments according to claim 8, characterized in that, The method further includes: Receive the state information before the fault injection primitive is injected, and the fault injection result after the fault injection primitive is injected, sent by the target path node; The status information includes at least one of the following: configuration file, forwarding table entry, or interface status.

10. The path-aware fault handling method for cloud computing environments according to claim 8, characterized in that, The method further includes: Based on the preset recovery strategy and / or the recovery information in the fault specification, a recovery command is sent to the target path node; The recovery instruction is used to instruct the distributed proxy component to perform a recovery operation corresponding to the fault injection primitive based on the state information before the fault injection primitive was injected.

11. The path-aware fault handling method for cloud computing environments according to any one of claims 1-4, characterized in that, The step of performing correlation analysis between the operational data and the fault injection primitives to generate the impact analysis results of the target fault includes: For each of the multiple path nodes, a correlation analysis is performed between the operational data and the fault injection primitives to determine the impact analysis parameters corresponding to the path node, and an impact score is determined based on the impact analysis parameters. The impact analysis parameters include: normalized performance degradation values ​​and durations of performance degradation for multiple key analysis indicators at the path node; numerical severity values ​​of abnormal events observed at the path node; and the propagation impact of the fault caused by the path node within the path. Based on the impact scores of each path node, the impact analysis results of the target fault are obtained.

12. The path-aware fault handling method for cloud computing environments according to any one of claims 1-4, characterized in that, The acquisition of the actual end-to-end network path of the target service traffic in the cloud-oriented network communication system includes: Obtain a network path view pre-constructed by fusing information from multiple sources; Based on the target service traffic, the network path view lookup determines the actual end-to-end network path.

Images