Fault detection method and device and computing equipment cluster

Abstract

The invention discloses a fault detection method, which comprises the following steps of: acquiring a plurality of configuration items, acquiring at least one fault mode in a fault mode library, mapping the plurality of configuration items and the at least one fault mode to obtain at least one detection rule, and carrying out fault detection on a plurality of services according to the at least one detection rule. According to the method, the configuration items of the multiple services can be integrated according to the fault modes in the fault mode library, the detection rule that each fault mode corresponds to one or more configuration items is obtained, and a system can comprehensively check complex fault modes caused by interaction of the multiple configuration items. The method can ensure that potential problems can be effectively identified and prevented under different service and configuration scenes. In addition, compared with direct fault injection detection, the method is more purposeful, and the root of the problem can be quickly positioned, so that the detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a fault detection method, device, and computing device cluster. Background Art

[0002] In today's rapidly evolving digital age, cloud-native technologies have become central to building and deploying modern applications. Cloud-native architectures, with their elasticity, scalability, and flexibility, provide enterprises with an efficient and agile software development and operations environment. In cloud-native environments, service configuration is a critical foundation for ensuring that services operate as expected. Configuration errors can lead to serious issues such as service unavailability, performance degradation, and even data leakage. Therefore, correct service configuration is crucial for achieving high availability, elasticity, and performance. Therefore, detecting and remediating service configuration defects is crucial for ensuring service stability and performance.

[0003] As the complexity of cloud-native services and middleware continues to grow, so too does the difficulty of detecting configuration errors and defects. With complex dependencies between services and numerous configuration parameters, errors in any one link can trigger a chain reaction. Furthermore, the dynamic nature of cloud environments requires frequent updates to service configurations to adapt to changing requirements, further complicating fault detection and remediation. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of the present application provide a fault detection method that enables the system to comprehensively examine complex fault modes caused by the interaction of multiple configuration items. Furthermore, the present application also provides a fault detection apparatus and computing device cluster corresponding to the fault detection method.

[0005] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0006] In a first aspect, an embodiment of the present application provides a fault detection method, which is applied to a management platform, the management platform is used to manage infrastructure, the basic settings include at least one node, and the at least one node is used to deploy multiple services. The method includes: obtaining multiple configuration items, each configuration item is a parameter for setting and adjusting the resources of one of the multiple services according to demand; obtaining at least one fault mode in a fault mode library, the fault mode refers to the failure mode or failure form of the service; mapping multiple configuration items with at least one fault mode to obtain at least one detection rule, the detection rule is used to indicate that the target fault mode corresponds to multiple target configuration items; and performing fault detection on multiple services according to at least one detection rule.

[0007] In this implementation, the method integrates configuration items for multiple services based on the failure modes in the failure mode library, generating detection rules for each failure mode corresponding to one or more configuration items. This allows the system to comprehensively examine complex failure modes caused by the interaction of multiple configuration items. This method ensures that potential issues can be effectively identified and prevented in different service and configuration scenarios. Furthermore, compared to direct fault injection testing, this method is more targeted and can quickly locate the root cause of the problem, thereby improving detection efficiency.

[0008] In one embodiment, mapping multiple configuration items with at least one failure mode to obtain at least one detection rule includes: obtaining a configuration document, where the configuration document is used to record configuration definitions of multiple services; mapping the multiple configuration items with at least one failure mode to obtain a mapping relationship between the multiple configuration items and the at least one failure mode; and obtaining at least one detection rule based on the configuration definitions of the multiple services and the mapping relationship.

[0009] In this embodiment, the failure mode may be caused by the combined effect of one or more unreasonable configuration items of the service, so the method can map one or more of the configuration items of the service to each failure mode, and generate corresponding detection rules based on the configuration document, so that each failure mode corresponds not only to one or more configuration items, but also to inspection logic, thresholds and conditions.

[0010] In one embodiment, fault detection is performed on multiple services based on at least one detection rule, including: obtaining configuration data and monitoring data of multiple services, where the configuration data is resource configuration information of multiple services, and the monitoring data is monitoring data of multiple services during operation; detecting whether at least one detection rule satisfies the configuration data and monitoring data of multiple services; when the detection rule satisfies the configuration data and monitoring data of multiple services, obtaining a first configuration item, which is a configuration item included in the detection rule that satisfies the configuration data and monitoring data of multiple services in at least one detection rule.

[0011] In this embodiment, the method can obtain resource configuration information of multiple services and monitoring data of multiple services during operation, and check multiple configuration items included in the detection rules in the rule base, thereby detecting deterministic fault configuration items and potential fault configuration items.

[0012] In one embodiment, the method also includes: constructing a fault triggering scenario based on monitoring data of multiple services; detecting whether the second configuration item triggers the fault triggering scenario to generate a fault, the second configuration item being a configuration item that does not meet the detection rules of the configuration data and monitoring data of multiple services; when the second configuration item triggers the fault triggering scenario to generate a fault, obtaining a third configuration item; when the second configuration item triggers the fault triggering scenario to generate a fault, obtaining a fourth configuration item.

[0013] In this implementation, the method leverages fault-trigger scenario construction and fault injection capabilities to enable the system to comprehensively test and analyze service configurations. This not only enhances the ability to detect potential issues but also covers complex failure modes that are difficult to detect using traditional, simple threshold matching methods. By simulating multiple scenarios, the method can deeply explore the interactions and impacts between configuration items, proactively identify and prevent potential failure points, and thus enhance the comprehensiveness of fault detection.

[0014] In one embodiment, the method also includes: generating a prompt template based on the first configuration item, the third configuration item and / or the fourth configuration item; generating a modification suggestion instruction based on the prompt template, at least one detection rule and the case in the case library, the modification suggestion instruction is used to instruct multiple services to modify the status of the configuration resource to a specified status, and the case is used to indicate the best operating method in a specific field or situation.

[0015] In this implementation, the method can provide targeted, customized resource configuration modification suggestions. This method uses a large language model as its inference engine, resource configuration documents, a fault pattern library, and a rule library as its knowledge base, and fault detection results as input data to output a reasonable configuration solution that meets the user's system requirements. This method provides users with intelligent configuration guidance, significantly reducing the cost of learning complex configuration knowledge and improving the reliability of user services. This allows even non-expert users to easily manage and maintain service resources, thereby enhancing user trust and satisfaction with cloud providers.

[0016] In a second aspect, an embodiment of the present application provides a fault detection device, including: a first processing module, used to obtain multiple configuration items, each configuration item is a parameter for setting and adjusting the resources of a service among multiple services according to demand; a second processing module, used to obtain at least one fault mode in a fault mode library, where the fault mode refers to the failure mode or failure form of the service; a third processing module, used to map multiple configuration items with at least one fault mode to obtain at least one detection rule, and the detection rule is used to indicate that the target fault mode corresponds to multiple target configuration items; a fourth processing module, used to perform fault detection on multiple services according to at least one detection rule.

[0017] In one embodiment, the third processing module is used to obtain a configuration document, which is used to record the configuration definitions of multiple services; map multiple configuration items with at least one failure mode to obtain a mapping relationship between the multiple configuration items and the at least one failure mode; and obtain at least one detection rule based on the configuration definitions and mapping relationships of the multiple services.

[0018] In one embodiment, the fourth processing module obtains configuration data and monitoring data of multiple services, where the configuration data is resource configuration information of multiple services, and the monitoring data is monitoring data during the operation of multiple services; detects whether at least one detection rule satisfies the configuration data and monitoring data of multiple services; and obtains a first configuration item when the detection rule satisfies the configuration data and monitoring data of multiple services, where the first configuration item is a configuration item included in the detection rule that satisfies the configuration data and monitoring data of multiple services in at least one detection rule.

[0019] In one embodiment, the fourth processing module is also used to construct a fault triggering scenario based on the monitoring data of multiple services; detect whether the second configuration item triggers the fault triggering scenario to generate a fault, and the second configuration item is a configuration item that does not meet the detection rules of the configuration data and monitoring data of multiple services; when the second configuration item triggers the fault triggering scenario to generate a fault, the third configuration item is obtained; when the second configuration item triggers the fault triggering scenario to generate a fault, the fourth configuration item is obtained.

[0020] In one embodiment, the fourth processing module is also used to generate a prompt template based on the first configuration item, the third configuration item and / or the fourth configuration item; and generate a modification suggestion instruction based on the prompt template, at least one detection rule and the case in the case library, the modification suggestion instruction is used to instruct multiple services to modify the status of the configuration resource to a specified status, and the case is used to indicate the best operating method in a specific field or situation.

[0021] In a third aspect, an embodiment of the present application provides a computing device, comprising: at least one memory; and at least one processor, the processor being configured to execute instructions stored in the memory, so that the computing device executes various possible implementations of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the various possible implementations of the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, enable the computing device to implement various possible implementation embodiments of the first aspect.

[0024] In a sixth aspect, an embodiment of the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the various possible implementations of the first aspect.

[0025] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the various possible implementations of the first aspect.

[0026] In an eighth aspect, an embodiment of the present application provides a computer program product comprising instructions, characterized in that the computer program product stores instructions that, when executed by a computing device cluster, enable the computing device cluster to implement various possible implementation embodiments of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0028] Figure 1 This is a schematic diagram of the structure of the fault detection system provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the process of constructing detection rules from the rule base provided in the embodiments of the present application;

[0030] Figure 3 This is a schematic diagram of the structure of the inspector provided in an embodiment of the present application;

[0031] Figure 4 This is a schematic diagram of the structure of the constructor provided in the embodiment of the present application;

[0032] Figure 5 This is a schematic diagram of the structure of the generator provided in the embodiment of the present application;

[0033] Figure 6 A schematic diagram of a scenario in which a user uses a fault detection system provided in an embodiment of the present application;

[0034] Figure 7 This is a flowchart of a fault detection method provided in an embodiment of the present application;

[0035] Figure 8 This is a structural diagram of a fault detection device provided in an embodiment of the present application;

[0036] Figure 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0037] Figure 10 A schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0038] Figure 11 This is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0040] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0041] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0042] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0044] Before introducing the technical solution protected by this application, several professional terms involved in the technical solution protected by this application are explained in advance, namely:

[0045] Cloud-native applications are applications designed and developed specifically for cloud environments. They leverage the advantages of services such as containerization, microservices architecture, dynamic orchestration, continuous integration and continuous deployment (CI / CD), declarative configuration, and elastic scaling to enable efficient operation and scalability in cloud environments. These features make cloud-native applications more efficient and flexible in deployment, scaling, and management, making them ideal for a variety of scenarios in modern cloud environments.

[0046] Next, the technical solution provided by this application is introduced.

[0047] Typically, during a test, a matching target detection rule is found within a detection rule set based on resource configuration information to check whether the target configuration item meets the rule requirements. A detection rule set contains multiple detection rules based on detection rule templates, which are used to check whether the target configuration item in the resource configuration information meets the requirements. However, existing service configuration fault detection methods mostly focus on matching a single configuration item with a pre-set detection rule, and fail to fully cover the complex fault modes caused by the interaction of multiple configuration items.

[0048] In view of this, an embodiment of the present application provides a fault detection method that can integrate the configuration items of multiple services based on the fault modes in the fault mode library to obtain detection rules for each fault mode corresponding to one or more configuration items, allowing the system to comprehensively check complex fault modes caused by the interaction of multiple configuration items. This method can ensure that potential problems can be effectively identified and prevented in different service and configuration scenarios. In addition, compared with direct fault injection detection, this method is more purposeful and can quickly locate the root cause of the problem, thereby improving detection efficiency.

[0049] Figure 1 This is a schematic diagram of the structure of the fault detection system provided in the embodiment of this application. Figure 1 As shown, the fault detection system 100 may include a rule base 110 , a collector 120 , a checker 130 , a constructor 140 and a generator 150 .

[0050] It is used to obtain at least one failure mode of multiple services to be detected from a failure mode library, and obtain configuration items of multiple services to be detected, and then map the configuration items of multiple services according to the at least one failure mode and configuration document to obtain at least one detection rule, and store the detection rules of multiple services locally to form a detection rule library.

[0051] Each service includes one or more configuration items. Configuration items are parameters used to set and adjust resources based on demand. Configuration items can include computing resource configuration, storage configuration, network configuration, security configuration, application service configuration, and so on. Computing resource configuration includes information such as the instance type, the number of CPU cores and memory size configured for the instance, and the operating system configured for the instance. Storage configuration includes information such as the data disk type and data disk storage performance. Network configuration includes information such as the virtual private cloud (VPC) and subnet, security group, and elastic Internet Protocol (IP) address.

[0052] Failure modes refer to the possible failure modes or failure forms of a service. They describe in detail the manifestation of the failure, its location, the scope of impact, and possible causes. A failure mode library typically stores failure modes for multiple services, such as service dependency failures, service load failures, and service node failures.

[0053] Configuration documents generally record the configuration definitions of multiple services, such as configuration purpose, configuration scope, configuration correct range, dependencies, etc.

[0054] In this application, a failure mode may be caused by the combined effect of one or more unreasonable configuration items, so the rule base 110 can map one or more configuration items of at least one service to each failure mode and generate corresponding detection rules based on the configuration document.

[0055] For example, Figure 2 As shown in the figure, service dependency failures in the failure mode library can include application programming interface (API) gateway failures, elastic scaling failures, and cache failures. Business load failures can include load imbalance, specification mismatch, and flow control failures. Business node failures can include system resource failures, process status anomalies, and application anomalies. Configuration documents can include gateway services, elastic scaling services, cache services, etc.

[0056] The rule base 110 can deeply analyze the possible failure modes in the system, identify the configuration items associated with them, and evaluate the impact of these configuration items on the system behavior. Among them, improper settings of some configuration items will directly cause failures. These are called "first configuration items", or more appropriately called "deterministic failure configuration items". Other configuration items need to be tested based on the current state of the system, or require specific inducing conditions or specific scenarios to determine their rationality and whether they trigger failures. These are called "second configuration items", or more appropriately called "potential failure configuration items". Based on these mapping relationships, the rule base 110 can construct detection rules. The rule base 110 uses failure modes as organizational units, allowing each failure mode to correspond to one or more configuration items, as well as inspection logic, thresholds, and conditions. The rule base 110 is a detection rule base integrated according to failure modes, which can be updated in real time and can be supplemented and modified accordingly based on new failure modes and new service configurations.

[0057] In an embodiment of the present application, the rule base 110 integrates the configuration items of multiple services based on the failure modes in the failure mode library to obtain detection rules for one or more configuration items corresponding to each failure mode, which allows the system to comprehensively check complex failure modes caused by the interaction of multiple configuration items.

[0058] For example, consider a network failure caused by a VPC configuration error. Imagine an enterprise deploys a business system using a VPC, which includes multiple subnets, routing tables, security groups, and other configuration items. The interaction of the following configuration items leads to a complex failure mode:

[0059] Subnet configuration: Enterprises deploy different services in different subnets, such as production environments and test environments. Each subnet has its own IP address range, gateway, and other configurations. Routing table configuration: The routing table determines the communication paths between subnets. For example, the production environment subnet needs to access the test environment subnet for data synchronization. Security group configuration: Security groups act as virtual firewalls, controlling traffic in and out of subnets or instances. Enterprises set different security group rules for different subnets to limit unnecessary access.

[0060] Failure Mode:

[0061] Configuration error: During a configuration update, an administrator mistakenly modified the routing table, preventing the production subnet from accessing the test subnet. Interaction: Because security group rules restrict direct access, the routing error caused data synchronization tasks to fail, impacting the normal operation of the production environment. Cascading failure: The production failure further impacted other systems that relied on its data, such as the data analytics platform, paralyzing parts of the entire business system.

[0062] This failure mode is caused by the interaction of multiple configuration items such as routing tables, subnets, and security groups. A single configuration item error may not lead to such serious consequences, but their interaction leads to complex system failures.

[0063] By covering these failure modes, the rule base 110 can ensure that potential problems can be effectively identified and prevented in different service and configuration scenarios. In addition, compared with direct fault injection detection, this method is more targeted and can quickly locate the root cause of the problem, thereby improving detection efficiency.

[0064] Collector 120 is used to obtain configuration data, such as the latest resource configuration information for multiple services, through APIs, command-line tools, or other means. It also collects runtime monitoring data for multiple services, such as performance metrics, log files, and event notifications. Collector 120 must be able to collect this data in real time or on a scheduled basis so that inspector 130 can use this information to detect potential faults.

[0065] The checker 130 is configured to perform configuration detection on multiple services according to at least one detection rule to obtain a first configuration item and a second configuration item.

[0066] The first configuration item refers to parameters that will directly cause system failures if they are improperly configured or exceed preset thresholds, such as insufficient memory allocation and excessive storage quota.

[0067] Secondary configuration items are parameters that may cause failures only under specific conditions or scenarios. These items typically do not cause issues under normal operation, but they may impact system stability under high load, special operating environments, or other specific conditions, such as autoscaling limits. Further verification of these secondary configuration items is required in specific scenarios.

[0068] For example, Figure 3 As shown, the checker 130 includes a rule processing module 131 and a rule matching module 132. The rule processing module 131 includes a rule parser 1311 and a data filter 1312. The rule parser 1311 is used to read detection rules from storage media such as the rule base 110 and the file system, and convert the detection rules into a format that the system can execute. The data filter 1312 can filter the configuration data and monitoring data collected by the collector 120 to obtain the data required by the rule base 110 and obtain the screening results.

[0069] The rule matching module 132 can apply the converted detection rules to the screening results to detect whether the screening results are consistent with the configuration items corresponding to the detection rules. If certain configuration items in the screening results are not in the configuration specified by the configuration items, it means that these configuration items are deterministic fault configuration items. In this case, the rule matching module 132 can use these configuration items as the first configuration items. If certain configuration items in the screening results are in the configuration specified by the configuration items, it means that these configuration items are potential fault configuration items. In this case, the rule matching module 132 can use the configuration items included in these detection rules as the second configuration items.

[0070] In an embodiment of the present application, the checker 130 can obtain resource configuration information of multiple services and monitoring data of multiple services during operation, and check multiple configuration items corresponding to the detection rules in the rule base, so as to detect deterministic fault configuration items and potential fault configuration items.

[0071] The constructor 140 is configured to analyze whether the second configuration item triggers a fault based on the detection rule and the monitoring data, and determine whether the second configuration item is the third configuration item or the fourth configuration item.

[0072] The third configuration item is more appropriately called a "confirmed fault configuration item," which refers to a configuration item that has been proven to cause faults in specific scenarios or after certain inductions. The fourth configuration item is more appropriately called a "non-critical fault configuration item," which, after fault injection testing, has been proven not to have a significant impact on the system in most cases. These configuration items can be considered optimization items, but may not be prioritized when resources are limited.

[0073] The constructor 140 first constructs a fault trigger scenario based on the detection rules and monitoring data. Then, the constructor 140 converts the fault trigger scenario into structured instructions, specifies the fault type, trigger conditions, affected system components, and expected abnormal performance, and generates executable fault injection instructions (such as simulating specific port blocking, forcibly terminating processes) and monitoring checkpoints (such as service response time, error log thresholds). Subsequently, the constructor 140 automatically arranges and analyzes the dependencies between instructions, merges scenarios that can be executed in parallel (such as injecting CPU overload and memory leaks at the same time), generates an ordered fault injection task sequence, and actually triggers the fault (such as shutting down service instances, creating network delays).

[0074] During this process, the constructor 140 collects the initial state of the system (such as configuration, resource occupancy) and creates a snapshot before fault injection. During injection, the constructor 140 monitors system behavior in real time (such as log changes, performance indicator fluctuations). After injection, the constructor 140 can determine whether the expected fault is triggered based on predefined thresholds (such as error rate > 5%, recovery time > 10 seconds). If the fault recurs stably in a specific scenario (such as a probability of more than 90% triggering a service crash), it is marked as the third configuration item. If no significant abnormalities are caused after injection or only occasional slight fluctuations occur (such as a brief jitter of resources but no impact on the service), it is classified as the fourth configuration item.

[0075] For example, Figure 4 As shown, the constructor 140 may include a scenario parser 141, a fault injector 142, and a check controller 143. The main responsibility of the scenario parser 141 is to parse, understand, and define the corresponding fault trigger scenario. Specifically, the scenario parser 141 first constructs the fault trigger scenario based on the detection rules and monitoring data. Then, the scenario parser 141 converts the fault trigger scenario into structured instructions, clearly defining the fault type, trigger conditions, affected system components, and expected abnormal performance, and generates executable fault injection instructions and monitoring checkpoints.

[0076] Fault injector 142 actually performs fault injection operations based on the instructions provided by scenario analyzer 141. Fault injector 142 may include an automatic orchestrator 1421 and an executor 1422. Automatic orchestrator 1421 analyzes the dependencies between instructions, merges scenarios that can be executed in parallel, and generates an ordered sequence of fault injection tasks, which are then passed to executor 1422 for execution. Executor 1422 actually triggers faults, such as shutting down service instances or creating network delays.

[0077] The inspection controller 143 may include a state recorder 1431, an injection controller 1432, and a result comparator 1433. The injection controller 1432 is responsible for executing fault injection and collecting, storing, and comparing system / application states. Before fault injection, the state recorder 1431 cooperates with the collector 120 to collect the initial state of the system and create a snapshot. During fault injection, the injection controller 1432 implements the fault injection behavior and uses the collector 120 to record the state during the experiment. After the fault injection is completed, the result comparator 1433 determines whether the expected fault is triggered based on a predefined threshold. If the fault recurs stably in a specific scenario, it is marked as a confirmed fault configuration item and needs to be repaired first or a fault tolerance mechanism is designed. If no significant anomalies are caused after the injection or only occasional minor fluctuations occur, it is classified as a non-critical fault configuration item and can be temporarily postponed for optimization, but potential risks need to be monitored. Finally, the state recorder 1431 rolls back to the initial state through a snapshot to ensure the isolation and repeatability of the experiment.

[0078] In the embodiments of the present application, Constructor 140 leverages the fault-trigger scenario construction process and fault injection capabilities to enable the system to comprehensively test and analyze service configurations. This not only improves the ability to detect potential issues, but also covers complex failure modes that are difficult to detect using traditional simple threshold matching methods. By simulating multiple scenarios, Constructor 140 can deeply explore the interactions and impacts between configuration items, proactively identify and prevent potential failure points, and thus enhance the completeness of fault detection.

[0079] The generator 150 is used to use the big model to input the first configuration item, the third configuration item and / or the fourth configuration item, as well as the configuration check rules of the rule library 110 and the best practice cases in the case library into the big model, and output modification suggestions for resource configuration.

[0080] The Case Library is a collection of successful cases, lessons learned, and industry standards. Each case is used to illustrate the best practices in a specific field or situation.

[0081] For example, Figure 5As shown, the generator 150 can prompt the generator 151 and the large language model 152. The prompt generator 151 generates a prompt template based on the first configuration item, the third configuration item and / or the fourth configuration item. A prompt template is a framework or format for guiding the generation of content, which usually contains some fixed structures and fillable variable parts. This type of template is very common in scenarios where natural language processing (NLP) and artificial intelligence (AI) generate content. In this application, the prompt template is used to extract relevant information from detection rules and cases to generate specific queries and instructions. The prompt template is generally based on experience to better ask the large model for improvement suggestions in the hope of getting a better answer. For example, a not-so-good prompt template is "A cluster, how to optimize?". A good reminder template is "There is a three-node cluster, each node is 4U8G, when the transaction volume per second (TPS) is 1000, the central processing unit (CPU) utilization of each node is 80% and the memory utilization is 50%."

[0082] Generator 150 can instruct prompt generator 151 to input the prompt template generated into large language model 152, as well as the detection rules from rule base 110 and best practice cases from the case library into large language model 152. Large language model 152 can retrieve relevant information and knowledge from the detection rules and cases based on the prompt template, and then generate customized resource configuration modification suggestions based on the retrieved information. A modification suggestion includes multiple operation instructions, each of which instructs the service to modify the state of a configured resource to a specified state.

[0083] In an embodiment of the present application, the generator 150 can provide targeted customized resource configuration modification suggestions. The generator 150 uses a large language model as an inference engine, takes resource configuration documents, fault mode libraries, and rule libraries as a knowledge base, uses fault detection results as input data, and outputs a reasonable configuration solution that conforms to the user's system. The generator 150 can provide users with intelligent configuration guidance, which not only greatly reduces the cost of users learning complex configuration knowledge, but also improves the reliability of user services. In this way, even non-professional users can easily manage and maintain service resources, thereby enhancing users' trust and satisfaction with cloud providers.

[0084] It should be understood that the functional modules and functional components involved in the above-mentioned fault detection system 100 can also be implemented through software or hardware, and the specific implementation can be determined according to actual circumstances and is not limited here. In addition, the functional modules and functional components involved in the above-mentioned fault detection system 100 can be arranged separately or integrated, and are not limited here.

[0085] The above is an introduction to the fault detection system 100 provided in the embodiment of the present application. It can be understood that the above-mentioned fault detection system 100 can be configured on a cloud management platform. For example, it can be deployed on at least one virtual machine or container instance, so that the cloud management platform can provide fault detection services. Of course, the fault detection system 100 can also be configured on a node other than the cloud management platform. For example, it can be deployed in at least one data center, or deployed on at least one server. The specific details can be determined according to the actual situation and are not limited here. Among them, the cloud management platform can provide pages related to public cloud services for users to remotely access public cloud services. In this embodiment, users can purchase the fault detection services that can be provided by the fault detection system 100 on the cloud management platform in advance. For ease of understanding, the interaction between the user and the cloud management platform is described below.

[0086] like Figure 6 As shown, the user's interaction with the cloud management platform primarily involves logging into the cloud management platform 600 through a client webpage, selecting and purchasing cloud services related to the fault detection system 100 (i.e., fault detection services) within the cloud management platform 600. After purchase, the user can then generate the fault detection system 100 on the cloud management platform 600 based on the functionality provided by the fault detection services. The cloud management platform 600 primarily manages the infrastructure for running the fault detection services. For example, the fault detection service infrastructure may include multiple data centers located in different regions, each of which includes multiple servers. Data centers may provide basic resources for the fault detection services, such as computing resources and storage resources. Therefore, when purchasing and using the fault detection services, users primarily pay for the resources used. When using the fault detection services, users can enter their requirements for the fault detection services through the configuration interface, application program interface (API), or user-interactive interface provided by the cloud management platform 600. The cloud management platform 600 then generates a fault detection service that matches the user's requirements (or other software / hardware, etc.).

[0087] In addition, some modules in the fault detection system 100 can be configured on the cloud side and others on the device side, thereby implementing fault detection services through device-cloud collaboration. In addition, the fault detection system 100 can also be configured entirely on the device side, which can be determined based on actual conditions and is not limited here.

[0088] The above is an introduction to the fault detection system provided by the embodiment of the present application. Next, based on the above content, the fault detection method provided by the embodiment of the present application is introduced.

[0089] For example, Figure 7 A flow chart of a fault detection method provided by an embodiment of the present application is shown. It is understood that the fault detection method can be executed by the above-mentioned fault detection system 100, and the specific implementation process is as follows:

[0090] Step S701: Acquire multiple configuration items.

[0091] Step S702: Obtain at least one failure mode in a failure mode library.

[0092] Configuration items are parameters used to set and adjust resources based on demand. Service configuration items can include computing resource configuration, storage configuration, network configuration, security configuration, application service configuration, etc.

[0093] Failure modes refer to the possible failure modes or failure forms of a service. They describe in detail the manifestation of the failure, its location, the scope of impact, and possible causes. A failure mode library typically stores failure modes for multiple services, such as service dependency failures, service load failures, and service node failures.

[0094] Step S703: Map multiple configuration items to at least one failure mode to obtain at least one detection rule.

[0095] Configuration documents generally record the configuration definitions of multiple services, such as configuration purpose, configuration scope, correct configuration range, dependencies, etc. In the present application, a failure mode may be caused by the combined effect of one or more unreasonable configuration items of a service, so the fault detection system 100 can map one or more of the configuration items of the service to each failure mode, and generate corresponding detection rules based on the configuration document. The fault detection system 100 stores the detection rules of multiple services in a library to form a detection rule library. The rule library uses failure modes as organizational units, and allows each failure mode to correspond to one or more configuration items, as well as inspection logic, thresholds, and conditions. The rule library is a detection rule library integrated according to failure modes, which can be updated in real time and can be supplemented and modified accordingly according to new failure modes and new service configurations.

[0096] Step S704: Perform fault detection on multiple services according to at least one detection rule.

[0097] The fault detection system 100 obtains configuration data, such as the latest resource configuration information of multiple services, and simultaneously collects monitoring data of multiple services during operation, such as performance indicators, log files, event notifications, etc. The fault detection system 100 detects whether the resource configuration and monitoring data of the service are consistent with the configuration items corresponding to the detection rules. If some configurations of the service are not in the configuration specified by the configuration items, it means that this part of the configuration items is a deterministic fault configuration item. At this time, the fault detection system 100 can use this part of the configuration items as the first configuration items. If some configurations of the service are in the configuration specified by the configuration items, it means that this part of the configuration items is a potential fault configuration item. At this time, the fault detection system 100 can use the configuration items included in this part of the detection rules as the second configuration items.

[0098] For the second configuration item, the fault detection system 100 first constructs a fault triggering scenario based on detection rules and monitoring data. The fault detection system 100 then converts the fault triggering scenario into structured instructions, specifying the fault type, triggering conditions, affected system components, and expected abnormal behavior. It then generates executable fault injection instructions and monitoring checkpoints. Subsequently, the fault detection system 100 automatically orchestrates and analyzes the dependencies between instructions, merging scenarios that can be executed in parallel to generate an ordered sequence of fault injection tasks and actually trigger the fault.

[0099] During this process, the fault detection system 100 captures the system's initial state before fault injection and creates a snapshot. During injection, the fault detection system 100 monitors system behavior in real time. After injection, the fault detection system 100 determines whether an expected fault has been triggered based on predefined thresholds. If the fault recurs stably under a specific scenario, it is marked as the third configuration item. If the injection does not cause significant anomalies or only occasional minor fluctuations, it is classified as the fourth configuration item.

[0100] In an embodiment of the present application, the method can integrate the configuration items of multiple services based on the failure modes in the failure mode library to obtain detection rules for each failure mode corresponding to one or more configuration items, allowing the system to comprehensively check complex failure modes caused by the interaction of multiple configuration items. This method can ensure that potential problems can be effectively identified and prevented in different service and configuration scenarios. In addition, compared with direct fault injection detection, this method is more targeted and can quickly locate the root cause of the problem, thereby improving detection efficiency.

[0101] Based on the above description, the present embodiment provides a fault detection device 800. Figure 8 As shown, the apparatus 800 includes:

[0102] The first processing module 810 is used to obtain multiple configuration items, each configuration item is a parameter for setting and adjusting the resources of one service among multiple services according to demand; the second processing module 820 is used to obtain at least one fault mode in the fault mode library, and the fault mode refers to the failure mode or failure form of the service; the third processing module 830 is used to map the multiple configuration items with the at least one fault mode to obtain at least one detection rule, and the detection rule is used to indicate that the target fault mode corresponds to multiple target configuration items, and the multiple target configuration items are derived from the configuration items of at least one service among the multiple services; the fourth processing module 840 is used to perform fault detection on multiple services according to the at least one detection rule.

[0103] In one embodiment, the third processing module 830 is used to obtain a configuration document, which is used to record the configuration definitions of multiple services; map multiple configuration items with at least one failure mode to obtain a mapping relationship between the multiple configuration items and the at least one failure mode; and obtain at least one detection rule based on the configuration definitions and mapping relationships of the multiple services.

[0104] In one embodiment, the fourth processing module 840 is used to obtain configuration data and monitoring data of multiple services, where the configuration data is resource configuration information of multiple services, and the monitoring data is monitoring data during the operation of multiple services; detect whether at least one detection rule satisfies the configuration data and monitoring data of multiple services; when the detection rule satisfies the configuration data and monitoring data of multiple services, obtain a first configuration item, which is a configuration item included in the detection rule that satisfies the configuration data and monitoring data of multiple services in at least one detection rule.

[0105] In one embodiment, the fourth processing module 840 is also used to construct a fault triggering scenario based on the monitoring data of multiple services; detect whether the second configuration item triggers the fault triggering scenario to generate a fault, and the second configuration item is a configuration item that does not meet the detection rules of the configuration data and monitoring data of multiple services; when the second configuration item triggers the fault triggering scenario to generate a fault, a third configuration item is obtained; when the second configuration item triggers the fault triggering scenario to generate a fault, a fourth configuration item is obtained.

[0106] In one embodiment, the fourth processing module 840 is also used to generate a prompt template based on the first configuration item, the third configuration item and / or the fourth configuration item; and generate a modification suggestion instruction based on the prompt template, at least one detection rule and the case in the case library, the modification suggestion instruction is used to instruct multiple services to modify the status of the configuration resource to a specified status, and the case is used to indicate the best operating method in a specific field or situation.

[0107] The first processing module 810, the second processing module 820, the third processing module 830, and the fourth processing module 840 can all be implemented via software or hardware. For example, the implementation of the first processing module 810 will be described below using the first processing module 810 as an example. Similarly, the implementation of the second processing module 820, the third processing module 830, and the fourth processing module 840 can refer to the implementation of the first processing module 810.

[0108] As an example of a software functional unit, the first processing module 810 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the first processing module 810 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0109] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0110] As an example of a hardware functional unit, the first processing module 810 may include at least one computing device, such as a server. Alternatively, the first processing module 810 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0111] The multiple computing devices included in the first processing module 810 can be distributed in the same region or in different regions. The multiple computing devices included in the first processing module 810 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first processing module 810 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0112] It should be noted that, in other embodiments, the first processing module 810 can be used to perform the following steps: Figure 7 In any step of the method shown, the second processing module 820 can be used to perform the following steps: Figure 7 In any step of the method shown, the third processing module 830 can be used to perform the following steps: Figure 7 In any step of the method shown, the fourth processing module 840 can be used to perform the following steps: Figure 7 In any step of the method shown, the steps that the first processing module 810, the second processing module 820, the third processing module 830 and the fourth processing module 840 are responsible for implementing can be specified as needed, and the steps as shown in the following are respectively implemented by the first processing module 810, the second processing module 820, the third processing module 830 and the fourth processing module 840. Figure 7 The different steps in the method shown implement the overall functionality of the apparatus 800 .

[0113] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 9As shown, computing device 900 includes a bus 910, a processor 920, a memory 930, and a communication interface 940. Processor 920, memory 930, and communication interface 940 communicate with each other via bus 910. Computing device 900 may be a server, a computer, a portable notebook, a cabinet, etc. It should be understood that this application does not limit the number of processors and memories in computing device 900.

[0114] The bus 910 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus 910 may include a path for transmitting information between various components of the computing device 900 (eg, the processor 920, the memory 930, and the communication interface 940).

[0115] The processor 920 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0116] The memory 930 may include a volatile memory, such as a random access memory (RAM). The memory 930 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0117] The memory 930 stores executable program codes, and the processor 920 executes the executable program codes to respectively implement the functions of the aforementioned multiple modules, such as the first processing module 810, the second processing module 820, the third processing module 830, and the fourth processing module 840, thereby implementing the following: Figure 7 That is, the memory 930 stores the method for executing Figure 7 Instructions for the method shown.

[0118] Alternatively, the memory 930 stores executable codes, and the processor 920 executes the executable codes to respectively implement the functions of the aforementioned modules, thereby achieving the following: Figure 7 That is, the memory 930 stores the method for executing Figure 7 Instructions for the method shown.

[0119] The communication interface 940 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0120] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0121] like Figure 10 As shown, the computing device cluster includes at least one computing device 900. The memory 930 in one or more computing devices 900 in the computing device cluster may store the same Figure 7 Instructions for the method shown.

[0122] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store a program for executing the following steps: Figure 7 In other words, the combination of one or more computing devices 900 can jointly execute instructions for performing the method shown in FIG. Figure 7 Instructions for the method shown.

[0123] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the first processing module 810, the second processing module 820, the third processing module 830, and the fourth processing module 840. In other words, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more of the first processing module 810, the second processing module 820, the third processing module 830, and the fourth processing module 840.

[0124] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 11 A possible implementation is shown. Figure 11As shown, two computing devices, computing device 900A and computing device 900B, are connected via a network. Specifically, the connection to the network is achieved through a communication interface in each computing device. In this possible implementation, the memory 930 in computing device 900A stores instructions for executing the functions of some modules in the first processing module 810 and the second processing module 820. Simultaneously, the memory 930 in computing device 900B stores instructions for executing the functions of other modules in the third processing module 830 and the fourth processing module 840.

[0125] Figure 11 The connection mode between the computing device clusters shown can be considered as provided in this application. Figure 7 The method shown requires a large amount of data storage, so it is considered to entrust the functions implemented by other modules in the above-mentioned first processing module 810, second processing module 820, third processing module 830 and fourth processing module 840 to be executed by the computing device 900B.

[0126] It should be understood that Figure 11 The functionality of the computing device 900A shown in FIG. 1 may also be implemented by multiple computing devices 900. Similarly, the functionality of the computing device 900B may also be implemented by multiple computing devices 900.

[0127] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Figure 9 and Figure 10 The difference is that the memory 930 of one or more computing devices 900 in the computing device cluster may store the same memory for executing the following Figure 7 Instructions for the method shown.

[0128] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store a program for executing the following steps: Figure 7 In other words, the combination of one or more computing devices 900 can jointly execute instructions for performing the method shown in FIG. Figure 7 Instructions for the method shown.

[0129] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions for executing partial functions of the computing device 900. That is, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more of the first processing module 810, the second processing module 820, the third processing module 830, and the fourth processing module 840 described above.

[0130] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the following instructions: Figure 7 The method shown.

[0131] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the following steps: Figure 7 The method shown.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A fault detection method, characterized in that: The method is applied to a management platform, the management platform is used to manage infrastructure, the infrastructure includes at least one node, and the at least one node is used to deploy multiple services. The method includes: Acquire multiple configuration items, each configuration item being a parameter for setting and adjusting resources of one of the multiple services according to demand; Acquire at least one failure mode in a failure mode library, where the failure mode refers to a failure mode or failure form that occurs in the multiple services; Mapping the multiple configuration items to the at least one failure mode to obtain at least one detection rule, where the detection rule is used to indicate that a target failure mode corresponds to multiple target configuration items; Fault detection is performed on the multiple services according to the at least one detection rule.

2. The method according to claim 1, characterized in that Mapping the multiple configuration items with the at least one fault mode to obtain at least one detection rule includes: Obtaining a configuration document, where the configuration document is used to record configuration definitions of the multiple services; Mapping the plurality of configuration items with the at least one failure mode to obtain a mapping relationship between the plurality of configuration items and the at least one failure mode; The at least one detection rule is obtained according to the configuration definitions of the multiple services and the mapping relationship.

3. The method according to claim 1 or 2, characterized in that The performing fault detection on the multiple services according to the at least one detection rule includes: Acquire configuration data and monitoring data of the multiple services, wherein the configuration data is resource configuration information of the multiple services, and the monitoring data is monitoring data of the multiple services during operation; detecting whether the at least one detection rule satisfies the configuration data and monitoring data of the plurality of services; When the detection rule satisfies the configuration data and monitoring data of the multiple services, a first configuration item is obtained, where the first configuration item is a configuration item included in the at least one detection rule that satisfies the configuration data and monitoring data of the multiple services.

4. The method according to claim 3, characterized in that The method further comprises: Constructing a fault triggering scenario based on the monitoring data of the multiple services; detecting whether a second configuration item triggers the fault triggering scenario to generate a fault, where the second configuration item is a configuration item included in a detection rule that does not satisfy the configuration data and monitoring data of the multiple services; When the second configuration item triggers the fault triggering scenario and a fault occurs, a third configuration item is obtained; When the second configuration item triggers the fault triggering scenario and a fault is generated, a fourth configuration item is obtained.

5. The method according to claim 3 or 4, characterized in that The method further comprises: Generate a prompt template according to the first configuration item, the third configuration item and / or the fourth configuration item; Based on the prompt template, the at least one detection rule and the cases in the case library, a modification suggestion instruction is generated, wherein the modification suggestion instruction is used to instruct the multiple services to modify the status of the configuration resources to a specified status, and the case is used to indicate the best operating method in a specific field or situation.

6. A fault detection device, characterized in that: include: A first processing module is configured to obtain a plurality of configuration items, each of which is a parameter for setting and adjusting resources of one of the plurality of services according to demand; A second processing module is configured to obtain at least one failure mode in a failure mode library, where the failure mode refers to a failure mode or failure form of the service; a third processing module, configured to map the multiple configuration items with the at least one failure mode to obtain at least one detection rule, wherein the detection rule is configured to indicate that a target failure mode corresponds to multiple target configuration items; A fourth processing module is configured to perform fault detection on the multiple services according to the at least one detection rule.

7. A computing device cluster, characterized in that: include: at least one computing device, each computing device comprising a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device performs the method according to any one of claims 1 to 5.

9. A computer program product comprising instructions, characterized in that The computer program product stores instructions, which, when executed by a computing device cluster, enable the computing device cluster to implement the method according to any one of claims 1 to 5.

Images