Fault processing method and related equipment

Abstract

The invention provides a fault processing method and related equipment. The method is applied to a cloud management platform. The method comprises the steps that under the condition that at least one of M services breaks down and multiple pieces of alarm information exist, first alarm information is provided for a tenant, and the first alarm information is generated by summarizing alarm information triggered according to the same source fault in the multiple pieces of alarm information according to a first operation and maintenance model; receiving first input information of a tenant; and in response to the first input information, providing first output information for the tenant. According to the method, at least one piece of alarm information triggered by the same root fault in a plurality of pieces of alarm information can be processed, so that one piece of valuable alarm information corresponding to each root fault is generated, the alarm storm is effectively inhibited, the reason of the fault can be provided for the tenant through natural language interaction with the tenant, and the user experience is improved. And thus, the fault processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and more specifically, to a fault handling method, a computing device, a cluster of computing devices, a computer program product, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of AI technology, intelligent operation and maintenance (O&M) technology has emerged. Intelligent O&M technology uses AI models to centrally manage O&M data, facilitating timely alerts when service failures occur and helping to determine the cause of the failure. However, due to the relatively low maturity of current intelligent O&M technology, it still requires O&M personnel with high levels of expertise, especially when dealing with numerous services and complex dependencies. For example, when a service fails, current intelligent O&M technology still requires manual assistance from O&M personnel for troubleshooting, leading to longer processing times and lower efficiency. Furthermore, in O&M scenarios involving alarm storms—the issuance of massive amounts of alarms in a short period—the sheer volume of alarm information forces O&M personnel to spend significant time sifting through the vast amount of data to identify valuable alarms, further reducing the efficiency of fault handling.

[0003] Therefore, improving the efficiency of fault handling has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a fault handling method, a computing device, a computing device cluster, a computer program product, and a computer-readable storage medium, which can effectively suppress alarm storms and improve the efficiency of determining the cause of faults, thereby improving the efficiency of fault handling.

[0005] Firstly, a fault handling method is provided. This method is applied to a cloud management platform for managing infrastructure providing cloud services. This infrastructure includes at least one computing node running M services, where M is a positive integer. The method includes: providing a first alarm message to a tenant when at least one of the M services fails and multiple alarm messages exist. This first alarm message indicates that at least one service has failed. The first alarm message is generated by summarizing alarm messages triggered by the same root cause fault from among the multiple alarm messages according to a first operation and maintenance model. This first operation and maintenance model is trained based on a historical fault database and deployment information of the M services. The multiple alarm messages are generated based on a set of operational information for the M services, which includes at least one operational message. The method also includes: receiving first input information from a tenant requesting the cause of the root cause fault corresponding to the first alarm message; and responding to the first input information by providing the tenant with first output information indicating the cause of the root cause fault corresponding to the first alarm message. This first output information is generated according to the first operation and maintenance model.

[0006] In this embodiment, when at least one of the M services fails, resulting in multiple alarm messages, the cloud management platform can process at least one alarm message triggered by the same root cause fault according to the first operation and maintenance model. This generates one valuable alarm message corresponding to each root cause fault, thereby reducing the number of alarm messages provided to the tenant and effectively suppressing alarm storms. Simultaneously, the cloud management platform can interact with the tenant using natural language through the first operation and maintenance model. Based on the tenant's input, it can provide the tenant with the cause of the fault, improving the efficiency of determining the cause and facilitating the tenant to resolve the fault based on the cause, thus improving the efficiency of fault handling.

[0007] In conjunction with the first aspect, in some implementation methods, the historical fault database includes at least one of the following: at least one historical operation and maintenance case, a fault mode library, or a diagnostic tree. The historical operation and maintenance case includes at least one of the following: historical fault alarm information, the location of the fault corresponding to the historical fault alarm information, the cause of the fault corresponding to the historical fault alarm information, and the handling measures for the fault corresponding to the historical fault alarm information. The fault mode library includes at least one of the following: fault identification information, fault location, fault cause, fault handling measures, fault level, and the correlation between faults. The diagnostic tree includes a first diagnostic tree and / or a second diagnostic tree. The first diagnostic tree is used to determine the root cause of the fault based on the fault alarm information, and the second diagnostic tree is used to determine the handling measures for the fault based on the root cause of the fault.

[0008] In conjunction with the first aspect, in some implementation methods, the deployment information of the M services includes at least one of the following: information about the computing node cluster where the computing nodes running the M services are located, the relationship between the M services, and the service topology of the M services.

[0009] In this embodiment of the application, a first operation and maintenance model is trained by using a historical fault database and deployment information of M services as training data. This first operation and maintenance model can classify, interpret, or summarize multiple alarm messages, thereby generating a first alarm message.

[0010] In conjunction with the first aspect, in some implementation methods, the runtime information set includes at least one of the following: a data indicator information set, a call chain information set, and a log information set. Specifically, the data indicator information set includes at least one data indicator during the operation of M services; the call chain information set includes call chain information of at least one request during processing in the operation of M services; and the log information set includes at least one log information during the operation of M services.

[0011] In conjunction with the first aspect, in some implementation methods, a first set of abnormal information is determined based on the set of operational information of M services and a second operation and maintenance model. The second operation and maintenance model is obtained by training based on the historical set of operational information of M services. The first set of abnormal information includes at least one abnormal information, and each abnormal information in the at least one abnormal information is operational information that meets the abnormal characteristics in the set of operational information. Based on the first set of abnormal information, multiple alarm information is determined.

[0012] In this embodiment, the real-time operation information set of M services is analyzed by the second operation and maintenance model to identify abnormal operation information in the operation information set and generate at least one alarm message, thereby facilitating the provision of important and valuable alarm information to tenants and effectively suppressing alarm storms.

[0013] In conjunction with the first aspect, in some implementations, the second operation and maintenance model includes at least one of the following: a first sub-model, a second sub-model, or a third sub-model. The method further includes: training a first initial model based on a set of historical data metrics information from M services to obtain a first sub-model; and / or training a second initial model based on a set of historical call chain information from M services to obtain a second sub-model; and / or training a third initial model based on a set of historical log information from M services to obtain a third sub-model.

[0014] The first sub-model is used to determine a first data indicator information set from the data indicator information sets of M services. This first data indicator information set includes at least one abnormal first data indicator, and it belongs to a first abnormal information set. The second sub-model is used to determine a first call chain information set from the call chain information sets of M services. This first call chain information set includes at least one abnormal call chain information, and it belongs to the first abnormal information set. The third sub-model is used to determine a first log information set from the log information sets of M services. This first log information set includes at least one abnormal log information, and it belongs to the first abnormal information set. These first, second, and third initial models are the foundation model (FM).

[0015] In this embodiment of the application, the basic model is trained by using the historical data indicator information set, call link information set, and log information set of M services as training data, thereby obtaining at least one of the following: a first sub-model for extracting abnormal data indicator information, a second sub-model for extracting abnormal call link information, or a third sub-model for extracting abnormal log information. This facilitates the analysis of a large amount of runtime data and the generation of alarm information.

[0016] In conjunction with the first aspect, in some implementation methods, an initial large model is trained based on a historical fault database and the deployment information of M services to obtain a first operation and maintenance model, which is a large language model (LLM).

[0017] In this embodiment, by adding the historical fault database and deployment information of M services to the LLM training corpus, the LLM is fine-tuned and enhanced by vertical domain knowledge and retrieval, thereby obtaining a first operation and maintenance model for generating the first alarm information and the first output information, which effectively suppresses alarm storms and improves the efficiency of fault handling.

[0018] In conjunction with the first aspect, in some implementation methods, the first operation and maintenance model searches in the historical fault database and / or the operation information set of M services based on the first input information to determine the first response information set, which includes at least one response information related to the first input information; the first operation and maintenance model generates the first output information based on the first response information set.

[0019] In this embodiment of the application, the first operation and maintenance model retrieves the response information related to the first alarm information from the historical fault database and / or the operation information set based on the first input information, thereby generating the first output information, and then providing the tenant with information related to the root cause fault corresponding to the first alarm information, helping the tenant to identify the root cause fault as soon as possible.

[0020] In conjunction with the first aspect, in some implementation methods, if the root cause fault corresponding to the first alarm information is not included in the historical fault database, at least one of the following shall be added to the historical fault database: the first alarm information, the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, and the handling measures for the root cause fault corresponding to the first alarm information.

[0021] In this embodiment, when the historical fault database includes the first alarm information or the root cause fault corresponding to the first alarm information, the first operation and maintenance model determines the relevant information of the root cause fault corresponding to the first alarm information by querying the historical fault database and directly feeds it back to the tenant, thereby improving the efficiency of fault handling. When the historical fault database does not include the first alarm information or the root cause fault corresponding to the first alarm information, the first operation and maintenance model generates the relevant information of the root cause fault corresponding to the first alarm information and stores it in the historical fault database, so that when the first alarm information reappears, the relevant information of the root cause fault corresponding to the first alarm information can be fed back to the tenant as soon as possible.

[0022] In conjunction with the first aspect, in some implementations, the first output information also includes: the location of the root cause fault corresponding to the first alarm information and / or the handling measures for the root cause fault corresponding to the first alarm information.

[0023] In this embodiment, the cloud management platform can provide tenants with the cause and / or location of the root cause of the fault corresponding to the first alarm information, and can also provide tenants with repair suggestions or generate fault reports, thereby improving the efficiency of fault handling.

[0024] In a second aspect, a computing device is provided. This device includes modules for implementing the first aspect or any possible implementation thereof.

[0025] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method of the first aspect or any implementation thereof described above.

[0026] Fourthly, a computer program product containing instructions is provided, which, when executed by a cluster of computer devices, causes the cluster of computer devices to perform the method described in the first aspect or any one of the implementations of the first aspect.

[0027] Fifthly, a computer-readable storage medium is provided, including computer program instructions, which, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the method described in the first aspect or any one of the implementations of the first aspect. Attached Figure Description

[0028] Figure 1 This is a schematic structural diagram of a fault handling system according to an embodiment of this application.

[0029] Figure 2 This is a schematic flowchart of a fault handling method according to an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of a first diagnostic tree according to an embodiment of this application.

[0031] Figure 4 This is a schematic diagram of a second diagnostic tree according to an embodiment of this application.

[0032] Figure 5 This is a schematic diagram of a second graphical interface according to an embodiment of this application.

[0033] Figure 6 This is a schematic flowchart illustrating a method for determining a first alarm message according to an embodiment of this application.

[0034] Figure 7 This is a schematic structural block diagram of a computing device according to an embodiment of the present application.

[0035] Figure 8 This is a schematic structural diagram of a computing device according to an embodiment of this application.

[0036] Figure 9 This is a schematic structural diagram of a computing device cluster according to an embodiment of this application.

[0037] Figure 10 This is a schematic diagram showing a network connection between computing devices 800A and 800B according to an embodiment of this application. Detailed Implementation

[0038] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0039] This application will present various aspects, embodiments, or features relating to a system comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0040] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0041] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0042] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0043] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0044] The method described in this application embodiment can be applied to various cloud management platforms. The cloud management platform manages the infrastructure providing cloud services, including at least one computing node. This computing node can be, for example, a processing unit, processor, virtual machine, container, computing device, etc. The at least one computing node runs M services, where M is a positive integer. At least one of the M services is a service provided by a cloud vendor or a service deployed by a tenant.

[0045] Figure 1 This is a schematic structural diagram of the fault handling system 100 provided in the embodiments of this application. Figure 1 The fault handling system 100 includes a cloud management platform 110. This cloud management platform 110 can be used to manage the infrastructure providing cloud services, including at least one data center (e.g., data center 120). The data center 120 may include at least one compute node cluster, each of which includes at least one compute node. For example, the data center 120 includes compute node clusters 130 and 140, where compute node cluster 130 includes compute nodes 131 and 132, and compute node cluster 140 includes compute nodes 141 and 142. Tenants can apply for access to resources in data center 120 through the cloud management platform 110. These tenants are public cloud tenants who have registered public cloud accounts and purchased public cloud resources.

[0046] In some embodiments, multiple computing nodes in a computing node cluster are directly connected or connected via a network, such as a wide area network or a local area network.

[0047] In some embodiments, the data center 120 may further include at least one storage node cluster, each storage node cluster including at least one storage node. This application embodiment does not limit the type of storage node; for example, the storage node may be a centralized storage node or a distributed storage node. Storage nodes are used to store tenant data, or to store data required and / or generated when performing the methods in this application embodiment. Exemplarily, multiple storage nodes in a storage node cluster are directly connected or connected via a network, such as a wide area network (WAN) or a local area network (LAN).

[0048] In some embodiments, the at least one data center is used to run M services, where M is a positive integer. The M services belong to a cloud vendor and / or a tenant.

[0049] The cloud management platform 110 provides a first alarm message to the tenant when at least one of the M services fails and multiple alarm messages exist. This first alarm message indicates that at least one service has failed. It is generated by summarizing alarm messages triggered by the same root cause failure from among the multiple alarm messages using a first operation and maintenance model. This first operation and maintenance model is trained based on a historical fault database and deployment information of the M services. The multiple alarm messages are generated based on a set of operational information for the M services, which includes at least one operational information.

[0050] In some embodiments, when a root cause failure occurs in at least one of the M services, that root cause failure may lead to at least one other failure, thereby generating at least one alarm message corresponding to each failure. In other words, the root cause failure is the fundamental reason for the generation of the at least one failure or the at least one alarm message.

[0051] For example, if a computing device used to run a service experiences a primary failure, it may result in at least one of the following: a computing instance (e.g., a virtual machine, container, etc.) running on the computing device fails; a service running on the computing device or the computing instance fails; or other services that call the service fail. This may trigger at least one of the following alarm messages: an alarm message indicating that the computing device has failed; an alarm message indicating that a computing instance (e.g., a virtual machine, container, etc.) running on the computing device has failed; an alarm message indicating that a service running on the computing device has failed; or an alarm message indicating that other services that call the service to perform a certain function have failed.

[0052] In some embodiments, the cloud management platform 110 inputs multiple alarm messages into the first operation and maintenance model to obtain the output of the first operation and maintenance model: at least one first alarm message. Each of the at least one first alarm message is generated based on alarm messages corresponding to the same root cause fault among the multiple alarm messages. For example, the first operation and maintenance model classifies, interprets, or summarizes the multiple alarm messages to generate the at least one first alarm message. In other words, the first operation and maintenance model is used to process multiple alarm messages and summarize alarm messages corresponding to the same root cause fault into one alarm message, thereby providing valuable alarm information to tenants, reducing the number of alarm messages provided to tenants, and thus effectively suppressing alarm storms. The historical fault database, deployment information of M services, and set of operational information are described in step 210.

[0053] In some embodiments, the cloud management platform 110 trains an initial large model based on a historical fault database and deployment information of M services to obtain the first operation and maintenance model. In other words, the cloud management platform 110 uses the historical fault database and deployment information of M services as training data to train the initial large model, so that the obtained first operation and maintenance model can classify, interpret, or summarize multiple alarm messages, thereby generating the first alarm message.

[0054] For example, the initial large model is an LLM.

[0055] In some embodiments, the cloud management platform 110 determines a first set of abnormal information based on a set of operational information for M services and a second operation and maintenance model. The second operation and maintenance model is trained based on historical operational information sets of the M services. The first set of abnormal information includes at least one abnormal piece of information, where each abnormal piece of information is operational information in the operational information set that meets abnormal characteristics. The cloud management platform 110 also determines multiple alarm messages based on the first set of abnormal information.

[0056] In some embodiments, the runtime information in the runtime information set that meets the abnormal characteristics includes: data indicators whose values ​​do not fall within a preset range, call chain information indicating call anomalies, and log information conforming to a preset format. The log information conforming to the preset format includes, for example, log information containing preset keywords and log information indicating runtime errors. This application embodiment does not limit the preset keywords; for example, it may include "error" or "warning." The abnormal characteristics can be predefined by the cloud management platform, tenant, or operations and maintenance personnel.

[0057] In some embodiments, the cloud management platform 110 trains an initial model based on a set of historical operational information for M services to obtain the second operation and maintenance model. In other words, the cloud management platform 110 uses the set of historical operational information for M services as training data to train the initial model, enabling the obtained second operation and maintenance model to extract operational information from the operational information set that conforms to abnormal characteristics. See details... Figure 2 or Figure 6 The description in the text.

[0058] The cloud management platform 110 is also used to receive first input information from the tenant. This first input information is used to request the cause of the root cause failure corresponding to the first alarm information. The cloud management platform 110 is also used to provide first output information to the tenant in response to the first input information. This first output information is used to indicate the cause of the root cause failure corresponding to the first alarm information. This first output information is generated based on a first operation and maintenance model. In other words, the cloud management platform 110 can engage in natural language dialogue with the tenant through the first operation and maintenance model, thereby providing the tenant with the cause of the root cause failure corresponding to the first alarm information.

[0059] In some embodiments, the cloud management platform 110 inputs first input information into a first operation and maintenance model, causing the first operation and maintenance model to search in a historical fault database and / or a set of operational information for M services based on the first input information, determine a first set of response information, and generate the first output information based on the first set of response information. The first set of response information includes at least one response information related to the first input information.

[0060] In some embodiments, the first output information includes at least one of the following: the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, and the handling measures for the root cause fault corresponding to the first alarm information.

[0061] In some embodiments, the first operation and maintenance model queries a historical fault database based on the first input information. If the first alarm information or the root cause fault corresponding to the first alarm information does not exist in the historical fault database, the first operation and maintenance model queries a set of operational information for M services, thereby determining the first response information set based on the historical fault database and / or the set of operational information. If the first alarm information or the root cause fault corresponding to the first alarm information exists in the historical fault database, the first operation and maintenance model determines the first response information set based on the relevant information of the root cause fault corresponding to the first alarm information included in the historical fault database. The first response information set includes relevant information of the root cause fault corresponding to the first alarm information. The relevant information of the root cause fault corresponding to the first alarm information includes at least one of the following: the alarm information corresponding to the root cause fault, the cause of the root cause fault, the location of the root cause fault, and the handling measures corresponding to the root cause fault.

[0062] In some embodiments, if the historical fault database does not include the first alarm information or the root cause fault corresponding to the first alarm information, at least one of the following shall be added to the historical fault database: the first alarm information, the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, the handling measures for the root cause fault corresponding to the first alarm information, and the first output information.

[0063] Optionally, the cloud management platform 110 can also receive second input information from the tenant and, in response to the second input information, provide the tenant with second output information. The second input information is used to request the cause of the root cause fault corresponding to the first alarm information. The content of the second input information differs from that of the first input information. The second output information is used to indicate the cause of the root cause fault corresponding to the first alarm information, and this second output information is generated based on the first operation and maintenance model. The content of the second output information may be the same as or different from that of the first output information; this embodiment of the application does not limit this. In other words, the cloud management platform 110 can engage in multi-turn natural language dialogue with the tenant through the first operation and maintenance model, thereby gradually determining the cause of the root cause fault corresponding to the alarm information based on the alarm information provided by the first operation and maintenance model, thereby improving the efficiency of fault handling.

[0064] Figure 1 When at least one of the M services fails, resulting in multiple alarm messages, the fault handling system 100 can, according to a first operation and maintenance model, process at least one alarm message triggered by the same root cause fault among these multiple alarm messages, generating one valuable alarm message corresponding to each root cause fault. This reduces the number of alarm messages provided to the tenant, thereby effectively suppressing alarm storms. Simultaneously, the fault handling system 100 can interact with the tenant using natural language through the first operation and maintenance model, providing the tenant with the cause of the fault based on the tenant's input information. This improves the efficiency of determining the cause of the fault, making it easier for the tenant to resolve the fault based on the cause, thus improving the efficiency of fault handling.

[0065] Figure 2 This is a schematic flowchart of the fault handling method provided in the embodiments of this application. Figure 2 The methods described can be executed by the cloud management platform, for example... Figure 1 The cloud management platform 110. Figure 2 The method includes the following steps.

[0066] 210. If at least one of the M services fails and multiple alarm messages exist, provide the tenant with the first alarm message.

[0067] When at least one root cause failure occurs in at least one of M services, each of these root cause failures may cause at least one other failure besides the root cause failure itself, resulting in the cloud management platform generating multiple alarm messages. That is, each of these multiple alarm messages corresponds to a failure, or each alarm message is triggered by a failure. The multiple alarm messages correspond to at least one root cause failure. Based on these multiple alarm messages and a first operation and maintenance model, the cloud management platform generates a first alarm message and provides it to the tenant. This first alarm message indicates that at least one service has failed. The first alarm message is generated by summarizing alarm messages triggered by the same root cause failure from among the multiple alarm messages according to the first operation and maintenance model. In other words, the cloud management platform processes multiple alarm messages through the first operation and maintenance model, thereby summarizing alarm messages corresponding to the same root cause failure into a single alarm message, thus providing valuable alarm information to the tenant, reducing the number of alarm messages provided to the tenant, and effectively suppressing alarm storms.

[0068] In some embodiments, an alarm storm refers to the generation of a large number of alarm messages in a short period of time, causing the number of alarm messages to exceed the limit that the operation and maintenance personnel can handle.

[0069] In some embodiments, the M services belong to a cloud vendor and / or a tenant. The M services run on at least one compute node.

[0070] Optionally, before step 210, the cloud management platform obtains a first operation and maintenance model. This first operation and maintenance model is trained based on a historical fault database and deployment information of M services.

[0071] In some embodiments, the historical fault database includes at least one of the following: at least one historical operation and maintenance case, a fault mode library, or a diagnostic tree. The historical operation and maintenance case includes at least one of the following: historical fault alarm information, the location of the fault corresponding to the historical fault alarm information, the cause of the fault corresponding to the historical fault alarm information, and the handling measures for the fault corresponding to the historical fault alarm information. The fault mode library includes at least one of the following: fault identification information, the alarm information corresponding to the fault, the location of the fault, the cause of the fault, the handling measures for the fault, the fault level, and the correlation between faults. The diagnostic tree includes a first diagnostic tree and / or a second diagnostic tree, wherein the first diagnostic tree is used to determine the root cause of the fault based on the fault alarm information, and the second diagnostic tree is used to determine the handling measures for the fault based on the root cause of the fault.

[0072] For example, a fault level is used to indicate the severity of a fault. The severity of a fault is determined based on at least one of the following: the semantics of the fault alarm message, the alarm frequency of the fault alarm message, the alarm period of the fault alarm message, the alarm time of the fault alarm message, and the type of fault. For example, a lower alarm frequency indicates a higher fault severity. Alternatively, a stronger periodicity of alarms indicates a lower fault severity. Alternatively, alarm messages occurring during peak business hours correspond to a higher fault severity. Alternatively, alarm messages that suddenly occur after the service has been running for a period of time correspond to a higher fault severity. Alternatively, application-level faults are considered more severe than operating system-level faults, network-level faults, and memory-level faults. Alternatively, the fault severity may be predefined by the cloud management platform, the tenant, or operations personnel.

[0073] For example, the association between faults is used to indicate the hierarchical relationship between multiple faults and / or whether multiple faults belong to the same type. For instance, if the occurrence of fault 1 is directly or indirectly related to fault 2, then there is a hierarchical relationship between fault 1 and fault 2; if the occurrence of fault 1 is unrelated to fault 2, then there is no hierarchical relationship between fault 1 and fault 2.

[0074] For example, the first diagnostic tree can be represented as a tree structure. The root node of the first diagnostic tree is used to indicate fault alarm information, the first-level leaf nodes are the direct causes that trigger the fault alarm information, the second-level leaf nodes are the direct causes that trigger the fault causes indicated by the first-level leaf nodes, and so on. That is, the leaf nodes of the first diagnostic tree represent the causes that directly or indirectly trigger the fault alarm information.

[0075] For example, the first diagnostic tree, such as Figure 3 As shown. Figure 3 This is a schematic diagram of the first diagnostic tree 300 provided in an embodiment of this application. In the first diagnostic tree 300, the root node is fault alarm information 1, the first-level leaf nodes include 3 leaf nodes, and the second-level leaf nodes include 3 leaf nodes. The 3 leaf nodes in the first-level leaf nodes are used to indicate fault cause 1, fault cause 2, and fault cause 3, respectively, which are the direct causes of triggering fault alarm information 1. The 3 leaf nodes in the second-level leaf nodes are used to indicate fault cause 1-1, fault cause 1-2, and fault cause 1-3, respectively, which are the direct causes of fault cause 1. That is, fault cause 1-1, fault cause 1-2, and fault cause 1-3 are the indirect causes of triggering fault alarm information 1. Figure 3 It can also be seen that the root cause of the fault alarm information 1 is at least one of the following: fault cause 1-1, fault cause 1-2, fault cause 1-3, fault cause 2, fault cause 3.

[0076] For example, the second diagnostic tree can be represented as a tree structure. The root node of the second diagnostic tree represents the cause of the fault, the first-level leaf nodes represent the direct measures to resolve the cause of the fault, the second-level leaf nodes represent the measures to further resolve the cause of the fault based on the first-level leaf nodes, and so on. That is, the leaf nodes of the second diagnostic tree represent the measures to directly or indirectly resolve the cause of the fault.

[0077] For example, the second diagnostic tree, such as Figure 4 As shown. Figure 4 This is a schematic diagram of the second diagnostic tree 400 provided in an embodiment of this application. In the second diagnostic tree 400, the root node is fault cause 1, the first-level leaf nodes include 3 leaf nodes, and the second-level leaf nodes include 3 leaf nodes. The 3 leaf nodes in the first-level leaf nodes are used to indicate processing measure 1, processing measure 2, and processing measure 3, respectively. These processing measures 1, 2, and 3 are direct processing measures to resolve fault cause 1. The 3 leaf nodes in the second-level leaf nodes are used to indicate processing measure 1-1, 1-2, and 1-3, respectively. These processing measures 1-1, 1-2, and 1-3 are further processing measures to resolve fault cause 1 based on processing measure 1. That is, these processing measures 1-1, 1-2, and 1-3 are indirect processing measures to resolve fault cause 1.

[0078] It should be understood that Figure 3 , Figure 4 For illustrative purposes only, the embodiments of this application do not limit the number of leaf nodes at each level in the first or second diagnostic tree, or the number of leaf nodes in each leaf node.

[0079] In some embodiments, the deployment information of the M services includes at least one of the following: information about the computing node cluster where the computing nodes running the M services are located, the association between the M services, and the service topology of the M services.

[0080] For example, the information of the computing node cluster where the computing nodes running M services are located includes at least one of the following: the identification information of the computing node cluster where the computing node running each of the M services is located, the information of the computing nodes that belong to the same computing node cluster among the computing nodes running the M services, etc.

[0081] For example, the compute node cluster is, for instance, a region-level or availability zone (AZ)-level compute device cluster.

[0082] For example, the relationships between the M services include at least one of the following: dependencies between the M services, and hierarchical relationships between the M services. The dependencies indicate information about other services that need to be invoked when running a service. The hierarchical relationships indicate information about at least one sub-service included in a service or information about the service to which a sub-service belongs. The sub-service belongs to the M services.

[0083] For example, the service topology of the M services is used to describe the dependencies between the services. The service topology includes at least one node and at least one edge connecting the at least one node. Each node represents a service, and an edge connecting two nodes represents a request between the two services. Different colors can also be used in the service topology to represent the health status of nodes or edges.

[0084] In some embodiments, the first operation and maintenance model is used to classify or reduce noise in the multiple alarm messages according to the fault levels corresponding to the multiple alarm messages, thereby generating a first alarm message. The first alarm message is generated based on at least one alarm message from the multiple alarm messages that corresponds to a fault of the same type or level.

[0085] In some embodiments, the first operation and maintenance model is used to determine the correlation between the multiple alarm messages based on the semantics of each alarm message, and then generate a first alarm message based on the correlation between the multiple alarm messages. The first alarm message is generated based on at least one alarm message that has a correlation among the multiple alarm messages.

[0086] For example, the correlation between alarm information is determined based on at least one of the alarm information's trigger time, trigger location, and semantic relevance. For instance, the first operation and maintenance model, based on the associated scenario, associates at least one alarm information within a preset time window into a set describing a specific problem in the current scenario according to the trigger time, trigger location, and semantic relevance, thereby generating the first alarm information based on this set.

[0087] In some embodiments, the cloud management platform directly obtains the pre-trained first operation and maintenance model. Alternatively, the cloud management platform trains the initial large model based on a historical fault database and deployment information of M services to obtain the first operation and maintenance model.

[0088] For example, the cloud management platform uses supervised fine-tuning (SFT) technology to train an initial large model using a historical fault database and deployment information of M services as training data, thereby obtaining a first operation and maintenance model. This first operation and maintenance model is used to classify, interpret, and summarize at least one alarm message, thereby aggregating at least one alarm message triggered by the same root cause fault into a single alarm message output.

[0089] For example, the cloud management platform trains the initial large model based on at least one of the following: a historical fault database, deployment information of M services, and a set of historical operation information of M services, to obtain the first operation and maintenance model.

[0090] For example, the initial large model is an LLM. This application embodiment does not limit the specific type of the LLM; any LLM in the prior art that can achieve the above functions can be applied to the method provided in this application embodiment.

[0091] In some embodiments, the historical fault database or the deployment information of M services can be represented in various forms, such as text, images, etc. This application embodiment does not limit the specific representation of the historical fault database or the deployment information of M services. When the historical fault database and / or the deployment information of M services are represented as images, the cloud management platform performs image-to-text processing on the images to obtain the text information in the images and the relationships between the text information. This text information and the relationships between the text information are then used as training data to train the initial large model.

[0092] Optionally, the multiple alarm messages are generated based on a set of operational information for M services, which includes at least one operational message.

[0093] In some embodiments, the runtime information set includes at least one of the following: a data metric set, a tracing set, and a logging set. The data metric set includes at least one data metric from the M service runtime processes. The tracing set includes tracing information for at least one request during processing in the M service runtime processes. The logging set includes at least one log entry from the M service runtime processes.

[0094] In some embodiments, the cloud management platform determines a first set of abnormal information based on a set of operational information for M services and a second operation and maintenance model. The second operation and maintenance model is trained based on historical operational information sets of the M services. The first set of abnormal information includes at least one abnormal piece of information, where each of the at least one abnormal piece of information is operational information in the operational information set that meets abnormal characteristics. The cloud management platform also determines multiple alarm messages based on the first set of abnormal information. See the description in step 610 for details.

[0095] In some embodiments, the runtime information set that meets the abnormal characteristics includes: data indicators whose values ​​do not fall within a preset range, call chain information indicating call anomalies, and log information conforming to a preset format. The log information conforming to the preset format includes, for example, log information containing preset keywords and log information indicating runtime errors. This application embodiment does not limit the preset keywords; for example, it may include "error" and "warning."

[0096] In some embodiments, before step 210, the cloud management platform obtains the pre-trained second operation and maintenance model. Alternatively, the cloud management platform trains the initial model based on a set of historical operational information for M services to obtain the second operation and maintenance model.

[0097] For example, the initial model is an FM. This application embodiment does not limit the specific type of the FM; any FM that can achieve the above functions in the prior art can be applied to the method provided in this application embodiment.

[0098] Optionally, the cloud management platform provides a first graphical interface to the tenant and displays the first alarm information in the first graphical interface, allowing the tenant to view the first alarm information through the first graphical interface. The specific form of the first graphical interface is not limited in this embodiment.

[0099] 220, Receive the tenant's first input information.

[0100] The cloud management platform receives the first input information from the tenant, which is used to request the cause of the root cause of the first alarm information.

[0101] Optionally, the cloud management platform provides a second graphical interface for the tenant, allowing the tenant to select, input, or upload the first input information within this second graphical interface. The specific form of this second graphical interface is not limited in this embodiment.

[0102] 230, in response to the first input information, provides the tenant with the first output information.

[0103] After receiving the first input information, the cloud management platform responds by providing the tenant with first output information. This first output information indicates the cause of the root cause of the fault corresponding to the first alarm information. This first output information is generated based on the first operation and maintenance model.

[0104] For example, the first input information is a question entered by the tenant, and the first output information is the answer to that question.

[0105] Optionally, the cloud management platform provides a third graphical interface for the tenant and displays the first output information in this third graphical interface, allowing the tenant to view the first output information through the third graphical interface. The specific form of this third graphical interface is not limited in this embodiment. Alternatively, the cloud management platform displays the first output information in a second graphical interface, allowing the tenant to view the first output information through the second graphical interface.

[0106] For example, the second graphical interface is as follows Figure 5 As shown. Figure 5 This is a schematic diagram of the second graphical interface 500 provided in an embodiment of this application. Figure 5 As shown, the second graphical interface 500 includes controls 510 and 520. Control 510 is used to display first input information entered by the tenant, and control 520 is used to display first output information in response to the first input information. For example, the second graphical interface 500 may also include a control 530, which is used to receive input from the tenant, such as first input information, second input information, etc.

[0107] Optionally, the cloud management platform inputs the first input information into the first operation and maintenance model to obtain the output of the first operation and maintenance model, i.e., the first output information. The first operation and maintenance model searches a historical fault database and / or a set of operational information for M services based on the first input information to determine a first set of response information. This first set of response information includes at least one response piece of information related to the first input information. The first operation and maintenance model also generates first output information based on the first set of response information.

[0108] In some embodiments, the first operation and maintenance model uses retrieval-augmented generation (RAG) technology to search in a historical fault database and / or a set of operational information for M services to determine a first response information set.

[0109] In some embodiments, the first operation and maintenance model queries a historical fault database based on the first input information. If the first alarm information or the fault corresponding to the first alarm information does not exist in the historical fault database, the first operation and maintenance model queries a set of operational information for M services to determine the first response information set. At least one response information in the first response information set is determined based on information related to the first alarm information in the historical fault database and / or the set of operational information. For example, the at least one response information is information related to the first alarm information in the historical fault database and / or the set of operational information, or the at least one response information is determined by summarizing information related to the first alarm information in the historical fault database and / or the set of operational information. Information related to the first alarm information in the historical fault database and / or the set of operational information includes at least one of the following: alarm information similar to the first alarm information in the historical fault database, related information of alarm information similar to the first alarm information in the historical fault database, operational information used to generate the first alarm information in the set of operational information, and alarm information related to the operational information used to generate the first alarm information in the set of operational information. If the first alarm information or the fault corresponding to the first alarm information exists in the historical fault database, the first operation and maintenance model determines a first response information set based on the relevant information of the first alarm information included in the historical fault database. The first response information set includes relevant information of the first alarm information. This relevant information includes at least one of the following: the root cause fault corresponding to the first alarm information, the cause of the root cause fault, the location of the root cause fault, and the corresponding handling measures.

[0110] In some embodiments, if the historical fault database does not include the first alarm information or the fault corresponding to the first alarm information, the cloud management platform will add at least one of the following to the historical fault database: the first alarm information, the root cause fault corresponding to the first alarm information, the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, the handling measures for the root cause fault corresponding to the first alarm information, and the first output information. In other words, the cloud management platform can expand the historical fault database so that when the first alarm information reappears, it can directly provide the tenant with the cause of the root cause fault corresponding to the first alarm information based on the information included in the historical fault database, thereby improving the efficiency of fault handling.

[0111] Optionally, the first output information includes at least one of the following: the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, and the handling measures for the root cause fault corresponding to the first alarm information.

[0112] Optionally, the cloud management platform can also receive second input information from the tenant and, in response to this second input information, provide the tenant with second output information. This second input information is used to request the cause of the root cause fault corresponding to the first alarm information. The content of this second input information differs from that of the first input information. The second output information is used to indicate the cause of the root cause fault corresponding to the first alarm information, and this second output information is generated based on the first operation and maintenance model. The content of this second output information may be the same as or different from that of the first output information; this embodiment of the application does not limit this. In other words, the cloud management platform can engage in multi-turn natural language dialogue with the tenant through the first operation and maintenance model, thereby gradually determining the cause of the root cause fault corresponding to the alarm information based on the alarm information provided by the first operation and maintenance model, thereby improving the efficiency of fault handling.

[0113] In some embodiments, after the cloud management platform inputs the first input information into the first operation and maintenance model, the first operation and maintenance model generates a first diagnostic tree and / or a second diagnostic tree corresponding to the first alarm information based on the deployment information of M services and a historical fault database. Based on the first diagnostic tree, the first operation and maintenance model, through natural language dialogue with the tenant, starts from the root node of the first diagnostic tree (i.e., the first alarm information) and performs layer-by-layer diagnosis through multiple rounds of natural language dialogue to determine the root cause of the fault corresponding to the first alarm information, and provides this information to the tenant. And / or, based on the second diagnostic tree, the first operation and maintenance model, through natural language dialogue with the tenant, starts from the root node of the second diagnostic tree (i.e., the root cause of the fault corresponding to the first alarm information) and performs layer-by-layer investigation through multiple rounds of natural language dialogue to determine the handling measures for the root cause of the fault corresponding to the first alarm information, and provides these measures to the tenant.

[0114] Figure 2 The method described above can handle situations where at least one of M services fails, resulting in multiple alarm messages. Based on the first operation and maintenance model, it processes at least one alarm message triggered by the same root cause failure, generating a single valuable alarm message corresponding to each root cause failure. This reduces the number of alarm messages provided to the tenant, effectively suppressing alarm storms. Simultaneously, the cloud management platform can interact with the tenant using natural language through the first operation and maintenance model. Based on the tenant's input, it provides the tenant with the cause of the failure, improving the efficiency of determining the cause and facilitating tenant resolution of the failure, thus enhancing the efficiency of fault handling.

[0115] In some embodiments, the method for determining the first alarm information in step 210 is as follows: Figure 6 As shown.

[0116] Figure 6 This is a schematic flowchart of a method for determining the first alarm information provided in an embodiment of this application. Figure 6The methods described can be executed by the cloud management platform, for example... Figure 1 The cloud management platform 110. Figure 6 The method includes the following steps.

[0117] 610, obtain the set of operation information for the first operation and maintenance model, the second operation and maintenance model, and M services.

[0118] Optionally, the cloud management platform directly obtains the pre-trained first operation and maintenance model. Alternatively, the cloud management platform trains the first operation and maintenance model based on a historical fault database and the deployment information of M services. Or, the cloud management platform trains the first operation and maintenance model based on a historical fault database, the deployment information of M services, and a set of historical operational information for the M services. M is a positive integer. The implementation method for the cloud management platform to train and obtain the first operation and maintenance model is described in step 210. The first operation and maintenance model, the historical fault database, the deployment information of the M services, and the set of operational information for the M services are described in step 210.

[0119] Optionally, the cloud management platform directly obtains the pre-trained second operation and maintenance model. Alternatively, the cloud management platform trains the second operation and maintenance model based on the historical operation information set of the M services. This second operation and maintenance model is described in step 210. The historical operation information set of the M services refers to the set of operation information generated by the M services during their historical operation.

[0120] In some embodiments, the second operation and maintenance model includes at least one of the following: a first sub-model, a second sub-model, or a third sub-model. The first sub-model is used to determine a first data indicator information set from a set of data indicator information for M services. This first data indicator information set includes at least one anomalous first data indicator, meaning that each first data indicator in the first data indicator information set exhibits anomalous characteristics. The second sub-model is used to determine a first call link information set from a set of call link information for M services. This first call link information set includes at least one anomalous call link information, meaning that each call link information in the first call link information set exhibits anomalous characteristics. The third sub-model is used to determine a first log information set from a set of log information for M services. This first log information set includes at least one anomalous log information, meaning that each log information in the first log information set exhibits anomalous characteristics.

[0121] For example, data indicators that meet the abnormal characteristics include data indicators whose values ​​do not fall within a preset range. Call chain information that meets the abnormal characteristics includes call chain information indicating a call exception. The meaning of "call exception" is similar to "call failure" and "call error," and they can be used interchangeably. Log information that meets the abnormal characteristics includes log information that conforms to a preset format. This log information conforming to the preset format includes, for example, log information containing preset keywords, log information indicating runtime errors, etc. This application embodiment does not limit the preset keywords; for example, it may include "error," "warning," etc.

[0122] For example, the above-mentioned abnormal characteristics can be predefined by the cloud management platform, tenants or operation and maintenance personnel, and the meaning of the abnormal characteristics in this application embodiment is not limited.

[0123] In some embodiments, the cloud management platform trains a first initial model based on a set of historical data metrics information of M services to obtain the first sub-model. And / or, the cloud management platform trains a second initial model based on a set of historical call chain information of M services to obtain the second sub-model. And / or, the cloud management platform trains a third initial model based on a set of historical log information of M services to obtain the third sub-model.

[0124] For example, the first initial model, the second initial model, and the third initial model are FMs. The specific type of FM is not limited in the embodiments of this application; any FM that can achieve the above functions in the prior art can be applied to the method provided in the embodiments of this application.

[0125] Optionally, the cloud management platform directly obtains a set of real-time operational information for the M services. Alternatively, the cloud management platform manages the operation of the M services, thereby generating a set of real-time operational information for the M services.

[0126] 620. Based on the second operation and maintenance model and the set of operation information of M services, determine the first set of abnormal information.

[0127] After obtaining the second operation and maintenance model and the real-time operational information set of M services, the cloud management platform inputs the real-time operational information set of the M services into the second operation and maintenance model to obtain the output of the second operation and maintenance model, namely, the first abnormal information set. In other words, during the real-time operation of the M services, the cloud management platform determines the abnormal information in the operational information set based on the operational information set of the M services, thereby facilitating the generation of multiple alarm messages.

[0128] In some embodiments, the cloud management platform inputs a set of real-time data metrics information for M services into a first sub-model to obtain the output of the first sub-model, i.e., obtains a first set of data metrics information. And / or, the cloud management platform inputs a set of real-time call chain information for M services into a second sub-model to obtain the output of the second sub-model, i.e., obtains a first set of call chain information. And / or, the cloud management platform inputs a set of real-time log information for M services into a third sub-model to obtain the output of the third sub-model, i.e., obtains a first set of log information. This first set of anomaly information includes at least one of the following: a first set of data metrics information, a first set of call chain information, and a first set of log information.

[0129] 630. Based on the first set of abnormal information, determine at least one alarm message.

[0130] After obtaining the first set of abnormal information, the cloud management platform analyzes and processes at least one abnormal information included in the first set of abnormal information to generate at least one alarm message.

[0131] In some embodiments, the cloud management platform inputs the first set of anomaly information into an Artificial Intelligence for Operations (AIOps) model to obtain the output of the AIOps model, i.e., to obtain at least one alarm message. This AIOps model is a model that empowers traditional Internet technology (IT) operations and maintenance management with big data, artificial intelligence, or machine learning technologies. The AIOps model is used for intelligent data analysis to achieve functions such as anomaly detection, root cause analysis of faults, and optimization suggestions. This application embodiment does not limit the specific type of AIOps model; any AIOps model in the prior art that can achieve the above functions can be applied to the method provided in this application embodiment.

[0132] 640. Based on the first operation and maintenance model and at least one alarm message, determine the first alarm message.

[0133] After receiving at least one alarm message, the cloud management platform inputs the alarm message into the first operation and maintenance model and obtains the output of the first operation and maintenance model, which is the first alarm message. This first alarm message is generated by the first operation and maintenance model by summarizing the alarm messages triggered by the same root cause fault from the at least one alarm message.

[0134] In some embodiments, the first operation and maintenance model performs alarm noise reduction, alarm deduplication, alarm merging, and alarm summary on the at least one alarm information, thereby providing the alarm analysis results, i.e. the first alarm information, to the tenant through natural language.

[0135] exist Figure 6 In this method, the cloud management platform uses a second operation and maintenance model to analyze the real-time operational information set of M services, thereby identifying abnormal operational information within the set and generating at least one alarm message based on this abnormal information. Simultaneously, the cloud management platform uses a first operation and maintenance model to process this at least one alarm message, generating a first alarm message. This reduces the number of alarm messages provided to tenants, ensuring that only important and valuable alarm messages are provided, effectively suppressing alarm storms and improving the efficiency of tenant fault handling.

[0136] Figure 7 This is a schematic structural diagram of the computing device provided in the embodiments of this application. Figure 7 The computing device 700 includes a transmitting module 710 and a receiving module 720. Figure 7 The computing device 700 in the middle can be applied to cloud management platforms, for example Figure 1 The cloud management platform in China.

[0137] The computing device 700 is used for execution Figure 2 In the method described above, the sending module 710 is used to provide a first alarm message to the tenant when at least one of the M services fails and multiple alarm messages exist. The receiving module 720 is used to receive the tenant's first input information and to execute... Figure 2 Step 220 in the process. The sending module 710 is also configured to provide first output information to the tenant in response to the first input information. The sending module 710 is configured to perform... Figure 2 Steps 210 and 230 in the text. See also the multiple alarm messages, the first alarm message, the first input message, and the first output message. Figure 2 The description in the text.

[0138] In some embodiments, when the computing device 700 is used to perform Figure 6 In the method described above, the computing device 700 further includes a processing module (not shown in the figure). This processing module is used to: obtain a first operation and maintenance model, a second operation and maintenance model, and a set of operational information for M services; determine a first set of abnormal information based on the second operation and maintenance model and the set of operational information for the M services; determine at least one alarm message based on the first set of abnormal information; and determine a first alarm message based on the first operation and maintenance model and at least one alarm message. The processing module is used to execute... Figure 6 Steps 610-640 are described in the document. The first operation and maintenance model, the second operation and maintenance model, the operational information set, and the first anomaly information set are described in detail below. Figure 2 or Figure 6 The description in the text.

[0139] Both the transmitting module 710 and the receiving module 720 can be implemented in software or in hardware. For example, the implementation of the transmitting module 710 will be described below. Similarly, the implementation of the receiving module 720 can be referenced from the implementation of the transmitting module 710.

[0140] As an example of a software functional unit, the sending module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the sending module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0141] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0142] As an example of a hardware functional unit, the transmitting module 710 may include at least one computing device, such as a server. Alternatively, the transmitting module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0143] The multiple computing devices included in the sending module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the sending module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the sending module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0144] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] It should be noted that the above embodiments of the device, when executing the above methods, are only illustrative examples of the division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the sending module 710 can be used to execute any step in the above methods, and the receiving module 720 can be used to execute any step in the above methods. The steps implemented by the sending module 710 and the receiving module 720 can be specified as needed, and the device can achieve all its functions by implementing different steps in the above methods through the sending module 710 and the receiving module 720 respectively.

[0146] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0147] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0148] Figure 8 This is a schematic structural block diagram of a computing device 800 provided in an embodiment of this application. The computing device 800 may be a server, a computer, or other device with computing capabilities. Figure 8 The computing device 800 shown includes at least one processor 810 and a memory 820.

[0149] It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0150] The processor 810 executes instructions in the memory 820, causing the computing device 800 to implement the method provided in this application. Alternatively, the processor 810 executes instructions in the memory 820, causing the computing device 800 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0151] Optionally, the computing device 800 also includes a communication interface 830. The communication interface 830 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 800 and other devices or communication networks.

[0152] Optionally, the computing device 800 also includes a system bus 840, wherein the processor 810, memory 820, and communication interface 830 are respectively connected to the system bus 840. The processor 810 can access the memory 820 through the system bus 840; for example, the processor 810 can perform data read / write or code execution in the memory 820 through the system bus 840. The system bus 840 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 840 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0153] In one possible implementation, the processor 810 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in the memory 820 or the cache of the processor 810.

[0154] Optionally, the processor 810 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, the processor 810 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, the processor 810 may be a central processing unit (CPU).

[0155] The memory 820 provides runtime space for processes in the computing device 800. For example, the memory 820 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 820. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 820 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0156] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 810, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 810 will load the data required for processing into RAM for computation, and then transfer the result back out after the computation is complete.

[0157] By way of example and not limitation, memory 820 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile storage medium may be, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 820 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0158] The structures of the computing device 800 listed above are merely illustrative and are not limited thereto. The computing device 800 in this application includes various hardware components in existing computer systems. For example, the computing device 800 also includes other memories besides the memory 820, such as disk storage. Those skilled in the art should understand that the computing device 800 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 800 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 800 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 8All the devices shown.

[0159] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0160] like Figure 9 As shown, the computing device cluster includes at least one computing device 800. The memory 820 of one or more computing devices 800 in the computing device cluster may store the same instructions for performing the methods described above.

[0161] In some possible implementations, the memory 820 of one or more computing devices 800 in the computing device cluster may also each store a portion of the instructions for executing the above-described method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions of the above-described method.

[0162] It should be noted that the memories 820 in different computing devices 800 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memories 820 of different computing devices 800 can implement the functions of one or more modules within the aforementioned device.

[0163] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 One possible implementation is shown. For example... Figure 10 As shown, the two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0164] It should be understood that Figure 10 The functions of the computing device 800A shown can also be performed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be performed by multiple computing devices 800.

[0165] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be a software or program product containing instructions capable of running on a computing device cluster or stored on any available medium. When run by the computing device cluster, it causes the computing device cluster to perform the methods provided above, or causes the computing device cluster to implement the functions of the apparatus provided above.

[0166] In this embodiment, a computer-readable storage medium is also provided. This computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described above.

[0167] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0168] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0170] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0172] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A fault handling method, characterized in that, The method is applied to a cloud management platform, which manages infrastructure providing cloud services. The infrastructure includes at least one computing node, which runs M services, where M is a positive integer. The method includes: If at least one of the M services fails and multiple alarm messages exist, a first alarm message is provided to the tenant. The first alarm message is used to indicate that the at least one service has failed. The first alarm message is generated by summarizing alarm messages triggered by the same root cause failure among the multiple alarm messages according to a first operation and maintenance model. The first operation and maintenance model is trained based on a historical fault database and the deployment information of the M services. The multiple alarm messages are generated based on the set of operation information of the M services. The set of operation information includes at least one operation message. Receive the first input information from the tenant, the first input information being used to request the cause of the root cause fault corresponding to the first alarm information; In response to the first input information, the first output information is provided to the tenant. The first output information is used to indicate the cause of the root cause of the fault corresponding to the first alarm information. The first output information is generated according to the first operation and maintenance model.

2. The method according to claim 1, characterized in that, The historical fault database includes at least one of the following: at least one historical operation and maintenance case, fault mode library or diagnostic tree; The historical operation and maintenance cases include at least one of the following: historical fault alarm information, the location of the fault corresponding to the historical fault alarm information, the cause of the fault corresponding to the historical fault alarm information, and the handling measures for the fault corresponding to the historical fault alarm information. The fault mode library includes at least one of the following: fault identification information, fault location, fault cause, fault handling measures, fault level, and correlation between faults. The diagnostic tree includes a first diagnostic tree and / or a second diagnostic tree. The first diagnostic tree is used to determine the root cause of the fault based on the fault alarm information, and the second diagnostic tree is used to determine the handling measures for the fault based on the root cause of the fault.

3. The method according to claim 1 or 2, characterized in that, The deployment information of the M services includes at least one of the following: information about the computing node cluster where the computing nodes running the M services are located, the association between the M services, and the service topology of the M services.

4. The method according to any one of claims 1 to 3, characterized in that, The set of operational information includes at least one of the following: a set of data indicator information, a set of call chain information, and a set of log information; The data indicator information set includes at least one data indicator during the operation of the M services, the call link information set includes call link information of at least one request during the processing of the M services, and the log information set includes at least one log information during the operation of the M services.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Based on the set of operational information of the M services and the second operation and maintenance model, a first set of abnormal information is determined. The second operation and maintenance model is obtained by training based on the historical set of operational information of the M services. The first set of abnormal information includes at least one abnormal piece of information, and each of the at least one abnormal piece of information is operational information that meets the abnormal characteristics in the set of operational information. Based on the first set of abnormal information, the multiple alarm messages are determined.

6. The method according to claim 5, characterized in that, The second operation and maintenance model includes at least one of the following: a first sub-model, a second sub-model, or a third sub-model, and the method further includes: Based on the historical data indicator information set of the M services, a first initial model is trained to obtain a first sub-model. The first sub-model is used to determine a first data indicator information set from the data indicator information set of the M services. This first data indicator information set includes at least one anomalous first data indicator, and the first data indicator information set belongs to a first anomalous information set; and / or Based on the historical call chain information set of the M services, a second initial model is trained to obtain a second sub-model. This second sub-model is used to determine a first call chain information set from the call chain information set of the M services. The first call chain information set includes at least one abnormal call chain information, and this first call chain information set belongs to the first abnormal information set; and / or, Based on the historical log information set of the M services, the third initial model is trained to obtain the third sub-model. The third sub-model is used to determine the first log information set from the log information set of the M services. The first log information set includes at least one abnormal log information, and the first log information set belongs to the first abnormal information set. The first initial model, the second initial model, and the third initial model are the basic model FM.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the historical fault database and the deployment information of the M services, the initial large model is trained to obtain the first operation and maintenance model, wherein the initial large model is a large language model (LLM).

8. The method according to any one of claims 1 to 7, characterized in that, In response to the first input information, first output information is provided to the tenant, including: The first operation and maintenance model searches the historical fault database and / or the set of operation information of the M services based on the first input information to determine a first response information set, wherein the first response information set includes at least one response information related to the first input information; The first operation and maintenance model generates the first output information based on the first response information set.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: If the historical fault database does not include the first alarm information or the root cause fault corresponding to the first alarm information, at least one of the following shall be added to the historical fault database: the first alarm information, the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, and the handling measures for the root cause fault corresponding to the first alarm information.

10. The method according to any one of claims 1 to 9, characterized in that, The first output information also includes: the location of the root cause fault corresponding to the first alarm information and / or the handling measures for the root cause fault corresponding to the first alarm information.

11. A computing device, characterized in that, The device is applied to a cloud management platform, which manages infrastructure providing cloud services. The infrastructure includes at least one computing node, which runs M services, where M is a positive integer. The device includes: A sending module is configured to provide a first alarm message to a tenant when at least one of the M services fails and multiple alarm messages exist. The first alarm message indicates that the at least one service has failed. The first alarm message is generated by summarizing alarm messages triggered by the same root cause failure among the multiple alarm messages based on a first operation and maintenance model. The first operation and maintenance model is obtained by training based on a historical fault database and the deployment information of the M services. The multiple alarm messages are generated based on a set of operation information of the M services, and the set of operation information includes at least one operation message. The receiving module is used to receive the first input information from the tenant, wherein the first input information is used to request the cause of the root cause of the fault corresponding to the first alarm information; The sending module is further configured to provide the tenant with first output information in response to the first input information. The first output information is used to indicate the cause of the root cause failure corresponding to the first alarm information. The first output information is generated according to the first operation and maintenance model.

12. The apparatus according to claim 11, characterized in that, The historical fault database includes at least one of the following: at least one historical operation and maintenance case, fault mode library or diagnostic tree; The historical operation and maintenance cases include at least one of the following: historical fault alarm information, the location of the fault corresponding to the historical fault alarm information, the cause of the fault corresponding to the historical fault alarm information, and the handling measures for the fault corresponding to the historical fault alarm information. The fault mode library includes at least one of the following: fault identification information, fault location, fault cause, fault handling measures, fault level, and correlation between faults. The diagnostic tree includes a first diagnostic tree and / or a second diagnostic tree. The first diagnostic tree is used to determine the root cause of the fault based on the fault alarm information, and the second diagnostic tree is used to determine the handling measures for the fault based on the root cause of the fault.

13. The apparatus according to claim 11 or 12, characterized in that, The deployment information of the M services includes at least one of the following: information about the computing node cluster where the computing nodes running the M services are located, the association between the M services, and the service topology of the M services.

14. The apparatus according to any one of claims 11 to 13, characterized in that, The set of operational information includes at least one of the following: a set of data indicator information, a set of call chain information, and a set of log information; The data indicator information set includes at least one data indicator during the operation of the M services; the call link information set includes call link information of at least one request during the processing of the M services; and the log information set includes at least one log information during the operation of the M services.

15. The apparatus according to any one of claims 11 to 14, characterized in that, The device further includes a processing module, the processing module being used for: Based on the set of operational information of the M services and the second operation and maintenance model, a first set of abnormal information is determined. The second operation and maintenance model is obtained by training based on the historical set of operational information of the M services. The first set of abnormal information includes at least one abnormal piece of information, and each of the at least one abnormal piece of information is operational information that meets the abnormal characteristics in the set of operational information. Based on the first set of abnormal information, the multiple alarm messages are determined.

16. The apparatus according to claim 15, characterized in that, The second operation and maintenance model includes at least one of the following: a first sub-model, a second sub-model, or a third sub-model, and the processing module is further used for: Based on the historical data indicator information set of the M services, the first initial model is trained to obtain the first sub-model. The first sub-model is used to determine the first data indicator information set from the data indicator information set of the M services. The first data indicator information set includes at least one abnormal first data indicator. The first data indicator information set belongs to the first abnormal information set. And / or, Based on the historical call chain information set of the M services, the second initial model is trained to obtain the second sub-model. The second sub-model is used to determine the first call chain information set from the call chain information set of the M services. The first call chain information set includes at least one abnormal call chain information. The first call chain information set belongs to the first abnormal information set. And / or, Based on the historical log information set of the M services, the third initial model is trained to obtain the third sub-model. The third sub-model is used to determine the first log information set from the log information set of the M services. The first log information set includes at least one abnormal log information, and the first log information set belongs to the first abnormal information set. The first initial model, the second initial model, and the third initial model are the basic model FM.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The device further includes a processing module, the processing module being used for: Based on the historical fault database and the deployment information of the M services, the initial large model is trained to obtain the first operation and maintenance model, wherein the initial large model is a large language model (LLM).

18. The apparatus according to any one of claims 11 to 17, characterized in that, The device further includes a processing module, which is used to run the first operation and maintenance model. The first operation and maintenance model searches the historical fault database and / or the set of operation information of the M services based on the first input information to determine a first response information set, wherein the first response information set includes at least one response information related to the first input information; The first operation and maintenance model generates the first output information based on the first response information set.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The device further includes a processing module, the processing module being used for: If the historical fault database does not include the first alarm information or the root cause fault corresponding to the first alarm information, at least one of the following shall be added to the historical fault database: the first alarm information, the cause of the root cause fault corresponding to the first alarm information, the location of the root cause fault corresponding to the first alarm information, and the handling measures for the root cause fault corresponding to the first alarm information.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The first output information also includes: the location of the root cause fault corresponding to the first alarm information and / or the handling measures for the root cause fault corresponding to the first alarm information.

21. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 10.

22. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 10.

Images