Alarm root cause positioning method of operation and maintenance platform

Abstract

The invention relates to an alarm root cause positioning method of an operation and maintenance platform. The method comprises the following steps: acquiring and processing monitoring data; constructing a software and hardware topological knowledge graph; analyzing call chain data; merging the maps; processing the alarm data, and classifying the alarm data; finding an alarm causal relationship; predicting time sequence performance index data; constructing a monitoring operation and maintenance knowledge base; carrying out alarm convergence and root cause positioning; and performing alarm interpretation and processing opinions. According to the method, the topological graph is created and configured on the basis of the CMDB, related information such as call chains, alarm causal relationships and faults of performance index classes is added, and the physical link logic relationships between the components and the virtual machine or the host machine where the components are located are obtained through the CMDB so as to discover the mutual call relationships of application and service levels between the components. And the monitoring operation and maintenance knowledge base is used for carrying out professional interpretation on alarms and abnormal events and giving expert-level disposal suggestions. The expandability is considered in the system design, and more data sources and algorithms can be conveniently integrated in the future.

Description

Technical Field

[0001] The present invention relates to a method for locating the root cause of an alarm on an operation and maintenance platform, an intelligent monitoring and operation and maintenance method for a computer network, and a method for quickly and accurately locating and analyzing alarms and abnormal events in a large-scale computer system. Background Art

[0002] As enterprise systems become increasingly complex, the relationships between systems and services become more complex. Traditional monitoring and alarm systems often have difficulty quickly and accurately locating the root cause of problems when faced with massive alarms. In addition, existing root cause analysis methods rely heavily on manual experience, which is inefficient and prone to errors. In addition, some fault root cause location methods pre-set a set of knowledge graphs of fault root causes by expert systems. The root cause location process is to locate the root cause by comparing the preset knowledge graphs of the expert system. The alarm rules are limited, and it is impossible to locate the root cause comprehensively and accurately. The granularity of the location is also relatively coarse, and it is impossible to be accurate and detailed.

[0003] With the advancement of technology, a common root cause location method is to build a topological relationship knowledge graph of IT equipment and virtual machines, components, and services running on it based on CMDB (Configuration Management Database) configuration information, and comprehensively judge the root cause of the fault based on the similarity calculation or causal relationship calculation results between alarms and the topological relationship knowledge graph between the equipment, components, and services to which the alarm belongs. This method has achieved good root cause location results. However, since it can only perform alarm convergence and root cause location for alarm events, it is impossible to consider and predict the trend of timing performance indicators, thereby digging out potential system abnormal risks that have not yet alarmed, and thus locating the root cause of those abnormal events. In addition, due to the lack of call chain monitoring, it is impossible to obtain the relevant call relationships between services and applications, and it is impossible to achieve more fine-grained root cause location at the service layer and application layer. In addition, for some root cause location results, especially log alarms of professional third-party equipment, they can only be located but no professional disposal opinions can be made, and they still rely on third-party experts to assist in solving them. Summary of the invention

[0004] In order to overcome the problems of the prior art, the present invention provides a method for locating the root cause of an alarm on an operation and maintenance platform. The method optimizes the root cause processing process by combining knowledge graphs and large language models, covering the monitoring of call chains, logs, and performance indicators. In addition to the service level, the root cause of hardware, virtual machine, and middleware failures can be located, improving the efficiency of alarm convergence and the accuracy of root cause location.

[0005] The object of the present invention is achieved by: a method for locating the root cause of an alarm of an operation and maintenance platform, wherein the operation and maintenance platform used in the method is arranged in the data layer, the core layer and the application layer, and comprises: Data access and processing module set in the data layer; data storage module set in the data layer; algorithm library module set in the core layer; model library module set in the core layer; knowledge graph construction module set in the core layer; AIOps module set in the core layer; fault and trend prediction module; service capability layer includes: monitoring operation and maintenance knowledge base module; large screen display and statistical report module; alarm interpretation, positioning and disposal suggestion module; The method comprises the following steps: Step 1, monitoring data collection and processing: configure the equipment of the monitoring platform, collect various monitoring data of the host machine, and access and process the monitoring data; the configuration and monitoring data mainly include the following six categories: CMDB, call chain data, time series performance indicator data, physical topology data, log information data and alarm information data; Step 2: Constructing the software and hardware topology knowledge graph: By analyzing the switch's routing table, MAC address table, and ARP table, as well as the physical device network connection data, we build a basic host device physical topology framework. Then, combined with CMDB data, we build a hierarchical device topology knowledge graph of routers-> switches-> hosts-> virtual machines-> services-> service groups-> applications. Step 3, call chain data analysis: call chain data is used to obtain the call relationship between services and applications, the call relationship between systems, the service / IP mapping relationship, and the logical connection relationship between middleware; for each user request, the call process between services has the same tracking number, and each span between services in the call process will generate a span number. Based on the same TraceId and progressive Spanid, the call chain relationship between services can be obtained; Step 4, merge graphs: merge the relationships obtained in steps 2 and 3 to obtain a complete device topology knowledge graph, and store it in a graph database; Step 5, process the alarm data and classify it: For the hardware monitoring data and alarm information at the switch layer, power environment and physical machine layer, the monitoring platform directly obtains the monitoring data, sets the alarm threshold and obtains the device alarm through the network protocol and private monitoring protocol. The alarm information in this part includes preset monitoring templates, monitoring items, alarm items, alarm trigger thresholds, alarm levels and alarm service information. This part accurately classifies the alarms through the preset monitoring templates. For the log alarm information, a large model fine-tuned on professional corpus is used in combination with the prompt word engineering to extract and classify the knowledge entities. Finally, the alarm classification results are mapped to the corresponding nodes of the device topology knowledge graph in step 4. Step 6, alarm causal relationship discovery: First, the apriori similarity algorithm is used to find alarm frequent items and alarm association rules from historical alarm data; secondly, the PC causal algorithm is used to discover the causal relationship between alarms from historical alarm data, and determine the causal edges and weights between alarms, merge alarm similarity relationships and causal relationships, and construct an alarm convergence knowledge graph, which includes the following sub-steps: Sub-step 6.1: Select appropriate alarm time slice and set sliding window: Set the alarm time slice to 2 minutes and the sliding time window width to 1 minute, so that two adjacent slices overlap for 1 minute; Sub-step 6.2: Perform apriori similarity calculation on the alarm information in the same slice, and use the apriori similarity algorithm to find the association rule list that meets the preset minimum support and minimum confidence from the historical alarm data that has occurred, as the alarm item association rule list, combined with the monitoring device information where the alarm is located, so as to construct the knowledge graph of the alarm information; Sub-step 6.3, performing PC causal discovery calculation for the alarm information in the same slice: using the PC causal algorithm to discover the causal relationship between alarms from historical alarm data, and determining the causal edges and their weights between alarms; Sub-step 6.4, constructing an alarm convergence knowledge graph: merging the alarm similarity relationship and the alarm causal relationship, constructing an alarm convergence knowledge graph, and using a graph database to store the alarm convergence knowledge graph content; Sub-step 6.5, repeating the above steps 5 and 6 regularly, processing the newly generated alarm information to discover new alarm convergence rules and expand them into the alarm convergence knowledge graph; Step 7, prediction of time series performance indicator data: Based on the transformer architecture, the time in the historical data of the time series performance indicator is processed and pre-trained to obtain a pre-trained basic model of performance indicators based on the equipment type. The pre-trained basic model and the actual performance indicator data of the project site are used to predict the time series performance indicator data; Step 8: Build a monitoring and operation knowledge base: Introduce technical documents and expert experience documents of third-party equipment, vectorize them, and use them as an expert knowledge base for alarm events and abnormal events. Use RAG technology of large language models to interpret events and give handling opinions, including the following sub-steps Sub-step 8.1: According to the type of the document, load different document content parsing tools to identify and read the text content in the document; Sub-step 8.2: Add the image recognition content and table parsing content to the document content; Sub-step 8.3: Process the logical association information between documents and use it as the document's meta data to enter the subsequent embedding process; Sub-step 8.4: Select a suitable embedding model to vectorize the document content and document meta data. Before vectorization, you need to preset the appropriate vectorization parameters chunk_size and chunk_overlap; Step 9: Alarm convergence and root cause location: Based on the knowledge graph built up to step 5 and the large language model, analyze the alarm data in real time, quickly converge the alarm and locate the root cause; The convergence of alarm data and root cause location are divided into the following sub-steps: Sub-step 9.1, alarm time slice granularity setting: real-time acquisition of 2-minute alarm data within the time slice; Sub-step 9.2, alarm classification: For the original alarm data, combined with specific alarm information and monitoring item information, the original alarm data is classified according to the alarm classification pre-set in the monitoring template. For log alarm information, it is necessary to use the industry-fine-tuned LLM to perform semantic understanding and alarm classification; Sub-step 9.3, alarm convergence: first query the alarm convergence knowledge graph, retain the cause alarm information that meets the alarm convergence rules within the time slice, and remove the result alarm information; then query the device topology knowledge graph to further converge the alarms that cannot continue to converge through the alarm convergence knowledge graph in units of applications; Sub-step 9.4, alarm causal construction: Based on the alarm convergence results, query the connection subgraph between all nodes under each application in the graph database according to the application level, and input the obtained results into networkx to obtain the final connection relationship between the nodes under a certain application; Sub-step 9.5, root cause path sorting: based on the alarm cause-effect graph generated above and the weights, the suspected paths are calculated and sorted to give the suspected root cause paths; Sub-step 9.6, determining the root cause: according to the preset root cause weight, the root cause paths with a weight lower than the weight are discarded, and the root cause paths with a weight higher than the weight are selected as the alarm root cause, with up to three root cause paths in descending order of weight; Step 10, alarm interpretation and disposal suggestions: for different root causes, further auxiliary tool analysis is performed using the Agent intelligent workflow and TOOl calling capabilities of the large language model; The following sub-steps are provided for alarm interpretation and handling suggestions: Sub-step 10.1, auxiliary tool analysis: write the alarm root cause as input into the LLM AI Agent analysis workflow, and the LLM AI Agent workflow calls the TOOL tool for auxiliary analysis according to the type of the alarm root cause; Sub-step 10.2, RAG interpretation and recommendation: The root cause of the alarm and the analysis results of the auxiliary tool in step 7 are used as the question of the RAG search question and answer, and the question is rewritten using LLM to add descriptions related to the alarm interpretation and disposal suggestions. Then, the RAG vector similarity calculation is performed to find the document slices with high similarity in the knowledge base, and rerank them. Then, the reranked slices are post-processed, and the post-processed slice content is used as the context to fill in the prompt words, and submitted to LLM together with the question, and finally the answer corresponding to the question is generated, that is, the alarm interpretation and disposal suggestion recommendation for the alarm root cause; The detailed steps of the RAG part algorithm include the following sub-steps: Sub-step 10.2.1 Question rewriting: Use the alarm root cause content and auxiliary tool analysis results as the RAG search question input, and rewrite the question to add descriptions of alarm interpretation and handling suggestions; Sub-step 10.2.2 Question retrieval: vectorize the rewritten question, and calculate the vector similarity of the slices in the knowledge base according to the preset TopK number, and retrieve the corresponding number of slices; Sub-step 10.2.3 Slice reordering: Use the preset reranker model to calculate the similarity scores of the slices retrieved in the previous step and reorder them in descending order of scores. Slices that exceed the preset TopN number are directly discarded; Sub-step 10.2.4 Slice post-processing: If the highest-scoring slice does not exceed the preset threshold min_score, it is considered that the vector similarity is too low, and LLM is used to further understand and dig deeper into the question intent, and try to add synonyms to the keywords in the question and rewrite them, and repeat steps 2 and 3 to increase the recall range of question retrieval; if it still does not exceed the preset threshold min_score, it is considered that there is no relevant knowledge in the knowledge base to refer to the alarm information, and LLM uses external search enhancement through web search and answers based on its own knowledge; if the highest slice score exceeds the threshold min_score, the slices above the min_score are padded with adjacent upper and lower slices, and then the slice content is added to the context of the prompt word and submitted to LLM for answering together with the question; Sub-step 10.2.5 Prompt word processing and LLM answer: According to the characteristics of the operation and maintenance industry, set appropriate prompt words, and combine them with the context and questions obtained in the fourth step, and submit them to the LLM for answering, so as to provide alarm interpretation and handling suggestions for the root cause alarm.

[0006] The advantages and beneficial effects of the present invention are: the present invention is based on multiple merging and optimization of physical topology data, CMDB configuration management data and call chain tracking data to build a complete software and hardware device topology knowledge map as the basis for root cause location. Compared with the traditional single CMDB method, it can not only represent the internal logical connection relationship between various applications, services, virtual machines and physical machines in the system, but also discover the call relationship between applications and services, as well as the physical connection relationship between the underlying network equipment and the host machine. The root cause location has a higher granularity and a wider range.

[0007] In addition, the present invention also uses a large model to extract and classify keywords of auxiliary knowledge entities from log alarm information, and combines preset monitoring templates of various underlying devices, virtual machines and middleware to achieve the classification of various types of alarm information, and map them to each node in the above-mentioned software and hardware device topology knowledge graph, so as to further improve the accuracy of root cause location.

[0008] Third, when locating the root cause, the present invention not only considers the software and hardware topology knowledge graph, but also considers the important influence of alarm convergence on root cause location. By analyzing the mutual causal relationship between historical alarm information, an alarm convergence knowledge graph is constructed. In the root cause location stage, the alarm convergence knowledge graph is first used to perform alarm convergence on massive alarm information to eliminate derived alarms, and then the software and hardware topology knowledge graph is used to perform further alarm convergence and root cause location, thereby reducing redundant calculations, accelerating the root cause location speed and improving accuracy.

[0009] Fourthly, the present invention performs time series prediction on performance indicators, thereby being able to discover alarms that have not been reported yet but will likely occur in the future.

[0010] Furthermore, the prior art first constructs a corpus and pre-trains a general LLM (Large Language Mode) based on the corpus. In this way, the upper limit of the ability to train LLM is limited by the pre-trained corpus. If there is a fault that is not in the pre-trained corpus, the root cause cannot be located, and the generated fault operation and maintenance report is not accurate enough. The approach of the present invention is based on a general open source LLM large model. By constructing a large operation and maintenance knowledge base, and by using RAG (Retrieval-Augmented Generation) on a special operation and maintenance knowledge base, the illusion of the large model LLM can be eliminated to the greatest extent and the knowledge shortcomings of the large model in operation and maintenance can be supplemented. There is no need for large-scale pre-training, etc., and the requirements for hardware computing power are relatively low. In addition, the knowledge base can also be easily updated and expanded dynamically, so that the capabilities of the large model will continue to rise.

[0011] The present invention also performs further analysis of the root cause positioning results by calling auxiliary tools. For example, in the case of temperature or smoke alarms, the tool calling capability of the large model will automatically open the relevant camera to capture images and videos, analyze them, and obtain further abnormal information such as whether a fire has occurred or the thermal imager is abnormal, thereby giving more accurate handling suggestions.

[0012] The automated alarm convergence and root cause location process of the present invention greatly improves the operation and maintenance efficiency. The knowledge graph and LLM large language model are combined to improve the accuracy of root cause location. The knowledge graph provides an intuitive alarm propagation path and enhances the interpretability of the results. The monitoring and operation knowledge base is used to interpret alarms and abnormal events at a professional level and give expert-level handling opinions. The system design takes scalability into consideration to facilitate the integration of more data sources and algorithms in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0014] Figure 1 is a system architecture diagram of the operation and maintenance platform used by the method described in Embodiment 1 of the present invention; Figure 2 is a flow chart of the method described in Embodiment 1 of the present invention; Figure 3 Schematic diagram of the CMDB main field design of the method described in the first embodiment of the present invention; Figure 4 is a schematic diagram of an ARP table example of the method described in Embodiment 1 of the present invention; Figure 5It is a topological knowledge graph of the monitored device of the method described in the first embodiment of the present invention; Figure 6 Schematic diagram of the call chain tracing process of the method described in Embodiment 1 of the present invention; Figure 7 This is a log alarm example of the method described in Embodiment 1 of the present invention; Figure 8 It is the log alarm classification corpus (jsonl structure) of the method described in the first embodiment of the present invention; Fig. 9 Schematic diagram of the process of discovering alarm similarity and alarm causal relationship according to the method of Embodiment 1 of the present invention; Fig.10 Schematic diagram of the alarm convergence and root cause location process of the method described in Embodiment 1 of the present invention; Fig.11 1 is a schematic diagram of the alarm interpretation and handling suggestion process of step 10 of the method according to the first embodiment of the present invention; Fig.12 It is a schematic diagram of the RAG retrieval enhancement generation process of sub-step 10.2 of step 10 of the method described in Example 1 of the present invention. DETAILED DESCRIPTION

[0015] Example: This embodiment is a method for locating the root cause of an alarm on an operation and maintenance platform. The system architecture of the operation and maintenance platform used in the method is as follows: Figure 1 The operation and maintenance platform is arranged in the data layer, core layer and application layer, and includes: (1) The data access and processing module set up in the data layer is responsible for collecting CMDB data and CMDB update data, call chain data and physical device network topology connection data, as well as performance monitoring indicator data and various alarm data, including log alarm information.

[0016] (2) Data storage module set in the data layer: including relational database, which is responsible for the storage of monitoring hosts, monitoring items, monitoring data and alarm data, etc.; and graph database, which is responsible for the storage of the subsequently constructed software and hardware topology knowledge graph and alarm convergence knowledge graph.

[0017] (3) The algorithm library module set in the core layer: including the alarm classification algorithm, which is responsible for classifying alarm information and mapping it to the knowledge graph nodes; as well as the alarm similarity algorithm and the causal relationship discovery algorithm, which are responsible for discovering the correlation and causal relationship between alarms and building the alarm convergence knowledge graph.

[0018] (4) Model library module set in the core layer: including time series data indicator prediction model, which is used to combine the pre-trained basic model and actual historical data to perform trend prediction on performance indicator data.

[0019] and a fault prediction model for predicting the trend of status-based faults based on historical fault data.

[0020] (5) The knowledge graph construction module set in the core layer is responsible for constructing and merging the device topology knowledge graph, and mapping the alarm classification data into the device topology knowledge graph.

[0021] (6) The AIOps (Artificial Intelligence for ITOperations) module set up in the core layer includes: The alarm convergence module is responsible for converging alarm information based on the causal relationship graph of the alarm; The root cause location engine is responsible for integrating real-time alarm events and abnormal events and combining them with knowledge graphs to locate the root cause; The fault and trend prediction module uses the fault prediction model and time series data indicator prediction model in the model library module to predict faults and trends and discover faults and indicator abnormal events that are likely to occur in the future.

[0022] (7) The service capability layer includes: The monitoring operation and maintenance knowledge base module is responsible for building and expanding the operation and maintenance local knowledge base, which is used to answer questions raised by users based on the support library; The large-screen display and statistical report module is used to uniformly display system data such as monitoring indicators, alarm information, topological relationships, and to analyze and compile statistics on the monitoring status of equipment and services and download reports.

[0023] The alarm interpretation, location and disposal suggestion module is used to locate the root cause and interpret the alarm for real-time alarm events, faults and trend prediction abnormal events, and call the auxiliary tool module to perform necessary auxiliary tool calls and further abnormal analysis for different root causes, and then the LLM large model combined with the knowledge base RAG gives professional disposal suggestions.

[0024] The steps of the method are as follows Figure 2 As shown, the following steps are included: Step 1, monitoring data collection and processing: configure the equipment of the monitoring platform, collect various monitoring data of the host machine, and access and process the monitoring data; the configuration and monitoring data mainly include the following six categories: CMDB, call chain data, timing performance indicator data, physical topology data, log information data and alarm information data.

[0025] CMDB, or configuration management database, is a logical database that contains information about the entire life cycle of configuration items and the relationships between configuration items. Figure 3 As shown in the design of the main fields of the CMDB data items, the logical relationship between the host machine, virtual machine, and the components, services, and even applications carried and run on them can be learned from the CMDB information. In addition, CMDB data also includes the ownership of some resources, such as the company, department, and individual, as well as contact information. Therefore, under normal circumstances, CMDB is combined with alarm information, and alarm reminders and other functions can be realized through alarm->device->contact when an alarm occurs.

[0026] Call chain data refers to call chain monitoring data, i.e., link tracing, which is an important part of APM (application performance monitoring). With the rapid increase in the number of microservice applications, the call relationship between services and service links has become more and more complicated. Through call chain data, the mutual call relationship between services can be obtained, especially in IT systems that are becoming increasingly large in scale and increasingly complex in mutual calls. Call chain data is the key data to clarify the call relationship between services. In addition, link tracing can capture exceptions and errors encountered by each request, as well as real-time information and valuable data.

[0027] Physical topology data refers to the fact that the operation and maintenance platform can obtain the routing table, MAC address table and ARP table of the monitored switch through the Simple Network Protocol (SNMP), thereby obtaining the physical network connection relationship between the switch and the IP address of the host in its subnet. The operation and maintenance platform is an IT equipment monitoring operation and maintenance platform built through a private protocol.

[0028] By combining CMDB data, call chain data and physical topology data, we can build a complete software and hardware topology network knowledge graph from routers, switches to services and applications. The entity relationships that can be constructed include physical connection relationships, host machines, bearers, aggregations, and mutual call relationships between services and applications.

[0029] Time series performance indicator data refers to the performance monitoring of different types of monitoring equipment by the operation and maintenance platform through preset monitoring equipment templates and the collection and storage of real-time performance indicator values ​​at preset time intervals.

[0030] Log monitoring information refers to the log information collected by the operation and maintenance platform for different types of devices, services, and applications using tools (such as logstash) according to preset monitoring device templates and application monitoring templates.

[0031] Alarm information refers to abnormal information generated in the above-mentioned monitoring data, such as changes in CMDB, abnormal call chains, changes in switch topology, abnormal performance indicator data of monitored devices, abnormal log information, etc. In addition, it also includes early warning information predicted by the fault and trend prediction module that has not occurred yet but is likely to occur in the future.

[0032] Step 2, hardware and software topology knowledge graph construction: By analyzing the switch's routing table, MAC address table and ARP table, as well as the physical device network connection data, a basic host device physical topology framework is built, and then combined with CMDB data, a layered device topology knowledge graph is constructed, including routers-> switches-> host-> virtual machines-> services (applications)-> service groups (application components)-> applications. The nodes in the knowledge graph include routers, switches, physical machines, virtual machines, software, systems, etc., and the edges represent the logical or physical connection relationship between them. The physical device network connection data is mainly obtained through the operation and maintenance platform.

[0033] The operation and maintenance platform can obtain the routing table, MAC address table and ARP table of the monitored switch through the Simple Network Protocol (SNMP), thereby obtaining the physical network connection relationship between the switch and the IP address of the host in its subnet. However, the host only has IP information, such as Figure 4 There is no application-level information such as the services and systems running on it, which requires CMDB information, such as Figure 3 As shown in the figure, the host information is expanded and extended to build a hierarchical device topology knowledge graph of router->switch->host->virtual machine->service (application)->service group->application, as shown in the figure. Figure 5 shown.

[0034] Step 3: Call chain data analysis: Figure 6 As shown in the figure, the call chain data is used to obtain the call relationship between services and applications, the call relationship between systems, the service / IP mapping relationship, the logical connection relationship between middleware, etc. For each user request, the call process between services has the same tracking number (TraceId), and each span between services in the call process will generate a span number (SpanId). Based on the same TraceId and progressive Spanid, the call chain relationship between services can be obtained. The call chain data is mainly obtained through some internal API interfaces.

[0035] Step 4: Merge graphs: Merge the relationships obtained in steps 1 and 2 to obtain a complete device topology knowledge graph, such as Figure 3 As shown, and stored in a graph database.

[0036] After completing steps 2, 3, and 4, a complete knowledge graph of software and hardware device topology is obtained. In this graph, the internal logic between applications, services, virtual machines, and physical machines in the system is displayed from a global perspective, as well as the calling relationship and logical connection relationship between applications and services, and the physical connection relationship of network devices. The nodes in the graph include applications, service groups, services, virtual machines, physical machines, access switches, core switches, aggregation switches, routers, etc. The relationships include composition, calling, logical connection, aggregation, operation (carrying), host machine, physical connection, etc.

[0037] In addition, when the device topology connection relationship changes, or the CMDB configuration information is updated, repeat steps 2, 3, and 4 to modify and update the hardware and software device topology knowledge graph and re-store the graph database.

[0038] Step 5, process the alarm data and classify it: For the hardware monitoring data and alarm information at the switch layer, power environment and physical machine layer, the monitoring platform directly obtains the monitoring data, sets the alarm threshold and obtains the device alarm through the network protocol and private monitoring protocol. This part of the alarm information includes preset monitoring templates, monitoring items, alarm items, alarm trigger thresholds, alarm levels and alarm services. This part can accurately classify the alarms through the preset monitoring templates, and finally map the alarm classification results to the corresponding nodes of the device topology knowledge graph in step 4.

[0039] In addition to using the monitoring platform to obtain virtual machine, system, and service-level alarms, especially log alarms, a large language model that has been fine-tuned and optimized with industry corpus is used to extract knowledge and classify alarm text information, and map the alarm information to the corresponding node of the knowledge graph in step 4.

[0040] Add different monitoring templates for different types of monitoring devices, virtual machines, services, middleware, etc., set different monitoring items and their categories, alarm thresholds, etc., so that the alarms for these monitoring items can be directly classified. However, the classification granularity of a considerable number of alarms is relatively coarse, which is not conducive to locating the root cause, especially log alarms. In the log alarm part, the alarm sources are relatively wide, including system alarms, service alarms, hardware alarms, and the alarm formats are diverse and the styles are different. It is difficult to classify these alarms in a more fine-grained manner.

[0041] This embodiment proposes a method for extracting knowledge and classifying alarm text information based on the LLM large language model, which greatly speeds up the classification of log alarm information and improves the accuracy of log classification.

[0042] Get the equipment monitoring template, the list of preset alarm categories and their descriptions, write the alarm categories and their descriptions into the prompt words, and use the LLM model fine-tuned with tens of thousands of manually annotated corpora to classify log alarms and mine knowledge entities, thereby assisting the improvement of the knowledge graph and automatic classification of alarms.

[0043] like Figure 7 The log alarm information of a service named gracd in one embodiment is shown. Through the preset log monitoring template, we can obtain the IP address and the device to which the log information belongs, but cannot locate it specifically on a specific service.

[0044] This embodiment is based on a large number of Figure 7 The service and application log information shown is constructed by manually constructing log operation and maintenance question and answer pairs (such as Figure 8 As shown in the figure), we build the log alarm operation and maintenance corpus, and fine-tune the open source LLM model with LORA, so that the LLM model can extract knowledge based on the log alarm information, and accurately identify the service, process, time, host name, alarm level, alarm content and other information. In this way, the log alarm information can be processed and classified. On the one hand, the log alarm information is refined to the service layer, and merged with the alarm classification information obtained by the preset alarm template into the software and hardware knowledge graph constructed in the previous step as the basis for the subsequent alarm root cause location; on the other hand, the structure of the alarm classification information obtained by the preset alarm template is uniformly processed, so that this part of the alarm information can be used together with the alarm information obtained by the preset monitoring template for the subsequent alarm processing process.

[0045] Step 6, alarm causal relationship discovery: the discovery of alarm similarity relationship and alarm causal relationship. At present, there are mainly two algorithms for alarm correlation discovery, one is alarm similarity calculation, and the other is causal relationship algorithm. The two algorithms have their own advantages. This embodiment uses the two algorithms in sequence, and finally merges the calculation results of the two algorithms. First, the apriori similarity algorithm is used to find alarm frequent items and alarm association rules from historical alarm data; secondly, the PC causal algorithm is used to discover the causal relationship between alarms from historical alarm data, and determine the causal edges and weights between alarms, merge the alarm similarity relationship and causal relationship, and construct an alarm convergence knowledge graph.

[0046] The specific steps include: Fig. 9 As shown: Sub-step 6.1: Select the appropriate alarm time slice and set the sliding window: Since the monitoring time interval is 1 minute, the alarm time slice is set to 2 minutes. At the same time, in order to avoid causal alarm information being divided into different time slices, the sliding time window width is set to 1 minute, so that the two adjacent slices have a 1 minute overlap.

[0047] Sub-step 6.2: Perform apriori similarity calculation on the alarm information in the same slice: Use the apriori similarity algorithm to find a list of association rules that meet the preset minimum support and minimum confidence from the historical data of alarms that have occurred, as the list of association rules for alarm items, and combine it with the information of the monitoring device where the alarm is located to construct a knowledge graph of the alarm information.

[0048] Sub-step 6.3, performing PC causal discovery calculation for the alarm information in the same slice: using the PC causal algorithm to discover the causal relationship between alarms from historical alarm data, and determine the causal edges and their weights between alarms.

[0049] Sub-step 6.4, constructing an alarm convergence knowledge graph: merging alarm similarity relationships and alarm causal relationships, constructing an alarm convergence knowledge graph, and using a graph database to store the contents of the alarm convergence knowledge graph.

[0050] Sub-step 6.5, repeat the above steps 5 and 6 regularly, process the newly generated alarm information to discover new alarm convergence rules and expand them into the alarm convergence knowledge graph.

[0051] Step 7, prediction of time series performance indicator data: Based on the improved transformer architecture, the time in the historical data of the time series performance indicators is processed and pre-trained to obtain a pre-trained basic model of performance indicators based on the equipment type. The pre-trained basic model and the actual performance indicator data of the project site are used to predict the time series performance indicator data.

[0052] This example uses the Transformer-based time series prediction model PatchTST to predict time series indicator data. PatchTST (Patch Time Series Transformer) is a Transformer-based time series prediction model, and its core innovations include the following aspects: 1. Patching operation: The time series is divided into multiple patches according to a certain window size and step size, and each patch is input into the model as a token. This operation reduces the computational complexity and memory usage, while enabling the model to better capture the local information of the time series.

[0053] 2. Channel independence: When processing multivariate time series, PatchTST processes the data of each variable separately to form independent channels, which are then input into the model separately. This approach allows the model to better capture the unique characteristics of each variable and improve the accuracy of predictions. It also means that the PatchTST model can predict different variables at the same time, which is important for multi-indicator monitoring operation and maintenance systems.

[0054] 3. Self-supervised learning: PatchTST enhances the model’s learning ability through self-supervised learning tasks, such as training by predicting patches that are randomly masked. This training method helps improve the generalization ability of the model.

[0055] These characteristics make the PatchTST model have significant advantages in processing long-series time series prediction tasks, especially in scenarios where local features and long-term dependencies need to be captured. This embodiment is based on the above characteristics of PatchTST, and pre-trains the same type of equipment monitored by multiple indicators based on historical data over the past year or more, thereby obtaining basic prediction models for various types of equipment for the prediction of time series indicator data.

[0056] At the same time, this embodiment also supports LoRa fine-tuning of the basic prediction model in actual projects by processing the historical monitoring data of the monitoring equipment and assigning additional LoRa weights, so that the basic prediction model has smaller loss and better performance in the actual monitoring data of the actual project.

[0057] Step 8, build a monitoring and operation knowledge base: Introduce technical documents and expert experience documents of third-party equipment, vectorize them, and use them as an expert knowledge base for alarm events and abnormal events. Use RAG technology of large language models to interpret events and give handling opinions.

[0058] Since the formats of monitoring and operation and maintenance technical documents are diverse, including pdf, word, excel, html, txt, etc., which also include pictures and tables, there are also logical links between documents. Therefore, the reasonable processing and vectorization of the documents imported into the knowledge base will directly determine the accuracy of the subsequent RAG process.

[0059] It includes the following sub-steps Sub-step 8.1: Depending on the type of document, load different document content parsing tools to identify and read the text content in the document.

[0060] Sub-step 8.2: The image recognition content and table parsing content need to be added to the document content.

[0061] For the image content, load the OCR recognition model to perform content recognition on it, and add it to the text content recognized in sub-step 8.1, and use <<<<image recognition content>>>> as a marker before and after the recognized content. For the table content, split the table content with |, and add the recognized content of the first row as the title content before the table content of each row, wrapping and marking it with []. Then add it to the recognized text content in sub-step 8.1.

[0062] Sub-step 8.3: The logical association information between documents needs to be processed and used as the document's meta data to enter the subsequent embedding process.

[0063] Sub-step 8.4: Select a suitable embedding model to vectorize the document content and document meta data. Before vectorization, you need to preset the appropriate vectorization parameters chunk_size (slice size) and chunk_overlap (slice overlap size).

[0064] Step 9: Alarm convergence and root cause location: Based on the knowledge graph built up to step 5 and combined with the large language model, the alarm data is analyzed in real time to quickly converge the alarm and locate the root cause.

[0065] The convergence of alarm data and the location of the root cause can be divided into the following main sub-steps: Fig.10 shown.

[0066] Sub-step 9.1, alarm time slice granularity setting: real-time acquisition of 2-minute alarm data within the time slice; Sub-step 9.2, alarm classification: For the original alarm data, combined with specific alarm information and monitoring items and other information, the original alarm data is classified according to the alarm classification pre-set in the monitoring template. For log alarm information, it is necessary to use the industry-fine-tuned LLM to perform semantic understanding and alarm classification.

[0067] Sub-step 9.3, alarm convergence: First, query the alarm convergence knowledge graph, retain the cause alarm information that meets the alarm convergence rules within the time slice, and remove the result alarm information; then query the device topology knowledge graph to further converge the alarms that cannot be further converged through the alarm convergence knowledge graph in units of applications (such as virtual machine or physical machine alarms, converge to virtual machines or physical machines), and the convergence format is as follows: Application 1: {Device topology knowledge graph node 1: [alarm type 1, alarm type 2...], Device topology knowledge graph node 2: [alarm type 1, alarm type 2...].

[0068] Sub-step 9.4, alarm causal construction: Based on the alarm convergence results, query the connection subgraph between all nodes under each application in the graph database according to the application level, and input the obtained results into networkx to obtain the final connection relationship between the nodes under a certain application, that is, the alarm causal graph.

[0069] Sub-step 9.5, root cause path sorting: based on the alarm cause-effect graph generated above and the weights, the suspected paths are calculated and sorted to give the suspected root cause paths.

[0070] Sub-step 9.6, determining the root cause: according to the preset root cause weight, the root cause paths with a weight lower than the weight are discarded, and the root cause paths with a weight higher than the weight are selected as the alarm root cause, with up to three root cause paths arranged in descending order of weight.

[0071] Step 10, alarm interpretation and disposal suggestions: for different root causes, further auxiliary tool analysis is performed using the Agent intelligent body workflow and TOOl calling capabilities of the large language model. The auxiliary tool analysis includes but is not limited to: for the timing performance indicator as the root cause of the abnormality, the indicator prediction described in step 6 is performed to further predict the abnormality that may occur in the next step; for the temperature and humidity, smoke, water immersion and other alarms of the dynamic environment as the root cause, the camera video stream is called or the infrared thermal imager image video stream is called, and the video stream is analyzed to obtain further abnormal information. Finally, the large language model is used to perform RAG query on the operation and maintenance knowledge base of step 7, interpret the abnormal conditions analyzed by the aforementioned alarm root cause and auxiliary tools, and give disposal suggestions. For the content that does not exist in the knowledge base, auxiliary tools such as webSearch are used for search enhancement, and the large language model combines its own knowledge to give disposal suggestions.

[0072] Therefore, when a new alarm event is received, the alarm event must be analyzed first, and the alarm convergence and root cause location must be quickly performed. Then, if necessary, the TOOl calling capability of LLM can be used to perform further auxiliary tool analysis based on the alarm root cause. Finally, the knowledge base interpretation and disposal opinions of the aforementioned root cause and auxiliary analysis results are recommended through RAG.

[0073] The interpretation and handling suggestions for alarms are divided into the following sub-steps, such as Fig.11 As shown: Sub-step 10.1, auxiliary tool analysis: write the alarm root cause as input into the AI ​​Agent analysis workflow of LLM, and the AI ​​Agent workflow of LLM calls the TOOL tool for auxiliary analysis according to the type of the alarm root cause. The auxiliary tool analysis includes but is not limited to: for the timing performance indicator as an alternative root cause, perform the indicator prediction described in step 6, and further predict the abnormality that may occur in the next step; for the temperature and humidity, smoke, water immersion and other alarms of the dynamic environment as alternative root causes, call the camera video stream and the infrared thermal imager image video stream, and perform image analysis on the video stream to obtain further abnormal information.

[0074] Sub-step 10.2, RAG interpretation and recommendation suggestions: Use the root cause of the alarm together with the analysis results of the auxiliary tools in the seventh step as the question of the RAG retrieval question, and use LLM to rewrite the question, add descriptions related to the alarm interpretation and handling suggestions, and then calculate the RAG vector similarity, find the document slices with high similarity in the knowledge base, and rerank them. Then post-process the re-ranked slices, and fill in the prompt words with the post-processed slice content as the context, and give it to LLM together with the question, and finally generate the answer corresponding to the question, that is, the alarm interpretation and handling suggestion recommendation for the alarm root cause.

[0075] The detailed steps of the RAG part of the algorithm include the following sub-steps (such as Fig.12 shown): Sub-step 10.2.1, problem rewriting: Use the alarm root cause content and auxiliary tool analysis results as the RAG search question input, and rewrite the question to add descriptions of alarm interpretation and handling suggestions.

[0076] Sub-step 10.2.2, question retrieval (Retrieve): vectorize the rewritten question, and calculate the vector similarity of the slices in the knowledge base according to the preset TopK number, and retrieve the corresponding number of slices.

[0077] Sub-step 10.2.3, slice reranking: Use the preset reranker model to calculate the similarity scores of the slices retrieved in the previous step and reorder them in descending order. Slices that exceed the preset TopN number are directly discarded.

[0078] Sub-step 10.2.4, post-slice processing: If the score of the slice with the highest score does not exceed the preset threshold min_score, it is considered that the vector similarity is too low, and LLM is used to further understand and dig deeper into the question intent, and try to add synonyms to the keywords in the question, and repeat steps 2 and 3 to increase the recall range of question retrieval. If it still does not exceed the preset threshold min_score, it is considered that there is no relevant knowledge in the knowledge base to refer to the alarm information, and LLM uses external search enhancement through web search and answers based on its own knowledge. If the highest score of the slice exceeds the threshold min_score, the slices above the min_score are filled with adjacent upper and lower slices, and then the slice content is added to the context of the prompt word, and submitted to LLM for answering together with the question. The adjacent upper and lower slices are slices that are logically adjacent to the slices obtained by the slice reranking in the third step. When slicing, the complete content may be divided into two upper and lower slices, and one may be recalled and the other may be discarded in the recall and reranking stages. At this time, it is necessary to consider retrieving the upper and lower adjacent slices of the reranked slice to prevent incomplete answers.

[0079] Sub-step 10.2.5, prompt word processing and LLM answer: According to the characteristics of the operation and maintenance industry, set appropriate prompt words, and combine them with the context and questions obtained in the fourth step, and submit them to the LLM for answering, so as to provide alarm interpretation and handling suggestions for the root cause alarm.

[0080] In addition, when new equipment model operation and maintenance knowledge or new operation and maintenance related company rules and regulations and other documents appear, repeat step 8 to identify and process this part of new content and add it to the knowledge base; when there is outdated content that needs to be cleaned up, this part of content can also be removed from the knowledge base through the interface. This allows the knowledge base to be dynamically adjusted and expanded to meet new alarm interpretation needs.

[0081] Finally, it should be noted that the above is only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred arrangement scheme, a person skilled in the art should understand that the technical solution of the present invention (such as the form of the monitored system, the relevant content of the database, the sequence of steps, etc.) can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for locating the root cause of an alarm on an operation and maintenance platform, wherein the operation and maintenance platform used in the method is arranged in a data layer, a core layer, and an application layer, and comprises: A data access and processing module provided in the data layer; A data storage module arranged in the data layer; An algorithm library module provided in the core layer; A model library module is provided in the core layer; The knowledge graph building module is set in the core layer; AIOps module set in the core layer; Fault and trend prediction module; service capability layer includes: monitoring and operation knowledge base module; large screen display and statistical report module; alarm interpretation, location and disposal suggestion module; Characterized in that the method comprises the following steps: Step 1, monitoring data collection and processing: configure the equipment of the monitoring platform, collect various monitoring data of the host machine, and access and process the monitoring data; the configuration and monitoring data mainly include the following six categories: CMDB, call chain data, time series performance indicator data, physical topology data, log information data and alarm information data; Step 2: Constructing the software and hardware topology knowledge graph: By analyzing the switch's routing table, MAC address table, and ARP table, as well as the physical device network connection data, we build a basic host device physical topology framework. Then, combined with CMDB data, we build a hierarchical device topology knowledge graph of routers-> switches-> hosts-> virtual machines-> services-> service groups-> applications. Step 3, call chain data analysis: call chain data is used to obtain the call relationship between services and applications, the call relationship between systems, the service / IP mapping relationship, and the logical connection relationship between middleware; for each user request, the call process between services has the same tracking number, and each span between services in the call process will generate a span number. Based on the same TraceId and progressive Spanid, the call chain relationship between services can be obtained; Step 4, merge graphs: merge the relationships obtained in steps 2 and 3 to obtain a complete device topology knowledge graph, and store it in a graph database; Step 5, process the alarm data and classify it: For the hardware monitoring data and alarm information at the switch layer, power environment and physical machine layer, the monitoring platform directly obtains the monitoring data, sets the alarm threshold and obtains the device alarm through the network protocol and private monitoring protocol. The alarm information in this part includes preset monitoring templates, monitoring items, alarm items, alarm trigger thresholds, alarm levels and alarm service information. This part accurately classifies the alarms through the preset monitoring templates. For the log alarm information, a large model fine-tuned on professional corpus is used in combination with the prompt word engineering to extract and classify the knowledge entities. Finally, the alarm classification results are mapped to the corresponding nodes of the device topology knowledge graph in step 4. Step 6, alarm causal relationship discovery: First, the apriori similarity algorithm is used to find alarm frequent items and alarm association rules from historical alarm data; secondly, the PC causal algorithm is used to discover the causal relationship between alarms from historical alarm data, and determine the causal edges and weights between alarms, merge alarm similarity relationships and causal relationships, and construct an alarm convergence knowledge graph, which includes the following sub-steps: Sub-step 6.1: Select appropriate alarm time slice and set sliding window: Set the alarm time slice to 2 minutes and the sliding time window width to 1 minute, so that two adjacent slices overlap for 1 minute; Sub-step 6.2: Perform apriori similarity calculation on the alarm information in the same slice, and use the apriori similarity algorithm to find the association rule list that meets the preset minimum support and minimum confidence from the historical alarm data that has occurred, as the alarm item association rule list, combined with the monitoring device information where the alarm is located, so as to construct the knowledge graph of the alarm information; Sub-step 6.3, performing PC causal discovery calculation for the alarm information in the same slice: using the PC causal algorithm to discover the causal relationship between alarms from historical alarm data, and determining the causal edges and their weights between alarms; Sub-step 6.4, constructing an alarm convergence knowledge graph: merging the alarm similarity relationship and the alarm causal relationship, constructing an alarm convergence knowledge graph, and using a graph database to store the alarm convergence knowledge graph content; Sub-step 6.5, repeating the above steps 5 and 6 regularly, processing the newly generated alarm information to discover new alarm convergence rules and expand them into the alarm convergence knowledge graph; Step 7, prediction of time series performance indicator data: Based on the transformer architecture, the time in the historical data of the time series performance indicator is processed and pre-trained to obtain a pre-trained basic model of performance indicators based on the equipment type. The pre-trained basic model and the actual performance indicator data of the project site are used to predict the time series performance indicator data; Step 8: Build a monitoring and operation knowledge base: Introduce technical documents and expert experience documents of third-party equipment, vectorize them, and use them as an expert knowledge base for alarm events and abnormal events. Use RAG technology of large language models to interpret events and give handling opinions, including the following sub-steps Sub-step 8.1: According to the type of the document, load different document content parsing tools to identify and read the text content in the document; Sub-step 8.2: Add the image recognition content and table parsing content to the document content; Sub-step 8.3: Process the logical association information between documents and use it as the document's meta data to enter the subsequent embedding process; Sub-step 8.4: Select a suitable embedding model to vectorize the document content and document meta data. Before vectorization, you need to preset the appropriate vectorization parameters chunk_size and chunk_overlap; Step 9: Alarm convergence and root cause location: Based on the knowledge graph built up to step 5 and the large language model, analyze the alarm data in real time, quickly converge the alarm and locate the root cause; The convergence of alarm data and root cause location are divided into the following sub-steps: Sub-step 9.1, alarm time slice granularity setting: real-time acquisition of 2-minute alarm data within the time slice; Sub-step 9.2, alarm classification: For the original alarm data, combined with specific alarm information and monitoring item information, the original alarm data is classified according to the alarm classification pre-set in the monitoring template. For log alarm information, it is necessary to use the industry-fine-tuned LLM to perform semantic understanding and alarm classification; Sub-step 9.3, alarm convergence: first query the alarm convergence knowledge graph, retain the cause alarm information that meets the alarm convergence rules within the time slice, and remove the result alarm information; then query the device topology knowledge graph to further converge the alarms that cannot continue to converge through the alarm convergence knowledge graph in units of applications; Sub-step 9.4, alarm causal construction: Based on the alarm convergence results, query the connection subgraph between all nodes under each application in the graph database according to the application level, and input the obtained results into networkx to obtain the final connection relationship between the nodes under a certain application; Sub-step 9.5, root cause path sorting: based on the alarm cause-effect graph generated above and the weights, the suspected paths are calculated and sorted to give the suspected root cause paths; Sub-step 9.6, determining the root cause: according to the preset root cause weight, the root cause paths with a weight lower than the weight are discarded, and the root cause paths with a weight higher than the weight are selected as the alarm root cause, with up to three root cause paths in descending order of weight; Step 10, alarm interpretation and disposal suggestions: for different root causes, further auxiliary tool analysis is performed using the Agent intelligent workflow and TOOl calling capabilities of the large language model; The following sub-steps are provided for alarm interpretation and handling suggestions: Sub-step 10.1, auxiliary tool analysis: write the alarm root cause as input into the LLM AI Agent analysis workflow, and the LLM AI Agent workflow calls the TOOL tool for auxiliary analysis according to the type of the alarm root cause; Sub-step 10.2, RAG interpretation and recommendation: The root cause of the alarm and the analysis results of the auxiliary tool in step 7 are used as the question of the RAG search question and answer, and the question is rewritten using LLM to add descriptions related to the alarm interpretation and disposal suggestions. Then, the RAG vector similarity calculation is performed to find the document slices with high similarity in the knowledge base, and rerank them. Then, the reranked slices are post-processed, and the post-processed slice content is used as the context to fill in the prompt words, and submitted to LLM together with the question, and finally the answer corresponding to the question is generated, that is, the alarm interpretation and disposal suggestion recommendation for the alarm root cause; The detailed steps of the RAG part algorithm include the following sub-steps: Sub-step 10.2.1 Question rewriting: Use the alarm root cause content and auxiliary tool analysis results as the RAG search question input, and rewrite the question to add descriptions of alarm interpretation and handling suggestions; Sub-step 10.2.2 Question retrieval: vectorize the rewritten question, and calculate the vector similarity of the slices in the knowledge base according to the preset TopK number, and retrieve the corresponding number of slices; Sub-step 10.2.3 Slice reordering: Use the preset reranker model to calculate the similarity scores of the slices retrieved in the previous step and reorder them in descending order of scores. Slices that exceed the preset TopN number are directly discarded; Sub-step 10.2.4 Slice post-processing: If the highest-scoring slice does not exceed the preset threshold min_score, it is considered that the vector similarity is too low, and LLM is used to further understand and dig deeper into the question intent, and try to add synonyms to the keywords in the question and rewrite them, and repeat steps 2 and 3 to increase the recall range of question retrieval; if it still does not exceed the preset threshold min_score, it is considered that there is no relevant knowledge in the knowledge base to refer to the alarm information, and LLM uses external search enhancement through webSearch and answers based on its own knowledge; if the highest slice score exceeds the threshold min_score, the slices above the min_score are filled with adjacent upper and lower slices, and then the slice content is added to the context of the prompt word and submitted to LLM for answering in combination with the question; Sub-step 10.2.5 Prompt word processing and LLM answer: According to the characteristics of the operation and maintenance industry, set appropriate prompt words, and combine them with the context and questions obtained in the fourth step, and submit them to the LLM for answering, so as to provide alarm interpretation and handling suggestions for the root cause alarm.

Images