A method for establishing an operation and maintenance alarm association knowledge base based on a large model

Abstract

This invention discloses a method for building an operation and maintenance alarm association knowledge base based on a large model, involving operation and maintenance data processing technology. The method includes: standardizing heterogeneous alarms through an alarm docking and adaptation module; filtering invalid and false alarm data from the structured alarm data; constructing a two-layer retrieval knowledge base based on explicit and implicit topology networks; performing association analysis between real-time alarm data and the knowledge base to generate a real-time alarm knowledge graph and matching it with historical graphs using a graph attention network; and dynamically updating the knowledge base based on alarm activity and changes in topology relationships. This invention provides unified structuring processing of alarm data, removing invalid and false alarm data, improving alarm effectiveness, saving operation and maintenance time, and increasing operation and maintenance efficiency. Simultaneously, the knowledge base built on a two-layer topology network, combined with a graph attention network, filters association graphs to assist in quickly locating the source of problems, improving alarm location accuracy and efficiency.

Description

Technical Field

[0001] This invention relates to the field of operation and maintenance data processing technology, specifically a method for establishing an operation and maintenance alarm association knowledge base based on a large model. Background Technology

[0002] With the rapid development of information technology, the scale and complexity of enterprise equipment systems are constantly increasing, and operation and maintenance work is facing unprecedented challenges. Traditional operation and maintenance methods mainly rely on manual monitoring and rule setting, which is not only inefficient, but also difficult to handle the rapid processing and accurate location of massive alarm information. Using knowledge base to set rules for automated operation and maintenance is an important means to liberate manpower, reduce personnel burden, and cope with alarm storms.

[0003] However, most existing knowledge bases are based on static rules or single-model reasoning, which are insufficient to cope with increasingly complex and variable operation and maintenance scenarios. For example, equipment failures may trigger chain reactions, and traditional knowledge bases lack multi-hop reasoning capabilities, failing to automatically link and correlate alarms. Furthermore, alarm data is scattered across multiple monitoring systems, making it difficult to unify data entry formats, leading to decreased accuracy of the knowledge base. In addition, alarm noise and a large number of false alarms consume significant operation and maintenance resources, resulting in decreased accuracy of real alarm warnings and reduced operation and maintenance efficiency. Therefore, we propose a method for linking operation and maintenance alarms to a knowledge base based on a large model. Summary of the Invention

[0004] The purpose of this invention is to provide a method for establishing an operation and maintenance alarm association knowledge base based on a large model to solve the above-mentioned problems.

[0005] This invention can be achieved through the following technical solution: a method for establishing an operation and maintenance alarm association knowledge base based on a large model, comprising the following steps:

[0006] Step 1: Standardize heterogeneous alarms through the alarm integration and adaptation module, including: obtaining multi-source operation and maintenance alarm logs and reconstructing them into JSON-formatted structured data containing core fields and tag fields;

[0007] Step 2: Filter invalid and false alarm data from the structured and coordinated alarm data;

[0008] Step 3: Construct a two-layer retrieval knowledge base based on explicit and implicit topology networks;

[0009] Step 4: Perform correlation analysis between real-time alarm data and knowledge base to generate a real-time alarm knowledge graph and match it with historical graphs using a graph attention network;

[0010] Step 5: Dynamically update the knowledge base based on alarm activity and topology changes.

[0011] A further technical improvement of the present invention is that: the standardization process in step one specifically includes:

[0012] Define core fields including UID, data source, alarm level, timestamp, and original alarm text; build a mapping template to match the alarm fields of each platform with the defined core fields;

[0013] The original alarm text is semantically parsed using the BERT model, and the device IP, the process in which it is located, the error code, and the alarm description are extracted as tag fields.

[0014] A further technical improvement of the present invention is that: the invalid data filtering in step two includes:

[0015] Validate field format and missing key values ​​using regular expressions;

[0016] Data that exceeds the normal range in the overall alarm data will be defined as invalid data.

[0017] The alarm data stream is segmented using a time window, the alarm density is calculated, and Word2Vec feature vector clustering is performed on the alarms in the high-density window to retain the earliest alarm data in the cluster.

[0018] A further technical improvement of the present invention is that: the false alarm data filtering in step two includes:

[0019] Periodic false alarm detection: The alarm timing characteristics are analyzed by Fourier transform to identify periodic spike signals, and after verification, they are added to the block list;

[0020] Traffic aggregation false alarm detection: Construct a formula to evaluate the probability of false alarms based on the distribution of source IP / destination IP, number of accesses and traffic volume, and filter alarms that exceed the threshold;

[0021] Threshold sensitivity false alarm detection: Based on the rate of change of the average value of the indicator in adjacent update cycles, the median value of the threshold is migrated and updated while keeping the threshold width unchanged.

[0022] A further technical improvement of this invention lies in the following: the formula for evaluating the probability of false alarms is:

[0023]

[0024] Where K represents the probability that an alarm generated within the time period [t, t+Δt] is a false alarm, a, b, c, and d are weighting coefficients, and e is a natural constant in mathematics. This represents the standard deviation of destination IP traffic within that time period; F represents the traffic volume level, U cpu Indicates CPU utilization, M usage t represents memory usage. w T represents the I / O wait time.av This represents the average access traffic to the destination IP, and L represents the integrity of the access link.

[0025] A further technical improvement of the present invention is that the explicit topology network construction in step three includes:

[0026] Based on CMDB, the physical connection relationship between physical devices and virtual resources is collected to form the associated topology of nodes and links;

[0027] The color scheme is dynamically rendered based on the node load, with warm colors used for high-load nodes and cool colors used for low-load nodes.

[0028] A further technical improvement of the present invention is that the implicit topology network construction includes: constructing a directed graph of service call relationships with service interfaces as nodes and call frequency as edge weights; and mapping and storing alarm types, error codes and associated maintenance measures in the nodes.

[0029] A further technical improvement of the present invention is that the association analysis in step four specifically includes: extracting node features of the real-time alarm knowledge graph and performing linear transformation through a learnable weight matrix; using a graph attention network to calculate the cosine similarity between real-time and historical graph nodes, and selecting the association graph with the highest similarity.

[0030] A further technical improvement of the present invention is that: the knowledge base update in step five includes: sorting according to alarm activity, filtering low-activity nodes and labeling erroneous data of high-activity nodes to update the training set; performing data augmentation on nodes with changes in topological relationships, including adding virtual alarm samples and adjusting edge weights.

[0031] A further technical improvement of this invention lies in the formula for selecting the historical alarm knowledge graph with the highest similarity: in, This represents the point similarity weighting coefficient.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention achieves structured processing of multi-source alarms by defining unified core fields and mapping templates through the construction of adaptation modules, which solves the pain point of scattered data formats and enables scattered data silos to be unified, facilitating data management and knowledge base construction.

[0034] 2. This invention uses regular expression verification, cluster deduplication, Fourier transform periodic detection, and dynamic threshold migration and update mechanism to accurately filter invalid and false alarm data, greatly reducing the data explosion caused by short-term alarm bursts, improving the effectiveness of alarms, saving a lot of operation and maintenance time, and improving operation and maintenance efficiency.

[0035] 3. This invention constructs a knowledge base through a two-layer topology network of explicit (physical connection) and implicit (service call), and combines it with a graph attention network to achieve real-time and historical alarm node alignment, thereby enhancing the accuracy of correlation analysis; and by calculating the node similarity between the real-time alarm knowledge graph and the historical database, it filters the correlation graph, assists in quickly locating the source of the problem, and improves the accuracy and efficiency of alarm location. Attached Figure Description

[0036] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the method execution flow of the present invention. Detailed Implementation

[0038] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0039] Please see Figure 1 As shown, a method for associating operation and maintenance alarms with a knowledge base based on a large model includes the following steps:

[0040] Step 1: Configure the alarm integration and adaptation module to standardize the processing of heterogeneous alarms from various platforms.

[0041] An alarm integration and adaptation module is built, which supports the protocol interfaces of current mainstream monitoring systems. It acquires operation and maintenance logs generated by various monitoring systems in real time, and then performs structured processing on the acquired operation and maintenance alarm data in different formats. Specifically:

[0042] The operation and maintenance alarm data is reconstructed using JSON format, and several unified core fields are defined, including UID, data source, alarm level, timestamp, and original alarm text. At the same time, a mapping template is built to match the alarm fields of each platform with the defined core fields, thereby deconstructing each operation and maintenance alarm data of each platform and reconstructing it according to the mapped core fields to obtain structured data in JSON format.

[0043] Subsequently, the semantic big data model (BERT model is used in this embodiment) is used to perform semantic parsing on the original alarm text (including word segmentation, part-of-speech tagging, rule matching, text entity recognition and relation extraction, etc.). The parsed semantic information is represented in JSON format, and then the device IP, process, error code and alarm description are extracted to form the tag field.

[0044] Finally, the key-value pairs in the core fields and tag fields are integrated into structured alarm data and marked as unified alarm data.

[0045] Step 2: Filter invalid and false alarm data from the overall alarm data.

[0046] (1) Filter invalid data

[0047] Invalid data refers to data that is clearly illogical or does not conform to rules, including data with missing key values, incorrect formatting, and values ​​that are outside the reasonable range;

[0048] Since the overall alarm data has been standardized in format, the format error mentioned here does not refer to a data structure error, but rather to the specific data format of the field object values ​​in the overall alarm data, such as the data format of device IP or timestamp. For format errors, regular expressions are directly constructed to perform format validation, and data that does not conform to the format is marked as invalid data and filtered out.

[0049] Similarly, when key fields or their corresponding values ​​are missing, regular expressions are used for validation and filtering.

[0050] For data that exceeds the reasonable range, such as certain fields in each overall alarm data exceeding their normal value range, for example, the timestamp is significantly higher than the current timestamp, then the data is considered invalid.

[0051] In addition, for a large number of identical or similar alarms that occur within the time neighborhood of time t0+Δt, correlation aggregation is performed to eliminate invalid alarms. Specifically:

[0052] The alarm data is sorted chronologically by timestamp, then a time window of a certain width W is set and configured according to... The alarm data stream is segmented by moving the data in increments of a certain size.

[0053] The system counts the number of alarm data and the types of device IPs involved in each time window. First, it calculates the alarm density based on the ratio of the number of alarm data in the time window to the width of the time window. If the alarm density is greater than a set threshold, subsequent operations are performed on the alarm data in that time window; otherwise, no operation is performed.

[0054] When the alarm density exceeds a set threshold, the device IP types of the alarms are clustered to group alarms of the same device IP into the same category, thus completing the classification in terms of spatial attributes; then the Word2Vec algorithm is used to construct a high-dimensional feature vector from the alarm data in the same category.

[0055] For the high-dimensional feature vectors obtained above, calculate the cosine similarity between any two vectors, and then calculate the cosine distance between any two vectors. The cosine distance = 1 - cosine similarity. Since the range of cosine similarity is [-1, 1], the range of cosine distance is [0, 2].

[0056] A distance threshold is set. When the cosine distance is less than the set distance threshold, it indicates that the similarity between the two vectors is extremely high. The two vectors are placed in the same cluster. The cosine distance between any two vectors in the same cluster is less than the set distance threshold. The earliest timestamp-corresponding overall warning data in the high-dimensional feature vector of the same cluster is taken as the representative warning data of the cluster, and other data in the cluster are eliminated.

[0057] (2) Filtering false alarm data

[0058] False alarm data includes periodic false alarms passively triggered by scheduled tasks, false alarms of traffic aggregation, and false alarms of threshold sensitivity;

[0059] For periodic false alarm data, a time-series feature array is constructed by selecting the device IP, alarm level, process, and error code of the alarm. This time-series feature array is then subjected to a Fourier transform to convert it from a time-domain signal to a frequency-domain signal, specifically a complex array containing amplitude and phase information at different frequencies. A spectrum diagram is plotted based on the above information to obtain periodic spike signals, and the corresponding period is selected as a candidate period. A sliding window is constructed based on the candidate period and the time sequence of the spike signal and is shifted over the sorted alarm data to verify whether there are periodic alarms. The alarm is automatically extracted and confirmed by the operation and maintenance personnel. Once confirmed, it is added to the false alarm report and the alarm is automatically masked.

[0060] To address false positives about traffic aggregation, we statistically analyze the source and destination IP distributions of the traffic and the number of accesses between IPs. False positives about traffic aggregation indicate that multiple source IPs are collectively accessing a single destination IP. This analysis is based on traffic volume (F) and CPU utilization (U). cpu ), memory usage (M usage ), I / O wait time (t) w ), average access traffic to the corresponding destination IP (T) av The probability of false alarms K is evaluated based on the integrity of the access link (L).

[0061] The evaluation formula is: This represents the probability that an alarm generated within the time period [t, t+Δt] is a false alarm, where a, b, c, and d are weighting coefficients, and e is a natural constant in mathematics. This represents the standard deviation of destination IP traffic within this time period; the value of Δt is dynamically set according to the alarm density, the higher the alarm density, the smaller the value of Δt.

[0062] When K is greater than the set threshold, alarms targeting the destination IP within the time period [t, t+Δt] are considered false alarms.

[0063] To address threshold-sensitive false alarms, which can easily trigger alarms due to the migration of relevant feature data caused by device or communication network aging or environmental changes, a fixed threshold can be fixed. A fixed update period is set, and the average value of the corresponding indicator within that update period and the rate of change of the average value in adjacent periods are calculated. The median value of the threshold is then updated based on this rate of change, thus adaptively updating the entire threshold.

[0064] Step 3: Building a knowledge base

[0065] A two-layer retrieval query network for the knowledge base is constructed based on structured, coordinated alert data from historical data.

[0066] First, an explicit topology network structure diagram is constructed based on entity connection relationships, and an implicit topology network structure diagram is constructed based on service call relationships.

[0067] The explicit topology network structure diagram includes collecting the physical connection relationships between physical devices (servers / switches / routers) and virtual resources (virtual machines / containers) through CMDB, and establishing the associated links between point objects (devices) and line objects (physical links);

[0068] In explicit topology networks, dynamic rendering based on color differentiation is performed, and nodes are labeled with colors according to their load levels. High-load nodes are marked with warm colors, such as red, and low-load nodes are marked with cool colors, such as blue.

[0069] Implicit topology network structure graph is a directed graph constructed based on service call relationships. The directed graph uses service interfaces (ports) as nodes and call frequency as edge weights, thus obtaining a network-like topology network.

[0070] At each node in the two-layer topology network, a mapping store contains various alarm types, error codes, and operational measures related to that node.

[0071] Step 4: Retrieve knowledge base and perform correlation analysis based on real-time alarm data.

[0072] The real-time alarm data is structured to obtain real-time unified alarm data. The field indicators of the unified alarm data are extracted. Based on the device association and service call relationship of the alarm object, a real-time alarm knowledge graph is generated, including explicit graph and implicit graph.

[0073] Based on the associated nodes and alarm types of the real-time alarm knowledge graph, historical alarm knowledge graphs of relevant or adjacent nodes are extracted from the knowledge base.

[0074] Feature extraction (such as alarm type, device attributes, etc.) is performed on the nodes and edges of the real-time alarm knowledge graph and the historical alarm knowledge graph. Then, a new weighted feature is obtained by linearly transforming different features using a learnable weight matrix P based on expert experience. The initial feature Q of node i is... i Mapped to PQ i ;

[0075] A node alignment model is constructed based on a graph attention network, employing a masked self-attention mechanism that calculates attention coefficients only between the target node and its first-order neighbors. For node pairs (i,j) in the real-time and historical graphs, weighted features of the two nodes are concatenated, and a single-layer feedforward neural network with the LeakyReLU activation function is applied to obtain asymmetric attention coefficients. The calculated attention coefficients are then used to perform a weighted summation on neighboring nodes to update the node representations, ultimately outputting the node's embedding vector. The model is trained with the goal of maximizing the similarity between node embedding vectors.

[0076] The node features of the real-time alarm knowledge graph and the historical alarm knowledge graph are input into the node alignment model to obtain the embedding vector. By calculating the cosine similarity between the embedding vectors, the set of nodes most similar to each node in the real-time alarm knowledge graph can be found in the historical alarm knowledge graph. Similarly, the real-time alarm knowledge graph and the historical alarm knowledge graph can be obtained, and the similarity of the relationship about the edges can be calculated.

[0077] Based on the similarity of nodes and edges, the historical alarm knowledge graph with the highest similarity is selected as the reference for subsequent analysis of the association graph. Specifically:

[0078]

[0079] in, This represents the point similarity weighting coefficient, and

[0080] Step 5: Adaptive updating of the knowledge base and model

[0081] When updating the knowledge base, the nodes are sorted according to their activity level in the previous update cycle. Nodes with low alarms are filtered out, and alarm data with incorrect associations in nodes with high alarms are manually labeled. The labeled data is then added to the training set to retrain the node alignment model.

[0082] Based on changes in the connection and invocation relationships of devices or services, the topology in the knowledge base is updated, and data augmentation is performed on the alarm data of related changed nodes and edges.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for establishing an operation and maintenance alarm association knowledge base based on a large model, characterized in that: Includes the following steps: Step 1: Standardize heterogeneous alarms through the alarm integration and adaptation module, including: obtaining multi-source operation and maintenance alarm logs and reconstructing them into JSON-formatted structured data containing core fields and tag fields; Step 2: Filter invalid and false alarm data from the structured and coordinated alarm data; Step 3: Construct a two-layer retrieval knowledge base based on explicit and implicit topology networks; Step 4: Perform correlation analysis between real-time alarm data and knowledge base to generate a real-time alarm knowledge graph and match it with historical graphs using a graph attention network; Step 5: Dynamically update the knowledge base based on alarm activity and topology changes; The false alarm data filtering described in step two includes: Periodic false alarm detection: The alarm timing characteristics are analyzed by Fourier transform to identify periodic spike signals, and after verification, they are added to the block list; Traffic aggregation false alarm detection: Construct a formula to evaluate the probability of false alarms based on the distribution of source IP / destination IP, number of accesses and traffic volume, and filter alarms that exceed the threshold; Threshold sensitivity false alarm detection: Based on the rate of change of the average value of the indicator in adjacent update cycles, the median value of the threshold is migrated and updated while keeping the threshold width unchanged.

2. The method for establishing an operation and maintenance alarm association knowledge base based on a large model according to claim 1, characterized in that, The standardization process described in step one specifically includes: Define core fields including UID, data source, alarm level, timestamp, and original alarm text; build a mapping template to match the alarm fields of each platform with the defined core fields; The original alarm text is semantically parsed using the BERT model, and the device IP, the process in which it is located, the error code, and the alarm description are extracted as tag fields.

3. The method for a knowledge base for operation and maintenance alarm association based on a large model according to claim 1, characterized in that, The invalid data filtering in step two includes: Validate field format and missing key values ​​using regular expressions; Data that exceeds the normal range in the overall alarm data will be defined as invalid data. The alarm data stream is segmented using a time window, the alarm density is calculated, and Word2Vec feature vector clustering is performed on the alarms in the high-density window to retain the earliest alarm data in the cluster.

4. The method for a knowledge base for operation and maintenance alarm association based on a large model according to claim 1, characterized in that, The formula for evaluating the probability of false alarms is: ； Where K represents the time period [ , The likelihood of a false alarm being generated within [the context of the alarm] is [not specified]. Here, e is the weighting coefficient, and e is the natural constant in mathematics. This represents the standard deviation of destination IP traffic within that time period; Indicates the flow rate level. Indicates the maximum flow rate level. Indicates CPU utilization. Indicates memory usage. Indicates the I / O wait time. This represents the average traffic to the destination IP. Indicates the integrity of the access link; This represents the historical average traffic to the destination IP.

5. The method for establishing an operation and maintenance alarm association knowledge base based on a large model according to claim 1, characterized in that, According to the method described in claim 1, the explicit topology network construction in step three includes: Based on CMDB, the physical connection relationship between physical devices and virtual resources is collected to form the associated topology of nodes and links; The color scheme is dynamically rendered based on the node load, with warm colors used for high-load nodes and cool colors used for low-load nodes.

6. The method for a knowledge base for operation and maintenance alarm association based on a large model according to claim 4, characterized in that, The implicit topology network construction includes: constructing a directed graph of service call relationships with service interfaces as nodes and call frequency as edge weights; and mapping and storing alarm types, error codes, and associated maintenance measures in the nodes.

7. The method for a knowledge base for operation and maintenance alarm association based on a large model according to claim 1, characterized in that, Step four, the association analysis specifically includes: extracting node features from the real-time alarm knowledge graph and performing linear transformation using a learnable weight matrix; using a graph attention network to calculate the cosine similarity between real-time and historical graph nodes, and selecting the association graph with the highest similarity.

8. The method for a knowledge base for operation and maintenance alarm association based on a large model according to claim 1, characterized in that, Step 5, knowledge base update, includes: sorting by alarm activity, filtering low-activity nodes and labeling erroneous data of high-activity nodes to update the training set; and performing data augmentation on nodes with changed topological relationships, including adding virtual alarm samples and adjusting edge weights.

9. The method for a knowledge base for operation and maintenance alarm association based on a large model according to claim 7, characterized in that, The formula for selecting the historical alert knowledge graph with the highest similarity is: ; in, This represents the point similarity weighting coefficient, and ; This represents the similarity between nodes in the real-time alarm knowledge graph and the historical alarm knowledge graph. This represents the edge similarity between the real-time alarm knowledge graph and the historical alarm knowledge graph. It represents the combined similarity between nodes and edges in the real-time alarm knowledge graph and the historical alarm knowledge graph.

Images