Method for associating operation and maintenance alarm with knowledge base based on large model

Abstract

The invention discloses a method for associating operation and maintenance alarms with a knowledge base based on a large model, and relates to an operation and maintenance data processing technology. Comprising the following steps: carrying out standardization processing on heterogeneous alarms through an alarm docking adaptation module; filtering invalid data and false alarm data of the structured overall alarm data; constructing a double-layer retrieval knowledge base based on the explicit topology network and the implicit topology network; performing association analysis on the real-time alarm data and a knowledge base, generating a real-time alarm knowledge graph, and matching a historical graph through a graph attention network; and dynamically updating the knowledge base according to the alarm activeness and the topological relation change, and the like. According to the method, unified structured processing is carried out on the alarm data, invalid data and false alarm data in the alarm data are removed, the alarm efficiency is improved, the operation and maintenance time is saved, and the operation and maintenance efficiency is improved; meanwhile, a knowledge base is constructed based on a double-layer topology network, a graph attention network is combined, an associated graph is screened, a problem source is rapidly positioned in an auxiliary mode, and the alarm positioning precision and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of operation and maintenance data processing, and in particular to a method for associating an operation and maintenance alarm with a knowledge base based on a large model. Background Art

[0002] With the rapid development of information technology, the scale and complexity of enterprise equipment systems are constantly increasing, and operations and maintenance are facing unprecedented challenges. Traditional operations and maintenance methods rely mainly on manual monitoring and rule setting, which is not only inefficient but also difficult to quickly process and accurately locate massive amounts of alarm information. Using knowledge bases to set rules for automated operations and maintenance is an important means to free up manual labor, reduce personnel burdens, and cope with alarm storms.

[0003] However, existing knowledge bases are mostly based on static rules or single-model reasoning, making them difficult to cope with increasingly complex and dynamic O&M scenarios. For example, equipment failures can trigger chain reactions, and traditional knowledge bases lack multi-hop reasoning capabilities, making it impossible to automatically link and correlate alarms. Furthermore, alarm data is scattered across multiple monitoring systems, making it difficult to standardize data entry formats, resulting in reduced knowledge base accuracy. Furthermore, alarm noise and a large number of false alarms consume significant O&M resources, resulting in lower warning accuracy and O&M efficiency for real alarms. To address this issue, we propose a method for building an O&M alarm correlation knowledge base based on a large model. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for associating operation and maintenance alarm knowledge base based on a large model to solve the above problems.

[0005] The present invention can be implemented by the following technical solution: A method for associating an operation and maintenance alarm knowledge base based on a large model, comprising the following steps:

[0006] Step 1: Standardize heterogeneous alarms through the alarm docking adaptation module, including: obtaining multi-source operation and maintenance alarm logs and reconstructing them into JSON format structured data containing core fields and tag fields;

[0007] Step 2: Filter invalid data and false alarm data from the structured coordinated alarm data;

[0008] Step 3: Construct a double-layer retrieval knowledge base based on the explicit topology network and the implicit topology network;

[0009] Step 4: Perform correlation analysis between real-time alarm data and the knowledge base to generate a real-time alarm knowledge graph and match it with the historical graph through the graph attention network;

[0010] Step 5: Dynamically update the knowledge base based on the alarm activity and topology relationship changes.

[0011] A further technical improvement of the present invention is that the first step of standardization processing specifically includes:

[0012] Define core fields including UID, data source, alarm level, timestamp, and original alarm text; build a mapping template to map the alarm fields of each platform to the defined core fields.

[0013] The original alarm text is semantically parsed using the BERT model, and the device IP, process, error code, and alarm description are extracted as label fields.

[0014] A further technical improvement of the present invention is that the invalid data filtering in step 2 includes:

[0015] Validate field format and key value missing using regular expressions;

[0016] The data in the coordinated alarm data that exceeds the normal value range is defined as invalid data;

[0017] The alarm data stream is cut into time windows, the alarm density is calculated, and the alarms in the high-density window are clustered using Word2Vec feature vectors, retaining the earliest alarm data in the cluster.

[0018] A further technical improvement of the present invention is that the false alarm data filtering in step 2 includes:

[0019] Periodic false alarm detection: Analyze alarm timing characteristics through Fourier transform, identify periodic spike signals, and add them to the shielding list after verification;

[0020] Traffic aggregation false alarm detection: Builds a false alarm probability evaluation formula based on source IP / destination IP distribution, number of visits, and traffic level, and filters out over-threshold alarms;

[0021] Threshold sensitivity false alarm detection: Based on the rate of change of the indicator average value in adjacent update cycles, the threshold median is migrated and updated while keeping the threshold width unchanged.

[0022] A further technical improvement of the present invention is that the false alarm possibility evaluation formula is:

[0023]

[0024] Where K represents the probability that the alarm generated in the time period [t, t+Δt] is a false alarm, a, b, c, d are weight coefficients, and e is a natural constant in mathematics. Indicates the standard deviation of the destination IP traffic within the time period; F indicates the traffic level, U cpu Indicates CPU utilization, M usage Indicates memory usage, t w Indicates the IO waiting time, Tav It represents the average access traffic of the destination IP address, and L represents the integrity of the access link.

[0025] A further technical improvement of the present invention is that the explicit topology network construction in step 3 includes:

[0026] Based on the CMDB, the physical connection relationship between physical devices and virtual resources is collected to form the association topology of nodes and links;

[0027] Dynamically render color labels based on node load, with high-load nodes using warm colors and low-load nodes using cool colors.

[0028] The further technical improvement of the present invention is that the implicit topology network construction includes: constructing a directed graph of service call relationships with service interfaces as nodes and call frequencies as edge weights; and storing alarm types, error codes and associated operation and maintenance measures in node mapping.

[0029] A further technical improvement of the present invention is that the association analysis in step four specifically includes: extracting node features of the real-time alarm knowledge graph and performing linear transformation through a learnable weight matrix; using a graph attention network to calculate the cosine similarity of real-time and historical graph nodes, and screening the highest similarity association graph.

[0030] A further technical improvement of the present invention is that the knowledge base update in step five includes: sorting according to alarm activity, filtering low-activity nodes and marking error data of high-activity nodes to update the training set; data enhancement of nodes with changed topological relationships, including adding virtual alarm samples and adjusting edge weights.

[0031] A further technical improvement of the present invention is that the formula for screening the historical alarm knowledge graph with the highest similarity is: in, Represents the point similarity coefficient,

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention implements structured processing of multi-source alarms by building an adaptation module to define unified core fields and mapping templates, solving the pain point of scattered data formats and unifying scattered data islands, facilitating data management and knowledge base construction.

[0034] 2. The present invention accurately filters invalid and false alarm data through regular verification, clustering deduplication, Fourier transform periodic detection, and dynamic threshold migration and update mechanism, greatly reducing the data explosion caused by short-term alarm outbreaks, improving the effectiveness of alarms, greatly saving operation and maintenance time, and improving operation and maintenance efficiency.

[0035] 3. The present invention constructs a knowledge base through an explicit (physical connection) and implicit (service call) two-layer topology network, and combines the graph attention network to achieve node alignment of real-time and historical alarms, thereby enhancing the accuracy of association analysis; and through the node similarity calculation of the real-time alarm knowledge graph and the historical library, the association graph is screened to assist in quickly locating the source of the problem, thereby improving the accuracy and efficiency of alarm positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0037] Figure 1 The figure is a schematic diagram of the execution flow of the method of the present invention. DETAILED DESCRIPTION

[0038] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0039] See also Figure 1 As shown in FIG, a method for associating an operation and maintenance alarm knowledge base based on a large model specifically includes the following steps:

[0040] Step 1: Set up the alarm docking adapter module to standardize the heterogeneous alarms of each platform

[0041] Build an alarm docking adapter module that supports the protocol interfaces of current mainstream monitoring systems, acquires the operation and maintenance logs generated by each monitoring system in real time, and then performs structured processing on the acquired operation and maintenance alarm data in different formats. Specifically:

[0042] Reconstruct the operation and maintenance alarm data in JSON format, defining several unified core fields, including UID, data source, alarm level, timestamp, and original alarm text. Simultaneously, build a mapping template to map each platform's alarm fields to the defined core fields. This allows each operation and maintenance alarm data on each platform to be deconstructed and reconstructed into structured data in JSON format according to the mapped core fields.

[0043] Subsequently, a semantic large model (BERT model is used in this embodiment) is used to perform semantic analysis on the original alarm text (including word segmentation, part-of-speech tagging, rule matching, text entity recognition, and relationship extraction). The parsed semantic information is expressed in JSON format, and the device IP, process, error code, and alarm description are extracted from it to form the label field.

[0044] Finally, the key-value pairs in the core fields and tag fields are integrated into structured alarm data and marked as coordinated alarm data.

[0045] Step 2: Filter invalid data and false alarm data in the coordinated alarm data

[0046] (1) Filter invalid data

[0047] Invalid data refers to data that obviously does not conform to logic or rules, including data with missing key values, incorrect formats, and values ​​outside the reasonable range;

[0048] Since the coordinated alarm data is standardized, the format error mentioned here does not refer to data structure errors, but to the specific data format of the field object value in the coordinated alarm data, such as the data format of the device IP or timestamp. For format errors, regular expressions are directly constructed to perform format verification. Data that does not conform to the format is marked as invalid data and filtered out.

[0049] Similarly, when the key field or the value of the corresponding field is missing, it is verified and filtered through regular expressions;

[0050] For data that exceeds the reasonable range, such as certain fields in each coordinated alarm data that exceed their normal value range, such as timestamps that significantly exceed the current timestamp, the data will be considered invalid.

[0051] In addition, for a large number of identical or similar alarms that occur within the time neighborhood of time t0 at time t0+Δt, correlation aggregation is performed to eliminate invalid alarms. Specifically:

[0052] The alarm data is sorted in the order of timestamps, and then a time window of a certain width W is set and the Move the alarm data stream by the step size of

[0053] Count the number of alarm data and the type of device IP addresses that generate alarms within each time window. First, calculate the alarm density based on the ratio of the number of alarm data within the time window to the time window width. When the alarm density exceeds the set threshold, perform subsequent operations on the alarm data within the time window; otherwise, do not perform any operations.

[0054] When the alarm density exceeds the set threshold, clustering is performed based on the device IP type of the alarm, and alarms from the same device IP are grouped together, thus completing the classification based on spatial attributes. The Word2Vec algorithm is then used to construct a high-dimensional feature vector from the coordinated alarm data in the same group.

[0055] Calculate the cosine similarity between any two high-dimensional feature vectors obtained above, and then calculate the cosine distance between any two vectors, and cosine distance = 1-cosine similarity. Since the value range of cosine similarity is [-1, 1], the value range of cosine distance is [0, 2].

[0056] A distance threshold is set. When the cosine distance is less than the set distance threshold, it means that the similarity between the two vectors is extremely high. The two vectors are placed in the same cluster. The cosine distance between any two vectors in the same cluster is less than the set distance threshold. The coordinated alarm data with the earliest corresponding timestamp in the high-dimensional feature vector of the same cluster is used as the representative warning data of the cluster, and other data in the cluster are eliminated.

[0057] (2) Filtering false positive data

[0058] False alarm data includes periodic false alarms passively triggered by scheduled tasks, traffic aggregation false alarms, and threshold sensitivity false alarms;

[0059] For periodic false alarm data, the device IP, alarm level, process, and error code of the alarm are selected to construct a time series feature array. The time series feature array is Fourier transformed to convert it from a time domain signal to a frequency domain signal, specifically a complex array containing amplitude and phase information at different frequencies. A spectrum diagram is drawn based on the above information to obtain periodic spike signals, and the corresponding period is obtained as a candidate period. A sliding window is constructed based on the candidate period and the timing of the spike signal. The sliding window is shifted on the sorted coordinated alarm data to verify whether there are periodic alarms. The alarm is automatically extracted and confirmed by the operation and maintenance personnel. After confirmation, it is added to the false alarm form and automatically blocked.

[0060] For traffic aggregation false positives, we count the source IP distribution and destination IP distribution of traffic, and count the number of visits between IPs. Traffic aggregation false positives are caused by multiple source IPs accessing a certain destination IP. Based on traffic magnitude (F), CPU utilization (U cpu ), memory usage (M usage ), IO waiting time (t w ), the average access traffic corresponding to the destination IP (T av ) and the integrity of the access link (L) to evaluate the false alarm probability K:

[0061] The evaluation formula is: Indicates the probability that the alarm generated in the time period [t, t+Δt] is a false alarm, where a, b, c, and d are weight coefficients, and e is a natural constant in mathematics. Indicates the standard deviation of the destination IP traffic within the time period. The value of Δt is dynamically set according to the alarm density. The higher the alarm density, the smaller the Δt value.

[0062] When K is greater than the set threshold, the alarm for the destination IP in the time period [t, t+Δt] is considered a false alarm;

[0063] For threshold sensitivity false alarms, due to aging of equipment or communication networks or environmental changes, the values ​​of relevant feature data may migrate, which may easily trigger a fixed threshold and cause an alarm. The threshold width is fixed, the update cycle is set, the average value of the corresponding indicator within the update cycle and the rate of change of the average value in adjacent cycles are calculated, and the median value of the threshold is migrated and updated according to the rate of change, thereby adaptively updating the entire threshold.

[0064] Step 3: Build a knowledge base

[0065] A two-layer retrieval query network is constructed based on the structured coordinated alarm data in historical data.

[0066] First, an explicit topology network structure diagram is constructed based on the entity connection relationship, and an implicit topology network structure diagram is constructed based on the service call relationship;

[0067] The explicit topology network structure diagram includes collecting the physical connection relationship between physical devices (servers / switches / routers) and virtual resources (virtual machines / containers) through the CMDB, and establishing the association link between point objects (devices) and line objects (physical links);

[0068] Dynamic rendering based on color differentiation is performed in the explicit topology network. Nodes are labeled according to their load level. High-load nodes use warm colors, such as red, and low-load nodes use cool colors, such as blue.

[0069] The implicit topology network structure diagram is a directed graph constructed based on the service call relationship. The directed graph uses the service interface (port) as the node and the call frequency as the edge weight, thus obtaining a chain-like topology network.

[0070] At each node in the two-layer topology network, a map stores various alarm types, error codes, and operation and maintenance measures related to the node;

[0071] Step 4: Search the knowledge base for correlation analysis based on real-time alarm data

[0072] Structural processing is performed on real-time alarm data to obtain real-time coordinated alarm data. Field indicators of the coordinated alarm data are extracted. Based on the device association relationship and service call relationship of the alarm object, a real-time alarm knowledge graph is generated, including explicit graphs and implicit graphs.

[0073] According to the associated nodes and alarm types of the real-time alarm knowledge graph, the historical alarm knowledge graph of related nodes or adjacent nodes is extracted from the knowledge base;

[0074] The nodes and edges of the real-time alarm knowledge graph and the historical alarm knowledge graph are subjected to feature extraction (such as alarm type, device attributes, etc.), and different features are linearly transformed through the weight matrix P based on expert experience and learnable to obtain new weighted features. The initial feature Q of node i is i Mapped to PQ i ;

[0075] A node alignment model is constructed based on a graph attention network, using a masked self-attention mechanism to calculate only the attention coefficients of the target node and its first-order neighboring nodes. For each node pair (i, j) in the real-time and historical graphs, an asymmetric attention coefficient is obtained by concatenating the weighted features of the two nodes and applying a single-layer feedforward neural network with a LeakyReLU activation function. The calculated attention coefficients are then used to perform a weighted sum of the neighboring nodes, updating the node representation and ultimately outputting the node embedding vector. The model is trained to maximize the similarity between the node embedding vectors.

[0076] The node features of the real-time alarm knowledge graph and the historical alarm knowledge graph are input into the node alignment model to obtain embedding vectors. By calculating the cosine similarity between the embedding vectors, for each node in the real-time alarm knowledge graph, the most similar node set in the historical alarm knowledge graph is found. Similarly, the real-time alarm knowledge graph and the historical alarm knowledge graph can be obtained to calculate the similarity of the edge relationships.

[0077] Based on the similarity of nodes and edges, the historical alarm knowledge graph with the highest similarity is selected as the reference for subsequent analysis of the associated graph. Specifically:

[0078]

[0079] in, represents the point similarity weight coefficient, and

[0080] Step 5: Adaptive update of knowledge base and model

[0081] When updating the knowledge base, the nodes are sorted according to their alert activity in the previous update cycle, low-alarm nodes are filtered, and the alarm data with associated errors in high-alarm nodes are manually annotated. The annotated data is then added to the training set to retrain the node alignment model.

[0082] According to the changes in the connection call relationship of devices or services, the topology structure in the knowledge base is updated and the alarm data of the relevant changed nodes and edges are enhanced.

[0083] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for associating an operation and maintenance alarm knowledge base based on a large model, characterized by: The steps include: Step 1: Standardize heterogeneous alarms through the alarm docking adaptation module, including: obtaining multi-source operation and maintenance alarm logs and reconstructing them into JSON format structured data containing core fields and tag fields; Step 2: Filter invalid data and false alarm data from the structured coordinated alarm data; Step 3: Construct a double-layer retrieval knowledge base based on the explicit topology network and the implicit topology network; Step 4: Perform correlation analysis between real-time alarm data and the knowledge base to generate a real-time alarm knowledge graph and match it with the historical graph through the graph attention network; Step 5: Dynamically update the knowledge base based on the alarm activity and topology relationship changes.

2. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 1 is characterized in that: The standardization process in step 1 specifically includes: Define core fields including UID, data source, alarm level, timestamp, and original alarm text; build a mapping template to map the alarm fields of each platform to the defined core fields. The original alarm text is semantically parsed using the BERT model, and the device IP, process, error code, and alarm description are extracted as label fields.

3. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 1 is characterized in that: The invalid data filtering in step 2 includes: Validate field format and key value missing using regular expressions; The data in the coordinated alarm data that exceeds the normal value range is defined as invalid data; The alarm data stream is cut into time windows, the alarm density is calculated, and the alarms in the high-density window are clustered using Word2Vec feature vectors, retaining the earliest alarm data in the cluster.

4. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 1 is characterized in that: The false positive data filtering described in step 2 includes: Periodic false alarm detection: Analyze alarm timing characteristics through Fourier transform, identify periodic spike signals, and add them to the shielding list after verification; Traffic aggregation false alarm detection: Builds a false alarm probability evaluation formula based on source IP / destination IP distribution, number of visits, and traffic level, and filters out over-threshold alarms; Threshold sensitivity false alarm detection: Based on the rate of change of the indicator average value in adjacent update cycles, the threshold median is migrated and updated while keeping the threshold width unchanged.

5. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 4 is characterized in that: The false alarm possibility evaluation formula is: Where K represents the probability that the alarm generated in the time period [t, t+Δt] is a false alarm, a, b, c, d are weight coefficients, and e is a natural constant in mathematics. Indicates the standard deviation of the destination IP traffic within the time period; F indicates the traffic level, U cpu Indicates CPU utilization, M usage Indicates memory usage, t w Indicates the IO waiting time, T av It represents the average access traffic of the destination IP address, and L represents the integrity of the access link.

6. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 1 is characterized in that: The method according to claim 1, wherein the explicit topology network construction in step (3) comprises: Based on the CMDB, the physical connection relationship between physical devices and virtual resources is collected to form the association topology of nodes and links; Dynamically render color labels based on node load, with high-load nodes using warm colors and low-load nodes using cool colors.

7. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 5 is characterized in that: The implicit topology network construction includes: constructing a directed graph of service call relationships with service interfaces as nodes and call frequencies as edge weights; and storing alarm types, error codes, and associated operation and maintenance measures in node mapping.

8. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 1 is characterized in that: The association analysis in step 4 specifically includes: extracting node features of the real-time alarm knowledge graph and performing linear transformation through a learnable weight matrix; using a graph attention network to calculate the cosine similarity of real-time and historical graph nodes, and screening the highest similarity association graph.

9. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 1, characterized in that: The knowledge base update in step 5 includes: sorting according to alarm activity, filtering low-activity nodes and marking error data of high-activity nodes to update the training set; and performing data enhancement on nodes with changed topological relationships, including adding virtual alarm samples and adjusting edge weights.

10. The method for an operation and maintenance alarm association knowledge base based on a large model according to claim 8, characterized in that: The formula for filtering the historical alarm knowledge graph with the highest similarity is: in, represents the point similarity weight coefficient, and

Images