Alarm suppression method and system based on semantic coupling distance distinction degree

Abstract

The invention provides an alarm suppression method and system based on semantic coupling distance discrimination, and aims to introduce context similarity calculation aiming at the problem that existing traditional word similarity calculation often only considers semantic items of words and neglects meanings of the words in specific contexts, so that the context similarity calculation can better understand the specific meanings of the words in different contexts and improve the accuracy of alarm suppression. And the calculation accuracy is enhanced. Compared with a traditional method, the method has the advantages that by combining word semantic similarity and context similarity calculation and optimizing a similarity algorithm of coupling co-occurrence distance distinction degree, deep semantic association between alarm texts can be recognized more accurately, the accuracy of alarm clustering is remarkably improved, redundant alarm output is effectively suppressed, and the method is suitable for popularization and application. And operation and maintenance personnel are helped to quickly position and process batch alarm problems, so that the alarm processing efficiency and reliability of the system are improved.

Description

Technical Field

[0001] The invention relates to an alarm suppression method and system based on semantic coupling distance discrimination, and belongs to the field of database intelligent operation and maintenance. Background Art

[0002] In the field of database operation and maintenance, the main task of the abnormal monitoring and alarm system is to continuously protect the stable operation of the database system. With the complexity of database business, the amount of alarm data has exploded, which directly leads to a large number of redundant alarms and false alarms faced by operation and maintenance personnel. The complicated alarm information greatly increases the workload of operation and maintenance personnel. The alarm suppression technology can reduce duplicate alarms and improve the efficiency of alarm processing, and has become an important part of the database intelligent operation and maintenance system.

[0003] Existing alarm suppression systems usually adopt aggregation strategies based on rule matching, similarity calculation and time window. In practical applications, the above systems mainly have the following problems: (1) Alarm data is often a mixture of structured and unstructured information. The sources of alarm information are diverse and the formats are not uniform, which makes the data cleaning and feature extraction process complicated. (2) In the process of alarm clustering, most systems rely on methods such as keyword matching or static word frequency and co-occurrence information to calculate alarm similarity. Although this method can reduce the recurrence of the same alarm to a certain extent, due to its limited understanding of the semantic association of alarms, it cannot effectively handle the semantic differences in the alarm text, resulting in false alarms and missed alarms.

[0004] Most of the existing similarity calculation methods use text similarity algorithms such as word frequency (TF-IDF), Jaccard coefficient or co-occurrence relationship. The main problems of these methods are:

[0005] (1) Ignoring semantic relationships: The above similarity calculation methods only focus on the surface information of the vocabulary in the warning text and cannot capture the deep semantic relationship between words. For example, two warnings may be different in vocabulary but highly similar in semantics. Existing methods cannot accurately identify these potential correlations, resulting in poor clustering of similar warnings.

[0006] (2) Lack of context: Existing similarity calculation methods ignore contextual information and rely only on the content of the individual alert text. The semantics of words in short texts may vary depending on the context. Existing similarity calculation methods are particularly inappropriate when processing short texts and structured data. Analyzing word frequency alone cannot produce accurate similarity results.

[0007] (3) False alarms and missed alarms: Since the similarity algorithm is not sensitive enough to the differences in alarm text, the alarm suppression system often generates a large number of false alarms when the alarm names are different but their semantics are similar. In addition, the diversity of alarms can easily cause the system to miss some related alarms, affecting the judgment and processing efficiency of operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Flowchart of the database alarm suppression system.

[0009] Figure 2 The flowchart of the short text similarity calculation method based on semantic coupling distance discrimination. Summary of the invention

[0010] In view of the duplicate alarm problem existing in the existing abnormal alarm system, the present invention realizes alarm suppression and convergence from the two dimensions of semantic similarity and time, helping users to find batch alarm problems more targeted. Based on this, the present invention proposes an alarm suppression method based on semantic coupling distance discrimination, which includes the following steps:

[0011] (1) Alarm data preprocessing, data cleaning and feature extraction;

[0012] (2) Alarm association clustering: two or more alarms with exactly the same alarm names are directly regarded as the same type of alarms; for alarms with different names but high semantic similarity, they are clustered using a short text similarity calculation method based on semantic coupling distance discrimination;

[0013] (3) adjusting the correlation threshold, dynamically adjusting the cluster similarity thresholds of various alarms based on the alarm data clustered in step (2);

[0014] (4) Convergence alarm output: filter alarms based on dynamic thresholds combined with clustering results, and output alarm results;

[0015] Furthermore, in step (2), the short text similarity calculation method based on semantic coupling distance discrimination comprises the following steps:

[0016] 2.1 Calculate the context word set C(w1) of word w1 and word w2 = {w p |p∈[qk,q+k]}, C(w2)={w p |||p∈[lk,l+k]}, where p and q are the position indices of words w1 and w2 in the sentence, p≠q, p≠l, and k is the context window size;

[0017] 2.2 Calculating comprehensive context similarity Among them, simw() is the semantic similarity between two words, and c is the context word;

[0018] 2.3 Calculate the similarity simw between two words f (w1,w2)=β·simw(w1,w2)+(1-β)·simw context (w1,w2), where β is the weight coefficient.

[0019] 2.4 Calculating Inline Relationships in, When CoR(t i ,t j )=0;d s For a specific text, and Term t i and term t j , and α is the weight for adjusting the contribution of semantic similarity.

[0020] Based on the above method, the present invention further proposes an alarm suppression system based on semantic coupling distance discrimination, which includes the following modules:

[0021] (1) Alarm data preprocessing module, which performs data cleaning and feature extraction;

[0022] (2) Alarm association clustering module: For two or more alarms with exactly the same alarm names, this module directly regards them as the same type of alarms; for alarms with different names but high semantic similarity, this module clusters them using a short text similarity calculation method based on semantic coupling distance discrimination;

[0023] (3) an association threshold adjustment module, which dynamically adjusts the similarity thresholds of various alarm clusters based on the alarm data clustered by the alarm association clustering module;

[0024] (4) Convergence alarm output module, which filters alarms based on dynamic thresholds combined with clustering results and outputs alarm results;

[0025] Furthermore, in the alarm association clustering module, the short text similarity calculation method based on semantic coupling distance discrimination comprises the following steps:

[0026] 2.1 Calculate the context word set C(w1) of word w1 and word w2 = {w p |p∈[qk,q+k]}, C(w2)={w p |||p∈[lk,l+k]}, where p and q are the position indices of words w1 and w2 in the sentence, p≠q, p≠l, and k is the context window size;

[0027] 2.2 Calculating comprehensive context similarity Among them, simw() is the semantic similarity between two words, and c is the context word;

[0028] 2.3 Calculate the similarity simw between two words f (w1,w2)=β·simw(w1,w2)+(1-β)·simw context (w1,w2), where β is the weight coefficient.

[0029] 2.4 Calculating Inline Relationships in, When CoR(t i ,t j )=0;d s For a specific text, and Term t i and term t j , and α is the weight for adjusting the contribution of semantic similarity.

[0030] The present invention introduces contextual similarity calculation in view of the fact that the existing traditional word similarity calculation often only considers the meaning of the word itself, while ignoring the meaning of the word in a specific context, so that it can better understand the specific meaning of the word in different contexts, thereby enhancing the accuracy of the calculation. Compared with the traditional method, the present invention combines word semantic similarity with contextual similarity calculation, and optimizes the similarity algorithm of coupled co-occurrence distance discrimination, which can more accurately identify the deep semantic association between alarm texts, significantly improve the accuracy of alarm clustering, effectively suppress redundant alarm output, and help operation and maintenance personnel quickly locate and handle batch alarm problems, thereby improving the alarm processing efficiency and reliability of the system. DETAILED DESCRIPTION

[0031] Example 1

[0032] In view of the problem that the similarity calculation method based on word frequency or co-occurrence information in the existing alarm system usually ignores the semantic relationship between words, resulting in inaccurate recognition of the relevance of the alarm text, redundant alarms and false alarms, the present invention proposes an alarm suppression method based on semantic coupling distance discrimination. The specific process is shown in Figure 1 .

[0033] (1) Alarm data preprocessing

[0034] Data cleaning: remove redundant information and standardize field formats (such as timestamp formatting).

[0035] Feature extraction: Receive alarm data from Kafka and extract key attribute features, such as alarm IP, monitoring indicators, monitoring objects, indicator keys, application names, and container nodes. Extract key feature information for subsequent clustering and similarity calculations. At the same time, extract relevant context information as input for subsequent semantic analysis.

[0036] (2) Alarm association clustering

[0037] In the alarm correlation clustering stage, the system divides the alarm data into two different clustering methods: wildcards with exactly the same characters and wildcards with high similarity.

[0038] Wildcards are identical: Two or more alarms with identical alarm names are directly considered as the same type of alarms. Based on wildcard matching, alarms with the same pattern are quickly aggregated to reduce the impact of duplicate alarms.

[0039] · High similarity wildcards: For alarms with different names but high semantic similarity, the present invention performs clustering through a short text similarity calculation method based on semantic coupling distance discrimination. This method combines word semantic similarity and context similarity, and introduces the meaning of words in a specific context into the short text similarity calculation. Compared with existing similarity calculation methods, the algorithm proposed in the present invention can more accurately identify the correlation between alarm texts, has stronger context understanding ability, deeper semantic association analysis ability, and is more suitable for similarity analysis of the brevity and structured characteristics of alarm texts in alarm systems, improves the accuracy of alarm clustering, and reduces false alarms and redundant alarms.

[0040] Specifically, the present invention proposes a short text similarity calculation method based on semantic coupling distance discrimination, which enhances the accuracy of short text similarity calculation by fully considering the specific meaning of words in different contexts. The specific process includes two parts: word semantic similarity calculation and short text similarity calculation.

[0041] 1) Calculation of word semantic similarity

[0042] Existing word similarity calculation methods are mostly based on static word vector models, which only consider the semantic information of the word itself, ignore the specific meaning of the word in different contexts, and cannot fully reflect the changes of words in different contexts, thus reducing the accuracy of similarity calculation. To address this problem, the present invention proposes a word similarity calculation method based on context window, which makes the similarity calculation more robust by combining the context information of the word.

[0043] Assuming that for words w1 and w2, a context window size k is defined, then the context word sets of w1 and w2 can be obtained as C(w1)={w p |p∈[qk,q+k] and p≠q}, C(w2)={wp |p∈[lk,l+k] and p≠l}, where p and q represent the position encodings of words w1 and w2 in the sentence respectively.

[0044] Define the comprehensive context similarity as

[0045] Among them, simw() represents the semantic similarity between two words. Through the semantic dictionary WordNet, the simw() function can calculate their similarity based on the synonymy, hyponymy and other semantic features of the words, and c represents the context word.

[0046] Finally, the context similarity is combined with the initial similarity to obtain the final similarity calculation formula between the two words:

[0047] simw f (w1,w2)=β·simw(w1,w2)+(1-β)·simw context (w1,w2)

[0048] Among them, β is a weight coefficient that controls the ratio of original similarity to contextual similarity.

[0049] 2) Short text similarity calculation

[0050] On the basis of word similarity calculation, the similarity between short texts is further calculated. Traditional short text similarity calculation methods, such as similarity based on word frequency or co-occurrence information, usually ignore the semantic relationship between words, resulting in information loss or inaccurate calculation. To solve this problem, the present invention adds the parameter of word semantic similarity to the original calculation of coupled co-occurrence distance discrimination. Given a short text set D = {d1, d2, ..., d m} and the term set T = {t1, t2, …, t n}. First, define a new inline relationship calculation as:

[0051]

[0052] in, when Define CoR(t i ,t j )=0;d s Indicates a specific text. and Respectively represent terms t i and term t j is the association weight of , and α is the weight for adjusting the contribution of semantic similarity.

[0053] Afterwards, normalize it Where UIaR(t i |t j ) represents the term t i and term t j The probability of co-occurrence, symmetric processing of the inline relationship is as follows:

[0054]

[0055] Finally, the coupling relationship between the two terms is defined by combining the inline and outline relationships, and the coupling distance discrimination is defined as:

[0056]

[0057] Among them, γ∈[0,1] represents the adjustment factor for adjusting the weights of internal and external relationships, CR(t i ,t j ) represents the term t i and term t j If the similarity between the two terms tends to be 0, it means that the correlation between the two terms is weaker, otherwise, the correlation between the two terms is stronger. i ,t j ) represents the external relationship between terms. Select CR(t i ,t j ) of the word pair (t i ,t j ) is put into the set M, and the method for calculating the semantic text similarity based on the coupling distance discrimination is:

[0058]

[0059] Among them, h(t i )={t j |t j ∈d2∧(t i ,t j )∈M},h(t i ) is represented by the term t i The set of terms whose similarity is greater than a threshold η.

[0060] In the above, the system will use the alarm correlation clustering method to filter the same or similar abnormal alarms within a time period, thereby solving the alarm redundancy problem caused by unnecessary repeated alarms over a period of time, further improving the work efficiency of operation and maintenance personnel and assisting in locating the root cause of faults, reducing the storage cost of alarm information and the communication costs caused by alarm notifications.

[0061] (3) Alarm threshold adjustment

[0062] The main purpose of the alarm threshold adjustment module is to improve the accuracy and flexibility of alarm clustering by dynamically adjusting the cluster similarity threshold, so as to effectively suppress redundant alarms and reduce false alarms. The specific steps are as follows:

[0063] 1) Initialization threshold setting

[0064] When the system is first run, a default clustering similarity threshold is set based on historical alarm data and field experience. Only two alarms with a similarity above the threshold will be clustered into the same type of alarm.

[0065] 2) Threshold adjustment based on historical data

[0066] Frequent alarm adjustment: The system analyzes historical alarm data, especially frequently occurring alarms. For alarm groups with high frequency and prone to false alarms, the clustering similarity threshold is gradually increased, so that the system can cluster such alarms more strictly and avoid redundant alarms.

[0067] Severe alarm adjustment: For alarms with higher severity, the similarity clustering threshold is appropriately lowered. In this way, even if the similarity between two alarms is slightly lower, the system can still quickly cluster them and trigger a timely response.

[0068] 3) Feedback mechanism and adaptive adjustment:

[0069] When an operator manually handles or ignores a certain type of alarm, the system will record these actions and automatically adjust the cluster similarity threshold of the corresponding alarm group. For example, when an operator ignores a certain type of false alarm multiple times, the system will increase the similarity threshold of this type of alarm, making it more difficult to trigger in the future.

[0070] (4) Convergence alarm output

[0071] The convergence alarm output module is designed to ensure that operation and maintenance personnel can handle alarms quickly and efficiently by clustering and optimizing alarm information output, avoiding the impact of excessive alarms or redundant information on decision-making.

[0072] Alarm summary generation: For each alarm group, the system generates a concise summary information that summarizes the core information of the alarm group, including alarm type, severity, number of similar alarms and frequency of occurrence, etc.

[0073] The system is designed with multiple levels of alarm views to display alarm correlation information at different levels, including top-level views and secondary views. The top-level view helps operation and maintenance personnel quickly understand the most important alarms; the secondary view allows users to view specific alarm information in each alarm group and understand the alarm details. For a specific flow chart, see Figure 2 .

[0074] Example 2

[0075] In the database alarm system, users hope to improve the efficiency of alarm processing by optimizing the associated clustering of alarms. Assume that a set of alarm data includes:

[0076] ·Alarm A: Database connection failed, unable to access user data

[0077] Alarm B: User data query timeout, database response slow

[0078] Warning C: Unable to connect to database, network problem

[0079] Warning D: The database read rate is too low, affecting user experience

[0080] Warning E: Data update failed, check the connection settings

[0081] The process of calculating the similarity between alarm A and alarm B using the predefined semantic similarity is as follows:

[0082] First, the keywords of alarm A and alarm B are:

[0083] Alarm A: Database, connection failure, access, user, data

[0084] Alarm B: User, Data, Query, Timeout, Database, Slow Response

[0085] Assume that the context window k = 2, obtain the context of each keyword, and calculate the semantic similarity simw. According to the context word set C(w) of each word, especially the two words on the left and right, calculate the context similarity simw context (w1,w2). Assuming β = 0.7, we can get the final word similarity. Finally, assuming the semantic similarity weights α = 0.5, γ = 0.7, η = 0.5, we can get the final similarity result CR(t i ,t j )

[0086]

[0087] Finally, the short text similarity between alarm A and alarm B is calculated as S CR (d1,d2)=0.83.

[0088] Use the predefined semantic similarity calculation method to convert each alarm into a vector representation and calculate the similarity score. The similarity matrix is ​​as follows:

[0089]

[0090]

[0091] Assuming the initial similarity threshold is 0.5, the preliminary clustering results based on the similarity matrix are as follows:

[0092] Cluster 1: Alarm A, Alarm B, Alarm C (similarity higher than 0.5)

[0093] Cluster 2: Alarm D, Alarm E (similarity higher than 0.5)

[0094] Therefore, the total number of alarms before clustering is 5, and the alarm categories after clustering are 2. These two types of alarms are submitted to the operation and maintenance personnel separately, avoiding repeated and unnecessary duplicate alarms.

[0095] However, after the initial clustering, based on the feedback from the operation and maintenance personnel and the calculation of the historical operation and maintenance data in the alarm threshold adjustment phase, the threshold was adjusted to 0.6, and the clustering result of the recalculated similarity matrix was:

[0096] Cluster 1: Alarm A, Alarm B (similarity higher than 0.6)

[0097] Cluster 2: Alarm C (independent, not clustered)

[0098] Cluster 3: Alarm D, Alarm E (similarity higher than 0.6)

[0099] Finally, the alarm categories after clustering are 3, which further improves the accuracy of clustering and reduces unnecessary alarm linkage. Moreover, if irrelevant alarms are incorrectly clustered, the system may issue unnecessary repair suggestions or trigger invalid automated processing.

[0100] The present invention also provides a device, which includes: a data acquisition device, a processor and a memory; the data acquisition device is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to execute any of the above methods.

[0101] The present invention further provides a computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used to execute any of the above methods.

[0102] The units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this application, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0104] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0105] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0106] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0107] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An alarm suppression method based on semantic coupling distance discrimination, characterized in that: The method comprises the following steps: (1) Alarm data preprocessing, data cleaning and feature extraction; (2) Alarm association clustering: two or more alarms with exactly the same alarm names are directly regarded as the same type of alarms; for alarms with different names but high semantic similarity, they are clustered using a short text similarity calculation method based on semantic coupling distance discrimination; (3) adjusting the correlation threshold, dynamically adjusting the cluster similarity thresholds of various alarms based on the alarm data clustered in step (2); (4) Converge alarm output, filter alarms based on dynamic thresholds combined with clustering results, and output alarm results.

2. The method for suppressing an alarm based on semantic coupling distance discrimination according to claim 1, characterized in that: In step (2), the short text similarity calculation method based on semantic coupling distance discrimination comprises the following steps: 2.1 Calculate the context word set C(w1) of word w1 and word w2 = {w p |p∈[qk,q+k]},C(w2)={w p |p∈[lk,l+k]}, where p and q are the position indexes of words w1 and w2 in the sentence, p≠q, p≠l, and k is the context window size; 2.2 Calculating comprehensive context similarity Among them, simw() is the semantic similarity between two words, and c is the context word; 2.3 Calculate the similarity simw between two words f (w1,w2)=β·simw(w1,w2)+(1-β)·simw context (w1, w2), where β is the weight coefficient. 2.4 Calculating Inline Relationships in, When CoR(t i , t j )=0;d s For a specific text, and Term t i and term t j , and α is the weight for adjusting the contribution of semantic similarity.

3. An alarm suppression system based on semantic coupling distance discrimination, characterized in that: The system includes the following modules: (1) Alarm data preprocessing module, which performs data cleaning and feature extraction; (2) Alarm association clustering module: For two or more alarms with exactly the same alarm names, this module directly regards them as the same type of alarms; for alarms with different names but high semantic similarity, this module clusters them using a short text similarity calculation method based on semantic coupling distance discrimination; (3) an association threshold adjustment module, which dynamically adjusts the similarity thresholds of various alarm clusters based on the alarm data clustered by the alarm association clustering module; (4) Convergence alarm output module, which filters alarms based on dynamic thresholds combined with clustering results and outputs alarm results.

4. The alarm suppression system based on semantic coupling distance discrimination as claimed in claim 3, characterized in that: In the alarm association clustering module, the short text similarity calculation method based on semantic coupling distance discrimination includes the following steps: 2.1 Calculate the context word set C(w1) of word w1 and word w2 = {w p |p∈[qk,q+k]},C(w2)={w p |p∈[lk,l+k]}, where p and q are the position indexes of words w1 and w2 in the sentence, p≠q, p≠l, and k is the context window size; 2.2 Calculating comprehensive context similarity Among them, simw() is the semantic similarity between two words, and c is the context word; 2.3 Calculate the similarity simw between two words f (w1,w2)=β·simw(w1,w2)+(1-β)·simw context (w1, w2), where β is the weight coefficient. 2.4 Calculating Inline Relationships in, When CoR(t i , t j )=0;d s For a specific text, and t i and term t j , and α is the weight for adjusting the contribution of semantic similarity.

Images