Abnormality detection method, device, electronic device, storage medium and program product

Abstract

The present invention provides an anomaly detection method, device, electronic device, storage medium and program product, which relate to the fields of big data, IT and software development technology. The embodiments of the present invention integrate data from multiple log sources and mine the correlation and complementarity between different log sources, so as to more accurately identify and locate abnormal events. Through multi-source log collaborative analysis, it can also help administrators quickly locate the root cause of abnormal events, providing strong support for fault recovery and prevention.

Description

Technical Field

[0001] Embodiments of the present invention relate to the fields of big data, IT and software development technology, and in particular to an anomaly detection method, device, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of information technology, various information systems and services have become increasingly complex, and the log data generated has also shown an explosive growth trend. These log data not only contain the status information of the system operation, but also hide potential anomalies and fault clues. At present, the main challenges faced by log anomaly detection include data representation, data instability, data imbalance, anomaly diversity, label availability, stream processing, large data volume, and model interpretability. These problems make it difficult for a single log analysis method to cope with complex and changing actual scenarios.

[0003] Common solutions to these problems include data preprocessing, feature extraction, and the application of machine learning algorithms. However, these methods can only process structured log data and are not very effective for unstructured or semi-structured log data. In addition, traditional anomaly detection methods lack collaborative analysis between different log sources and cannot fully utilize the advantages of multi-source logs. Summary of the invention

[0004] Embodiments of the present invention provide an anomaly detection method, device, electronic device, storage medium, and program product to at least partially solve the above-mentioned problems.

[0005] A first aspect of an embodiment of the present invention provides an anomaly detection method, the method comprising: Obtaining log data from at least two log sources, wherein the log data carries a global common identifier; Based on the global common identifier, the log data is aggregated to obtain multiple log data sets, each log data set includes log data from multiple different sources but belonging to the same business process; Based on the log data set corresponding to the business process where the abnormal event occurs, an abnormal event set is constructed, the support of each single event in the abnormal event set is calculated, and based on the support and support threshold of each single event, an abnormal candidate set containing multiple event combinations is constructed; Generate abnormal link information based on a combination of multiple events in the abnormal candidate set; An abnormality detection message is generated based on the abnormal link information.

[0006] Optionally, an abnormal event set is constructed based on a log data set corresponding to an abnormal business process, the support of each single event in the abnormal event set is calculated, and an abnormal candidate set of a multi-event combination is constructed based on the support and support threshold of each single event, including: Determine the support degree of each single event based on the proportion of times each single event appears in the abnormal event set; Construct an abnormal candidate set containing multiple dual events based on single events whose support is greater than the support threshold; Determine the support degree of each double event based on the ratio of the number of times that two events in each double event appear consecutively in the abnormal event set; Based on the dual events whose support is greater than the support threshold, an abnormal candidate set containing multiple multi-events is constructed, and the support of multiple multi-events is calculated until there is only one multi-event whose support is greater than the support threshold, and an abnormal candidate set containing a combination of multiple events is obtained.

[0007] Optionally, before aggregating the log data, the method further includes: Performing structured analysis on the log data from the at least two log sources, and converting the log data from different sources into a unified format; Filter meaningless log data; De-duplicate log data; Fill missing or invalid values ​​in log data.

[0008] Optionally, the method further comprises: Detect abnormal events; In case an abnormal event is detected, an abnormal alarm message is generated.

[0009] Optionally, generating abnormal link information based on a combination of multiple events in the abnormal candidate set includes: Generate an event association rule based on a combination of multiple events in the abnormal candidate set, and obtain a directed graph according to the event association rule; Based on the directed graph, determining a key propagation path; Abnormal link information is generated based on the key propagation path.

[0010] Optionally, generating an event association rule based on a combination of multiple events in the abnormal candidate set, and obtaining a directed graph according to the event association rule, including: Create virtual nodes based on multiple events in the multi-event combination; Calculate the confidence level of the second event occurring when the first event occurs in each of the two events among multiple events; When the confidence level is greater than the confidence level threshold, the first event is associated with the second event, a directed edge is established between the virtual nodes corresponding to the first event and the second event, and the confidence level is used as a weight of the directed edge; Based on the directed graph, a key propagation path is determined, including: Path discovery is performed based on directed graphs. Taking the abnormal event as the starting node, the shortest path to other nodes is found to determine the propagation path of each abnormal event. The key propagation path is determined based on the weights of each directed edge included in each abnormal event propagation path.

[0011] A second aspect of an embodiment of the present invention provides an abnormality detection device, the device comprising: A data collection unit, used to obtain log data from at least two log sources, wherein the log data carries a global common identifier; A feature extraction unit, configured to aggregate the log data based on the global common identifier to obtain a plurality of log data sets, each of which includes log data from a plurality of different sources but belonging to the same business process; A collaborative analysis unit is used to construct an abnormal event set based on a log data set corresponding to a business process where an abnormal event occurs, calculate the support of each single event in the abnormal event set, and construct an abnormal candidate set containing a combination of multiple events based on the support and support threshold of each single event; and generate abnormal link information based on the combination of multiple events in the abnormal candidate set; The abnormality warning unit is used to generate an abnormality detection message based on the abnormal link information.

[0012] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the anomaly detection method as described in the first aspect of the present invention when executed.

[0013] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the anomaly detection method as described in the first aspect of the present invention.

[0014] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is used by a processor to implement the steps in the anomaly detection method described in the first aspect of the present invention.

[0015] In the anomaly detection method proposed in the embodiment of the present invention, log data from different sources (such as server, client, middleware, database, etc.) are innovatively integrated, and structured integration of multi-source logs is achieved through unified log parsing rules and standardized format conversion, effectively eliminating format differences and semantic inconsistencies between multi-source logs, forming a unified log data pool, and significantly improving the comprehensiveness and accuracy of anomaly detection.

[0016] In an embodiment of the present invention, a collaborative analysis method for multi-source logs is performed based on a cross-source log event correlation analysis method. By performing a comprehensive analysis on multi-source log data, abnormal patterns that cannot be revealed by a single log source can be discovered, thereby improving the accuracy and sensitivity of anomaly detection.

[0017] Therefore, in the embodiment of the present invention, by integrating the data of multiple log sources and exploring the correlation and complementarity between different log sources, abnormal events can be more accurately identified and located. Through collaborative analysis of multi-source logs, administrators can also quickly locate the root cause of abnormal events, providing strong support for fault recovery and prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 is a flow chart of an anomaly detection method provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a process for constructing anomaly candidate sets in an anomaly detection method provided in an embodiment of the present invention; Figure 3 is an example diagram of a directed graph of abnormal events in the abnormality detection method provided in an embodiment of the present invention; Figure 4 It is a schematic diagram of the framework structure of the implementation environment of the anomaly detection method provided by the embodiment of the present invention; Figure 5 It is a specific flowchart of the anomaly detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] In the embodiments of the present invention, the inventors have explored and studied related technologies and found that traditional log anomaly detection methods often only analyze a single log source, ignoring the correlation and complementarity between multi-source logs, resulting in limited accuracy and efficiency of anomaly detection.

[0022] For example: A log processing method is disclosed in the related art, in which the description information of the error log needs to be obtained; the description information of all error logs is standardized; all error logs are clustered by a clustering model according to the standardized description information to obtain multiple clustering results; the category features of each clustering result are marked according to the standardized description information; and the log analysis results of the corresponding error log set are generated according to the category features. It can be seen that in this method, the clustering accuracy of the clustering model for the error log is improved mainly by using the description information of the error log as the model feature of the clustering model, so as to obtain a more accurate log analysis result.

[0023] Another example: A method for automatically detecting abnormal logs of a system based on a graph attention network is disclosed in the related art. In this method, first, it is necessary to construct corresponding log undirected graphs for all log sample data that need to be judged, and then the log undirected graphs are used to obtain the graph embedding vectors of each log sample data based on the graph attention network; then all the graph embedding vectors are input into the graph attention network to learn semantic features to obtain the first graph representation features of each log undirected graph; then all the first graph representation features are input into the feature projection network, and all the first graph representation features are enhanced using the feature projection technology to obtain the second graph representation features; finally, all the second graph representation features are classified using the fully connected network to obtain the abnormal log category. It can be seen that in this method, the graph representation features of the logs are mainly constructed to classify the logs, so as to improve the accuracy of automatic detection of abnormal logs.

[0024] In the above two exemplary related technologies, analysis is performed on a single log source, without considering the correlation and complementarity between multi-source logs.

[0025] Therefore, the embodiment of the present invention proposes an anomaly detection method based on multi-source log collaboration, which integrates data from multiple log sources and uses a collaborative analysis algorithm to mine the correlation and complementarity between different log sources, thereby achieving comprehensive and accurate detection of abnormal events.

[0026] Specifically, Figure 1 As shown, it shows a flowchart of a method for detecting anomalies according to an embodiment of the present invention, the method comprising the following steps: S101, obtaining log data from at least two log sources, wherein the log data carries a global common identifier.

[0027] S102 , aggregating the log data based on the global common identifier to obtain multiple log data sets, each log data set including multiple log data from different sources but belonging to the same business process.

[0028] S103, constructing an abnormal event set based on the log data set corresponding to the business process where the abnormal event occurs, calculating the support of each single event in the abnormal event set, and constructing an abnormal candidate set containing a combination of multiple events based on the support of each single event and the support threshold.

[0029] S104: Generate abnormal link information based on a combination of multiple events in the abnormal candidate set.

[0030] S105: Generate an abnormality detection message based on the abnormal link information.

[0031] In the embodiment of the present invention, the log data from different sources may be: system logs, application logs, database logs, network logs, etc.

[0032] In an embodiment of the present invention, the method is applied to an anomaly detection device, which mainly includes a data acquisition unit, a data preprocessing unit, a feature extraction unit, a collaborative analysis unit, an anomaly warning unit, and a data storage unit.

[0033] The data collection unit is used to collect log data from multiple log sources. The log sources can be defined and configured. The collected content includes but is not limited to system logs, application logs, database logs, network logs, etc. The logs can be collected in real time or in batches, and the encrypted transmission and storage of data are supported. The data receiving unit is responsible for receiving the log data reported by the data collection unit and storing them in order according to the time series. The data preprocessing unit is responsible for cleaning, formatting and standardizing the collected log data to eliminate noise and inconsistency in the data. The feature extraction unit is responsible for aggregating all relevant log entries through the global common identifier of the preprocessed log data. The collaborative analysis unit is responsible for constructing event sets through log data and automatically generating association rules. The abnormal warning unit is responsible for sending warning information of detected abnormal events to the alarm contact through a visual interface, SMS, email, instant messaging tools, etc. The data storage unit is responsible for storing the collected logs, automatically generated association rules, identified abnormal warning information, and relevant information of the alarm contact.

[0034] Specifically, in the embodiment of the present invention, a log collection adapter is configured in the data collection unit, and log data from different sources can be collected respectively. In the embodiment of the present invention, different data collection units can be used to collect log data from different sources respectively. Before collecting log data, log source information, log type, collection frequency, collection method, data encryption method, data transmission method, etc. can also be configured for each data collection unit.

[0035] In the embodiment of the present invention, the log collection adapter can be used to collect and adapt non-post-fragmented data and semi-structured data, and extract log information according to rule or feature matching.

[0036] In the embodiments of the present invention, the collection methods can be divided into real-time collection, incremental collection, and batch collection. Real-time collection refers to monitoring the log source and pulling the newly generated log data at a set frequency. Incremental collection is to transmit only the newly added log entries since the last collection to reduce network bandwidth consumption. Batch is to collect data at one time for historical logs stored in the file system, cloud storage or database by specifying the collection range.

[0037] In the embodiment of the present invention, the data encryption method requires key decryption before log collection for certain encrypted storage logs. In this case, a corresponding data encryption method can be configured for the data collection unit.

[0038] In the embodiment of the present invention, the global common identifier is an identifier that can associate log data from different sources that belong to the same business process or transaction, such as a user ID, a business identifier, and the like.

[0039] In the embodiment of the present invention, a unified global common identifier rule can be pre-configured for log generation devices corresponding to log data from different sources, so that log data from different sources can carry a unified global common identifier for subsequent collaborative analysis.

[0040] In the embodiment of the present invention, before executing step S103, all log entries containing the global common identifier can be merged together and arranged and summarized in time series. Further, the log entries are subjected to event standardization, and event labels are set for each type of event to prepare for multi-source log collaborative analysis.

[0041] In the embodiment of the present invention, support refers to the frequency of occurrence of a certain item set (which may include one or more data) in a data set.

[0042] Specifically, step S103 includes the following sub-steps: S1031, determining the support degree of each single event based on the proportion of times each single event appears in the abnormal event set.

[0043] S1032: construct an abnormal candidate set including multiple dual events based on single events whose support is greater than a support threshold.

[0044] S1033: Determine the support degree of each double event based on the ratio of the number of times that two events in each double event appear consecutively in the abnormal event set.

[0045] S1034, constructing an abnormal candidate set including multiple multi-events based on the dual events whose support is greater than the support threshold, and calculating the support of the multiple multi-events until there is only one multi-event whose support is greater than the support threshold, thereby obtaining an abnormal candidate set including the combination of multiple events.

[0046] In the embodiment of the present invention, constructing an event set refers to constructing a predefined event set based on abnormal events. The abnormal events may include: network problems, slow application response, database timeout, etc. Specifically, after the abnormal event occurs, the log data set corresponding to the business process of the abnormal event can be determined based on the global common identifier, so as to construct the abnormal event set. Assuming that the determined event is Event 1 , Event 2, ...... , Event n , then the initial single event set is: C1={Event 1 , Event 2, ...... , Event n } .

[0047] In the embodiment of the present invention, in step S301, the support degree is calculated for each single event item in the initial event set, and the calculation formula is: (1) Where C is the total number of event sets, y is the number of times event A appears in the total event set, Represents the support of event A.

[0048] In the embodiment of the present invention, a support threshold, min_sup, can be preset. >min_sup, then event A is considered frequent and a frequent item set is obtained.

[0049] Further, based on frequent itemsets (single events with support greater than the support threshold), anomaly candidate sets containing multiple dual events are constructed.

[0050] In the embodiment of the present invention, the abnormal candidate set is a set of event sets that meet the support threshold. Each time a new abnormal candidate set is constructed, it is necessary to generate a candidate set by connecting the event items in the previous frequent item set. At the same time, it is necessary to meet the requirement that in the new item set generated by the connection, any two consecutive sub-items (which are also continuous in the original set) must appear in the frequent item set of the previous stage. Specifically: C k Depend on C k−1 The size of the generated new item set is k. L k−1 Indicates size k−1 The frequent itemsets of C k The construction method is: C k ={X∪Y|X∈L k−1 ,Y∈L k−1 ,X≠Y,andX ∣X∣−1 =Y 1 } (2) Among them, X ∣X∣−1 represents the last element of set X, and Y1 is the first element of set Y. Frequent item sets at each stage L k Used to construct the candidate set C of the next order k+1 , until no new frequent itemsets can be generated.

[0051] In the embodiment of the present invention, frequent item sets and association rules are mined based on the Apriori algorithm. The Apriori algorithm is based on an assumption: if an item set is frequent, then all its non-empty subsets must also be frequent. This assumption is called the Apriori property. Based on this property, the algorithm can effectively reduce the number of candidate item sets that need to be checked.

[0052] In the embodiment of the present invention, the candidate set refers to a set of itemsets that may become frequent itemsets during the mining process. Each step of the algorithm generates a new candidate set, which is constructed based on the frequent itemsets found in the previous step.

[0053] In the embodiment of the present invention, the support threshold is a threshold defined by the user according to actual needs, which is used to determine whether an item set is considered frequent. If the number of times an item set appears in all events exceeds this threshold, then it is considered frequent.

[0054] In the Apriori algorithm, new candidate sets are generated from the previous frequent itemset through a concatenation operation. For example, if there are two frequent itemsets {A} and {B}, then their concatenation can generate a new candidate itemset {A, B}. When generating a new candidate itemset, it is necessary to ensure that any two consecutive sub-items in the new itemset (i.e., items that are adjacent in the new itemset) must also be consecutive in the frequent itemset of the previous stage. This means that if a candidate itemset containing three items is to be generated, then any two consecutive items among the three items must have appeared in the previous frequent itemset.

[0055] This rule ensures that the generation of candidate itemsets is meaningful, because if a subset of an itemset is not frequent, then the itemset itself cannot be frequent. In this way, the algorithm can effectively narrow the search space and reduce the amount of computation.

[0056] For ease of understanding, the following combination Figure 2 The specific steps of constructing anomaly candidate sets in the anomaly detection method provided in the embodiment of the present invention are described as follows: In an embodiment of the present invention, a support threshold can be set according to actual needs, and then a set of single event items can be constructed. The support of the abnormal event set is calculated for each single event item, and an abnormal event candidate set is constructed based on single event items whose support is greater than the support threshold. When there are multiple dual event items that meet the support threshold in the abnormal event candidate set, an abnormal event candidate set including multiple multi-event items is further constructed until there are no new multi-event items that meet the support threshold, and association rules are automatically generated based on the finally generated abnormal candidate set including multi-event combinations.

[0057] For example, assume that the following simplified log data is set, see Table 1, which records the events recorded by various components or application systems of user U123 within a certain period of time: Table 1

[0058] Assuming the support threshold is set, min_sup = 0.3, it means that at least 30% of the transactions need to contain a certain itemset to be considered frequent.

[0059] Next, starting with a single event item, create C1, the set of all possible single event items: C1={NetworkIssue, AppSlow, DBTimeout, OSOverload}; Calculate the support for each event item in C1: NetworkIssue: appears 4 times in T1, T3, T6, and T10, with a support of 4 / 10 = 0.4; AppSlow: Appears 5 times in T2, T3, T7, T8, and T10, with a support value of 0.5; DBTimeout: appears 3 times in T4, T7, and T8, with a support of 0.3; OSOverload: Appears 3 times in T5, T6, and T9, and the support is also 0.3; Since the support threshold is 0.3, all items meet the requirements of frequent itemsets.

[0060] Based on C1, construct an abnormal event candidate set C2 containing two event items. Since the support meets the conditions, C2={NetworkIssue,AppSlow},{NetworkIssue,DBTimeout}, {NetworkIssue,OSOverload},{AppSlow,DBTimeout}, {AppSlow,OSOverload},{DBTimeout,OSOverload} The next step is to calculate the support of C2 to determine the frequent itemsets: {NetworkIssue, AppSlow}: Appears 3 times in T3, T8, and T10, with a support value of 0.3; {NetworkIssue, DBTimeout}: appears in T8, 1 time, support is 0.1 (does not meet min_sup); {NetworkIssue, OSOverload}: Appears twice in T6 and T10, with a support of 0.2 (does not meet min_sup); {AppSlow, DBTimeout}: Appears twice in T7 and T8, with a support of 0.2 (min_sup is not met); {AppSlow, OSOverload}: does not exist; {DBTimeout, OSOverload}: does not exist; Therefore, only {NetworkIssue, AppSlow} satisfies the condition of frequent itemset.

[0061] Since all combinations except {NetworkIssue, AppSlow} in C2 do not meet the support threshold, there will be no higher-order frequent itemsets. Assuming that there are more options that meet the support threshold, we need to continue to build C3 (a candidate item set containing three event items), and so on.

[0062] In an embodiment of the present invention, based on the above steps S1031~S1034, an abnormal candidate set including a multi-event combination can be obtained. The multi-event combination represents that the abnormal event has a high correlation with multiple events in the multi-event combination. Therefore, the correlation relationship between each event of the multiple events included in the multi-event combination can be further analyzed.

[0063] Specifically, step S104 includes the following sub-steps: S1041, generating event association rules based on the combination of multiple events in the abnormal candidate set, and obtaining a directed graph according to the event association rules.

[0064] S1042: Determine a key propagation path based on the directed graph.

[0065] S1043: Generate abnormal link information based on the key propagation path.

[0066] In an embodiment of the present invention, each event in a multi-event group can be analyzed to obtain event association rules. Each event is used as a node, and an edge is obtained based on the event association rules to construct a directed graph, so that the propagation path of the abnormal event can be determined simply and clearly, and each path can be further analyzed to obtain the key propagation path, thereby determining the abnormal link information.

[0067] Specifically, step S1041 includes the following sub-steps: S10411, creating a virtual node based on multiple events in the multi-event combination.

[0068] S10412, calculating the confidence level of the occurrence of the second event when the first event occurs in every two events among the multiple events.

[0069] S10413: When the confidence level is greater than the confidence level threshold, associate the first event with the second event, establish a directed edge between virtual nodes corresponding to the first event and the second event, and use the confidence level as a weight of the directed edge.

[0070] In the embodiment of the present invention, the confidence level is used in the association rule to measure the probability of a subsequent event occurring when a previous event occurs.

[0071] Specifically, confidence can be used to represent conditional probability. Event i ⇒Eventj , that is, in Event i In case of occurrence Event j Probability of occurrence: (3) in Indicated in Under the conditions that occur, The probability of occurrence, setting the significance level to α, then the confidence level is 1-α, so when the conditional probability If the confidence level is greater than 1-α, then and Establish event correlation rules.

[0072] In an embodiment of the present invention, multiple events in a multi-event combination can be combined in pairs, and the probability of another event occurring when a certain event in the pairwise combination occurs is calculated, thereby obtaining multiple confidences, and then according to the confidence threshold, it is determined that the events with confidences greater than the confidence threshold are associated, and then the events can be used as virtual nodes, and directed edges can be constructed between the virtual nodes corresponding to the associated events, and the corresponding confidences can be used as the weights of the directed edges. Based on the weights of each directed edge, the weight of the corresponding abnormal time propagation path can be calculated, and then the path with the largest weight can be used as the key propagation path.

[0073] In this case, the step S1042 includes the following sub-steps: S10421, perform path discovery based on a directed graph, take the abnormal event as the starting node, find the shortest path to other nodes, and determine the propagation path of each abnormal event.

[0074] S10422, determining a key propagation path based on the weights of each directed edge included in each abnormal event propagation path.

[0075] For ease of understanding, the following example is used to explain: Based on the frequent itemsets {NetworkIssue, AppSlow}, association rules can be generated and confidence can be calculated.

[0076] Rule 1: NetworkIssue → AppSlow Support is: sup(NetworkIssue∪AppSlow)=0.3 The confidence level is:

[0077] This indicates that when network problems occur, there is a high probability that the application will respond slowly, providing valuable information for system monitoring and exception handling.

[0078] Furthermore, it is assumed that the following association rules can be constructed from the multi-event combinations obtained based on the analysis of the abnormal candidate set: Rule 1: (NetworkIssue) → (AppSlow); Rule 2: (AppSlow) → (OSOverload); Rule 3: (NetworkIssue) → (DBTimeout); Rule 4: (DBTimeout) → (AppSlow Based on the above rules, a directed graph is constructed to show the propagation relationship of abnormal status, and confidence can also be added as a weight value, such as Figure 3 shown.

[0079] Based on this directed graph, anomaly propagation path analysis is performed. The starting anomaly point is NetworkIssue, and there are two direct paths: NetworkIssue → AppSlow; NetworkIssue → DBTimeout; There is one indirect path: NetworkIssue → AppSlow → OSOverload, this path shows that the network issue affects the responsiveness of the application, which further causes the operating system resources to be overloaded.

[0080] Further critical path identification is performed based on the weights of directed edges: Path 1: NetworkIssue → AppSlow → OSOverload. This path reveals the chain reaction of network problems, which not only affects users’ use of applications, but may also cause a significant increase in system resource usage.

[0081] Path 2: NetworkIssue → DBTimeout. This path directly points out the direct impact of network problems on database operations, which may cause data processing failure.

[0082] In the embodiment of the present invention, the Dijkstra algorithm or the Floyd-Warshall algorithm may also be used to implement the identification of the critical path.

[0083] In the embodiment of the present invention, in step S104, the event association rule is converted into a directed graph, and n association rules are automatically obtained. Based on the events in each rule, a virtual node is created, and the association in the association rule is converted into a directed edge. The weight of the edge is the confidence level of 1-α, so as to construct a directed graph. Further path discovery based on the directed graph and finding the shortest path from the starting anomaly to other nodes can help quickly identify the key propagation path. At the same time, based on the weight of the edge, the high probability path of the anomaly propagation is identified, and then the propagation path of the abnormal event is generated.

[0084] In the embodiment of the present invention, after step S101 and before step S102, the method further includes the following steps: S11, performing structured analysis on the log data from the at least two log sources, and converting the log data from different sources into a unified format.

[0085] S12, filter meaningless log data.

[0086] S13, performing deduplication processing on the log data.

[0087] S14, filling in missing values ​​or invalid values ​​in the log data.

[0088] In an embodiment of the present invention, the raw log data collected from different sources can be structured and parsed, and the logs from different sources can be converted into a unified format, including time format standardization, field name standardization, and coding format standardization. Further filter meaningless log entries and perform denoising. Further deduplicate the log entries and merge duplicate records. Further process the missing or invalid log values, set default values ​​or use interpolation methods. Further perform outlier detection processing, identify abnormal or outlier data, determine whether it is a real anomaly or data error, and make corrections or markings.

[0089] Specifically, in the embodiment of the present invention, the method further includes the following steps: S21, detecting abnormal events.

[0090] S22: When an abnormal event is detected, an abnormal alarm message is generated.

[0091] In an embodiment of the present invention, after an abnormal event is detected, an abnormal alarm message (without path information) is generated. After the abnormal link information is analyzed, an abnormal detection message is generated for a secondary alarm. The abnormal alarm message can notify the alarm contact of the abnormal event in the first time, thereby improving the timeliness of the abnormal alarm, and the abnormal detection message provides the propagation path of the abnormal event to the alarm contact, thereby understanding the propagation path of the abnormal event from a global perspective, quickly finding the source of the abnormal event, and improving the efficiency of abnormal event processing.

[0092] In an embodiment of the present invention, the abnormal alarm is divided into two stages. The first non-associated alarm is issued when an abnormal event occurs, and the secondary associated alarm is issued after the abnormal event propagation path is generated. The first non-associated alarm is to notify the alarm contact of the alarm event as soon as possible, intervene in the abnormal status component investigation and recovery, and the secondary associated alarm provides the propagation path of the abnormal event to the alarm contact, understands the propagation path of the abnormal event from a global perspective, quickly finds the source of the abnormal event, quickly resolves the abnormal event, and restores the service status of the associated components.

[0093] For ease of understanding, the following is based on Figure 4 The anomaly detection method proposed in the embodiment of the present invention is further explained. Figure 4 A schematic diagram showing the framework structure of the implementation environment of the embodiment of the present invention is shown. Figure 3 The relationship between the data acquisition unit, the abnormality detection device and the Dashboard in the embodiment of the present invention is described as follows: In an embodiment of the present invention, a data collection unit collects log data, a log collection adapter is added to the collection unit to increase the collection adaptation capability of the collection unit, data is received by a data receiving unit, a data preprocessing unit preprocesses the log data, feature data is further extracted by a feature extraction unit for aggregation, automatic event association rule generation is completed by a collaborative analysis unit, and the event is saved in a data storage unit, and finally a propagation path of an abnormal event is generated by an abnormal warning unit, and an alarm message is sent.

[0094] In related technologies, the data collection unit actively collects various system logs and actively reports them to the unified log service. The unified log service stores them in groups and displays them to operation and maintenance personnel or users through the dashboard interface. Multi-source logs are stored in groups and are not related to each other, so the correlation value of multi-source logs cannot be brought into play. In this application, by integrating data from multiple log sources and exploring the correlation and complementarity between different log sources, abnormal events can be more accurately identified and located. Through collaborative analysis of multi-source logs, administrators can also quickly locate the root cause of abnormal events and provide strong support for fault recovery and prevention.

[0095] Further, based on Figure 5 The anomaly detection method proposed in the embodiment of the present invention is further explained. Figure 5 The specific flow chart of the abnormality detection method according to the embodiment of the present invention is shown as follows: Figure 5 As shown: In an embodiment of the present invention, log data can be collected from multiple log sources through a data collection unit, and the configuration information of each data collection unit includes but is not limited to the log source ID, log type (file, syslog, etc.), log extraction format, collection cycle, encryption, key information, etc.

[0096] In the embodiment of the present invention, for the encrypted log data, it is necessary to collect the required information after the data collection unit decrypts it.

[0097] In the embodiment of the present invention, for the log data encrypted by the transmission channel, a reliable transmission protocol such as SSL / TLS, SFTP, etc. can be configured for the data collection unit to collect the data and send it to the data receiving unit.

[0098] In the embodiment of the present invention, for batch collection of log data, for the file system, it is necessary to specify the file directory, collect the file range, and collect according to certain rules. For example, if the file name is date_random number.txt, then the date range can be specified to collect log information within the specified date range. The rules can be configured and matched according to the file suffix and file name rules.

[0099] The following takes the file system log as an example Figure 5 To illustrate, in the embodiment of the present invention, the log data collection process specifically includes the following steps: S1, the data collection unit reads the collection configuration information and determines whether it is real-time collection. If it is configured as real-time collection, it directly enters the data collection link S3, otherwise it enters S2.

[0100] S2 reads the batch collection interval, such as log files from January 2024 to the present, and matches the log file names that meet the configuration requirements and adds them to the collection queue.

[0101] S3, the data collection unit reads the log file and determines whether the log file is encrypted. If it is encrypted, it first enters S4 for decryption, otherwise, it directly enters S5.

[0102] S4, the data collection unit reads the configuration file decryption key, decrypts the log file, and enters S5 after decryption.

[0103] S5, matches the content to be collected through the configured log collection adapter.

[0104] S6, transmitting the content to the data receiving unit through the configured preset address.

[0105] In the embodiment of the present invention, the data receiving unit pushes the collected log data to the data preprocessing unit for structured analysis, data cleaning, and standardization processing.

[0106] The following takes the file system log as an example Figure 5 To illustrate, in the embodiment of the present invention, the log data preprocessing process specifically includes the following steps: S7, the data preprocessing unit, performs structured analysis on the original log text and extracts key fields (such as timestamp, event type, source IP, user ID, operation type, error code, etc.). It supports multiple analysis strategies such as regular expressions, template matching, JSON / XML analysis, etc., to adapt to different log formats.

[0107] S8, the data preprocessing unit cleans the log data, including but not limited to deduplication processing, missing value processing, and outlier detection and correction.

[0108] In the embodiment of the present invention, deduplication processing refers to determining whether a record is duplicated through a variety of identification methods and removing duplicate log records, which can be achieved through hash value comparison, primary key check, etc.

[0109] Missing value processing refers to filling missing or invalid key fields (such as using median, mode, top / bottom value filling, default value filling, etc.) or marking them as missing.

[0110] Outlier detection and correction refers to the identification and processing of outliers that significantly deviate from the normal range through statistical analysis of numeric fields. This can be done by using methods such as Z-score and box plots, and can be corrected using preset rules.

[0111] S9, the data preprocessing unit standardizes the log data, including but not limited to: timestamp conversion, encoding conversion, and field standardization.

[0112] Among them, timestamp conversion is to convert inconsistent timestamp formats in various source logs into a standard format (such as UTC time) to facilitate time series analysis. Encoding conversion means ensuring that all text data uses a unified character encoding (such as UTF-8) to avoid garbled characters. Field standardization is to normalize synonymous or near-synonymous field names and enumeration values, establish a vocabulary mapping table across source logs, and ensure data consistency.

[0113] In the embodiment of the present invention, the pre-processed log data enters the feature extraction unit, which aggregates all related log entries together by identifying a global common identifier.

[0114] Taking user ID as an example, if the access is set to a Web service, the front-end logging data, application server logs, database server logs, payment gateway logs, etc. can be associated.

[0115] According to the time series, the event flow of the user ID in the specified time period can be obtained. Through the feature extraction unit, all log entries containing the user ID are merged together and arranged and summarized in chronological order.

[0116] Furthermore, the log entries are processed by event standardization, and event labels are set for each type of event. For example, abnormal events are abstracted into event items, such as NetworkIssue (network problem), AppSlow (slow application response), DBTimeout (database timeout), OSOverload (operating system overload). Then, according to the above event label classification, event labels are set for each log entry.

[0117] For abnormal events that have been identified in the log, the first unrelated alarm can be issued through the abnormal warning unit.

[0118] Furthermore, based on the collaborative analysis unit, a multi-source log collaborative analysis is performed on the abnormal event set corresponding to the abnormal event, and association rules are automatically generated. A directed graph is constructed based on the association rules. The abnormal propagation path is analyzed based on the directed graph, and the key path is identified. The abnormal warning unit issues a secondary correlation alarm.

[0119] Specifically, the abnormal warning unit can obtain the alarm contact information from the data storage unit, and notify the generated critical path information to the alarm contact through the configured alarm channel, which includes but is not limited to email, SMS, phone, WeChat, etc.

[0120] In the embodiments of the present invention, log data from different sources (such as server, client, middleware, database, etc.) are innovatively integrated, and structured integration of multi-source logs is achieved through unified log parsing rules and standardized format conversion, effectively eliminating format differences and semantic inconsistencies between multi-source logs, forming a unified log data pool, and significantly improving the comprehensiveness and accuracy of anomaly detection.

[0121] In the embodiment of the present invention, a cross-source log event correlation analysis method is established, and an innovative collaborative analysis method is established. By comprehensively analyzing multi-source log data, abnormal patterns that cannot be revealed by a single log source can be discovered, thereby improving the accuracy and sensitivity of anomaly detection.

[0122] In an embodiment of the present invention, a secondary associated alarm mechanism is proposed after the first unrelated alarm and the abnormal event propagation path are generated. The first unrelated alarm is to notify the alarm contact of the alarm event as soon as possible, emphasizing the timeliness of the abnormal alarm, while the secondary associated alarm provides the propagation path of the abnormal event to the alarm contact, understands the propagation path of the abnormal event from a global perspective, quickly finds the source of the abnormal event, and improves the efficiency of abnormal event processing.

[0123] The anomaly detection method provided by the embodiment of the present invention focuses on data integration and analysis at the log level, and does not specifically target the call link between services. The purpose of this association is to build a unified view across multiple systems for a specific transaction or session, so that operation and maintenance personnel or developers can view all log information related to a business operation in one interface or query.

[0124] In the anomaly detection method provided in the embodiment of the present invention, cross-source log association is not limited to the call link between services, but also includes all components that generate logs, such as the front-end, back-end, database and even external services (such as payment gateways), providing a more comprehensive business perspective. In addition to technical performance and failure issues, this method is also suitable for analyzing a wider range of scenarios such as user behavior paths and business process anomalies because it covers every touchpoint of the user journey. In the embodiment of the present invention, cross-source log association helps to build a more sophisticated business indicator and data analysis system. Based on the full-link log data, enterprises can better perform user behavior analysis, system performance evaluation and business process optimization.

[0125] The anomaly detection method provided by the embodiment of the present invention can help monitor the system status and quickly identify node failures or performance bottlenecks in a distributed system.

[0126] Based on the anomaly detection method provided by the embodiment of the present invention, potential security threats, such as intrusion attempts or abnormal traffic patterns, can be detected by analyzing the logs of network devices.

[0127] Based on the anomaly detection method provided by the embodiment of the present invention, the cloud service platform can improve the service quality of the cloud platform and improve the stability and security of the system by analyzing user behavior and system logs.

[0128] In summary, in the embodiments of the present invention, by integrating data from different sources, more comprehensive and accurate anomaly detection can be provided. In addition, the method can achieve real-time monitoring and rapid response, reducing the impact of system failures. Through automated anomaly detection, the workload of manual troubleshooting is reduced, thereby reducing the operating costs of the enterprise.

[0129] Based on the same inventive concept, an embodiment of the present invention further provides an abnormality detection device, the device comprising: A data collection unit, used to obtain log data from at least two log sources, wherein the log data carries a global common identifier; A feature extraction unit, configured to aggregate the log data based on the global common identifier to obtain a plurality of log data sets, each of which includes log data from a plurality of different sources but belonging to the same business process; A collaborative analysis unit is used to construct an abnormal event set based on a log data set corresponding to a business process where an abnormal event occurs, calculate the support of each single event in the abnormal event set, and construct an abnormal candidate set containing a combination of multiple events based on the support and support threshold of each single event; and generate abnormal link information based on the combination of multiple events in the abnormal candidate set; The abnormality warning unit is used to generate an abnormality detection message based on the abnormal link information.

[0130] Optionally, the collaborative analysis unit is used to: Determine the support degree of each single event based on the proportion of times each single event appears in the abnormal event set; Construct an abnormal candidate set containing multiple dual events based on single events whose support is greater than the support threshold; Determine the support degree of each double event based on the ratio of the number of times that two events in each double event appear consecutively in the abnormal event set; Based on the dual events whose support is greater than the support threshold, an abnormal candidate set containing multiple multi-events is constructed, and the support of multiple multi-events is calculated until there is only one multi-event whose support is greater than the support threshold, and an abnormal candidate set containing a combination of multiple events is obtained.

[0131] Optionally, the device further includes: a data preprocessing unit, configured to: Performing structured analysis on the log data from the at least two log sources, and converting the log data from different sources into a unified format; Filter meaningless log data; De-duplicate log data; Fill missing or invalid values ​​in log data.

[0132] Optionally, the device further comprises: An anomaly detection unit, used to detect abnormal events; The abnormal warning unit is also used to generate an abnormal warning message when an abnormal event is detected.

[0133] Optionally, the collaborative analysis unit is used to: Generate an event association rule based on a combination of multiple events in the abnormal candidate set, and obtain a directed graph according to the event association rule; Based on the directed graph, determining a key propagation path; Abnormal link information is generated based on the key propagation path.

[0134] Optionally, the collaborative analysis unit is used to: Create virtual nodes based on multiple events in the multi-event combination; Calculate the confidence level of the second event occurring when the first event occurs in each of the two events among multiple events; When the confidence level is greater than the confidence level threshold, the first event is associated with the second event, a directed edge is established between the virtual nodes corresponding to the first event and the second event, and the confidence level is used as a weight of the directed edge; Based on the directed graph, a key propagation path is determined, including: Path discovery is performed based on directed graphs. Taking the abnormal event as the starting node, the shortest path to other nodes is found to determine the propagation path of each abnormal event. The key propagation path is determined based on the weights of each directed edge included in each abnormal event propagation path.

[0135] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the anomaly detection method described in any of the above embodiments are implemented.

[0136] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the anomaly detection method described in any of the above embodiments are implemented.

[0137] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the steps of the anomaly detection method described in any of the above embodiments when executed by a processor.

[0138] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0139] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0140] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0144] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0145] The above is a detailed introduction to an anomaly detection method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in the field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An anomaly detection method, characterized in that: The method comprises: Obtaining log data from at least two log sources, wherein the log data carries a global common identifier; Based on the global common identifier, the log data is aggregated to obtain multiple log data sets, each log data set includes log data from multiple different sources but belonging to the same business process; Based on the log data set corresponding to the business process where the abnormal event occurs, an abnormal event set is constructed, the support of each single event in the abnormal event set is calculated, and based on the support and support threshold of each single event, an abnormal candidate set containing multiple event combinations is constructed; Generate abnormal link information based on a combination of multiple events in the abnormal candidate set; An abnormality detection message is generated based on the abnormal link information.

2. The anomaly detection method according to claim 1, characterized in that: Based on the log data set corresponding to the business process where the abnormality occurs, an abnormal event set is constructed, the support of each single event in the abnormal event set is calculated, and an abnormal candidate set of multiple event combinations is constructed based on the support and support threshold of each single event, including: Determine the support degree of each single event based on the proportion of times each single event appears in the abnormal event set; Construct an abnormal candidate set containing multiple dual events based on single events whose support is greater than the support threshold; Determine the support degree of each double event based on the ratio of the number of times that two events in each double event appear consecutively in the abnormal event set; Based on the dual events whose support is greater than the support threshold, an abnormal candidate set containing multiple multi-events is constructed, and the support of multiple multi-events is calculated until there is only one multi-event whose support is greater than the support threshold, and an abnormal candidate set containing a combination of multiple events is obtained.

3. The anomaly detection method according to claim 1, characterized in that: Before aggregating the log data, the method further includes: Performing structured analysis on the log data from the at least two log sources, and converting the log data from different sources into a unified format; Filter meaningless log data; De-duplicate log data; Fill missing or invalid values ​​in log data.

4. The anomaly detection method according to claim 1, characterized in that: The method further comprises: Detect abnormal events; In case an abnormal event is detected, an abnormal alarm message is generated.

5. The anomaly detection method according to any one of claims 1 to 4, characterized in that: Generate abnormal link information based on the combination of multiple events in the abnormal candidate set, including: Generate an event association rule based on a combination of multiple events in the abnormal candidate set, and obtain a directed graph according to the event association rule; Based on the directed graph, determining a key propagation path; Abnormal link information is generated based on the key propagation path.

6. The anomaly detection method according to claim 5, characterized in that: Generate an event association rule based on the combination of multiple events in the abnormal candidate set, and obtain a directed graph according to the event association rule, including: Create virtual nodes based on multiple events in the multi-event combination; Calculate the confidence level of the second event occurring when the first event occurs in each of the two events among multiple events; When the confidence level is greater than the confidence level threshold, the first event is associated with the second event, a directed edge is established between the virtual nodes corresponding to the first event and the second event, and the confidence level is used as a weight of the directed edge; Based on the directed graph, a key propagation path is determined, including: Path discovery is performed based on directed graphs. Taking the abnormal event as the starting node, the shortest path to other nodes is found to determine the propagation path of each abnormal event. The key propagation path is determined based on the weights of each directed edge included in each abnormal event propagation path.

7. An abnormality detection device, characterized in that: The device comprises: A data collection unit, used to obtain log data from at least two log sources, wherein the log data carries a global common identifier; A feature extraction unit, configured to aggregate the log data based on the global common identifier to obtain a plurality of log data sets, each of which includes log data from a plurality of different sources but belonging to the same business process; A collaborative analysis unit is used to construct an abnormal event set based on a log data set corresponding to a business process where an abnormal event occurs, calculate the support of each single event in the abnormal event set, and construct an abnormal candidate set containing a combination of multiple events based on the support and support threshold of each single event; and generate abnormal link information based on the combination of multiple events in the abnormal candidate set; The abnormality warning unit is used to generate an abnormality detection message based on the abnormal link information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the anomaly detection method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the anomaly detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the anomaly detection method described in any one of claims 1 to 6 are implemented.

Images