Intelligent log analysis and anomaly detection method and system for computing power host management platform

Abstract

The invention discloses an intelligent log analysis and anomaly detection method and system for a computing power host management platform, and relates to the technical field of log analysis, the method comprises the following steps: realizing multi-source data acquisition through Fluentd and EventExporter, and storing standardized log and cluster event information into Elasticsearch; the method comprises the following steps: analyzing an original log to generate a log template, vectorizing the log template, and clustering the vectorized log template by using a DBSCAN algorithm to obtain a log fingerprint database; template extraction and fingerprint matching are carried out on the real-time logs to obtain log fingerprint frequency, the log fingerprint frequency is input into a pre-trained Isolation Forest model, and the Isolation Forest model carries out prediction and recognizes sudden increase abnormity by comparing log template frequency in a log fingerprint database statistical time window; and on the basis of the constructed alarm causal map, using a Random Walk algorithm and a PageRank algorithm to carry out association analysis on anomalies, and recommending root causes. According to the method, the abnormal mode can be automatically identified from the log, and the potential problem is pre-warned.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis, and specifically to an intelligent log analysis and anomaly detection method and system for a computing power host management platform. Background Art

[0002] In the modern digital computing power support system, many computing power host management systems and platforms generally adopt Kubernetes as the core orchestration tool at the bottom layer to meet the growing computing needs. With its powerful container orchestration and cluster management capabilities, Kubernetes efficiently coordinates and manages a large number of computing power hosts and the terminal devices connected thereto, ensuring the stable operation of the entire computing system and the reasonable allocation of resources.

[0003] However, with the booming development of the business, the scale of the computing power cluster continues to expand. A small cluster that might originally consist of only dozens of nodes gradually evolves into a large cluster with hundreds, thousands, or even more nodes. In this process, the scale of log data also grows exponentially in an explosive manner. Among this vast amount of log data, there are multiple key types.

[0004] 1. Container logs: As log records closely related to containerized applications, container logs detail the running status of application programs within each container. From the application startup process, to every key operation during runtime, to possible error reports, container logs retain all of these. For example, when a container running a data analysis task encounters a memory overflow error while processing large-scale data, the container log will precisely record details such as the time when the error occurred, the specific code lines involved, and the resource usage status within the container at that time. These details are extremely crucial for developers to locate the root cause of faults within the application program.

[0005] 2. Audit logs: Audit logs play the role of a "supervisor" in the system, and they detail all access operations to system resources. Whether a user logs in to the system to initiate a computing task, or an administrator makes configuration adjustments to a computing power host, the audit log will record the initiator of the operation, the initiation time, the operation object, and the specific operation content. For example, if an administrator modifies the CPU resource allocation of a certain computing power host at a specific time, the audit log will clearly record this process for subsequent security audits and operation tracing to ensure the compliance and security of system operations.

[0006] 3. Application Logs: Application logs focus on the behavior of each specific application during operation. For a 3D rendering application running on a computing power host, the application logs will record information such as the submission time of the rendering task, the rendering progress, the rendering algorithm used, and any image quality issues or computing bottlenecks encountered during the rendering process. These logs provide direct data basis for optimizing application performance and enhancing user experience.

[0007] 4. System Logs: System logs reflect the operating status of the entire computing power host management system from a macroscopic level. It covers information such as the startup and shutdown of the operating system, the status changes of hardware devices (such as hard disk failure warnings, network interface abnormalities, etc.), and system-level resource usage statistics (such as the overall usage of CPU, memory, and disk I / O). For example, when the system detects a read / write error on the hard disk of a certain computing power host, the system log will record this event in a timely manner, reminding the operation and maintenance personnel to perform equipment maintenance to prevent data loss or system failures.

[0008] These rich and diverse log data are the key windows to insight into the system operating status, just like the "health barometer" of the system, which can help developers and operation and maintenance personnel keenly detect potential problems in the system and quickly take effective solutions. However, the massive log data also brings unprecedented challenges. Traditional manual analysis methods gradually expose many drawbacks in the face of such a large amount of data.

[0009] 1. High storage and retrieval costs. The massive log data requires a large amount of storage space. As time goes by, the cost of storing these log data continues to climb, not only requiring the purchase of more storage devices, but also involving a series of expenses such as equipment maintenance and power consumption. Moreover, when developers or operation and maintenance personnel need to retrieve specific information from this massive log, the traditional retrieval method is extremely inefficient. It may take hours or even days to search line by line in the huge log files, which undoubtedly greatly increases the labor cost and time cost.

[0010] 2. Key issues are easily submerged. In the information explosion of massive log data, the key information that is truly valuable for problem-solving is easily covered up by a large amount of irrelevant or secondary information. For example, when analyzing the cause of a system failure, a large number of normal operation log records may interfere with the identification of key error information, making it difficult for developers and operation and maintenance personnel to quickly and accurately locate the problem, resulting in a delay in problem-solving and affecting the normal operation of the system.

[0011] 3. Lack of real-time performance. In today's fast-paced digital business environment, a quick response to system problems is crucial. However, the traditional manual analysis method cannot meet this real-time requirement. From problem discovery to manual intervention for analysis and finally to finding a solution, this process often takes a long time. During this period, the system may have suffered greater losses, such as service interruption, data loss, etc. For example, when the system is under a malicious attack, due to the lack of real-time performance in manually analyzing logs, the attack behavior may not be detected in time and protective measures cannot be taken, resulting in a serious threat to system security.

[0012] 4. Low efficiency and easy omission in manual analysis. Manually analyzing log data not only consumes a large amount of time and energy, but also, due to the limited attention and energy of people, it is very easy to get tired and make omissions when facing a large amount of data for a long time. Some important error messages or abnormal information may be missed, thus unable to comprehensively and accurately analyze system problems, affecting the stability and reliability of the system. Summary of the Invention

[0013] In view of the requirements and deficiencies in the current technological development, the present invention provides an intelligent log analysis and anomaly detection method and system for a computing power host management platform, which can automatically identify abnormal patterns from logs, give early warnings of potential problems, improve the speed of problem discovery, and enhance the level of operation and maintenance automation and the efficiency of fault handling.

[0014] In the first aspect, for an intelligent log analysis and anomaly detection method of a computing power host management platform of the present invention, the technical solution adopted to solve the above technical problems is as follows:

[0015] An intelligent log analysis and anomaly detection method for a computing power host management platform, which includes the following steps:

[0016] S1. Implement multi-source data collection through Fluentd and EventExporter, and store the standardized logs and cluster event information in Elasticsearch;

[0017] S2. Parse the original logs to generate log templates, vectorize the log templates, and use the DBSCAN algorithm to cluster the vectorized log templates to obtain a log fingerprint library;

[0018] S3. Extract templates and match fingerprints from real-time logs to obtain the log fingerprint frequency, and input it into a pre-trained Isolation Forest model. The Isolation Forest model predicts and identifies sudden increases in anomalies by comparing the log template frequencies within the statistical time window of the log fingerprint library.

[0019] S4. Based on the constructed alarm causal graph, use the Random Walk algorithm and the PageRank algorithm to perform correlation analysis on the anomalies and recommend the root causes.

[0020] Optionally, execute step S2, use the Drain3 algorithm to extract fixed patterns from the original logs and generate log templates.

[0021] Use the TF-IDF technology to convert the log templates into numerical vectors and store them in the feature library for subsequent clustering analysis.

[0022] Use the DBSCAN clustering algorithm to cluster the log templates in the feature library to obtain a log fingerprint library.

[0023] Optionally, before executing step S3, pre-train the Isolation Forest model using historical frequency data to learn the normal frequency distribution.

[0024] When executing step S3, first perform template extraction and fingerprint matching on the real-time logs to obtain the frequencies of the log fingerprints. Subsequently, count the frequency data of the real-time logs and input them into the trained Isolation Forest model. The Isolation Forest model predicts by comparing the frequencies of the log templates appearing within the time window in the log fingerprint library and identifies sudden abnormal situations.

[0025] Optionally, execute step S4. Based on the constructed alarm causal graph, use the Random Walk algorithm to generate event sequences, capture the associations between events, use the PageRank algorithm to calculate the importance of events, identify key events, and finally combine the event sequences and importance scores to recommend the root cause events leading to the anomalies and output the impact paths.

[0026] In a second aspect, for an intelligent log analysis and anomaly detection system of a computing power host management platform according to the present invention, the technical solution adopted to solve the above technical problems is as follows:

[0027] An intelligent log analysis and anomaly detection system of a computing power host management platform, which includes:

[0028] A data collection and storage module, which is used to achieve multi-source data collection through Fluentd and EventExporter and store the standardized logs and cluster event information in Elasticsearch.

[0029] A log processing module, which is used to parse the original logs to generate log templates, vectorize the log templates, and use the DBSCAN algorithm to cluster the vectorized log templates to obtain a log fingerprint library.

[0030] A log prediction module is used to perform template extraction and fingerprint matching on real-time logs to obtain the log fingerprint frequency, and input it into a pre-trained Isolation Forest model. The Isolation Forest model predicts and identifies sudden increases in anomalies by comparing the log template frequencies within a statistical time window in the log fingerprint library;

[0031] An anomaly correlation module is used to perform correlation analysis on anomalies and recommend root causes based on the constructed alarm causal graph, using the Random Walk algorithm and the PageRank algorithm.

[0032] Optionally, the involved log processing module first uses the Drain3 algorithm to extract fixed patterns from the original logs to generate log templates, then uses the TF-IDF technique to convert the log templates into numerical vectors and store them in the feature library, and finally uses the DBSCAN clustering algorithm to cluster the log templates in the feature library to obtain the log fingerprint library.

[0033] Optionally, the involved log prediction module specifically includes:

[0034] A model pre-training unit is used to pre-train the Isolation Forest model using historical frequency data to learn the normal frequency distribution;

[0035] A log processing unit is used to perform template extraction and fingerprint matching on real-time logs to obtain the frequency of log fingerprint occurrences;

[0036] The Isolation Forest model is used to predict and identify sudden increases in anomalies by comparing the log fingerprint library and counting the frequencies of log templates within a time window.

[0037] Optionally, based on the constructed alarm causal graph, the involved anomaly correlation module first uses the Random Walk algorithm to generate event sequences to capture the correlations between events, then uses the PageRank algorithm to calculate the importance of events to identify key events, and finally combines the event sequences and importance scores to recommend the root cause events leading to anomalies and output the impact paths.

[0038] The beneficial effects of an intelligent log analysis and anomaly detection method and system for a computing power host management platform according to the present invention compared with the prior art are:

[0039] 1. The present invention can automatically identify abnormal patterns from logs and give early warnings of potential problems, improving the speed of problem discovery, and enhancing the level of operation and maintenance automation and the efficiency of fault handling;

[0040] 2. By processing the original logs, on the one hand, the present invention templates and clusters the logs, greatly reducing the amount of data processed in subsequent steps, and combines machine learning and graph-related algorithms to achieve automated analysis and root cause recommendation; on the other hand, real-time processing and detection enable rapid early warning and response to system anomalies, realizing timely, accurate, and automated log-based monitoring and early warning and fault diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG. Figure 1 is a flowchart of the method according to the first embodiment of the present invention;

[0042] FIG. Figure 2 is a block diagram of module connections according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to make the technical solutions, the technical problems to be solved, and the technical effects of the present invention clearer and more understandable, the following describes the technical solutions of the present invention clearly and completely in conjunction with specific embodiments.

[0044] Embodiment 1:

[0045] Combined with FIG. Figure 1 , this embodiment proposes an intelligent log analysis and anomaly detection method for a computing power host management platform, which includes the following steps:

[0046] S1. Implement multi-source data collection through Fluentd and EventExporter, and store the standardized logs and cluster event information in Elasticsearch.

[0047] As a very powerful and flexible data collector, Fluentd has extensive data access capabilities. It can connect to numerous different types of data sources. Whether it is the running logs generated inside the application, the system logs generated by the server operating system, or the status logs during the transmission of network devices, Fluentd can easily connect and obtain this data. Fluentd collects the original data through a variety of input plugins, adapting to the interface specifications and data formats of different data sources. For example, for applications running based on containerization technology, Fluentd can directly obtain log data from the container running environment through specific container log collection plugins, and can handle log collection tasks under multiple container orchestration platforms such as Docker and Kubernetes.

[0048] EventExporter focuses on the collection of cluster event information. In a cluster environment, various events are constantly occurring, such as the addition of new nodes, the failure of existing nodes, and the allocation and scheduling of resources. EventExporter can keenly capture these cluster events and convert them into data forms that can be analyzed and stored. It interacts closely with the cluster management system, obtaining real-time cluster status change information by calling relevant API interfaces or listening to specific event channels. Taking the Kubernetes cluster as an example, EventExporter can establish a connection with the Kubernetes API Server and listen in real time for events such as the creation, update, and deletion of various resource objects (such as Pods, Services, Deployments, etc.) in the cluster, collecting in full the detailed information of these events, including the time when the event occurred, the resource objects involved, the type of the event, and the specific description.

[0049] It should be noted that after Fluentd completes the collection of various types of log data and EventExporter completes the collection of cluster event information, these raw data cannot be directly stored and need to be standardized first. Because the data formats and structures generated by different data sources often vary, in order to facilitate subsequent storage, query, and analysis, they must be converted into a unified standard format. Fluentd and EventExporter play important roles in this process. They use their respective filter plugins to clean, transform, and standardize the collected data. For example, for the timestamp field in the log data, it is uniformly formatted into a specific time standard, and the key information in the log content is extracted and structured, so that all data follows a consistent specification in terms of format and content organization.

[0050] The data after standardization is finally stored in Elasticsearch. Elasticsearch is a distributed and highly scalable search engine and data storage tool, which is very suitable for storing and managing massive amounts of log and event data. Fluentd and EventExporter send the standardized log and cluster event information to the Elasticsearch cluster through corresponding output plugins. Elasticsearch will build indexes for these data so that subsequent data retrieval and query can be carried out quickly. During the storage process, Elasticsearch uses its distributed storage architecture to disperse the data across multiple nodes, which not only improves the reliability of storage but also enables efficient data read and write operations. At the same time, the powerful query language and data analysis functions of Elasticsearch enable developers, operation and maintenance personnel, and other relevant personnel to conveniently extract valuable information from the stored data, providing strong support for system troubleshooting, performance optimization, security auditing, and other tasks.

[0051] S2. First, use the Drain3 algorithm to extract fixed patterns from the original logs to generate log templates. The Drain3 algorithm can analyze the original logs in detail. Its core principle is to identify the fixed patterns in the logs by understanding the structure and semantics of the log text. For example, in the access logs of a server, although the specific time, IP address, and other information in each access record may vary, the basic format of the log and the key operation description part often have certain fixed patterns. The Drain3 algorithm will scan each original log, and through the set rules and pattern matching mechanism, identify the variable parts (such as specific timestamps, dynamically generated IDs, etc.) as variables, and extract the fixed parts (such as the module name to which the log belongs, the keywords of the operation type, etc.), thus generating log templates. These log templates are a high-level summary and abstraction of the original logs, greatly simplifying the structure of the log data and providing a more regular basis for subsequent processing.

[0052] Subsequently, the TF-IDF (Term Frequency-Inverse Document Frequency) technique is used to convert the log templates into numerical vectors and store them in the feature library for subsequent clustering analysis. The TF-IDF technique quantifies and evaluates the words in each log template. Among them, the term frequency (TF) measures how frequently a certain word appears in the current log template. The more times it appears, the higher the term frequency. The inverse document frequency (IDF) reflects the rarity of a certain word in the entire set of log templates. The fewer log templates a word appears in, the higher its inverse document frequency. By multiplying the term frequency by the inverse document frequency, the TF-IDF value of each word in the log template is obtained. In this way, all the words in a log template are assigned corresponding TF-IDF values, and these values form a numerical vector, thus converting the log template from text form to numerical form. After the conversion is completed, these numerical vectors are stored in the feature library. The feature library is like a warehouse specifically for storing the feature information of log templates, providing rich data resources for subsequent clustering analysis and facilitating ready access and processing at any time.

[0053] Finally, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm is used to cluster the log templates in the feature library to obtain the log fingerprint library. The DBSCAN algorithm clusters based on the density of data points. It divides data points that are density-connected in space into the same class. In the space of the log template numerical vectors, the DBSCAN algorithm traverses each vector in the feature library and determines the density relationship between vectors according to pre-set density parameters (such as neighborhood radius and minimum number of points). If the number of a group of vectors within a certain neighborhood range meets the requirement of the minimum number of points, then these vectors are considered to be in the same density region and are thus grouped into the same cluster. Each cluster represents a group of log templates with similar characteristics, and these clustering results form the log fingerprint library. Each cluster in the log fingerprint library can be regarded as a unique "fingerprint", which represents a class of logs with similar patterns and characteristics. In this way, the massive and complex original logs are effectively classified and summarized, enabling the rapid identification and processing of log data with the same pattern in subsequent log analysis work, improving the analysis efficiency and accuracy.

[0054] S3. Pre-train an Isolation Forest model using historical frequency data to learn the normal frequency distribution. This process specifically includes: ① Data collection: Collect a large amount of historical log data, which should cover log information in different time periods and different business scenarios to ensure the comprehensiveness and representativeness of the data. ② Frequency calculation: Process the collected historical logs, extract log fingerprints through preset algorithms and rules, and count their occurrence frequencies. ③ Model training: Use the calculated historical frequency data as input and provide it to the Isolation Forest model for training; during the training process, the Isolation Forest model will construct multiple isolation trees. Each isolation tree will randomly select a feature and a split point to divide the data into different subsets, and this process will be repeated continuously until each subset contains only one data point or reaches the preset stop condition; through learning a large amount of historical data, IsolationForest can understand the distribution characteristics of log fingerprint frequencies under normal circumstances, such as the mean, variance, and fluctuation range of frequencies, thereby establishing a model of normal frequency distribution.

[0055] After the Isolation Forest model is trained, template extraction and fingerprint matching are performed on real-time logs to obtain the frequency of log fingerprints; the frequency data of real-time logs are statistically analyzed and input into the trained Isolation Forest model; the Isolation Forest model predicts and identifies sudden abnormal situations by comparing the log fingerprint library and counting the frequency of log templates within a time window. This process specifically includes: ① Template extraction and fingerprint matching: Analyze the real-time generated logs, use preset algorithms (such as regular expression matching, natural language processing techniques, etc.) to extract fixed patterns from the logs to generate log templates, and then match these log templates with a pre-defined log fingerprint library to assign corresponding log fingerprints to each log template; in this way, the frequency of each log fingerprint in real-time logs can be counted. ② Data input: Input the statistically obtained real-time log frequency data into the already trained Isolation Forest model. These real-time frequency data represent the actual situation of the current logs, and the trained Isolation Forest model will evaluate these data based on the normal frequency distribution pattern learned previously. ③ Abnormal prediction and identification: After receiving the real-time frequency data, the Isolation Forest model will compare it with the log fingerprint library and count the frequency of log templates within the set time window; the Isolation Forest model will judge the abnormality degree of each data point according to the path length of the data point in the isolation tree. The shorter the path, the easier it is for the data point to be isolated, and the more likely it is to be an abnormal point; through the analysis and judgment of the real-time frequency data, the Isolation Forest model can predict whether there are sudden abnormal situations. Once it is found that the frequency of a certain log fingerprint has increased significantly within a short period of time, exceeding the normal frequency range learned by the Isolation Forest model, it will be identified as an abnormal situation and an alarm will be issued in a timely manner for relevant personnel to conduct further investigation and processing.

[0056] Through the above steps, first let the Isolation Forest model learn the normal log frequency distribution, and then apply it to the monitoring of real-time logs, which can effectively identify sudden abnormal situations in the logs and provide strong support for the stable operation and security guarantee of the system.

[0057] S4. Based on the constructed alarm causal graph, use the Random Walk algorithm to generate event sequences, capture the associations between events, use the PageRank algorithm to calculate the importance of events, identify key events, and finally combine the event sequences and importance scores to recommend the root cause events leading to abnormalities and output the impact paths.

[0058] In this step, constructing an alarm causal graph is a complex but crucial step, which requires collecting and integrating various alarm data, event information generated during the operation of the system, and the potential causal relationships between them. Through in-depth mining of historical data, in-depth understanding of the system architecture and business logic, and the application of data mining techniques (such as association rule mining), the causal associations between different events and alarms are determined. For example, too high CPU usage of the server may lead to slower application response, which in turn triggers an alarm for user request timeout, and these causal relationships will be clearly depicted in the alarm causal graph. The nodes in the graph represent various events or alarms, while the edges represent the causal connections between them, thus forming a network structure that intuitively reflects the alarm logic relationship of the system.

[0059] In this step, after constructing the alarm causal graph, first use the Random Walk algorithm to generate event sequences. The core idea of the Random Walk algorithm is to perform random walks in this network structure of the graph. Specifically, starting from an initial node in the graph (which can be any alarm or event node), randomly select an edge connected to the current node with a certain probability and move along this edge to the next node, and continuously repeat this process to generate an event sequence. In each movement process, the probability of selecting different edges is not completely random, but is related to factors such as the tightness of the causal relationship represented by the edge and the historical occurrence frequency. For example, if the two events connected by a certain edge frequently show a causal association in historical data, then the probability of selecting this edge during the random walk process will be relatively high. By executing the Random Walk algorithm multiple times, a large number of different event sequences can be generated. These event sequences reflect the various possible association paths between events in the alarm causal graph and help to comprehensively capture the intricate connections between events.

[0060] The PageRank algorithm is then used to evaluate the importance of each event node in the alarm causal graph. In an alarm causal graph, the importance of an event node depends not only on the number of nodes pointing to it (i.e., in-degree), but more importantly, on the importance of these nodes themselves. Specifically, the PageRank algorithm assigns an initial importance score to each node based on the graph structure. Then, through continuous iterative calculations, each node transfers its importance score to its connected nodes in a certain proportion. Nodes that are pointed to by more important nodes and have higher scores transferred by their nodes will have higher final PageRank scores, indicating that the event is more important within the entire system. For example, in a multi-level alarm causal graph, an event node at the core of a critical business process will receive a relatively high PageRank score, indicating its high importance within the system, as it is pointed to by numerous related upstream and downstream event nodes, which themselves are also highly important. The PageRank algorithm's calculations clearly identify which events are critical within the entire alarm causal network.

[0061] Finally, after generating event sequences and calculating their importance scores, the event sequences and importance scores are combined to recommend root cause events and output impact paths. Each generated event sequence is evaluated based on the importance scores of each event within the sequence. Generally, events that frequently appear in multiple event sequences and have high importance scores are more likely to be the root cause of the anomaly. Because these events are at key points in numerous causal paths, they have a significant impact on the overall system operation. Once a potential root cause event is identified, the causal edges from the root cause event in the alarm causal graph can be traced back to identify the paths that affect other events. For example, if a server hardware failure is determined to be the root cause of a system anomaly, the alarm causal graph clearly illustrates how this hardware failure triggers a chain reaction, such as causing a service process to crash, affecting the normal operation of related application modules, and ultimately generating alarms on multiple user terminals. Visualizing these causal relationships forms the impact path. By outputting the impact path, operations and maintenance personnel can intuitively understand how the root cause event gradually spreads in the system and triggers a series of abnormal alarms, thereby providing strong support for quickly locating and solving problems, greatly improving the efficiency of system troubleshooting and repair.

[0062] Example 2:

[0063] Combined with attachment Figure 2, this embodiment proposes an intelligent log analysis and anomaly detection system for a computing power host management platform, which includes:

[0064] A data collection and storage module, which is used to realize multi-source data collection through Fluentd and EventExporter, and store standardized logs and cluster event information in Elasticsearch;

[0065] A log processing module, which is used to parse the original logs to generate log templates, vectorize the log templates, and use the DBSCAN algorithm to cluster the vectorized log templates to obtain a log fingerprint library;

[0066] A log prediction module, which is used to extract templates and match fingerprints for real-time logs to obtain the frequency of log fingerprints, and input it into a pre-trained Isolation Forest model. The Isolation Forest model predicts and identifies sudden increases in anomalies by comparing the frequency of log templates within a statistical time window in the log fingerprint library;

[0067] An anomaly correlation module, which is used to perform correlation analysis on anomalies and recommend root causes based on the constructed alarm causal graph using the Random Walk algorithm and the PageRank algorithm.

[0068] In this embodiment, the involved log processing module first uses the Drain3 algorithm to extract fixed patterns from the original logs to generate log templates, then uses the TF-IDF technology to convert the log templates into numerical vectors and store them in the feature library, and finally uses the DBSCAN clustering algorithm to cluster the log templates in the feature library to obtain a log fingerprint library.

[0069] In this embodiment, the involved log prediction module specifically includes:

[0070] A model pre-training unit, which is used to pre-train the Isolation Forest model using historical frequency data to learn the normal frequency distribution;

[0071] A log processing unit, which is used to extract templates and match fingerprints for real-time logs to obtain the frequency of the appearance of log fingerprints;

[0072] The Isolation Forest model, which is used to predict and identify sudden increases in anomalies by comparing the log fingerprint library and counting the frequency of log templates within a time window.

[0073] In this embodiment, the abnormal association module is based on the constructed alarm causal graph. First, the RandomWalk algorithm is used to generate an event sequence to capture the associations between events. Subsequently, the PageRank algorithm is used to calculate the importance of events to identify key events. Finally, combining the event sequence and the importance scores, the root cause events leading to the abnormality are recommended, and the impact path is output.

[0074] In summary, by using the intelligent log analysis and anomaly detection method and system for a computing power host management platform of the present invention, the efficiency and accuracy of log analysis are improved through automated means, so as to quickly discover key problems and locate the root cause, ensuring stable operation and efficient operation and maintenance.

[0075] The above specific application examples have elaborated in detail the principle and implementation manner of the present invention. These embodiments are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by those skilled in the art of this technical field without departing from the principle of the present invention shall fall within the patent protection scope of the present invention.

Claims

1. An intelligent log analysis and anomaly detection method for a computing power host management platform, characterized in that , including the following steps: S1. Implement multi-source data collection through Fluentd and EventExporter, and store the standardized logs and cluster event information in Elasticsearch; S2. Parse the original logs to generate log templates, vectorize the log templates, and use the DBSCAN algorithm to cluster the vectorized log templates to obtain a log fingerprint library; S3. Extract the template and match the fingerprint for real-time logs to obtain the log fingerprint frequency, and input it into the pre-trained Isolation Forest model. The Isolation Forest model predicts and identifies sudden increase anomalies by comparing the log template frequencies within the time window in the log fingerprint library; S4. Based on the constructed alarm causal graph, use the Random Walk algorithm and PageRank algorithm to perform correlation analysis on the anomalies and recommend the root cause.

2. The intelligent log analysis and anomaly detection method of a computing power host management platform according to claim 1, characterized in that, Execute step S2, and use the Drain3 algorithm to extract fixed patterns from the original logs to generate log templates; Use the TF-IDF technique to convert the log templates into numerical vectors and store them in the feature library for subsequent clustering analysis; Use the DBSCAN clustering algorithm to cluster the log templates in the feature library to obtain a log fingerprint library.

3. The intelligent log analysis and anomaly detection method of a computing power host management platform according to claim 1, characterized in that, Before executing step S3, pre-train the Isolation Forest model using historical frequency data to learn the normal frequency distribution; When executing step S3, first extract the template and match the fingerprint for real-time logs to obtain the frequency of the log fingerprint appearance. Subsequently, count the frequency data of the real-time logs and input it into the trained Isolation Forest model. The Isolation Forest model predicts and identifies sudden increase anomalies by comparing the log template frequencies within the time window in the log fingerprint library.

4. The intelligent log analysis and anomaly detection method of a computing power host management platform according to claim 1, characterized in that, Execute step S4. Based on the constructed alarm causal graph, use the Random Walk algorithm to generate an event sequence to capture the associations between events, use the PageRank algorithm to calculate the importance of events, identify key events, and finally combine the event sequence and importance scores to recommend the root cause event leading to the anomaly and output the impact path.

5. An intelligent log analysis and anomaly detection system for a computing power host management platform, characterized in that, It includes: A data collection and storage module for implementing multi-source data collection through Fluentd and EventExporter and storing the standardized logs and cluster event information in Elasticsearch; A log processing module for parsing the original logs to generate log templates, vectorizing the log templates, and using the DBSCAN algorithm to cluster the vectorized log templates to obtain a log fingerprint library; A log prediction module for extracting the template and matching the fingerprint for real-time logs to obtain the log fingerprint frequency, and inputting it into the pre-trained Isolation Forest model. The Isolation Forest model predicts and identifies sudden increase anomalies by comparing the log template frequencies within the time window in the log fingerprint library; Anomaly correlation module, which is used to perform anomaly correlation analysis and recommend root causes based on the constructed alarm causal graph, using the Random Walk algorithm and the PageRank algorithm.

6. The intelligent log analysis and anomaly detection system of a computing power host management platform according to claim 5, characterized in that, The log processing module first uses the Drain3 algorithm to extract fixed patterns from the original logs to generate log templates, then uses the TF-IDF technology to convert the log templates into numerical vectors and stores them in the feature library, and finally uses the DBSCAN clustering algorithm to cluster the log templates in the feature library to obtain a log fingerprint library.

7. The intelligent log analysis and anomaly detection system of a computing power host management platform according to claim 5, characterized in that, The log prediction module specifically includes: Model pre-training unit, which is used to pre-train the Isolation Forest model with historical frequency data to learn the normal frequency distribution; Log processing unit, which is used to perform template extraction and fingerprint matching on real-time logs to obtain the frequency of the appearance of log fingerprints; Isolation Forest model, which is used to predict by comparing the log fingerprint library, counting the frequency of the appearance of log templates within a time window, and identifying sudden abnormal situations.

8. The intelligent log analysis and anomaly detection system of a computing power host management platform according to claim 5, characterized in that, Based on the constructed alarm causal graph, the anomaly correlation module first uses the Random Walk algorithm to generate an event sequence to capture the association between events, then uses the PageRank algorithm to calculate the importance of events to identify key events, and finally combines the event sequence and the importance score to recommend the root cause event that leads to the anomaly and output the impact path.

Images