Log analysis method, system, device and storage medium

Abstract

The present application relates to the field of log data processing and analysis, and discloses a log analysis method, system, device and storage medium. The method includes: collecting log data from multiple source devices, cleaning and structuring the log data; generating a dynamic topology network based on the interaction relationship between devices in the log data, in which nodes represent devices and edges represent device interactions; extracting topology characteristics based on the dynamic topology network, including the connectivity of the topology structure, the ring structure and the complexity characteristics of the network; analyzing the changes in network topology characteristics within different time scales, and identifying abnormal patterns in log data; constructing a dynamic causal network between devices, extracting the abnormal propagation chain, and locating the root cause device or event that caused the abnormality. The present invention can adapt to multiple log formats and multi-source data, support real-time analysis and anomaly detection, improve the intelligence and efficiency of system log analysis, and provide strong technical support for system troubleshooting and operation and maintenance management.

Description

Technical Field

[0001] The present invention relates to the technical field of log data processing and analysis, and in particular to a log analysis method, system, device and storage medium. Background Art

[0002] With the rapid development of information technology, large-scale distributed systems and complex network environments have become an integral part of modern data centers and enterprise IT architectures. Within these environments, the massive amounts of log data generated by system operations have become a critical source of data for monitoring, maintenance, and troubleshooting. However, existing log analysis methods and tools face numerous challenges in processing this complex log data.

[0003] Existing technologies typically rely on simple keyword searches and time series analysis to process log data. These methods can meet certain basic needs, but they struggle to adapt to the real-time nature and complexity of large-scale log data. On the one hand, traditional methods often lack the ability to model the complex interactions between devices and are unable to fully characterize the dynamic behavior of the system. On the other hand, when faced with complex anomaly scenarios caused by the collaboration or linkage of multiple devices in a distributed system, existing technologies have limited correlation analysis capabilities and struggle to identify potential anomaly propagation paths. In addition, existing log analysis tools have a low degree of automation, and most rely on manual analysis, resulting in inefficient troubleshooting and problem location, which seriously affects system stability and operational efficiency.

[0004] In practical applications, due to the heterogeneity of log data sources and the diversity of formats, existing technologies lack unified pre-processing capabilities during the data processing phase, making it impossible to effectively integrate log data from different devices. Furthermore, the anomaly detection methods used in current log analysis systems are often limited to identifying anomalies at a single scale, failing to comprehensively analyze both short-term sudden anomalies and long-term trending anomalies, resulting in the omission of certain anomaly patterns. Furthermore, existing systems generally lack the ability to analyze anomaly propagation paths and root cause devices. Operations and maintenance personnel can only rely on empirical inferences when troubleshooting, further increasing the difficulty and cost of system maintenance.

[0005] Therefore, existing technologies have significant deficiencies in real-time, accuracy, intelligence, and comprehensiveness. There is an urgent need for a log analysis method that can efficiently process log data, comprehensively identify abnormal patterns, and accurately locate the root causes of abnormalities. Summary of the Invention

[0006] To address the shortcomings of the existing technology, the present invention provides a log analysis method, system, device and storage medium, which efficiently construct a dynamic topology network from multi-source log data. Through topological feature extraction, multi-scale anomaly detection and causal analysis, it can accurately identify system anomaly patterns and locate the root cause of the anomaly, such as the device or event.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a log analysis method, comprising the following steps:

[0008] Collect log data from multiple source devices, clean and structure the log data, and extract timestamps, event types, device identifiers, and description information;

[0009] Based on the interaction relationship between devices in log data, a dynamic topology network is generated, where nodes represent devices and edges represent device interactions. The network evolves dynamically as the time window changes.

[0010] Extract topological characteristics based on dynamic topological networks, including connectivity, ring structure, and network complexity.

[0011] Analyze changes in network topology characteristics over different time scales, and combine local and global anomaly detection to identify abnormal patterns in log data;

[0012] Build a dynamic causal network between devices, extract the anomaly propagation chain, and locate the root cause device or event that causes the anomaly.

[0013] Preferably, the steps of constructing the dynamic topology network specifically include:

[0014] Map the source device of each log data to a node in the dynamic network;

[0015] Map the interaction relationships between devices in the log data into network edges, where the weight of the edge is determined by the number of associated events or interaction intensity in the log;

[0016] The log data is segmented based on the set time window to generate a dynamic topology network sequence that evolves over time.

[0017] Preferably, the step of extracting the topological characteristics specifically includes:

[0018] Assign weights to the edges of a dynamic topology network and construct a filtering sequence to generate a subgraph of the network;

[0019] Perform persistent homology analysis on network subgraphs to extract network topological characteristics, including connectivity, ring structure, and holes;

[0020] Calculate topological invariants based on topological properties, including Betti numbers and network topological complexity indicators.

[0021] Preferably, the step of identifying abnormal patterns in log data specifically includes:

[0022] Calculate the topological characteristic vector in each time window, and calculate the local anomaly difference based on the topological characteristic changes between time windows;

[0023] Calculate the global anomaly difference based on the changing trend of the overall topological characteristic vector in the dynamic network;

[0024] The differences between local anomalies and global anomalies are comprehensively considered, and the abnormal patterns in the log data are determined by setting thresholds.

[0025] Preferably, the step of locating the root cause device or event causing the anomaly specifically includes:

[0026] Based on the interaction relationship and dynamic topology network between devices in the log data, a dynamic causal network between devices is constructed;

[0027] Extract the anomaly propagation chain from the causal network and locate the anomaly propagation path through the causal relationship between devices;

[0028] Combine the anomaly scores of devices in the propagation chain to determine the root cause device or event that caused the anomaly.

[0029] Preferably, the root cause device or event is determined based on the following conditions:

[0030] In the anomaly propagation chain, the device or event that first experiences an anomaly and has the highest correlation is the root cause;

[0031] The inter-device associations in the dynamic causal network are estimated by conditional probability, and the root cause priorities of the devices are determined by maximum likelihood estimation.

[0032] Preferably, the method further comprises:

[0033] Generate visual displays based on log analysis results, including system health dashboards, anomaly trend charts, and root cause analysis reports;

[0034] The content of the visual display includes real-time updated network topology characteristic changes, anomaly detection results and device status statistics.

[0035] The present invention also provides a log analysis system, comprising:

[0036] Data collection module, used to collect log data from multiple devices and pre-process the log data;

[0037] Dynamic network building module, used to generate dynamic topological networks that evolve over time;

[0038] Topology feature extraction module, used to extract the topology features of dynamic networks;

[0039] Anomaly detection module, used to detect local and global anomalies in log data at different time scales;

[0040] Root cause analysis module, used to locate the root cause device or event of the anomaly based on the anomaly propagation path;

[0041] The visualization module is used to display log analysis results, including network topology characteristics, abnormal trends, and root cause analysis information.

[0042] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0043] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0044] The present invention provides a log analysis method, system, device, and storage medium. It has the following beneficial effects:

[0045] 1. The present invention can effectively process log data from multiple devices and in multiple formats. Through structured preprocessing, it converts different types of log data into standardized input, adapts to the heterogeneous data environment in complex systems, and improves the versatility and applicability of log analysis.

[0046] 2. By constructing a dynamic topology network, the present invention can clearly describe the interactive relationships between devices and their dynamic changes over time, comprehensively characterize the status and correlation characteristics of system operation, and provide structured basic data support for anomaly detection and analysis.

[0047] 3. The present invention combines the methods of local anomaly detection and global anomaly detection, which can capture different patterns of short-term sudden anomalies and long-term trend anomalies, provide comprehensive anomaly recognition capabilities, and effectively improve the coverage and detection accuracy of system anomalies.

[0048] 4. Through the construction of a dynamic causal network and the extraction of propagation chains, the present invention can reveal the propagation mechanism of abnormal events between devices, accurately locate the root cause device or event that causes the abnormality, provide efficient support for system troubleshooting and problem location, and reduce the time and cost of fault handling.

[0049] 5. This invention supports the visualization of analysis results, including topology, abnormal trends, and root cause devices, providing operation and maintenance personnel with an intuitive and easy-to-understand overview of system health status and a basis for problem location, thereby improving the efficiency and accuracy of operation and maintenance work. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the method flow of the present invention;

[0051] Figure 2 Schematic diagram of the system structure of the present invention;

[0052] Figure 3Schematic diagram of the computer device structure of the present invention.

[0053] Among them, 100, data acquisition module; 200, dynamic network construction module; 300, topology feature extraction module; 400, anomaly detection module; 500, root cause analysis module; 600, visualization module; 40, computer equipment; 41, processor; 42, memory; 43, storage medium. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Please see the attached Figure 1 The present invention provides a log analysis method that, by introducing dynamic topology networks and high-order topology analysis techniques, combines multi-scale anomaly detection with causal network analysis. This method addresses the shortcomings of traditional log analysis methods in terms of real-time performance, relevance, and anomaly detection, enabling accurate identification of abnormal patterns in log data and root cause location. The specific implementation steps of the present invention are described in detail below.

[0056] like Figure 1 As shown, the log analysis method may include the following steps:

[0057] S1. Collect log data from multiple source devices and perform preprocessing.

[0058] S2, build a dynamic topology network based on log data;

[0059] S3, extracting topological features based on dynamic topological network;

[0060] S4, analyze changes in network topology characteristics and identify abnormal patterns in log data;

[0061] S5. Build a dynamic causal network between devices to locate the source of anomalies.

[0062] Regarding step S1, in this embodiment, step S1 primarily involves the collection, cleaning, and structuring of log data. The present invention provides basic data support for subsequent analysis steps by standardizing log data from multiple sources. The specific implementation of this step is described in detail below.

[0063] In this embodiment, log data primarily originates from a variety of devices, including but not limited to servers, switches, routers, application logs, virtualization platform logs, and container logs. Log data may be structured, semi-structured, or unstructured. Common formats include plain text logs, JSON files, XML files, and system-specific binary log formats.

[0064] Alternatively, log data can be collected using distributed log collection tools. For example, tools like Fluentd and Logstash can collect log data from multiple sources and format it uniformly.

[0065] It should be noted that during the log data collection process, you can set the real-time collection mode or batch collection mode according to the log generation speed and storage capacity. For example:

[0066] The real-time collection mode is suitable for scenarios where high-frequency logs are generated, and log data is directly passed to subsequent modules through streaming processing.

[0067] The batch collection mode is suitable for logs of non-critical systems, which are collected regularly and uploaded for unified processing.

[0068] In one possible implementation, log data cleaning involves removing duplicates, removing noise, and filtering out invalid information from the original log data. For example, for log entries containing duplicate timestamps and content, redundant records can be identified and deleted using a hash algorithm or direct string comparison.

[0069] Specifically, log data cleaning may include the following:

[0070] Invalid log filtering: For example, removing log records that contain empty values or meaningless fields.

[0071] Log time alignment: For device logs with clock deviation, you can use a time synchronization algorithm (such as timestamp calibration based on the NTP protocol) to align timestamps.

[0072] Abnormal character processing: Some logs may contain garbled or special characters. Regular expressions can be used to clean or replace these abnormal characters.

[0073] In some embodiments, the description field of the log can be cleaned using natural language processing (NLP) technology. Specifically, stop words (such as "is," "a," and "the") can be removed from the log description and capitalization can be standardized to facilitate subsequent structured extraction of the description information.

[0074] For example, the structured processing of log data can be parsed based on the following fields:

[0075] Timestamp( ) : Parses the time information in log records and converts it to a unified time format, such as the ISO 8601 standard (YYYY-MM-DDTHH:MM:SSZ). Note that if a log lacks a timestamp, it can be supplemented with the file creation time or other related information.

[0076] Event Type (

[0077] ): Extracts the event type of the log by matching keywords or regular expressions. For example, matching "ERROR" marks error type logs, and matching "WARNING" marks warning type logs.

[0078] Device Identification (

[0079] ): For logs in multi-device scenarios, extract the device identifier by parsing the log source information (such as IP address, host name, or device ID).

[0080] Description information ( ): Extract the detailed description field in the log and retain the complete information content for subsequent analysis.

[0081] Alternatively, descriptive information can be extracted using word segmentation techniques to break long text into key phrases for subsequent pattern matching or classification tasks. For example, for the log entry "Memory usage exceeds 90%," the word segmentation results might be "Memory," "usage," "exceeds," and "90%."

[0082] It's important to note that structured storage of log data can be done using either relational databases (such as MySQL and PostgreSQL) or non-relational databases (such as Elasticsearch and MongoDB). One possible implementation involves storing log data using an Elasticsearch-based index structure, which supports efficient full-text searches and complex queries.

[0083] In some embodiments, to further improve processing efficiency, a data stream processing framework (such as Apache Kafka) can be used to transmit structured log data to the storage module in real time.

[0084] For better explanation, the specific operations of the present invention in actual application scenarios are as follows:

[0085] Assume that the following logs are collected from two servers (Server_A and Server_B):

[0086] [20xx-12-30 10:23:45] [ERROR][Server_A] CPU utilization exceeds 95%.

[0087] [20xx-12-30 10:24:05] [INFO][Server_B] Memory utilization stable.

[0088] After cleaning and structuring, the standardized log data obtained is:

[0089] [ { "timestamp": "2024-12-30T10:23:45Z", "event_type": "ERROR", "source_device": "Server_A", "description": "CPU utilization exceeds 95%"},{ "timestamp": "2024-12-30T10:24:05Z", "event_type": "INFO", "source_device":"Server_B", "description": "Memory utilization stable"}] These structured data provide standardized input for subsequent steps.

[0090] It is understandable that the quality of log data collection and cleaning directly affects the accuracy and real-time performance of subsequent analysis results. In this embodiment, each step of the pre-processing stage can be implemented in a programmatic manner and supports distributed deployment to meet the processing needs of large-scale log data.

[0091] It should be noted that the log data preprocessing method of the present invention is adaptable to a variety of log sources and formats, providing strong versatility and flexibility for log analysis.

[0092] In step S2, this embodiment constructs a dynamic topology network to abstract the inter-device relationships in the log data into a network structure that evolves over time. This network structure accurately describes the interactions between devices and their dynamic changes, providing data support for subsequent anomaly detection and analysis.

[0093] In this embodiment, the construction of a dynamic topology network begins by mapping devices into nodes and the interactions between devices into edges based on device identifiers, timestamps, and interaction information extracted from log data. It should be noted that a dynamic topology network not only depicts static relationships between devices but also introduces a time dimension, enabling the network to evolve dynamically as time windows are divided.

[0094] Alternatively, a collection of nodes in a dynamic topology network It can be generated from a set of device identifiers extracted from log data. For example, in a log record, the source device (such as a server, switch, etc.) of each log can be directly mapped to a node. The generation of nodes does not depend on the device type or log format and can adapt to data input from a variety of heterogeneous devices.

[0095] Specifically, the edge set in a dynamic topology network

[0096] Established through interaction events between devices. The presence of and There is an interaction relationship within a certain time range. Exemplarily, the interaction relationship can be determined by the following methods:

[0097] If there are direct communication events between two devices (such as data transmission logs, request and response logs), an edge can be established between the devices.

[0098] If a collaboration event between devices appears in the log record (such as an alarm event jointly triggered by multiple devices), an edge can be established for these devices.

[0099] In one possible implementation, the edge weights Used to quantify the intensity of the interaction between devices. For example, the edge weight can be defined as the number of interaction events between two devices in the log record: in, is an indicator function, which takes the value 1 when the condition is met. is the time window. It should be noted that by setting the weight threshold , edges with low interaction strength can be filtered out, thereby reducing the complexity of the network.

[0100] In terms of dynamics, in this embodiment, the log data is segmented according to the timestamp, and a fixed time window method is used to divide the time slices. A static topology network is generated in each time slice.

[0101] For example, the width of the time window can be set to one minute, five minutes, or other time scales. The specific selection can be adjusted according to the frequency of log data generation and system requirements. By combining static networks of multiple time slices in a time sequence, a complete dynamic topology network sequence is formed: Among them, each time slice corresponds to a static network ,The overall network evolves over time, depicting the dynamic interaction relationship between system devices.

[0102] In some embodiments, in order to capture the complex linkage behaviors between devices, the dynamic topology network can also introduce the concept of high-order simplex complex. High-order simplex complex not only describes the pairwise interactions between devices, but also can describe the joint behavior between multiple devices. For example: when the log record shows three devices 、 、 At the same time, they participate in an event, which can be represented as a three-dimensional simplex in the network. .

[0103] Higher-order simplex complex

[0104] The construction can be achieved by: in, is the threshold for filtering weak interactions Represents the dimension of the simplex.

[0105] It should be noted that the introduction of high-order simplex complexes can more comprehensively capture the complex correlation characteristics between devices and is suitable for scenarios where multiple devices frequently collaborate, such as cluster management in data centers or multi-device linkage analysis in network attack incidents.

[0106] In one possible implementation, the construction of a dynamic topology network can further consider the attribute characteristics of the relationships between devices. For example, by adding interaction type labels (such as data transmission, alarm triggering, etc.) to each edge, richer semantic information can be provided for subsequent analysis. These attribute characteristics can be extracted by parsing the log description field. For example:

[0107] For the log "Server_A sends data to Server_B", you can Add property "Data Transfer".

[0108] For the log "Switch_1 triggers alarm affecting Switch_2", you can Add the attribute "Alarm Trigger".

[0109] It can be understood that the construction of a dynamic topology network provides structured input for topology feature extraction and anomaly detection in subsequent steps. The dynamic topology network construction method of the present invention is highly flexible and adaptable, capable of adapting to a variety of log data sources and complex interaction modes. Furthermore, the constructed dynamic network can intuitively present the relationships between devices in the system, providing important basic data support for anomaly analysis.

[0110] It should be noted that by adjusting the size of the time window, the threshold of the edge weight, and the dimension of the simplex, the dynamic topology network construction method of the present invention can flexibly adapt to the needs of different scenarios, such as smaller time windows in high-frequency log scenarios or higher-dimensional simplex modeling in multi-device linkage scenarios. This designed network can fully reflect the system status and interaction patterns, providing support for log analysis in complex systems.

[0111] Regarding step S3, in this embodiment, by extracting the topological characteristics of the dynamic topology network, a quantitative description of the global and local characteristics of the network is achieved, providing basic data support for subsequent anomaly detection and analysis.

[0112] In this embodiment, topology feature extraction uses a dynamic topology network as input and comprehensively characterizes the system's dynamic behavior by calculating the network's topological structural characteristics, including connectivity, ring structure, and network complexity. It should be noted that this topology feature extraction method is based on persistent homology analysis and can effectively capture the dynamic changes in topological characteristics.

[0113] As an alternative, topological feature extraction first constructs a filter function based on the weights of nodes and edges in the dynamic topology network and generates a multi-scale network subgraph sequence. Specifically, the filter function sorts the edges in the network and introduces them in order from low to high weight, thereby gradually constructing the network subgraph: in, is the filtering parameter, which indicates the filtering threshold of the current subgraph.

[0114] It can be understood that by gradually increasing the filtering threshold, a network sequence from sparse to dense can be generated, thereby capturing the topological properties of the network at different scales.

[0115] In one possible implementation, by performing persistent homology analysis on the filtered network sequence, the following two key topological features can be extracted:

[0116] 0-dimensional feature (connectivity): represents the number of connected components in the network, that is, the number of disconnected subgraphs in the current network.

[0117] 1-dimensional characteristic (ring structure): represents the number of closed paths in the network, that is, the rings formed in the network.

[0118] Specifically, persistent homology calculates the generation and extinction time of topological features to form a persistent barcode, which is used to quantify the stability of the features. For example:

[0119] For a connected component, the generation time represents the filtering threshold for its first appearance, and the extinction time represents the filtering threshold for it to be incorporated into other components.

[0120] For a ring structure, the generation time represents the filtering threshold for its first appearance, and the extinction time represents the filtering threshold for the ring to be filled or broken.

[0121] As an option, topological feature extraction can further quantify network features based on topological invariants. Specifically, the following topological invariants are calculated in this embodiment:

[0122] Betti number: used to quantify the number of topological features of different dimensions. For k-dimensional topological features, the Betti number calculation formula is: in, is the boundary operator of the k-dimensional chain complex, which represents the boundary information of the topological features. For example:

[0123] Represents the number of connected components in the network.

[0124] Indicates the number of rings in the network.

[0125] Topological entropy: used to measure the topological complexity of the network, defined as: in, It indicates the proportion of topological features in each dimension. It should be noted that the higher the topological entropy, the more complex the network topology.

[0126] In some embodiments, the result of topological feature extraction may be represented by a feature vector, for example: in, Indicates time The number of connected components, represents the number of rings, These feature vectors can fully reflect the changes in the dynamic network in the time dimension and provide quantitative input for anomaly detection.

[0127] For example, for a dynamic topology network

[0128] If the number of connected components decreases significantly while the number of rings increases significantly during the filtering process, this may indicate an abnormality in the coordination of multiple devices in the system. For example, in a distributed system, the coordination of multiple devices may lead to high load or network congestion, which manifests itself in a significant change in the network structure in the topology characteristics.

[0129] It should be noted that, in order to further improve the computational efficiency of topological feature extraction, the persistent homology analysis can be parallelized in this embodiment. For example:

[0130] In a multi-core computing environment, persistent barcodes can be computed in parallel for different time slices of the network.

[0131] In large-scale network scenarios, topological characteristics can be calculated in parallel for different areas of the network.

[0132] Through the above methods, topological feature extraction can not only accurately reflect the global and local characteristics of dynamic networks, but also maintain high computational efficiency in large-scale data scenarios.

[0133] It is understood that topological feature extraction is a key step in this method. By calculating persistent homology and topological invariants, it can accurately capture the complex correlation information in dynamic networks. These topological features provide quantitative support for subsequent anomaly detection and can also reveal the dynamic evolution of the system through the comparison of global and local characteristics.

[0134] In step S4, this embodiment analyzes dynamic topological changes at different time scales, combining local and global anomaly detection to accurately identify abnormal patterns in log data. Multi-scale anomaly detection enables comprehensive monitoring of system operation status at both micro and macro levels.

[0135] In this embodiment, multi-scale anomaly detection uses the topological characteristics of the dynamic topological network as input, analyzes the changes in topological characteristics within each time window, determines the occurrence of local anomalies and global anomalies, and calculates the comprehensive anomaly index to provide accurate anomaly identification results for subsequent anomaly propagation path analysis.

[0136] As an option, the time scale for anomaly detection can be adjusted based on the specific application scenario. Specifically, in a high-frequency log environment, the time window can be set to seconds, while in a low-frequency log environment, the time window can be set to minutes or longer. By flexibly adjusting the time scale, this embodiment can adapt to log analysis scenarios of varying scales.

[0137] Specifically, the multi-scale anomaly detection method in this embodiment includes the following core steps:

[0138] Local anomaly detection

[0139] In one possible implementation, local anomaly detection analyzes topological feature changes between adjacent time windows. By comparing topological feature vectors of adjacent windows and calculating local differences, it can identify abnormal fluctuations within a short period of time.

[0140] The calculation formula of local difference is: in, Is the time window The topological characteristic vector of represents the Euclidean distance.

[0141] As an option, the criterion for local anomaly detection can be set by setting a threshold When the local difference When the threshold is exceeded, the time window is determined There are local abnormalities.

[0142] For example, in some embodiments, if the topological characteristics of the system change significantly in a short period of time, such as a sudden decrease in the number of connected components ( decrease) or a surge in the number of ring structures ( These changes can be captured through local anomaly detection.

[0143] Global anomaly detection

[0144] In one possible implementation, global anomaly detection identifies abnormal trends over a longer period of time by analyzing the difference between the current time window and the global average characteristics. Specifically, by comparing the time window The topological characteristic vector and the global average topological characteristic vector , and calculate the global difference.

[0145] The calculation formula of global difference is: in, is the mean of the topological properties of all time windows, is the total number of time windows.

[0146] It should be noted that the judgment threshold of global anomaly detection It can be set according to the tolerance range of the system. When the threshold is exceeded, the time window is determined There is a global exception.

[0147] Abnormality index calculation

[0148] In this embodiment, in order to integrate the detection results of local anomalies and global anomalies, an anomaly index is defined as a metric for overall anomaly detection. The calculation formula for the anomaly index is: in, and are the standard deviations of local and global differences, respectively, used for normalization.

[0149] It is understandable that the higher the anomaly index, the more serious the overall anomaly within the current time window. By analyzing the dynamic changes of the anomaly index, we can effectively capture potential problems that may exist in the system.

[0150] In some embodiments, changes in the topology characteristics of a dynamic topology network may reflect abnormalities in the system's operating state. For example:

[0151] In a distributed system, if some nodes lose connectivity, the number of connected components may increase significantly ( boost), which can be captured by local anomaly detection.

[0152] In a collaborative system, if multiple nodes form an abnormal collaborative ring, the number of ring structures may increase ( Improvement), which can be captured by global anomaly detection.

[0153] It should be noted that in order to improve the computational efficiency of multi-scale anomaly detection, the following optimization measures can be introduced in this embodiment:

[0154] In local anomaly detection, the sliding window technology is used to calculate only the differences between adjacent time windows to reduce repeated calculations.

[0155] In global anomaly detection, the incremental update technique is used to update only the global average feature vector without recalculating the mean for all windows.

[0156] Through the above optimization measures, this embodiment can maintain high detection efficiency in large-scale log data scenarios.

[0157] It can be understood that multi-scale anomaly detection achieves accurate identification of abnormal patterns in log data by combining a dual analysis approach of local and global anomalies. Based on a dynamic topology network, the multi-scale anomaly detection method of the present invention can effectively capture both short-term fluctuations and long-term trends in the system, providing important input for subsequent anomaly propagation path analysis and root cause location.

[0158] In step S5, this embodiment constructs a dynamic causal network between devices to analyze the anomaly propagation chain and locate the root device or event that caused the anomaly. Building on the anomaly detection results from step S4, this step further reveals the cause and propagation path of the anomaly, providing precise guidance for system maintenance and problem remediation.

[0159] In this embodiment, the core of anomaly propagation path analysis is to extract causal relationships between devices using log data and a dynamic topology network. The construction of a dynamic causal network can reveal inter-device dependencies and the transmission mechanism of abnormal events. Alternatively, the construction of causal relationships can be based on conditional probability inference, analyzing the interaction patterns between devices to identify the causal chain between abnormal events.

[0160] Specifically, in this embodiment, the nodes of the dynamic causal network correspond to devices or events, and the edges represent causal relationships. For example, if an anomaly in device A triggers an anomaly in device B, a directed edge is established in the causal network from device A to device B.

[0161] Construction of causal networks

[0162] In one possible implementation, the causal network is implemented through conditional probabilities Indicates the causal relationship between devices. The calculation of conditional probability can be based on the time and event association information in the log data. For example: in, Representation device Following the abnormality, the device The number of unusual events, Representation device The total number of anomaly events.

[0163] Alternatively, weights can be assigned to edges in a causal network to quantify the strength of the causal relationship. For example, edge weights can be defined as a normalized form of conditional probability values to screen paths with strong causal relationships.

[0164] Abnormal propagation chain extraction

[0165] Specifically, in this embodiment, the extraction of the abnormal propagation chain is based on the directed path in the causal network. By performing a depth-first search or a breadth-first search on the abnormal event nodes and their causal relationships, the complete abnormal propagation chain can be extracted.

[0166] It should be noted that the result of the abnormal propagation chain extraction is in the form of: in, It is the strength threshold of the causal relationship, which is used to filter out paths with weak correlations. It can be understood that by setting an appropriate threshold, we can focus on the most important anomaly propagation paths and avoid overly complex causal networks.

[0167] In some embodiments, if a propagation chain in a causal network has multiple branches, the main path of anomaly propagation can be identified by using metrics such as cumulative edge weight or path length. For example, if a propagation chain has the largest cumulative edge weight, it can be determined to be the main path of anomaly propagation.

[0168] Root Cause Location

[0169] In this embodiment, root cause location identifies the root cause device or event that caused the anomaly based on the starting point node in the anomaly propagation chain. Specifically, the criteria for determining the root cause device are as follows:

[0170] If the device Located at the starting point of the abnormal propagation chain, and the weights of its related edges are all higher than the set threshold , then determine The root cause device.

[0171] If there are multiple starting devices in the anomaly propagation chain, the device with the highest conditional probability is selected as the root cause device.

[0172] For example, the calculation formula for root cause location is: It should be noted that this formula maximizes the device The impact on subsequent abnormal events is used to determine the most likely root cause device.

[0173] In some embodiments, application scenarios of dynamic causal networks and anomaly propagation chains include:

[0174] Tracing the source of device failures in distributed systems: If server A's CPU overload causes a load balancer exception, which in turn causes a connection timeout on server B, the causal network can identify the propagation chain from server A to server B and locate server A as the root cause.

[0175] Attack path analysis in network attack incidents: If firewall records show that attack traffic from IP1 causes router anomalies and further affects internal servers, the causal network can identify the attack path and lock IP1 as the root cause of the anomaly.

[0176] In one possible implementation, to improve the efficiency of abnormal propagation path analysis, the following optimization measures may be adopted in this embodiment:

[0177] Incremental construction of causal networks: For newly generated log data, only the affected parts of the causal network are updated, avoiding recalculation of the relationships between all devices.

[0178] Parallelized path search: For large-scale causal networks, propagation paths from different starting nodes can be searched in parallel, significantly shortening the computation time.

[0179] Through the above optimization measures, this embodiment can adapt to high-frequency log data scenarios and complete the analysis of abnormal propagation paths in a relatively short time.

[0180] This embodiment, by constructing a dynamic causal network and combining propagation chain extraction and root cause location methods, can accurately identify the propagation mechanism and root cause device of system anomalies. It can be understood that the anomaly propagation path analysis method of the present invention not only locates the cause of the anomaly problem, but also provides clear path guidance for system troubleshooting.

[0181] The log data analysis method provided by the present invention collects, processes, analyzes, and detects anomalies in log data, thereby enabling monitoring of system operating status and identifying and tracing potential problems. The following is the overall workflow of the method of the present invention:

[0182] 1. Collection and preprocessing of log data

[0183] The first step of the present invention is to collect raw log data from multiple data sources (such as servers, switches, routers, etc.). This data may come from a variety of formats and types of logs, including plain text logs, JSON format logs, or XML logs.

[0184] Preprocess the collected log data, including:

[0185] Clean invalid log records (such as meaningless debugging information or redundant records).

[0186] Extract key fields from the logs, such as timestamp, event type, device ID, and event description information.

[0187] Unify log data in different formats into a structured format to provide standardized input for subsequent analysis.

[0188] 2. Construction of dynamic topology network

[0189] By analyzing the device interaction relationships in log data, a dynamic network topology is constructed, including:

[0190] Map the devices extracted from the logs to nodes in the network.

[0191] The interaction behaviors between devices are mapped to edges in the network, and weights are assigned to the edges according to the interaction strength.

[0192] The log data is divided into multiple time slices according to the time window, and a static topology network is constructed for each time slice to form a time-serialized dynamic topology network.

[0193] The dynamic topology network can intuitively present the relationship between devices in the system and its changes over time, laying the foundation for subsequent feature extraction and anomaly detection.

[0194] 3. Extraction of topological features

[0195] Based on the constructed dynamic topology network, the topological characteristics of the network are extracted, mainly including:

[0196] Describes the characteristics of network structure, such as connectivity and ring structure, and is used to characterize the basic topology of the network.

[0197] By quantifying the indicators of network complexity, the overall state changes of the system can be further reflected.

[0198] The extracted topological features are updated in the time dimension as the network changes dynamically, providing a quantitative basis for subsequent anomaly detection.

[0199] 4. Multi-scale anomaly detection

[0200] Detect abnormal patterns in log data by analyzing changes in network topology characteristics at different time scales, including:

[0201] Local anomaly detection: Analyze topological characteristic changes within adjacent time windows to identify short-term sudden anomalies.

[0202] Global anomaly detection: Analyze the difference between the current time window and the overall system historical status to identify long-term trend anomalies.

[0203] Combining local and global detection results, the comprehensive anomaly index is calculated and the degree of anomaly is quantified to provide abnormal node and time window information for subsequent anomaly propagation path analysis.

[0204] 5. Analysis of abnormal propagation paths

[0205] Based on the results of anomaly detection, a dynamic causal network between devices is constructed to analyze the propagation mechanism of anomalies. Specifically, the following are involved:

[0206] Extract anomaly propagation chains and identify causal relationships between devices or events.

[0207] By tracing the starting point of the transmission chain, the root cause device or event that caused the anomaly can be located.

[0208] Finally, an abnormal propagation path is generated, providing a clear logical basis for system troubleshooting and problem location.

[0209] 6. Output and Application

[0210] Output the analysis results, including:

[0211] Visual display of system operation status, such as device topology, health status, and abnormal trends.

[0212] Reports on abnormal propagation chains and root cause devices provide decision support for operation and maintenance personnel.

[0213] The method of this invention implements intelligent analysis of system operating status and problem location through a comprehensive process from log data collection and processing to anomaly detection and root cause location. The entire process has clear steps and tight logical relationships, making it applicable to log data analysis needs in a variety of system scenarios.

[0214] In summary, this method collects, cleans, and structures log data from multiple sources to construct a topological network that dynamically changes over time. It then extracts the network's topological characteristics and performs multi-scale anomaly detection, identifying local and global anomaly patterns within the system. Based on a dynamic causal network, it analyzes the propagation paths of anomalies and accurately locates the root cause of the anomaly, the device or event. This method is adaptable to multiple log formats and multi-source data input for complex systems, supporting real-time analysis and dynamic monitoring, and providing efficient and accurate technical support for system troubleshooting and operational optimization.

[0215] The log analysis system described below and the log analysis method described above can refer to each other.

[0216] Please see the attached Figure 2 The present invention also provides a log analysis system, comprising:

[0217] The data collection module 100 is used to collect log data from multiple devices and pre-process the log data;

[0218] A dynamic network construction module 200 is used to generate a dynamic topology network that evolves over time;

[0219] A topology feature extraction module 300 is used to extract topology features of a dynamic network;

[0220] Anomaly detection module 400, for detecting local and global anomalies in log data at different time scales;

[0221] A root cause analysis module 500 is used to locate the root cause device or event of the anomaly based on the anomaly propagation path;

[0222] The visualization module 600 is used to display log analysis results, including network topology characteristics, abnormal trends and root cause analysis information.

[0223] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, so they will not be repeated here.

[0224] Please see the attached Figure 3 The present invention further provides a computer device 40, comprising: a processor 41 and a memory 42, wherein the memory 42 stores a computer program executable by the processor, and when the computer program is executed by the processor, the above method is performed.

[0225] The present invention further provides a storage medium 43 on which a computer program is stored. When the computer program is run by the processor 41 , the above method is executed.

[0226] Among them, the storage medium 43 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0227] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A log analysis method, characterized in that: The following steps are involved: Collect log data from multiple source devices, clean and structure the log data, and extract timestamps, event types, device identifiers, and description information. Based on the interaction relationships between devices in the log data, generate a dynamic topology network with nodes representing devices and edges representing device interactions, where the network dynamically evolves over time windows. Extract topological characteristics based on the dynamic topology network, including the connectivity of the topological structure, the ring structure, and the complexity characteristics of the network. Analyze changes in network topology characteristics at different time scales, and combine local and global anomaly detection to identify abnormal patterns in log data. Construct a dynamic causal network between devices, extract the anomaly propagation chain, and locate the root cause device or event that caused the anomaly. The topological property extraction steps specifically include: assigning weights to edges of a dynamic topological network and constructing a filtering sequence to generate a subgraph of the network; performing persistent homology analysis on the network subgraph to extract the topological properties of the network, including connectivity, ring structure, and void characteristics; and calculating topological invariants based on the topological properties, including Betti numbers and network topological complexity indicators. The step of identifying abnormal patterns in the log data specifically includes: calculating a topological characteristic vector within each time window, and calculating a local abnormal difference based on the change in the topological characteristics between the time windows; calculating a global abnormal difference based on the change trend of the overall topological characteristic vector in the dynamic network; and determining abnormal patterns in the log data by combining the local abnormalities and the global abnormal differences and setting a threshold; The step of locating the root cause device or event that caused the anomaly specifically includes: constructing a dynamic causal network between devices based on the interaction relationships between devices and the dynamic topology network in the log data; extracting the anomaly propagation chain from the causal network and locating the anomaly propagation path through the causal relationships between devices; and determining the root cause device or event that caused the anomaly based on the anomaly scores of the devices in the propagation chain; The determination of the root cause device or event is based on the following conditions: in the anomaly propagation chain, the device or event that occurs the earliest and has the highest correlation is the root cause; the correlation between devices in the dynamic causal network is estimated through conditional probability, and the root cause priority of the device is determined by maximum likelihood estimation.

2. The log analysis method according to claim 1, characterized in that: The steps for constructing the dynamic topology network specifically include: mapping the source device of each log data into a node of the dynamic network; mapping the interaction relationship between devices in the log data into the edge of the network, and the weight of the edge is determined by the number of associated events or interaction intensity in the log; segmenting the log data based on a set time window to generate a dynamic topology network sequence that evolves over time.

3. The log analysis method according to claim 1, wherein: The method further includes generating a visual display based on the log analysis results, including a system health dashboard, anomaly trend charts, and root cause analysis reports; the content of the visual display includes real-time updated network topology characteristic changes, anomaly detection results, and device status statistics.

4. A log analysis system for executing the log analysis method according to any one of claims 1 to 3, characterized in that: include: Data collection module, used to collect log data from multiple devices and pre-process the log data; Dynamic network construction module, used to generate dynamic topological networks that evolve over time; topological feature extraction module, used to extract topological features of dynamic networks; anomaly detection module, used to detect local and global anomalies in log data at different time scales; The root cause analysis module is used to locate the root cause device or event of the anomaly based on the anomaly propagation path; the visualization module is used to display log analysis results, including network topology characteristics, anomaly trends and root cause analysis information.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images