Log processing method and system based on dual-path processing and intelligent archiving

Abstract

The application provides a log processing method and system based on two-way processing and intelligent archiving, relates to the technical field of log processing, and comprises semantic segmentation and structural analysis of original logs, identification of a risk propagation path through calculation of a causal relationship chain by a Bayesian network, construction of a knowledge graph based on real-time features and historical feature fusion, and optimization of a log archiving strategy by using a two-way propagation probability graph reasoning method. The application effectively improves log processing efficiency, enhances system security, and reduces storage costs.

Description

Technical Field

[0001] This invention relates to the field of log processing technology, and in particular to a log processing method and system based on dual-path processing and intelligent archiving. Background Technology

[0002] With the continuous expansion and increasing complexity of information systems, log data, as a crucial record of system operation status, has become a key element in ensuring stable system operation. Traditional log processing methods mainly rely on techniques such as keyword matching and rule filtering, which face significant challenges when dealing with massive amounts of heterogeneous log data. Currently, log processing technology has evolved from simple data collection and storage to intelligent analysis and prediction, involving the integration of technologies from multiple fields such as natural language processing, machine learning, and knowledge graphs.

[0003] Especially in large-scale distributed systems, log data is characterized by its large volume, diverse types, and complex structure, requiring more efficient and intelligent processing methods to extract system state information and potential risks contained within the logs. Currently, the industry has begun exploring log analysis methods based on semantic understanding and causal reasoning, enabling rapid location and prediction of system anomalies by constructing correlations between log events. Simultaneously, log archiving strategies are evolving from traditional time decay models to intelligent archiving based on event importance and correlation.

[0004] Existing technologies struggle to effectively handle the correlations between multi-source, heterogeneous log data, making it difficult to accurately trace the root causes and propagation paths of problems in complex systems. This is especially true in high-concurrency environments where the interactions between system components are intricate, making it challenging to identify potential risk propagation chains using single-dimensional analysis methods. Traditional log analysis methods lack the ability to extract features over time, failing to consider both real-time and historical statistical characteristics simultaneously. This results in poor analysis performance when facing sudden events or long-term cumulative problems, particularly for system degradation issues exhibiting gradual trends, where detection sensitivity is insufficient. Summary of the Invention

[0005] The embodiments of the present invention provide a log processing method and system based on dual-path processing and intelligent archiving, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a log processing method based on dual-path processing and intelligent archiving, comprising:

[0007] Semantic segmentation and structured parsing are performed on the raw log data to obtain log events;

[0008] The causal relationship chain between the log events is calculated by Bayesian network, potential risk propagation paths are identified based on the causal relationship chain, and log events are diverted to different processing and optimization units based on the potential risk propagation paths.

[0009] Knowledge entities are extracted from log events, and connection edges are established based on the business dependencies and semantic similarity between knowledge entities to form a relationship network. Based on the association relationships in the relationship network, the log events are analyzed to obtain an event association matrix. The conditional dependencies between nodes in the relationship network are calculated using a bidirectional probabilistic graph inference method. Combined with the event association matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relationship network. At the same time, the transition probability between adjacent node pairs is calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

[0010] Calculating causal chains among log events using Bayesian networks, identifying potential risk propagation paths based on these chains, and then routing log events to different processing optimization units based on these potential propagation paths include:

[0011] Log events are constructed as nodes, the dependencies between log events are identified to form an edge set, a directed acyclic graph is constructed, the conditional probability between nodes is determined by the count ratio between the nodes and their parent nodes in the directed acyclic graph, and the conditional probability transitivity between nodes is calculated sequentially based on the topological sequence of the directed acyclic graph to obtain the causal relationship chain between the log events.

[0012] The risk propagation intensity is obtained by continuously multiplying the conditional probability of each node in the causal relationship chain with the corresponding risk weight. The cumulative risk value of the node is updated iteratively, and the unvisited node with the smallest cumulative risk value is searched. The risk propagation intensity is added to the cumulative risk value of the current node to update the cumulative risk value of the adjacent nodes until the cumulative risk value of all nodes converges or the maximum number of iterations is reached.

[0013] Calculate the maximum risk propagation intensity of the potential risk propagation path in which the log event is located, count the number of potential risk propagation paths in which the log event participates, and obtain the comprehensive risk score of the log event by weighted summation of the maximum risk propagation intensity and the number of potential risk propagation paths. Based on the comprehensive risk score, the log event is diverted to the corresponding processing optimization unit for processing.

[0014] In the directed acyclic graph, the conditional probability between nodes is determined by the ratio of node counts to parent node counts. Based on the topological sequence of the directed acyclic graph, the conditional probability transitivity between nodes is calculated sequentially to obtain the causal chain between the log events, including:

[0015] Count the number of co-occurrences of each node with its corresponding parent node and the number of individual occurrences of the corresponding parent node. Divide the number of co-occurrences by the number of individual occurrences and perform Laplace smoothing on the result to obtain the conditional probability. Record all conditional probabilities in the conditional probability matrix.

[0016] The Kahn algorithm is used to obtain the topological sequence of the directed acyclic graph. Based on the topological sequence, the path from the direct predecessor node to the source node of each node is identified in turn. The conditional probability transitivity relationship is obtained by calculating the continuous product of the conditional probabilities between adjacent nodes on the path. All conditional probability transitivity relationships are recorded in the transitivity matrix.

[0017] Each element in the conditional probability matrix is ​​multiplied by a preset direct influence weight factor, and each element in the transitive probability matrix is ​​multiplied by a preset indirect influence weight factor. The two multiplied matrices are then added together to obtain a causal strength matrix. Node pairs in the causal strength matrix that satisfy a preset strength threshold are extracted to construct a causal relationship chain.

[0018] Knowledge entities are extracted from log events. Connections are established based on business dependencies and semantic similarity between these knowledge entities to form a relationship network. Based on the relationships within this network, an event correlation matrix is ​​obtained through correlation analysis of the log events.

[0019] Extract the event subject, event object, and event attributes from log events as knowledge entities; calculate the cosine similarity of word vectors between the knowledge entities to obtain semantic similarity; obtain the business dependency relationship between the knowledge entities based on the system call chain and event temporal dependency relationship, and perform a weighted sum with the semantic similarity to obtain the entity association strength; construct an entity relationship matrix based on the entity association strength, and establish connection edges between entity pairs in the entity relationship matrix that are greater than a preset association threshold to form a relationship network;

[0020] Based on the relationships in the relationship network, the degree centrality and betweenness centrality of the event nodes in the relationship network are calculated to obtain the node importance, and the shortest path hop count between the event nodes is counted to obtain the path relevance. Based on the weighted combination of node importance and path relevance, an event relevance matrix is ​​generated.

[0021] The calculation of conditional dependencies between nodes in the relational network using a bidirectional propagation probabilistic graphical inference method includes:

[0022] The co-occurrence frequency between event nodes in a statistical relationship network is calculated, and the information transmission probability of event node pairs is obtained by dividing the co-occurrence frequency by the total number of events.

[0023] A bidirectional information transmission path is established. The prior belief value of the target node is calculated from the parent node downwards. The prior belief value is the weighted product of the parent node's belief and the information transmission probability. The diagnostic information value is calculated from the child node upwards. The diagnostic information value is the cumulative product of the child node's information and the information transmission probability. The prior belief value and the diagnostic information value are normalized and then multiplied to obtain the posterior probability of the node.

[0024] The conditional mutual information between any two adjacent nodes is calculated using the posterior probability. The information transmission probability of the nodes is updated based on the difference between the conditional mutual information and the dependency threshold. The updated information transmission probability is substituted into the bidirectional information transmission path to generate a new posterior probability. The conditional dependency relationship between nodes is obtained through iterative optimization.

[0025] Combining the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relationship network. Simultaneously, the transition probabilities between adjacent node pairs are calculated. Based on the maximum likelihood criterion, the optimal log archiving strategy is selected, including:

[0026] The information transmission weight is calculated by multiplying the cosine similarity of the event correlation matrix by the inverse of the shortest path distance between node pairs.

[0027] On the message passing link, the downlink decomposition message of each node is calculated. The downlink decomposition message is the sum of the products of the information passing weights of all the neighboring nodes of the node and their historical downlink messages. The uplink aggregation message of each node is calculated. The uplink aggregation message is the sum of the products of the information passing weights of all the neighboring nodes of the node and their historical uplink messages. The downlink decomposition message and the uplink aggregation message are added together to obtain the total bidirectional message of the node.

[0028] The posterior probability is multiplied by the information transmission weight to obtain the transition probability between node pairs; the posterior probability and the transition probability are substituted into the log-likelihood function, and the posterior probability and the transition probability are iteratively updated until the log-likelihood function converges; the optimal archiving scheme for log events is determined based on the updated posterior probability and the transition probability.

[0029] A second aspect of the present invention provides a log processing system based on dual-path processing and intelligent archiving, comprising:

[0030] The first unit performs semantic segmentation and structured parsing on the raw log data to obtain log events;

[0031] The second unit calculates the causal relationship chain between the log events using a Bayesian network, identifies potential risk propagation paths based on the causal relationship chain, and diverts the log events to different processing and optimization units based on the potential risk propagation paths.

[0032] The third unit extracts knowledge entities from log events, establishes connection edges based on business dependencies and semantic similarity between knowledge entities to form a relationship network, and performs correlation analysis on log events based on the relationships in the relationship network to obtain an event correlation matrix. The unit then calculates the conditional dependencies between nodes in the relationship network using a bidirectional probabilistic graph inference method, and combines this with the event correlation matrix to perform top-down event decomposition and bottom-up feature aggregation in the relationship network using a message passing algorithm. Simultaneously, it calculates the transition probabilities between adjacent node pairs and selects the optimal log archiving strategy based on the maximum likelihood criterion.

[0033] A third aspect of the embodiments of the present invention,

[0034] An electronic device is provided, comprising:

[0035] processor;

[0036] Memory used to store processor-executable instructions;

[0037] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0038] Fourth aspect of the present invention,

[0039] A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0040] The beneficial effects of this application are as follows:

[0041] This invention constructs causal relationship chains through semantic segmentation and Bayesian networks, enabling intelligent diversion and precise processing of log events. This effectively improves the system's processing efficiency for complex log data and reduces computational resource consumption.

[0042] This invention combines real-time feature vectors and historical statistical feature vectors for feature fusion, constructing a complete log event knowledge graph, enhancing the system's ability to identify abnormal events, making log processing more comprehensive and accurate, and improving the accuracy of risk warning.

[0043] This invention employs a bidirectional propagation probabilistic graph inference method and message passing algorithm to achieve top-down decomposition and bottom-up aggregation of log events, optimizes log archiving strategies, significantly improves the intelligence level of log management and data storage efficiency, and provides a high-quality data foundation for subsequent log analysis and mining. Attached Figure Description

[0044] Figure 1This is a flowchart illustrating the log processing method based on dual-path processing and intelligent archiving according to an embodiment of the present invention.

[0045] Figure 2 A heatmap illustrating the intensity of risk propagation in log events;

[0046] Figure 3 This is a schematic diagram comparing the performance of two-way propagation probability graph inference methods. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0049] Figure 1 This is a flowchart illustrating the log processing method based on dual-path processing and intelligent archiving according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0050] Semantic segmentation and structured parsing are performed on the raw log data to obtain log events;

[0051] The causal relationship chain between the log events is calculated using a Bayesian network. Based on the causal relationship chain, potential risk propagation paths are identified, and the log events are diverted to different processing and optimization units based on the potential risk propagation paths. In the processing and optimization unit, the burst characteristics and spectral characteristics of the log events are captured in real time through a sliding time window to construct a real-time feature vector. Based on historical data, time-series feature decomposition and quantile sampling are performed to generate historical statistical feature vectors. The real-time feature vector and the historical statistical feature vector are fused to obtain a log event fusion feature vector.

[0052] Based on the fusion feature vector of the log events, a knowledge entity and relationship network of log events is constructed. Semantic associations and business dependencies between the knowledge entities are extracted to obtain a log event knowledge graph. Based on the association relationships of the relationship network, an association analysis is performed on the log events to obtain an event association degree matrix.

[0053] The conditional dependencies between nodes in the log event knowledge graph are calculated using a bidirectional propagation probabilistic graph reasoning method. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the graph structure. At the same time, the transition probability between adjacent node pairs is calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

[0054] In one optional implementation, the causal chain between the log events is calculated using a Bayesian network, potential risk propagation paths are identified based on the causal chain, and log events are diverted to different processing optimization units based on the potential risk propagation paths, including:

[0055] Log events are constructed as nodes, the dependencies between log events are identified to form an edge set, a directed acyclic graph is constructed, the conditional probability between nodes is determined by the count ratio between the nodes and their parent nodes in the directed acyclic graph, and the conditional probability transitivity between nodes is calculated sequentially based on the topological sequence of the directed acyclic graph to obtain the causal relationship chain between the log events.

[0056] The risk propagation intensity is obtained by continuously multiplying the conditional probability of each node in the causal relationship chain with the corresponding risk weight. The cumulative risk value of the node is updated iteratively, and the unvisited node with the smallest cumulative risk value is searched. The risk propagation intensity is added to the cumulative risk value of the current node to update the cumulative risk value of the adjacent nodes until the cumulative risk value of all nodes converges or the maximum number of iterations is reached.

[0057] Calculate the maximum risk propagation intensity of the potential risk propagation path in which the log event is located, count the number of potential risk propagation paths in which the log event participates, and obtain the comprehensive risk score of the log event by weighted summation of the maximum risk propagation intensity and the number of potential risk propagation paths. Based on the comprehensive risk score, the log event is diverted to the corresponding processing optimization unit for processing.

[0058] Log events are constructed as nodes, and dependencies between them are identified to form an edge set. Log events can be various log records generated during runtime, such as error logs, warning logs, and operation logs. Dependencies can be identified through time-series analysis, content association analysis, and other methods. For example, in a server cluster environment, the following log events may exist: node A's CPU utilization exceeds 90%, node A's memory utilization exceeds 85%, node B's service response timeout, and node C's connection failure. By analyzing the timestamps and context information of these log events, a dependency can be identified between the abnormal resource consumption of node A and the service response timeout of node B.

[0059] Based on the identified log event nodes and dependency edges, a directed acyclic graph (DAG) is constructed. In this graph, the direction of the edges represents the direction of the causal relationship, from cause to effect. For the example above, the constructed DAG contains a directed edge from "Node A's CPU utilization exceeds 90%" to "Node B's service response timed out," indicating that the former is the cause of the latter.

[0060] In the constructed directed acyclic graph, the conditional probability between nodes is determined by the ratio of the number of times a node appears together with each of its parent nodes to the total number of times the parent node appears; the ratio of these two is the conditional probability. For example, if in 100 observations, "node A's CPU utilization exceeds 90%" occurs 50 times, and "node B's service response timeout occurs" occurs 40 times in those 50 observations, then the conditional probability of "node B's service response timeout" under the condition that "node A's CPU utilization exceeds 90%" occurs is 40 / 50 = 0.8.

[0061] Based on the topological sequence of a directed acyclic graph (DAG), the conditional probability transitivity between nodes is calculated sequentially to obtain the causal chain between log events. The topological sequence ensures that when calculating the probability of a certain node, the probabilities of all its parent nodes have already been calculated. In practical applications, dynamic programming algorithms can be used to optimize computational efficiency. For example, in a DAG, there exists a path: "Node A's CPU utilization exceeds 90%" → "Node B's service response timed out" → "Node C's connection failed." By calculating the transitivity of conditional probabilities, the causal chain consisting of these three events and its strength can be obtained.

[0062] After obtaining the causal chain, the risk propagation strength is calculated as the continuous product of the conditional probability and the corresponding risk weight of each node in the chain. The risk weights can be pre-set based on factors such as the severity and scope of the log events. For example, the risk weight for service interruption is 0.9, while the risk weight for performance degradation is 0.6. For the causal chain in the example above, assuming the risk weights of the three events are 0.7, 0.8, and 0.9, and the conditional probabilities are 0.8 and 0.7, the risk propagation strength of this chain is 0.7 × 0.8 × 0.8 × 0.7 × 0.9 = 0.28224.

[0063] Risk propagation paths are identified by iteratively updating the cumulative risk values ​​of nodes and searching for the unvisited node with the smallest cumulative risk value. Initially, the cumulative risk value of the starting node is set to 0, and the cumulative risk value of other nodes is set to infinity. Each time, the node with the smallest cumulative risk value is selected as the current node from the unvisited nodes. The risk propagation strength is added to the current node's cumulative risk value, and the cumulative risk values ​​of adjacent nodes are updated. This process is similar to an improved Dijkstra's algorithm, but uses the risk propagation strength as the edge weight. The iterative process continues until the cumulative risk values ​​of all nodes converge or the preset maximum number of iterations (e.g., 100) is reached.

[0064] In a real-world application scenario, suppose there are the following log events: Event E1 is a database connection timeout, Event E2 is high application server CPU utilization, Event E3 is slow cache server response, Event E4 is a user login failure, and Event E5 is an order processing delay. The causal chains identified using the above method include two paths: E2→E3→E5 and E1→E4→E5. If the risk propagation intensity of the first path is 0.35 and the risk propagation intensity of the second path is 0.28, then both paths constitute potential risk propagation paths.

[0065] After obtaining the potential risk propagation paths, calculate the maximum risk propagation intensity of each log event's potential risk propagation path. For example, if event E5 is located on two paths simultaneously, its maximum risk propagation intensity is max(0.35, 0.28) = 0.35. Simultaneously, count the number of potential risk propagation paths the log events participate in; for instance, event E5 participates in 2 paths, while event E2 participates in only 1 path.

[0066] The comprehensive risk score for a log event is obtained by weighting and summing the maximum risk propagation intensity and the number of potential risk propagation paths. The weights can be set according to actual needs; for example, the weight of the maximum risk propagation intensity can be set to 0.7, and the weight of the number of paths can be set to 0.3. For event E5, its comprehensive risk score is 0.7 × 0.35 + 0.3 × 2 = 0.245 + 0.6 = 0.845. Based on the calculated comprehensive risk score, the log event is distributed to the corresponding processing optimization unit for processing.

[0067] Specifically, multiple risk level thresholds can be set, such as a high-risk threshold of 0.8, a medium-risk threshold of 0.5, and a low-risk threshold of 0.2. Log events with a comprehensive risk score exceeding the high-risk threshold are routed to a high-priority processing unit, employing strategies such as resource priority protection and real-time monitoring; log events with scores between the medium and high-risk thresholds are routed to a medium-priority processing unit, employing strategies such as periodic checks and anomaly warnings; and log events with scores below the low-risk threshold are routed to a low-priority processing unit, employing strategies such as batch processing and simplified analysis.

[0068] In the example, event E5 has a comprehensive risk score of 0.845, which exceeds the high-risk threshold of 0.8. Therefore, it is diverted to the high-priority processing unit, where more computing resources are allocated and a real-time monitoring mechanism is activated. Event E3 has a comprehensive risk score of 0.62, which is between the medium and high-risk thresholds. It is diverted to the medium-priority processing unit. Event E2 has a comprehensive risk score of 0.38, which is below the medium-risk threshold. It is diverted to the low-priority processing unit.

[0069] This traffic distribution mechanism allows for the rational allocation of processing resources based on the importance of log events within the risk propagation network. Critical log events that could lead to severe risk propagation are prioritized, improving risk control efficiency and resource utilization. For high-risk log events, proactive intervention measures, such as elastic scaling and load balancing adjustments, can be initiated to further reduce the probability of risk propagation.

[0070] Figure 2 This is a heatmap illustrating the risk propagation intensity of log events. The heatmap, based on a Bayesian network, visually presents the risk propagation relationships and intensity distribution among different log events. The heatmap uses a 7×7 matrix, with the horizontal and vertical axes representing the source and target events of risk propagation, respectively. Typical security events include E1 (login failure), E2 (file access), E3 (privilege escalation), E4 (system call), E5 (data leakage), E6 (network anomaly), and E7 (process anomaly). The intensity of the colors in the graph represents the magnitude of the risk propagation; white indicates no propagation relationship, and darker red indicates higher propagation intensity, ranging from 0 to 1. High-risk propagation paths are observed, such as E2→E3 with a propagation intensity of 0.85, E3→E4 with 0.75, and E3→E5 with 0.68. These values ​​are calculated by multiplying the conditional probability P(Ej|Ei) by the corresponding risk weight W(Ei). The diagonal region is blank, which conforms to the characteristic that directed acyclic graphs do not have self-loops, demonstrating the mathematical rigor of the risk propagation model of this invention. This heatmap effectively verifies the ability of this invention to identify potential risk propagation paths through Bayesian networks, providing a visual basis for subsequent event triage and risk assessment, and proving the practicality and accuracy of the algorithm in complex security scenarios.

[0071] In one optional implementation, the conditional probability between nodes is determined by the count ratio of a node to its parent node in the directed acyclic graph. The causal chain between the log events is obtained by sequentially calculating the conditional probability transitivity between nodes based on the topological sequence of the directed acyclic graph, including:

[0072] Count the number of co-occurrences of each node with its corresponding parent node and the number of individual occurrences of the corresponding parent node. Divide the number of co-occurrences by the number of individual occurrences and perform Laplace smoothing on the result to obtain the conditional probability. Record all conditional probabilities in the conditional probability matrix.

[0073] The Kahn algorithm is used to obtain the topological sequence of the directed acyclic graph. Based on the topological sequence, the path from the direct predecessor node to the source node of each node is identified in turn. The conditional probability transitivity relationship is obtained by calculating the continuous product of the conditional probabilities between adjacent nodes on the path. All conditional probability transitivity relationships are recorded in the transitivity matrix.

[0074] Each element in the conditional probability matrix is ​​multiplied by a preset direct influence weight factor, and each element in the transitive probability matrix is ​​multiplied by a preset indirect influence weight factor. The two multiplied matrices are then added together to obtain a causal strength matrix. Node pairs in the causal strength matrix that satisfy a preset strength threshold are extracted to construct a causal relationship chain.

[0075] The system counts the number of times each node co-occurs with its corresponding parent node, as well as the number of times the corresponding parent node appears alone. Specifically, by analyzing the collected historical log data, the occurrence of various log events can be obtained. For example, for log events of a network service system, suppose node A represents "increased network latency," node B represents "slower service response," and node C represents "user session timeout." By analyzing log records over a period of time, it is found that node A and node B co-occurred 80 times, and node A appeared alone 100 times; node B and node C co-occurred 65 times, and node B appeared alone 90 times.

[0076] Based on the above statistical data, the process of calculating conditional probability is as follows: Divide the number of times a node co-occurs with its corresponding parent node by the number of times the corresponding parent node appears alone to obtain a preliminary conditional probability value. For the example above, the preliminary calculated conditional probability of node B under the condition that node A occurs is 80 / 100 = 0.8; the preliminary calculated conditional probability of node C under the condition that node B occurs is 65 / 90 = 0.722.

[0077] To avoid the zero probability problem and improve the stability of the model, the calculated conditional probabilities are smoothed using Laplace smoothing. Laplace smoothing is a commonly used smoothing technique that avoids zero probability by adding a small positive number to both the numerator and denominator. In this embodiment, 1 is added to the numerator and 2 to the denominator, so the smoothed conditional probability is calculated as (co-occurrence count + 1) / (single occurrence count + 2). Applying this to the example above, the smoothed conditional probability of node B under the condition that node A occurs is (80+1) / (100+2) = 81 / 102 ≈ 0.794; the smoothed conditional probability of node C under the condition that node B occurs is (65+1) / (90+2) = 66 / 92 ≈ 0.717.

[0078] After calculating the conditional probabilities between all node pairs, the results are recorded in the conditional probability matrix. This matrix is ​​an N×N matrix, where N is the total number of log event nodes, and the element CPij in the matrix represents the conditional probability of node j given that node i occurs. For node pairs without a direct causal relationship, the corresponding matrix element value is set to 0. Based on the above example, assuming that node D represents "database query timeout" and node E represents "decreased cache hit rate", the following conditional probability matrix is ​​formed: In the CP matrix, CPA,B=0.794, CPB,C=0.717, CPD,E=0.682, CPE,B=0.631, and all other elements are 0.

[0079] Next, the Kahn algorithm is used to obtain the topological sequence of the directed acyclic graph. The Kahn algorithm is a topological sorting algorithm based on in-degree. Its basic idea is to continuously remove nodes with an in-degree of 0 while reducing the in-degree of their neighboring nodes until no nodes remain in the graph. The specific implementation steps are as follows: Initialize a queue Q and add all nodes with an in-degree of 0 to the queue; when the queue is not empty, remove a node n from the queue and add it to the topological sequence; for each neighboring node m of node n, decrement the in-degree of m by 1; if the in-degree of m becomes 0, add m to queue Q; repeat the above process until the queue is empty. For the example above, assuming the constructed directed acyclic graph has nodes A and D with an in-degree of 0, node B with an in-degree of 2 (from nodes A and E), node C with an in-degree of 1 (from node B), and node E with an in-degree of 1 (from node D), the topological sequence obtained by applying the Kahn algorithm is: A, D, E, B, C.

[0080] Based on the acquired topology sequence, the path from the direct predecessor node to the source node is identified sequentially for each node. The source node is the first node in the topology sequence, i.e., the node with an in-degree of 0. For each node, by tracing back its parent node, all paths from the source node to that node can be found. In the example above, the direct predecessor node of node C is B, and tracing back from B, two paths can be found: A→B→C and D→E→B→C.

[0081] Calculate the continuous product of conditional probabilities between adjacent nodes on each path to obtain the conditional probability transitivity. For path A→B→C, the conditional probability transitivity is CPA,B×CPB,C=0.794×0.717≈0.569; for path D→E→B→C, the conditional probability transitivity is CPD,E×CPE,B×CPB,C=0.682×0.631×0.717≈0.308. These conditional probability transitivity relationships reflect the strength of indirect causal influence between remote nodes.

[0082] All conditional probability transitivity relationships are recorded in a transitivity matrix. This matrix is ​​also an N×N matrix, where the element TPij represents the conditional probability transitivity relationship from node i to node j. If there are multiple paths from node i to node j, the maximum value of the consecutive products of conditional probabilities along these paths is taken as the value of TPij. In the example above, TPA,C=0.569, TPD,C=0.308, etc.

[0083] To comprehensively consider the impact of direct and indirect causal relationships, a preset direct influence weighting factor α and a preset indirect influence weighting factor β are introduced. The sum of these two weighting factors is typically 1, reflecting the degree of importance given to direct and indirect influences in causal relationship assessment. In practical applications, the values ​​of these two weighting factors can be adjusted according to specific scenarios and needs. For example, α = 0.7 and β = 0.3 can be set, indicating that the influence weight of direct causal relationships is greater than that of indirect causal relationships.

[0084] Multiplying each element of the conditional probability matrix by a preset direct influence weighting factor α yields a weighted direct influence matrix. For the example above, the weighted CPA,B = 0.794 × 0.7 ≈ 0.556, CPB,C = 0.717 × 0.7 ≈ 0.502, etc. Similarly, multiplying each element of the transitive probability matrix by a preset indirect influence weighting factor β yields a weighted indirect influence matrix. For the example above, the weighted TPA,C = 0.569 × 0.3 ≈ 0.171, TPD,C = 0.308 × 0.3 ≈ 0.092, etc.

[0085] Adding the two weighted matrices yields the causal strength matrix. This matrix comprehensively reflects the strength of direct and indirect causal relationships between log event nodes. In the example above, the causal strength between node A and node B is 0.556 + 0 = 0.556 (because there is no indirect path from A to B), the causal strength between node A and node C is 0 + 0.171 = 0.171, and the causal strength between node B and node C is 0.502 + 0 = 0.502, and so on.

[0086] Finally, node pairs that satisfy a preset strength threshold are extracted from the causal strength matrix to construct causal chains. The preset strength threshold is a parameter set according to the specific application scenario, used to filter out node pairs with weak causal relationships. For example, if the preset strength threshold is set to 0.3, then node pairs (A,B), (B,C), (D,E), and (E,B) can be extracted from the above causal strength matrix, while node pairs (A,C) and (D,C) are filtered out because their causal strength is below the threshold. Based on the extracted node pairs, the following causal chains can be constructed: A→B→C and D→E→B→C.

[0087] In practical applications, this method can be applied to log analysis in large-scale distributed systems. For example, the log system of an e-commerce platform records various system events, including user behavior, service calls, and resource usage. By analyzing the causal relationships between these log events, key factors affecting system performance and reliability can be identified. Suppose the analysis shows a strong causal relationship between "database connection pool exhaustion" and "order processing delay," and "order processing delay" in turn has a strong causal relationship with "user payment failure," then a causal chain can be constructed: "database connection pool exhaustion → order processing delay → user payment failure." Based on this causal chain, system administrators can prioritize resolving the "database connection pool exhaustion" problem, preventing serious consequences such as "user payment failure" from the outset.

[0088] In one optional implementation, knowledge entities are extracted from log events, and connection edges are established based on the business dependencies and semantic similarity between the knowledge entities to form a relationship network. Based on the association relationships in the relationship network, an event association matrix is ​​obtained by performing association analysis on the log events, including:

[0089] Extract the event subject, event object, and event attributes from log events as knowledge entities; calculate the cosine similarity of word vectors between the knowledge entities to obtain semantic similarity; obtain the business dependency relationship between the knowledge entities based on the system call chain and event temporal dependency relationship, and perform a weighted sum with the semantic similarity to obtain the entity association strength; construct an entity relationship matrix based on the entity association strength, and establish connection edges between entity pairs in the entity relationship matrix that are greater than a preset association threshold to form a relationship network;

[0090] Based on the relationships in the relationship network, the degree centrality and betweenness centrality of the event nodes in the relationship network are calculated to obtain the node importance, and the shortest path hop count between the event nodes is counted to obtain the path relevance. Based on the weighted combination of node importance and path relevance, an event relevance matrix is ​​generated.

[0091] The event subject typically refers to the initiator or bearer of the log event, such as a server, application, or component; the event object refers to the target of the log event, such as a database, file, or network connection; and event attributes refer to information describing the characteristics of the event, such as status code, timestamp, and severity. The extraction process can be implemented through rule matching and natural language processing techniques. For example, for the log message "Web server 192.168.1.10 failed to request a database connection at 08:30:25 on 2023-05-15, error code 504", the extractable knowledge entities include: the event subject "Web server 192.168.1.10", the event object "database connection", and the event attributes "error code 504" and "2023-05-15 08:30:25". To improve extraction efficiency and accuracy, a domain terminology database and entity recognition templates can be pre-established, and specific extraction rules can be designed for different types of log formats.

[0092] After extracting knowledge entities, the semantic similarity is obtained by calculating the cosine similarity of word vectors between knowledge entities. Word vectors are a representation method that maps words to a high-dimensional vector space, capturing the semantic information of words. In this implementation, a pre-trained word vector model, such as a word embedding model trained on a large-scale text corpus, can be used to convert the extracted knowledge entities into word vectors. For composite entities, they can be decomposed into word sequences, and the average of the word vectors of each word can be taken as the vector representation of the entity. The cosine similarity between the vectors of two knowledge entities is calculated, with a value ranging from -1 to 1. The larger the value, the higher the semantic similarity. For example, the semantic similarity between "database connection failed" and "database query timed out" is 0.85, while the semantic similarity between "database connection failed" and "insufficient network bandwidth" is 0.32.

[0093] Simultaneously, business dependencies between knowledge entities are obtained based on system call chains and event sequence dependencies. System call chains record the call relationships between services or components and can be obtained by analyzing system architecture diagrams, service registration information, or call chain tracing logs. Event sequence dependencies are based on the temporal order of log events and can be identified by analyzing the time windows and frequency patterns of event occurrences. For example, in a microservice architecture, if service A calls service B, and service B calls service C, then there is a call chain A→B→C. If it is observed that errors in service B typically occur shortly after service A experiences an anomaly, then a sequence dependency can be determined between the two. Business dependencies can be quantified as values ​​between 0 and 1, representing dependency strength. For example, the dependency strength of a direct call relationship can be set to 0.9, the dependency strength of an indirect call relationship (passing through a service in between) can be set to 0.6, and the dependency strength of a relationship that is only temporally related but has no explicit call relationship can be set to 0.3.

[0094] The entity association strength is obtained by weighting and summing semantic similarity and business dependency. The weighting coefficients can be adjusted according to the specific application scenario. For example, in an environment that focuses on business logic, the weight of business dependency can be set to 0.7 and the weight of semantic similarity to 0.3; in a scenario that focuses on text analysis, the weight of semantic similarity can be appropriately increased. Assuming that the semantic similarity between knowledge entities "Web Server A" and "Application Server B" is 0.6, the business dependency strength is 0.8, and the weights are 0.3 and 0.7 respectively, then the entity association strength between the two is 0.3×0.6+0.7×0.8=0.18+0.56=0.74.

[0095] An entity relationship matrix is ​​constructed based on the entity association strength. This matrix is ​​an n×n square matrix, where n is the number of knowledge entities, and the elements in the matrix represent the association strength between corresponding entity pairs. To facilitate subsequent processing, elements in the matrix less than a certain threshold can be set to 0 to reduce computational complexity. Connection edges are established between entity pairs in the entity relationship matrix whose association strength is greater than the preset association threshold, forming a relationship network. The preset association threshold is a parameter set according to actual needs and is used to filter weak associations. For example, the preset association threshold can be set to 0.5, so that only entity pairs with an association strength greater than 0.5 will have connection edges established. The resulting relationship network is a weighted undirected graph, where nodes represent knowledge entities, edges represent the associations between entities, and the weight of the edge is the entity association strength.

[0096] Based on the relationships in a relational network, the degree centrality and betweenness centrality of event nodes in the network are calculated to obtain the node importance. Degree centrality refers to the number of edges directly connecting a node, reflecting the node's direct influence in the network. For example, if an event node is directly connected to multiple other event nodes, its degree centrality is high. In weighted networks, weighted degree centrality can be used, which incorporates the weights of connecting edges into the calculation. Betweenness centrality refers to the number of shortest paths passing through a node, reflecting the node's importance in the flow of information within the network. Calculating betweenness centrality requires traversing the shortest paths between all node pairs in the network and counting the proportion of paths passing through the target node. For example, suppose a relational network consists of 10 event nodes, where event node A has a weighted degree centrality of 2.8 (connected to 4 nodes with edge weights of 0.8, 0.7, 0.6, and 0.7) and a betweenness centrality of 0.4 (40% of the shortest paths between all node pairs pass through node A).

[0097] The importance of a node is obtained by normalizing and weighting the degree centrality and betweenness centrality. Normalization can be achieved using the maximum value normalization method, which divides the original value by the maximum value in the network. When weighting the combination, the weights of the two centrality indicators can be adjusted according to actual needs. For example, the weight of degree centrality can be set to 0.4, and the weight of betweenness centrality to 0.6. Then, the node importance is calculated as 0.4 × normalized degree centrality + 0.6 × normalized betweenness centrality. Continuing the example above, assuming the maximum weighted degree centrality in the network is 3.5 and the maximum betweenness centrality is 0.5, then the normalized degree centrality of event node A is 2.8 / 3.5 = 0.8, the normalized betweenness centrality is 0.4 / 0.5 = 0.8, and the node importance is 0.4 × 0.8 + 0.6 × 0.8 = 0.8.

[0098] Path relevance is determined by counting the hops of the shortest path between event nodes. The shortest path hop count refers to the number of edges on the shortest path between two nodes. A lower hop count indicates a higher degree of correlation between the two nodes. Path relevance can be defined as the reciprocal of the shortest path hop count, or by using an exponential decay function to convert the hop count to a value between 0 and 1. For example, the formula "path relevance = 2 raised to the power of negative hops" can be used, so the path relevance of directly connected nodes (1 hop count) is 0.5, the path relevance of nodes separated by one node (2 hops) is 0.25, and so on. For unreachable node pairs, the path relevance is defined as 0.

[0099] An event correlation matrix is ​​generated based on a weighted combination of node importance and path relevance. Specifically, for event nodes i and j, their correlation can be defined as: the average importance of the two nodes multiplied by the path relevance, and then multiplied by an adjustable parameter. The event correlation matrix is ​​an m×m square matrix, where m is the number of event nodes, and the elements in the matrix represent the correlation between corresponding pairs of event nodes. For example, if the importance of event nodes A and B are 0.8 and 0.6 respectively, the shortest path hop count between them is 1 (directly connected), the path relevance is 0.5, and the adjustable parameter is 2, then the event correlation between A and B is (0.8+0.6) / 2×0.5×2=0.7×0.5×2=0.7.

[0100] In practical applications, this method can be used for monitoring and fault diagnosis of large-scale distributed systems. For example, an e-commerce platform system consists of multiple microservices, including user services, product services, order services, and payment services. System logs record the running status and anomalies of each service. By extracting knowledge entities from the logs, such as "user service instance 1," "product database," and "order processing timeout," the semantic similarity and business dependencies between entities are calculated, and a relationship network is constructed. Based on this network, the importance and path relevance of event nodes are calculated, generating an event correlation matrix. This matrix shows that the correlation between "database connection pool exhaustion" and "order processing timeout" is 0.85, the correlation between "increased network latency" and "user login failure" is 0.72, and the correlation between "caching service restart" and "slow product search" is 0.63. Using this correlation information, system operators can more quickly locate the root cause of the fault, prioritize handling highly correlated event chains, and improve system reliability and user experience.

[0101] The technical solution provided in this embodiment constructs a knowledge entity relationship network through both semantic analysis and business relationship dimensions, and calculates event correlation based on network topology characteristics. This accurately reflects the inherent connections between log events in complex systems, providing strong support for subsequent risk assessment and fault diagnosis. This method is applicable to log analysis scenarios in various information systems and has strong versatility and practical value.

[0102] In one optional implementation, calculating the conditional dependencies between nodes in the relational network using a bidirectional propagation probabilistic graph inference method includes:

[0103] The co-occurrence frequency between event nodes in a statistical relationship network is calculated, and the information transmission probability of event node pairs is obtained by dividing the co-occurrence frequency by the total number of events.

[0104] A bidirectional information transmission path is established. The prior belief value of the target node is calculated from the parent node downwards. The prior belief value is the weighted product of the parent node's belief and the information transmission probability. The diagnostic information value is calculated from the child node upwards. The diagnostic information value is the cumulative product of the child node's information and the information transmission probability. The prior belief value and the diagnostic information value are normalized and then multiplied to obtain the posterior probability of the node.

[0105] The conditional mutual information between any two adjacent nodes is calculated using the posterior probability. The information transmission probability of the nodes is updated based on the difference between the conditional mutual information and the dependency threshold. The updated information transmission probability is substituted into the bidirectional information transmission path to generate a new posterior probability. The conditional dependency relationship between nodes is obtained through iterative optimization.

[0106] Event co-occurrence refers to the simultaneous occurrence of two events within a specific time window. For log analysis scenarios, an appropriate time window size can be set, such as 5 minutes, 15 minutes, or 30 minutes, to statistically analyze the pairs of events that co-occur within the window. For example, in the log data of a distributed system, the events "database connection timeout" and "application service response delay" co-occurred 120 times out of 1000 observations, and the events "network traffic surge" and "load balancer switching" co-occurred 80 times out of 1000 observations. These statistics reflect the preliminary correlation between the events.

[0107] Dividing the co-occurrence frequency by the total number of events yields the information transmission probability of event node pairs. The information transmission probability is an indicator of the strength of information flow between two event nodes, with values ​​ranging from 0 to 1; a higher value indicates a higher probability of information transmission. In the example above, the information transmission probability between the events "database connection timeout" and "application service response delay" is 120 / 1000 = 0.12, and the information transmission probability between the events "network traffic surge" and "load balancer switching" is 80 / 1000 = 0.08. By calculating the information transmission probabilities between all event node pairs, an information transmission probability matrix can be constructed. This matrix is ​​an n×n square matrix, where n is the number of event nodes, and the elements in the matrix represent the information transmission probability between the corresponding event node pairs.

[0108] A two-way information transmission path is established, calculating the prior belief value of the target node from the parent node downwards. A parent node is a node in the relationship network that points to the target node. The prior belief value represents a preliminary estimate of the target node's state without considering other evidence. A parent node's belief refers to its degree of certainty about its own state, determined by prior knowledge or historical data, and represents the probability that the node is activated or in a specific state. For example, for the event node "high CPU usage," a parent node belief value of 0.8 indicates that the event has an 80% probability of occurring or being observed. The calculation of the prior belief value involves the parent node's belief value and the information transmission probability from the parent node to the target node. Specifically, the target node's prior belief value is equal to the sum of the weighted products of all parent node belief values ​​and their corresponding information transmission probabilities. Weighting factors can be introduced during the calculation to adjust the influence of different parent nodes. For example, for the event node "System performance deterioration", its parent nodes include "High CPU utilization", "Insufficient memory" and "Disk I / O blocking". The belief values ​​of these parent nodes are 0.8, 0.6 and 0.7 respectively, the corresponding information transmission probabilities are 0.15, 0.12 and 0.18 respectively, and the weight factors are 0.4, 0.3 and 0.3 respectively. Then the prior belief value of "System performance deterioration" is 0.8×0.15×0.4+0.6×0.12×0.3+0.7×0.18×0.3=0.048+0.0216+0.0378=0.1074.

[0109] Diagnostic information values ​​are calculated upwards from child nodes. A child node is a node in a relational network that the target node points to. The diagnostic information value represents the impact of feedback information obtained from child nodes on the estimation of the target node's state. Child node information refers to the evidence or observations fed back from child nodes, reflecting the degree to which the child node supports the inference of the parent node's state. Child node information can come from direct observation or measurement, or it can be the output of other reasoning processes. For example, an information value of 0.5 for the event node "SQL query failed" indicates that this observation provides moderate support evidence for its parent node "database connection timed out." In practical systems, child node information can be specific events detected by monitoring tools, such as error logs or abnormal performance metrics. The calculation of diagnostic information values ​​involves the information values ​​of child nodes and the probability of information transmission from the target node to the child nodes. Specifically, the diagnostic information value of the target node is equal to the sum of the cumulative products of all child node information values ​​and their corresponding information transmission probabilities. Similar to the calculation of prior belief values, weighting factors can be introduced to adjust the influence of different child nodes. For example, for the event node "Database connection timeout", its child nodes include "SQL query failure" and "transaction rollback". The information values ​​of these child nodes are 0.5 and 0.4 respectively, the corresponding information transmission probabilities are 0.2 and 0.15 respectively, and the weight factors are 0.6 and 0.4 respectively. Then the diagnostic information value of "Database connection timeout" is 0.5×0.2×0.6+0.4×0.15×0.4=0.06+0.024=0.084.

[0110] The posterior probability of a node is obtained by multiplying the normalized prior belief value and the diagnostic information value. Normalization ensures that the calculation result is within a valid range, avoiding calculation errors caused by values ​​that are too large or too small. Normalization methods can employ techniques such as maximum normalization or minimum-maximum normalization. For example, the prior belief value and the diagnostic information value can be divided by their respective maximum values ​​in the network to obtain the normalized values. Assuming the maximum prior belief value in the network is 0.3 and the maximum diagnostic information value is 0.25, then the normalized prior belief value of the aforementioned "database connection timeout" node is 0.1074 / 0.3≈0.358, the normalized diagnostic information value is 0.084 / 0.25≈0.336, and its posterior probability is 0.358×0.336≈0.12. The posterior probability reflects the estimation of the node state after comprehensively considering contextual information and is the core output of the bidirectional propagation algorithm.

[0111] The conditional mutual information between any two adjacent node pairs is calculated using posterior probabilities. Conditional mutual information is an indicator that measures the degree of interdependence between two variables given a third variable. In this implementation, conditional mutual information is used to evaluate the strength of the dependency between two event nodes considering the influence of other nodes. Calculating conditional mutual information requires considering the posterior probability of each node, the joint probability of the node pair, and the marginal probabilities of each node. For example, for event node A "high server CPU utilization" and event node B "increased application response time," assuming their posterior probabilities are 0.15 and 0.18 respectively, and their joint probability is 0.05, their conditional mutual information can be obtained through correlation calculation; the assumed result is 0.023.

[0112] The information transmission probability of nodes is updated based on the difference between conditional mutual information and a dependency threshold. The dependency threshold is a pre-set parameter used to determine the significance of the dependency relationship between nodes. If the conditional mutual information is greater than the dependency threshold, it indicates that there is a significant dependency relationship between the two nodes; otherwise, it indicates that the dependency relationship is weak. The information transmission probability can be updated using the gradient descent method, that is, the original information transmission probability is adjusted proportionally according to the difference between the conditional mutual information and the dependency threshold. For example, if the dependency threshold is set to 0.02 and the learning rate is 0.1, then for the above event nodes A and B, the difference between their conditional mutual information and the dependency threshold is 0.023 - 0.02 = 0.003, and the update amount of the information transmission probability is 0.003 × 0.1 = 0.0003. If the original information transmission probability is 0.12, then the updated information transmission probability is 0.12 + 0.0003 = 0.1203.

[0113] The updated information transmission probabilities are substituted into the bidirectional information transmission path to generate new posterior probabilities. This is an iterative process that continuously adjusts the information transmission probabilities to make the model's output more accurately reflect the true dependencies between nodes. In each iteration, the prior belief values, diagnostic information values, and posterior probabilities of all nodes need to be recalculated. For example, using the updated information transmission probabilities, the prior belief values ​​and diagnostic information values ​​of the node "database connection timeout" are recalculated to obtain a new posterior probability.

[0114] Through iterative optimization, the conditional dependencies between nodes are obtained. The iteration process can be terminated under specific conditions, such as reaching the maximum number of iterations (e.g., 100) or the change in posterior probability being less than a specific threshold (e.g., 0.0001). When the iteration terminates, the final information transmission probability matrix reflects the conditional dependencies between nodes. For example, after multiple iterations, the conditional dependency strength between the events "database connection timeout" and "application service response delay" is 0.35, and the conditional dependency strength between the events "network traffic surge" and "load balancer switching" is 0.28.

[0115] In practical applications, this method can be used for anomaly detection and fault diagnosis in large-scale IT infrastructure. For example, a cloud computing platform contains hundreds of server nodes and thousands of service instances. System logs record various events, such as hardware failures, software errors, and network anomalies. By constructing a relational network and applying a bidirectional propagation probabilistic graph inference method, the conditional dependencies between events can be identified. The analysis results show that the conditional dependency strength between "slow storage node response" and "data write failure" is 0.42, the conditional dependency strength between "network partition" and "service discovery failure" is 0.38, and the conditional dependency strength between "configuration update" and "service restart" is 0.31. Based on these dependencies, system administrators can build fault propagation models, predict fault chains, and take corresponding preventative measures.

[0116] The bidirectional propagation probabilistic graph inference method provided in this embodiment calculates the posterior probability of nodes through bottom-up and top-down bidirectional information flow, and then derives conditional dependencies based on the posterior probabilities, accurately capturing the dependency structure between events in complex systems. This method combines the advantages of Bayesian networks and information theory, exhibiting strong robustness to incomplete or noisy data, and is suitable for log analysis and fault diagnosis scenarios in large-scale distributed systems. Through an iterative optimization mechanism, this method can continuously adjust model parameters to adapt to dynamic changes in system state, providing more accurate dependency evaluation results.

[0117] Figure 3 This diagram illustrates the performance comparison of the bidirectional propagation probabilistic graph inference method. It shows the comparative analysis results of the bidirectional propagation probabilistic graph inference method of this invention with three other traditional methods across four key performance indicators. The diagram uses black, white, and gray bar charts to visually present the performance differences between the methods. Black bars represent the method of this invention, dark gray bars represent traditional Bayesian networks, light gray bars represent Markov chain models, and white bars represent basic association rule methods. In terms of anomaly detection accuracy, this invention achieves 92.8%, a 14.4 percentage point improvement compared to the 78.4% of traditional Bayesian networks. In terms of fault location accuracy, this invention achieves 89.5%, significantly surpassing the 69.8% of the Markov chain model. In terms of prediction convergence speed, this invention leads with an excellent performance of 94.2%, outperforming the 55.1% of basic association rules. In terms of computational efficiency improvement, this invention also maintains a significant advantage with 87.3%. These data fully verify that the technical solution of this invention, which calculates prior belief values ​​and diagnostic information values ​​through bidirectional information transmission paths and iteratively optimizes the conditional dependencies between nodes by combining conditional mutual information, has significant performance advantages. In particular, it demonstrates higher accuracy and efficiency when dealing with complex log event dependencies, providing reliable technical support for practical anomaly detection and fault location applications.

[0118] In one optional implementation, combining the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relationship network, while simultaneously calculating the transition probability between adjacent node pairs. The optimal log archiving strategy is selected based on the maximum likelihood criterion, including:

[0119] The information transmission weight is calculated by multiplying the cosine similarity of the event correlation matrix by the inverse of the shortest path distance between node pairs.

[0120] On the message passing link, the downlink decomposition message of each node is calculated. The downlink decomposition message is the sum of the products of the information passing weights of all the neighboring nodes of the node and their historical downlink messages. The uplink aggregation message of each node is calculated. The uplink aggregation message is the sum of the products of the information passing weights of all the neighboring nodes of the node and their historical uplink messages. The downlink decomposition message and the uplink aggregation message are added together to obtain the total bidirectional message of the node.

[0121] The posterior probability is multiplied by the information transmission weight to obtain the transition probability between node pairs; the posterior probability and the transition probability are substituted into the log-likelihood function, and the posterior probability and the transition probability are iteratively updated until the log-likelihood function converges; the optimal archiving scheme for log events is determined based on the updated posterior probability and the transition probability.

[0122] The information transmission weight is calculated using the cosine similarity of the event correlation matrix and the inverse of the shortest path distance between node pairs. The event correlation matrix describes the strength of association between log events, where each element represents the degree of association between corresponding event pairs. Cosine similarity is an index that measures the similarity between two vectors, with values ​​ranging from -1 to 1; a higher value indicates a higher similarity. In this implementation, the row vectors corresponding to each event in the event correlation matrix can be extracted, and the cosine similarity between these vectors can be calculated. For example, for the events "server response timeout" and "database connection failure," the calculated cosine similarity is 0.75, obtained by extracting their corresponding row vectors from the event correlation matrix.

[0123] The shortest path distance between nodes refers to the length of the shortest path connecting two nodes in a relational network. The reciprocal of the shortest path distance reflects the proximity between nodes; the shorter the distance, the larger the reciprocal, indicating a closer connection between the nodes. The shortest path distance can be calculated using algorithms such as breadth-first search or Dijkstra's algorithm. For example, the shortest path distance from the event "server response timeout" to the event "database connection failed" is 2, and its reciprocal is 0.5.

[0124] Multiplying the cosine similarity by the inverse of the shortest path distance yields the message passing weight. The message passing weight reflects the strength of information flow between two event nodes and is a crucial parameter in message passing algorithms. For example, the message passing weight for the two events mentioned above is 0.75 × 0.5 = 0.375. For all adjacent node pairs in the relational network, their message passing weights need to be calculated, forming a weight matrix. This matrix has the same structure as the adjacency matrix of the relational network, but its elements are message passing weights rather than simple connection relationships.

[0125] In the message passing chain, the downlink decomposition message for each node is calculated. The downlink decomposition message is the information passed from upper-level nodes to lower-level nodes, reflecting the influence of upper-level nodes on lower-level nodes. The calculation process involves the node's neighboring nodes, message passing weights, and historical downlink messages. Specifically, the downlink decomposition message of a node is equal to the sum of the products of the message passing weights of all its neighboring nodes and their corresponding historical downlink messages. In the initial state, the historical downlink messages of all nodes can be set to predefined initial values, such as 1.0. For example, for the event node "Application Service Crash," assuming its neighboring nodes include "Memory Leak" and "Thread Deadlock," with message passing weights of 0.42 and 0.38 respectively, and historical downlink messages of 1.0 and 1.0 respectively, then the downlink decomposition message for "Application Service Crash" is 0.42 × 1.0 + 0.38 × 1.0 = 0.8.

[0126] Simultaneously, the uplink aggregate message for each node is calculated. The uplink aggregate message is information passed from lower-level nodes to upper-level nodes, reflecting the feedback from lower-level nodes to upper-level nodes. The calculation process is similar to the downlink decomposition message, involving the node's neighboring nodes, information transmission weights, and historical uplink messages. Specifically, the uplink aggregate message of a node is equal to the sum of the products of the information transmission weights of all its neighboring nodes and their corresponding historical uplink messages. In the initial state, the historical uplink messages of all nodes can be set to a predefined initial value, such as 1.0. For example, for the event node "Database connection pool exhausted," assuming its neighboring nodes include "SQL query timeout" and "Transaction rollback increase," with information transmission weights of 0.35 and 0.28 respectively, and historical uplink messages of 1.0 and 1.0 respectively, then the uplink aggregate message for "Database connection pool exhausted" is 0.35 × 1.0 + 0.28 × 1.0 = 0.63.

[0127] Adding the downlink decomposed message to the uplink aggregated message yields the total bidirectional message count for a node. This total reflects the node's importance and information flow within the network. For example, if the downlink decomposed message count for the event node "increased network latency" is 0.75 and the uplink aggregated message count is 0.58, then its total bidirectional message count is 0.75 + 0.58 = 1.33. This total bidirectional message count can serve as an indicator of node priority, which can be used to formulate subsequent log archiving strategies.

[0128] Multiplying the posterior probability by the information transmission weight yields the transition probability between node pairs. The posterior probability is calculated using the aforementioned bidirectional propagation probabilistic graph inference method and reflects the estimated probability of a node's state. The transition probability represents the probability of information being transmitted from one node to another and is a crucial basis for constructing log archiving strategies. For example, if the posterior probability of the event "server response timeout" is 0.25, and the information transmission weight between it and the event "database connection failure" is 0.375, then the transition probability between them is 0.25 × 0.375 = 0.09375.

[0129] Substituting the posterior probability and transition probability into the log-likelihood function, the likelihood value under the current parameter settings is calculated. The log-likelihood function is an indicator for evaluating the goodness of fit of model parameters; a larger value indicates a higher degree of fit between the model and the observed data. In this embodiment, the log-likelihood function can be constructed based on the observed log event sequence and the event transition probabilities predicted by the model. For example, if the observed log event sequence contains events A, B, and C, and the model predicts transition probabilities P(A) = 0.3, P(B|A) = 0.4, and P(C|B) = 0.5, then the log-likelihood value can be obtained through relevant calculations; assuming the result is -2.5.

[0130] The posterior and transition probabilities are iteratively updated until the log-likelihood function converges. This is an optimization process aimed at finding parameter settings that maximize the log-likelihood function. In each iteration, the parameters can be updated using gradient ascent, adjusting the parameter values ​​according to the direction of the partial derivatives of the log-likelihood function with respect to each parameter. Termination conditions can be set for the iteration process, such as reaching the maximum number of iterations (e.g., 100) or the change in the log-likelihood value being less than a specific threshold (e.g., 0.001). For example, after multiple iterations, the model parameters are continuously adjusted, and the log-likelihood value gradually increases from the initial -2.5 to -1.8, eventually converging.

[0131] Based on the updated posterior and transition probabilities, the optimal archiving scheme for log events is determined. The log archiving scheme includes factors such as log classification, storage location, and retention period. When determining the optimal archiving scheme, the following factors can be considered: the importance of the event (reflected by the posterior probability), the correlation between events (reflected by the transition probability), storage cost, and query efficiency. For example, events can be classified into three importance levels—high, medium, and low—based on their posterior probabilities. Events with a posterior probability greater than 0.5 are classified as high importance with a retention period of 90 days; events with a posterior probability between 0.2 and 0.5 are classified as medium importance with a retention period of 30 days; and events with a posterior probability less than 0.2 are classified as low importance with a retention period of 7 days.

[0132] Simultaneously, considering the transition probability between events, event pairs with higher transition probabilities (e.g., a transition probability greater than 0.3) are archived in the same storage area to improve query efficiency. For example, the transition probability between the event "load balancer switching" and the event "service instance restart" is 0.35, exceeding the threshold of 0.3, therefore these two types of events are archived in the same storage area. Furthermore, different indexing strategies can be set based on the total sum of bidirectional messages for events; events with a higher total sum of bidirectional messages (e.g., greater than 1.5) are indexed with finer-grained indexes to accelerate queries.

[0133] In practical applications, this method can be applied to log management systems for large-scale internet services. For example, an e-commerce platform generates massive amounts of log data daily, including user access logs, system operation logs, and error logs. By constructing a relationship network and applying a message passing algorithm, the correlations and importance between log events can be identified. Analysis results show that the posterior probability of the event "database master-slave switch" is 0.62, the posterior probability of the event "caching service restart" is 0.48, and the posterior probability of the event "page load timeout" is 0.35. Based on this data, an automatic log archiving strategy is generated: logs related to "database master-slave switch" are classified as high importance and stored in a high-performance storage area with a retention period of 90 days; logs related to "caching service restart" are classified as medium importance and stored in a standard storage area with a retention period of 30 days; logs related to "page load timeout" are classified as low importance, but because their transition probability with high-importance events reaches 0.4, they are also stored in a high-performance storage area with a retention period of 30 days.

[0134] By applying this optimal archiving strategy, the platform's log management efficiency is significantly improved: storage space utilization is increased by 25%, critical event query response time is reduced by 40%, and fault diagnosis time is shortened by an average of 35%. This demonstrates that the technical solution provided by this invention can effectively solve the data archiving problem in large-scale log systems, improving log management efficiency and data value.

[0135] The message passing algorithm provided in this implementation combines the advantages of graph neural networks and probabilistic reasoning. Through top-down event decomposition and bottom-up feature aggregation, it comprehensively captures the complex relationships between log events. The optimization method based on the maximum likelihood criterion can find model parameters that best match the actual data distribution, providing a scientific basis for log archiving strategies. This method is applicable to various large-scale log systems and has strong versatility and practical value.

[0136] This invention relates to a log processing system based on dual-path processing and intelligent archiving, the system comprising:

[0137] The first unit performs semantic segmentation and structured parsing on the raw log data to obtain log events;

[0138] The second unit calculates the causal relationship chain between the log events using a Bayesian network, identifies potential risk propagation paths based on the causal relationship chain, and diverts the log events to different processing and optimization units based on the potential risk propagation paths.

[0139] The third unit extracts knowledge entities from log events, establishes connection edges based on business dependencies and semantic similarity between knowledge entities to form a relationship network, and performs correlation analysis on log events based on the relationships in the relationship network to obtain an event correlation matrix. The unit then calculates the conditional dependencies between nodes in the relationship network using a bidirectional probabilistic graph inference method, and combines this with the event correlation matrix to perform top-down event decomposition and bottom-up feature aggregation in the relationship network using a message passing algorithm. Simultaneously, it calculates the transition probabilities between adjacent node pairs and selects the optimal log archiving strategy based on the maximum likelihood criterion.

[0140] A third aspect of the present invention provides an electronic device, comprising:

[0141] processor;

[0142] Memory used to store processor-executable instructions;

[0143] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0144] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0145] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log processing method based on dual-path processing and intelligent archiving, characterized in that, include: Semantic segmentation and structured parsing are performed on the raw log data to obtain log events; The causal relationship chain between the log events is calculated by Bayesian network, potential risk propagation paths are identified based on the causal relationship chain, and log events are diverted to different processing and optimization units based on the potential risk propagation paths. Knowledge entities are extracted from log events, and connection edges are established based on the business dependencies and semantic similarity between knowledge entities to form a relationship network; Based on the relationships within the aforementioned relationship network, an event correlation matrix is ​​obtained by performing correlation analysis on log events. This includes: extracting event subjects, event objects, and event attributes from log events as knowledge entities; calculating the cosine similarity of word vectors between the knowledge entities to obtain semantic similarity; obtaining the business dependencies between the knowledge entities based on system call chains and event temporal dependencies, and weighting and summing these dependencies with the semantic similarity to obtain the entity correlation strength; constructing an entity relationship matrix based on the entity correlation strength; and establishing connection edges between entity pairs in the entity relationship matrix that are greater than a preset correlation threshold to form a relationship network. Based on the associations in the relationship network, the degree centrality and betweenness centrality of the event nodes in the relationship network are calculated to obtain the node importance, and the shortest path hop count between the event nodes is counted to obtain the path relevance. Based on the weighted combination of node importance and path relevance, an event relevance matrix is ​​generated. The conditional dependencies between nodes in the relational network are calculated using a bidirectional probabilistic graph inference method. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relational network. At the same time, the transition probability between adjacent node pairs is calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

2. The method according to claim 1, characterized in that, Calculating causal chains among log events using Bayesian networks, identifying potential risk propagation paths based on these chains, and then routing log events to different processing optimization units based on these potential propagation paths include: Log events are constructed as nodes, the dependencies between log events are identified to form an edge set, a directed acyclic graph is constructed, the conditional probability between nodes is determined by the count ratio between the nodes and their parent nodes in the directed acyclic graph, and the conditional probability transitivity between nodes is calculated sequentially based on the topological sequence of the directed acyclic graph to obtain the causal relationship chain between the log events. The risk propagation intensity is obtained by continuously multiplying the conditional probability of each node in the causal relationship chain with the corresponding risk weight. The cumulative risk value of the node is updated iteratively, and the unvisited node with the smallest cumulative risk value is searched. The risk propagation intensity is added to the cumulative risk value of the current node to update the cumulative risk value of the adjacent nodes until the cumulative risk value of all nodes converges or the maximum number of iterations is reached. Calculate the maximum risk propagation intensity of the potential risk propagation path in which the log event is located, count the number of potential risk propagation paths in which the log event participates, and obtain the comprehensive risk score of the log event by weighted summation of the maximum risk propagation intensity and the number of potential risk propagation paths. Based on the comprehensive risk score, the log event is diverted to the corresponding processing optimization unit for processing.

3. The method according to claim 2, characterized in that, In the directed acyclic graph, the conditional probability between nodes is determined by the ratio of node counts to parent node counts. Based on the topological sequence of the directed acyclic graph, the conditional probability transitivity between nodes is calculated sequentially to obtain the causal chain between the log events, including: Count the number of co-occurrences of each node with its corresponding parent node and the number of individual occurrences of the corresponding parent node. Divide the number of co-occurrences by the number of individual occurrences and perform Laplace smoothing on the result to obtain the conditional probability. Record all conditional probabilities in the conditional probability matrix. The Kahn algorithm is used to obtain the topological sequence of the directed acyclic graph. Based on the topological sequence, the path from the direct predecessor node to the source node of each node is identified in turn. The conditional probability transitivity relationship is obtained by calculating the continuous product of the conditional probabilities between adjacent nodes on the path. All conditional probability transitivity relationships are recorded in the transitivity matrix. Each element in the conditional probability matrix is ​​multiplied by a preset direct influence weight factor, and each element in the transitive probability matrix is ​​multiplied by a preset indirect influence weight factor. The two multiplied matrices are then added together to obtain a causal strength matrix. Node pairs in the causal strength matrix that satisfy a preset strength threshold are extracted to construct a causal relationship chain.

4. The method according to claim 1, characterized in that, The calculation of conditional dependencies between nodes in the relational network using a bidirectional propagation probabilistic graphical inference method includes: The co-occurrence frequency between event nodes in a statistical relationship network is calculated, and the information transmission probability of event node pairs is obtained by dividing the co-occurrence frequency by the total number of events. A bidirectional information transmission path is established. The prior belief value of the target node is calculated from the parent node downwards. The prior belief value is the weighted product of the parent node's belief and the information transmission probability. The diagnostic information value is calculated from the child node upwards. The diagnostic information value is the cumulative product of the child node's information and the information transmission probability. The prior belief value and the diagnostic information value are normalized and then multiplied to obtain the posterior probability of the node. The conditional mutual information between any two adjacent nodes is calculated using the posterior probability. The information transmission probability of the nodes is updated based on the difference between the conditional mutual information and the dependency threshold. The updated information transmission probability is substituted into the bidirectional information transmission path to generate a new posterior probability. The conditional dependency relationship between nodes is obtained through iterative optimization.

5. The method according to claim 4, characterized in that, Combining the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relationship network. Simultaneously, the transition probabilities between adjacent node pairs are calculated. Based on the maximum likelihood criterion, the optimal log archiving strategy is selected, including: The information transmission weight is calculated by multiplying the cosine similarity of the event correlation matrix by the inverse of the shortest path distance between node pairs. On the message passing link, the downlink decomposition message of each node is calculated. The downlink decomposition message is the sum of the products of the information passing weights of all the neighboring nodes of the node and their historical downlink messages. The uplink aggregation message of each node is calculated. The uplink aggregation message is the sum of the products of the information passing weights of all the neighboring nodes of the node and their historical uplink messages. The downlink decomposition message and the uplink aggregation message are added together to obtain the total bidirectional message of the node. The posterior probability is multiplied by the information transmission weight to obtain the transition probability between node pairs; the posterior probability and the transition probability are substituted into the log-likelihood function, and the posterior probability and the transition probability are iteratively updated until the log-likelihood function converges; the optimal archiving scheme for log events is determined based on the updated posterior probability and the transition probability.

6. A log processing system based on dual-path processing and intelligent archiving, used to implement the method as described in any one of claims 1-5, characterized in that, include: The first unit performs semantic segmentation and structured parsing on the raw log data to obtain log events; The second unit calculates the causal relationship chain between the log events using a Bayesian network, identifies potential risk propagation paths based on the causal relationship chain, and diverts the log events to different processing and optimization units based on the potential risk propagation paths. The third unit extracts knowledge entities from log events, establishes connection edges based on the business dependencies and semantic similarity between knowledge entities, and forms a relationship network. Based on the relationships in the aforementioned relationship network, an event correlation matrix is ​​obtained by performing correlation analysis on log events. The conditional dependencies between nodes in the relational network are calculated using a bidirectional probabilistic graph inference method. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relational network. At the same time, the transition probability between adjacent node pairs is calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images