Log processing method and system based on two-way processing and intelligent archiving

Abstract

The invention provides a log processing method and system based on two-way processing and intelligent archiving, and relates to the technical field of log processing, and the method comprises the steps: carrying out the semantic segmentation and structural analysis of an original log, calculating a causal relationship chain through a Bayesian network, recognizing a risk propagation path, constructing a knowledge graph based on the fusion of real-time features and historical features, and carrying out the intelligent archiving. And a log archiving strategy is optimized by adopting a two-way propagation probability graph reasoning method. According to the method, the log processing efficiency is effectively improved, the system security is enhanced, and the storage cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of log processing, and in particular to a log processing method and system based on dual-path processing and intelligent archiving. Background Art

[0002] As information systems continue to expand in scale and become increasingly complex, log data, as a crucial record of system operational status, has become crucial for ensuring stable system operation. Traditional log processing methods, which primarily rely on techniques such as keyword matching and rule filtering, face significant challenges in handling massive amounts of heterogeneous log data. Currently, log processing technology has evolved from simple data collection and storage to intelligent analysis and prediction, integrating technologies from multiple fields, including natural language processing, machine learning, and knowledge graphs.

[0003] Especially in large-scale distributed systems, log data is characterized by large volumes, diverse types, and complex structures. This requires more efficient and intelligent processing methods to mine the system status information and potential risks contained in the logs. The industry has begun exploring log analysis methods based on semantic understanding and causal reasoning. By building correlations between log events, these methods enable rapid location and prediction of system anomalies. Simultaneously, log archiving strategies are evolving from traditional time decay models to intelligent archiving based on event importance and relevance.

[0004] Existing technologies struggle to effectively process the correlations between heterogeneous log data from multiple sources, making it difficult to accurately track the root causes and propagation paths of issues in complex systems. This is especially true in highly concurrent environments, where interactions between system components are complex and intricate. Relying solely on single-dimensional analysis methods makes it difficult to identify potential risk propagation links. Traditional log analysis methods lack the ability to extract features across the time dimension and are unable to simultaneously consider both real-time and historical statistical features. This results in poor analytical results for sudden events or long-term, cumulative issues, particularly those exhibiting gradual system degradation trends, where detection sensitivity is insufficient. Summary of the Invention

[0005] The embodiments of the present invention provide a log processing method and system based on dual-path processing and intelligent archiving, which can solve the problems in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a log processing method based on dual-path processing and intelligent archiving, comprising: Perform semantic segmentation and structural analysis on the original log data to obtain log events; Calculating the causal relationship chain between the log events through a Bayesian network, identifying potential risk propagation paths based on the causal relationship chain, and diverting the log events to different processing optimization units based on the potential risk propagation paths; Knowledge entities are extracted from log events, and connection edges are established based on the business dependencies and semantic similarities between the knowledge entities to form a relational network. Based on the association relationships in the relational network, the log events are subjected to correlation analysis to obtain an event correlation matrix. The conditional dependencies between nodes in the relational network are calculated using a probabilistic graph inference method with bidirectional propagation. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relational network. The transition probabilities between adjacent node pairs are calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

[0007] Calculating the causal relationship chain between the log events through a Bayesian network, identifying potential risk propagation paths based on the causal relationship chain, and diverting the log events to different processing optimization units based on the potential risk propagation paths includes: Log events are constructed as nodes, dependencies between the log events are identified to form an edge set, a directed acyclic graph is constructed, conditional probabilities between nodes are determined in the directed acyclic graph by using a statistical ratio of counts between nodes and their parent nodes, and conditional probability transfer relationships between nodes are sequentially calculated based on a topological sequence of the directed acyclic graph to obtain a causal chain between the log events; The continuous product of the conditional probability of each node on the causal chain and the corresponding risk weight is used as the risk propagation strength. The cumulative risk value of the node is iteratively updated and the unvisited node with the smallest cumulative risk value is searched. The cumulative risk value of the adjacent nodes is updated by adding the risk propagation strength to the cumulative risk value of the current node until the cumulative risk values ​​of all nodes converge or the maximum number of iterations is reached, thereby obtaining the potential risk propagation path. Calculate the maximum risk propagation intensity of the potential risk propagation path where the log event is located, count the number of potential risk propagation paths involved in the log event, and take the weighted sum of the maximum risk propagation intensity and the number of potential risk propagation paths to obtain a comprehensive risk score for the log event. According to the comprehensive risk score, the log event is diverted to the corresponding processing optimization unit for processing.

[0008] Determining the conditional probability between nodes in the directed acyclic graph by using the count statistics ratio of the node to the parent node, and sequentially calculating the conditional probability transfer relationship between nodes based on the topological sequence of the directed acyclic graph to obtain the causal relationship chain between the log events includes: Count the number of co-occurrences of each node with its corresponding parent node and the number of independent occurrences of the corresponding parent node, divide the co-occurrence number by the independent occurrence number, perform Laplace smoothing on the result to obtain the conditional probability, and record all conditional probabilities in a conditional probability matrix; A Kahn algorithm is used to obtain a topological sequence of the directed acyclic graph, and based on the topological sequence, a path from the direct predecessor node of each node to the source node is sequentially identified, and a conditional probability transfer relationship is obtained by calculating the continuous product of conditional probabilities between adjacent nodes on the path, and all conditional probability transfer relationships are recorded in a transfer probability matrix; Multiply each element in the conditional probability matrix by a preset direct influence weight factor, multiply each element in the transfer probability matrix by a preset indirect influence weight factor, add the two multiplied matrices together to obtain a causal strength matrix, and extract the node pairs that meet the preset strength threshold in the causal strength matrix to construct a causal relationship chain.

[0009] Knowledge entities are extracted from log events, and connection edges are established based on the business dependencies and semantic similarities between the knowledge entities to form a relationship network. Based on the association relationships in the relationship network, the log events are analyzed for correlation to obtain an event correlation matrix including: Extract event subjects, event objects, and event attributes from log events as knowledge entities; calculate the cosine similarity of word vectors between the knowledge entities to obtain semantic similarity, obtain the business dependency between the knowledge entities based on the system call chain and event timing dependency, and perform weighted summation with the semantic similarity to obtain entity association strength; construct an entity relationship matrix based on the entity association strength, and establish connecting edges between entity pairs in the entity relationship matrix that are greater than a preset association threshold to form a relationship network; Based on the association relationships in the relationship network, the degree centrality and betweenness centrality of the event nodes in the relationship network are calculated to obtain the node importance, the number of shortest path hops between the event nodes is counted to obtain the path relevance, and based on the weighted combination of the node importance and the path relevance, an event relevance matrix is ​​generated.

[0010] Calculating the conditional dependency between nodes in the relationship network using a probabilistic graph reasoning method using bidirectional propagation includes: Counting the co-occurrence frequency between event nodes in the relationship network, and dividing the co-occurrence frequency by the total number of events to obtain the information transmission probability of the event node pair; Establish a bidirectional information transmission path, calculate the prior belief value of the target node from the parent node downward, the prior belief value is the weighted product of the parent node belief and the information transmission probability, calculate the diagnostic information value from the child node upward, the diagnostic information value is the cumulative product of the child node information and the information transmission probability, and multiply the normalized prior belief value and the diagnostic information value to obtain the posterior probability of the node; The conditional mutual information between any adjacent node pair is calculated by using the posterior probability, the information transmission probability of the node is updated based on the difference between the conditional mutual information and a dependence threshold value, the updated information transmission probability is substituted into the bidirectional information transmission path to generate a new posterior probability, and the conditional dependence relationship between nodes is obtained through iterative optimization.

[0011] In combination with the event correlation matrix, event decomposition from top to bottom and feature aggregation from bottom to top are performed in the relationship network through a message passing algorithm, the transition probability between adjacent node pairs is calculated, and the optimal log archiving strategy is selected based on a maximum likelihood criterion, including: The information transmission weight is calculated by using the product of the cosine similarity of the event correlation matrix and the reciprocal of the shortest path distance between node pairs. On the message passing link, the downlink decomposition message of each node is calculated, the downlink decomposition message being the sum of the product of the information transmission weight of all adjacent nodes of the node and the historical downlink message thereof, the uplink aggregation message of each node is calculated, the uplink aggregation message being the sum of the product of the information transmission weight of all adjacent nodes of the node and the historical uplink message thereof, and the bidirectional message sum of the node is obtained by adding the downlink decomposition message and the uplink aggregation message. The transition probability between node pairs is obtained by multiplying the posterior probability by the information transmission weight, the posterior probability and the transition probability are substituted into a log likelihood function, the posterior probability and the transition probability are iteratively updated until the log likelihood function converges, and the optimal archiving scheme of the log event is determined based on the updated posterior probability and the transition probability.

[0012] In a second aspect of the embodiment of the present application, a log processing system based on double-path processing and intelligent archiving is provided, including: A first unit performs semantic segmentation and structured analysis on original log data to obtain log events. A second unit calculates a causal relationship chain between the log events through a Bayesian network, identifies a potential risk propagation path based on the causal relationship chain, and shunts the log events to different processing optimization units based on the potential risk propagation path. A third unit extracts knowledge entities from the log events, establishes connection edges according to business dependence relationships and semantic similarities between the knowledge entities, forms a relationship network, performs correlation analysis on the log events based on the association relationships of the relationship network to obtain an event correlation matrix, calculates the conditional dependence relationships between nodes in the relationship network through a bidirectional propagation probabilistic graph reasoning method, in combination with the event correlation matrix, performs event decomposition from top to bottom and feature aggregation from bottom to top in the relationship network through a message passing algorithm, calculates the transition probability between adjacent node pairs, and selects the optimal log archiving strategy based on a maximum likelihood criterion.

[0013] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0014] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0015] The beneficial effects of this application are as follows: The present invention constructs a causal chain through semantic segmentation and Bayesian network, realizes the intelligent diversion and precise processing of log events, effectively improves the system's processing efficiency for complex log data, and reduces computing resource consumption.

[0016] The present invention combines real-time feature vectors and historical statistical feature vectors for feature fusion, constructs a complete log event knowledge graph, enhances the system's ability to identify abnormal events, makes log processing more comprehensive and accurate, and improves the accuracy of risk warnings.

[0017] The present invention adopts a bidirectional probabilistic graph reasoning method and a message passing algorithm to realize the top-down decomposition and bottom-up aggregation of log events, optimize the log archiving strategy, significantly improve the intelligence level of log management and data storage efficiency, and provide a high-quality data foundation for subsequent log analysis and mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of a flow chart of a log processing method based on dual-path processing and intelligent archiving according to an embodiment of the present invention; Figure 2 This is a thermal diagram of the risk propagation intensity of log events; Figure 3 Schematic diagram of the performance comparison of bidirectional propagation probabilistic graph inference methods. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] Figure 1 FIG. 1 is a flow chart of a log processing method based on dual-path processing and intelligent archiving according to an embodiment of the present invention. Figure 1 As shown, the method includes: Perform semantic segmentation and structural analysis on the original log data to obtain log events; The causal relationship chain between the log events is calculated through a Bayesian network, and the potential risk propagation path is identified based on the causal relationship chain. The log events are then diverted to different processing optimization units based on the potential risk propagation path. In the processing optimization unit, the burst characteristics and spectral characteristics of the log events are captured in real time through a sliding time window to construct a real-time feature vector. Time series feature decomposition and quantile sampling are performed based on historical data to generate a historical statistical feature vector. The real-time feature vector and the historical statistical feature vector are then fused to obtain a log event fusion feature vector. Constructing a knowledge entity and a relationship network of log events based on the fusion feature vector of the log events, extracting semantic associations and business dependencies between the knowledge entities to obtain a log event knowledge graph, and performing a correlation analysis on the log events based on the association relationships in the relationship network to obtain an event correlation matrix; The conditional dependency relationships between nodes in the log event knowledge graph are calculated through a probabilistic graph reasoning method with bidirectional propagation. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the graph structure. At the same time, the transition probability between adjacent node pairs is calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

[0022] In an optional embodiment, calculating the causal relationship chain between the log events through a Bayesian network, identifying potential risk propagation paths based on the causal relationship chain, and diverting the log events to different processing optimization units based on the potential risk propagation paths includes: Log events are constructed as nodes, dependencies between the log events are identified to form an edge set, a directed acyclic graph is constructed, conditional probabilities between nodes are determined in the directed acyclic graph by using a statistical ratio of counts between nodes and their parent nodes, and conditional probability transfer relationships between nodes are sequentially calculated based on a topological sequence of the directed acyclic graph to obtain a causal chain between the log events; The continuous product of the conditional probability of each node on the causal relationship chain and the corresponding risk weight is taken as the risk propagation strength. The cumulative risk value of the node is updated by iteration, and the cumulative risk value of the adjacent node is updated by adding the risk propagation strength to the cumulative risk value of the current node, until the cumulative risk values of all nodes converge or reach the maximum iteration number, to obtain the potential risk propagation path. The maximum risk propagation strength of the potential risk propagation path where the log event is located is calculated, the number of potential risk propagation paths in which the log event participates is counted, and the maximum risk propagation strength and the number of potential risk propagation paths are weighted and summed to obtain the comprehensive risk score of the log event. The log event is shunted to the corresponding processing optimization unit for processing according to the comprehensive risk score.

[0023] The log events are constructed as nodes, and the dependency relationships between the log events are identified to form a set of edges. The log events can be various log records generated during running, such as error logs, warning logs, operation logs, etc. The dependency relationships can be identified through time sequence analysis, content association analysis, etc. For example, in a server cluster environment, there are the following log events: the CPU usage of node A exceeds 90%, the memory usage of node A exceeds 85%, the service response of node B is timed out, and the connection of node C fails, etc. By analyzing the timestamps and context information of these log events, it can be identified that there is a dependency relationship between the resource occupation exception of node A and the service response timeout of node B.

[0024] Based on the identified log event nodes and dependency relationship edges, a directed acyclic graph is constructed. In the graph, the direction of the edge represents the direction of the causal relationship, from the cause to the result. For the above example, the directed acyclic graph constructed contains a directed edge from "the CPU usage of node A exceeds 90%" to "the service response timeout of node B", indicating that the former is the cause of the latter.

[0025] In the constructed directed acyclic graph, the conditional probability between nodes is determined by the count ratio of the node and the parent node. Specifically, for each node in the graph, the number of times that the node and each parent node appear together is counted, and the total number of times that the parent node appears. The ratio of the two is the conditional probability. For example, if in 100 observations, "the CPU usage of node A exceeds 90%" appears 50 times, and "the service response timeout of node B" appears 40 times in these 50 times, then the conditional probability of "the service response timeout of node B" under the condition of "the CPU usage of node A exceeds 90%" is 40 / 50=0.8.

[0026] Based on the topological sequence of a directed acyclic graph, the conditional probability transfer relationships between nodes are calculated sequentially to obtain the causal chain between log events. The topological sequence ensures that when calculating the probability of a node, the probabilities of all its parent nodes have already been calculated. In practical applications, dynamic programming algorithms can be used to optimize computational efficiency. For example, in a directed acyclic graph, there is a path: "Node A's CPU utilization exceeds 90%" → "Node B's service response timed out" → "Node C's connection failed." By calculating the conditional probability transfer, the causal chain formed by these three events and its strength can be determined.

[0027] After obtaining the causal chain, the risk propagation intensity is calculated by multiplying the conditional probability of each node in the chain by the corresponding risk weight. Risk weights can be pre-set based on factors such as the severity and impact of the log event. For example, the risk weight of service interruption is 0.9, while the risk weight of performance degradation is 0.6. For the causal chain in the above example, assuming the risk weights of the three events are 0.7, 0.8, and 0.9, respectively, and the conditional probabilities are 0.8 and 0.7, respectively, the risk propagation intensity of the chain is 0.7 × 0.8 × 0.8 × 0.7 × 0.9 = 0.28224.

[0028] The risk propagation path is identified by iteratively updating the cumulative risk values ​​of nodes and searching for unvisited nodes with the lowest cumulative risk value. Initially, the cumulative risk value of the starting node is set to 0, and that of all other nodes is set to infinity. Each time, the node with the lowest cumulative risk value is selected from the unvisited nodes as the current node. The risk propagation strength is added to the cumulative risk value of the current node to update the cumulative risk values ​​of the adjacent nodes. This process is similar to the modified Dijkstra algorithm, but uses the risk propagation strength as the edge weight. The iterative process continues until the cumulative risk values ​​of all nodes converge or a preset maximum number of iterations (e.g., 100) is reached.

[0029] In a real-world application scenario, consider the following log events: Event E1 represents a database connection timeout, Event E2 represents high application server CPU usage, Event E3 represents a slow cache server response, Event E4 represents a user login failure, and Event E5 represents an order processing delay. The causal chain identified using the above method includes two paths: E2 → E3 → E5 and E1 → E4 → E5. If the risk propagation strength of the first path is 0.35 and the risk propagation strength of the second path is 0.28, both paths constitute potential risk propagation paths.

[0030] After obtaining the potential risk propagation paths, calculate the maximum risk propagation intensity for each log event's potential risk propagation path. For example, event E5 is located on two paths simultaneously, and its maximum risk propagation intensity is max(0.35, 0.28) = 0.35. Also, count the number of potential risk propagation paths that the log event participates in. For example, event E5 participates in two paths, while event E2 participates in only one path.

[0031] The maximum risk propagation intensity and the number of potential risk propagation paths are weighted and summed to obtain the comprehensive risk score for the log event. Weights can be set based on actual needs. For example, the weight for the maximum risk propagation intensity can be set to 0.7, and the weight for the number of paths can be set to 0.3. For event E5, its comprehensive risk score is 0.7 × 0.35 + 0.3 × 2 = 0.245 + 0.6 = 0.845. Based on the calculated comprehensive risk score, the log event is diverted to the corresponding processing optimization unit for processing.

[0032] Specifically, multiple risk level thresholds can be set, such as a high-risk threshold of 0.8, a medium-risk threshold of 0.5, and a low-risk threshold of 0.2. Log events with a comprehensive risk score exceeding the high-risk threshold are diverted to a high-priority processing unit, where strategies such as resource priority protection and real-time monitoring are adopted; log events with scores between the medium and high risk thresholds are diverted to a medium-priority processing unit, where strategies such as regular inspections and abnormal warnings are adopted; and log events with scores below the low-risk threshold are diverted to a low-priority processing unit, where strategies such as batch processing and simplified analysis are adopted.

[0033] In this example, the comprehensive risk score of event E5 is 0.845, which exceeds the high-risk threshold of 0.8. Therefore, it is diverted to the high-priority processing unit, which is allocated more computing resources and has a real-time monitoring mechanism activated. The comprehensive risk score of event E3 is 0.62, which is between the medium and high risk thresholds and is diverted to the medium-priority processing unit. The comprehensive risk score of event E2 is 0.38, which is below the medium-risk threshold and is diverted to the low-priority processing unit.

[0034] This diversion mechanism allows for the rational allocation of processing resources based on the importance of log events within the risk propagation network, prioritizing critical log events that could lead to serious risk propagation, thereby improving risk prevention and control efficiency and resource utilization. For high-risk log events, proactive intervention measures, such as elastic capacity expansion and load balancing adjustments, can be initiated to further reduce the probability of risk propagation.

[0035] Figure 2This is a heat map of the risk propagation intensity of log events. The figure shows the heat map of the risk propagation intensity of log events based on the Bayesian network of the present invention, which intuitively presents the risk propagation relationship and intensity distribution between different log events. The heat map adopts the form of a 7×7 matrix. The horizontal and vertical coordinates represent the source events and target events of risk propagation, including typical security events such as E1 (login failure), E2 (file access), E3 (privilege escalation), E4 (system call), E5 (data leakage), E6 (network anomaly), and E7 (process anomaly). The color depth in the figure represents the magnitude of the risk propagation intensity. White indicates no propagation relationship, and the darker the red, the higher the propagation intensity. The value range is from 0 to 1. It can be observed that the propagation intensity of E2→E3 is 0.85, E3→E4 is 0.75, and E3→E5 is 0.68, which are high-risk propagation paths. These values ​​are the risk propagation intensity calculated by multiplying the conditional probability P(Ej|Ei) and the corresponding risk weight W(Ei). The diagonal area is blank, consistent with the characteristic of a directed acyclic graph (DAG) that has no self-loops, demonstrating the mathematical rigor of the proposed risk propagation model. This heat map effectively demonstrates the proposed method's ability to identify potential risk propagation paths through a Bayesian network, providing a visual basis for subsequent event triage and risk assessment, and demonstrating the algorithm's practicality and accuracy in complex security scenarios.

[0036] In an optional embodiment, determining the conditional probability between nodes in the directed acyclic graph by the count statistical ratio of the node to the parent node, and sequentially calculating the conditional probability transfer relationship between the nodes based on the topological sequence of the directed acyclic graph to obtain the causal relationship chain between the log events includes: Count the number of co-occurrences of each node with its corresponding parent node and the number of independent occurrences of the corresponding parent node, divide the co-occurrence number by the independent occurrence number, perform Laplace smoothing on the result to obtain the conditional probability, and record all conditional probabilities in a conditional probability matrix; A Kahn algorithm is used to obtain a topological sequence of the directed acyclic graph, and based on the topological sequence, a path from the direct predecessor node of each node to the source node is sequentially identified, and a conditional probability transfer relationship is obtained by calculating the continuous product of conditional probabilities between adjacent nodes on the path, and all conditional probability transfer relationships are recorded in a transfer probability matrix; Multiply each element in the conditional probability matrix by a preset direct influence weight factor, multiply each element in the transfer probability matrix by a preset indirect influence weight factor, add the two multiplied matrices together to obtain a causal strength matrix, and extract the node pairs that meet the preset strength threshold in the causal strength matrix to construct a causal relationship chain.

[0037] Count the number of times each node co-occurs with its corresponding parent node, as well as the number of times the corresponding parent node appears alone. Specifically, by analyzing the collected historical log data, the occurrence of various types of log events can be obtained. For example, for the log events of a certain network service system, suppose there is node A representing "increased network delay", node B representing "slowed service response", and node C representing "user session timeout". By analyzing the log records over a period of time, it is statistically found that the number of times node A and node B co-occur is 80 times, and the number of times node A appears alone is 100 times; the number of times node B and node C co-occur is 65 times, and the number of times node B appears alone is 90 times.

[0038] Based on the above statistical data, the process for calculating conditional probabilities is as follows: the number of co-occurrences between a node and its corresponding parent node is divided by the number of individual occurrences of the corresponding parent node to obtain a preliminary conditional probability value. For the above example, the preliminary calculated conditional probability of node B given the occurrence of node A is 80 / 100 = 0.8; the preliminary calculated conditional probability of node C given the occurrence of node B is 65 / 90 = 0.722.

[0039] In order to avoid the zero probability problem and improve the stability of the model, the calculated conditional probability is Laplace smoothed. Laplace smoothing is a commonly used smoothing technique that avoids zero probability by adding a small positive number to the numerator and denominator. For this embodiment, 1 can be added to the numerator and 2 to the denominator, that is, the smoothed conditional probability is calculated as (co-occurrence number + 1) / (single occurrence number + 2). Applied to the above example, the smoothed conditional probability of node B under the condition that node A occurs is (80+1) / (100+2)=81 / 102≈0.794; the smoothed conditional probability of node C under the condition that node B occurs is (65+1) / (90+2)=66 / 92≈0.717.

[0040] After calculating the conditional probabilities between all node pairs, the results are recorded in a conditional probability matrix. This matrix is ​​an N×N matrix, where N is the total number of log event nodes. Each element CPij in the matrix represents the conditional probability of node j given the occurrence of node i. For node pairs without a direct causal relationship, the corresponding matrix element value is set to 0. Based on the above example, assuming that node D represents "database query timeout" and node E represents "cache hit rate reduction," the following conditional probability matrix is ​​formed: In the CP matrix, CPA,B = 0.794, CPB,C = 0.717, CPD,E = 0.682, CPE,B = 0.631, and all other elements are 0.

[0041] Next, we use the Kahn algorithm to obtain the topological sequence of the directed acyclic graph. The Kahn algorithm is an in-degree-based topological sorting algorithm. Its basic idea is to continuously remove nodes with an in-degree of 0 while simultaneously reducing the in-degrees of their adjacent nodes until no nodes remain in the graph. The specific implementation steps are as follows: Initialize a queue Q and add all nodes with an in-degree of 0 to the queue. When the queue is not empty, remove a node n from the queue and add it to the topological sequence. For each adjacent node m of node n, decrement m's in-degree by 1. If m's in-degree becomes 0, add m to queue Q. Repeat this process until the queue is empty. For the above example, assume that in the constructed directed acyclic graph, nodes A and D have an in-degree of 0, node B has an in-degree of 2 (from nodes A and E), node C has an in-degree of 1 (from node B), and node E has an in-degree of 1 (from node D). Applying the Kahn algorithm yields the following topological sequence: A, D, E, B, C.

[0042] Based on the obtained topological sequence, we sequentially identify the paths from each node's direct predecessor to the source node. The source node is the first node in the topological sequence, that is, the node with an in-degree of 0. For each node, we can find all paths from the source node to that node by backtracking to its parent node. In the above example, the direct predecessor of node C is B. Backtracking from B, we can find two paths: A→B→C and D→E→B→C.

[0043] The conditional probability transfer relationships are calculated by continuously multiplying the conditional probabilities between adjacent nodes on each path. For path A→B→C, the conditional probability transfer relationship is CPA,B×CPB,C=0.794×0.717≈0.569; for path D→E→B→C, the conditional probability transfer relationship is CPD,E×CPE,B×CPB,C=0.682×0.631×0.717≈0.308. These conditional probability transfer relationships reflect the strength of indirect causal influence between remote nodes.

[0044] All conditional probability transfer relationships are recorded in the transfer probability matrix. This matrix is ​​also an N×N matrix, where the element TPij represents the conditional probability transfer relationship from node i to node j. If there are multiple paths from node i to node j, the maximum value of the continuous product of the conditional probabilities of these paths is taken as the value of TPij. In the above example, TPA,C = 0.569, TPD,C = 0.308, etc.

[0045] To comprehensively consider the impact of both direct and indirect causal relationships, we introduce a preset direct impact weighting factor α and a preset indirect impact weighting factor β. The sum of these two weighting factors is typically 1, reflecting the relative importance given to direct and indirect impacts in causal relationship assessment. In practice, the values ​​of these two weighting factors can be adjusted based on specific scenarios and requirements. For example, setting α to 0.7 and β to 0.3 would indicate that the impact of direct causality is weighted more heavily than that of indirect causality.

[0046] Multiply each element in the conditional probability matrix by the preset direct influence weight factor α to obtain the weighted direct influence matrix. For the above example, the weighted CPA,B = 0.794 × 0.7 ≈ 0.556, CPB,C = 0.717 × 0.7 ≈ 0.502, and so on. Similarly, multiply each element in the transfer probability matrix by the preset indirect influence weight factor β to obtain the weighted indirect influence matrix. For the above example, the weighted TPA,C = 0.569 × 0.3 ≈ 0.171, TPD,C = 0.308 × 0.3 ≈ 0.092, and so on.

[0047] Adding the two weighted matrices together yields the causal strength matrix. This matrix comprehensively reflects the strength of direct and indirect causal relationships between log event nodes. In the above example, the causal strength between nodes A and B is 0.556 + 0 = 0.556 (because there is no indirect path from A to B), the causal strength between nodes A and C is 0 + 0.171 = 0.171, and the causal strength between nodes B and C is 0.502 + 0 = 0.502, and so on.

[0048] Finally, node pairs that meet the preset strength threshold in the causal strength matrix are extracted to construct a causal chain. The preset strength threshold is a parameter set according to the specific application scenario and is used to filter out node pairs with weak causal relationships. For example, if the preset strength threshold is set to 0.3, the node pairs (A, B), (B, C), (D, E), and (E, B) can be extracted from the above causal strength matrix, while the node pairs (A, C) and (D, C) are filtered out because their causal strength is lower than the threshold. Based on the extracted node pairs, the following causal chains can be constructed: A→B→C and D→E→B→C.

[0049] In practical applications, this method can be applied to log analysis in large-scale distributed systems. For example, the log system of an e-commerce platform records various system events, including user behavior, service calls, and resource usage. By analyzing the causal relationships between these log events, key factors affecting system performance and reliability can be identified. Suppose the analysis results show a strong causal relationship between "database connection pool exhaustion" and "order processing delays," which in turn has a strong causal relationship with "user payment failures." A causal chain can be constructed: "database connection pool exhaustion → order processing delays → user payment failures." Based on this causal chain, system operations personnel can prioritize resolving the "database connection pool exhaustion" issue, preventing serious consequences like "user payment failures" from occurring at the source.

[0050] In an optional embodiment, knowledge entities are extracted from log events, and connecting edges are established based on the business dependencies and semantic similarities between the knowledge entities to form a relationship network; based on the association relationships in the relationship network, the log events are subjected to correlation analysis to obtain an event correlation matrix including: Extract event subjects, event objects, and event attributes from log events as knowledge entities; calculate the cosine similarity of word vectors between the knowledge entities to obtain semantic similarity, obtain the business dependency between the knowledge entities based on the system call chain and event timing dependency, and perform weighted summation with the semantic similarity to obtain entity association strength; construct an entity relationship matrix based on the entity association strength, and establish connecting edges between entity pairs in the entity relationship matrix that are greater than a preset association threshold to form a relationship network; Based on the association relationships in the relationship network, the degree centrality and betweenness centrality of the event nodes in the relationship network are calculated to obtain the node importance, the number of shortest path hops between the event nodes is counted to obtain the path relevance, and based on the weighted combination of the node importance and the path relevance, an event relevance matrix is ​​generated.

[0051] The event subject generally refers to the initiator or carrier of the log event, such as a server, an application, a component, etc.; the event object refers to the action object of the log event, such as a database, a file, a network connection, etc.; and the event attribute refers to information describing the characteristics of the event, such as a status code, a timestamp, a severity, etc. The extraction process can be achieved through rule matching and natural language processing techniques. For example, for the log message "Web server 192.168.1.10 failed to request a database connection at 2023-05-15 08:30:25, error code 504", the extractable knowledge entities include: event subject "Web server 192.168.1.10", event object "database connection", event attribute "error code 504" and "2023-05-15 08:30:25". To improve the efficiency and accuracy of extraction, a domain terminology library and entity recognition template can be established in advance, and specific extraction rules can be designed for different types of log formats.

[0052] After extracting the knowledge entities, the semantic similarity between the knowledge entities is obtained by calculating the cosine similarity of the word vectors. Word vectors are a representation method that maps words to high-dimensional vector spaces, which can capture the semantic information of words. In this embodiment, a pre-trained word vector model, such as a word embedding model trained on a large-scale text corpus, can be used to convert the extracted knowledge entities into word vectors. For composite entities, they can be decomposed into a sequence of single words, and the average of the word vectors of each single word is taken as the vector representation of the entity. The cosine similarity of the vectors of two knowledge entities is calculated, with a value range of -1 to 1, and the larger the value, the higher the semantic similarity. For example, the semantic similarity between "database connection failure" and "database query timeout" is 0.85, while the semantic similarity between "database connection failure" and "network bandwidth shortage" is 0.32.

[0053] At the same time, the business dependency relationship between knowledge entities is obtained according to the system call chain and the event timing dependency relationship. The system call chain records the calling relationship between services or components, which can be obtained by analyzing the system architecture diagram, service registration information or call chain tracking log. The event timing dependency relationship is based on the time sequence of log events, which can be identified by analyzing the time window and frequency pattern of event occurrence. For example, in a microservice architecture, if service A calls service B, and service B calls service C, there is a call chain A→B→C. If it is observed that the error of service B usually occurs shortly after the exception of service A, it can be determined that there is a timing dependency relationship between them. The business dependency relationship can be quantified as a value between 0 and 1, representing the dependency strength. For example, the dependency strength of direct calling relationship can be set to 0.9, the dependency strength of indirect calling relationship (intermediate through a service) can be set to 0.6, and the dependency strength of only timing correlation but no explicit calling relationship can be set to 0.3.

[0054] The entity association strength is calculated by weighting semantic similarity and business dependency. The weighting coefficient can be adjusted based on the specific application scenario. For example, in an environment focused on business logic, the weight of business dependency can be set to 0.7, and the weight of semantic similarity can be set to 0.3. In scenarios primarily focused on text analysis, the weight of semantic similarity can be appropriately increased. Assuming that the semantic similarity of the knowledge entities "Web Server A" and "Application Server B" is 0.6 and the business dependency strength is 0.8, with weights of 0.3 and 0.7, respectively, the entity association strength between the two entities is 0.3 × 0.6 + 0.7 × 0.8 = 0.18 + 0.56 = 0.74.

[0055] An entity relationship matrix is ​​constructed based on the entity association strength. The matrix is ​​an n×n square matrix, where n is the number of knowledge entities, and the elements in the matrix represent the association strength between corresponding entity pairs. To facilitate subsequent processing, elements in the matrix that are less than a certain threshold can be set to 0 to reduce computational complexity. Connecting edges are established between entity pairs in the entity relationship matrix that are greater than the preset association threshold to form a relationship network. The preset association threshold is a parameter set according to actual needs and is used to filter weak associations. For example, the preset association threshold can be set to 0.5, so that connecting edges will be established between entity pairs with entity association strength greater than 0.5. The formed relationship network is a weighted undirected graph, where nodes represent knowledge entities, edges represent association relationships between entities, and the weight of the edge is the entity association strength.

[0056] Based on the associations within a relational network, the degree centrality and betweenness centrality of event nodes in the relational network are calculated to determine node importance. Degree centrality refers to the number of edges directly connected to a node, reflecting the node's direct influence in the network. For example, if an event node has direct connections to multiple other event nodes, its degree centrality is high. In weighted networks, weighted degree centrality can be used, incorporating the weights of the connecting edges into the calculation. Betweenness centrality refers to the number of shortest paths passing through the node, reflecting the node's importance in the flow of information within the network. Calculating betweenness centrality requires traversing the shortest paths between all pairs of nodes in the network and counting the proportion of paths that pass through the target node. For example, in a relational network consisting of 10 event nodes, event node A has a weighted degree centrality of 2.8 (connected to four nodes with edge weights of 0.8, 0.7, 0.6, and 0.7, respectively) and a betweenness centrality of 0.4 (40% of the shortest paths between all pairs of nodes pass through node A).

[0057] Degree centrality and betweenness centrality are normalized and weighted to form a combined value to obtain node importance. Normalization can be performed using maximum normalization, which divides the original value by the maximum value in the network. When performing the weighted combination, the weights of the two centrality metrics can be adjusted based on actual needs. For example, the weight of degree centrality can be set to 0.4 and the weight of betweenness centrality to 0.6. The node importance is calculated as 0.4 × normalized degree centrality + 0.6 × normalized betweenness centrality. Continuing with the above example, assuming the maximum weighted degree centrality in the network is 3.5 and the maximum betweenness centrality is 0.5, the normalized degree centrality of event node A is 2.8 / 3.5 = 0.8, the normalized betweenness centrality is 0.4 / 0.5 = 0.8, and the node importance is 0.4 × 0.8 + 0.6 × 0.8 = 0.8.

[0058] Path correlation is calculated by counting the number of hops in the shortest path between event nodes. The shortest path hop count refers to the number of edges on the shortest path between two nodes. Fewer hops indicate a higher degree of correlation between the two nodes. Path correlation can be defined as the inverse of the shortest path hop count, or by using an exponential decay function to convert the hop count to a value between 0 and 1. For example, the formula "path correlation = 2 raised to the negative power of the hop count" can be used. Thus, the path correlation for directly connected nodes (1 hop count) is 0.5, the path correlation for nodes separated by one node (2 hops) is 0.25, and so on. For unreachable node pairs, the path correlation is defined as 0.

[0059] An event correlation matrix is ​​generated based on a weighted combination of node importance and path relevance. Specifically, for event nodes i and j, their relevance can be defined as the average of the two node importances multiplied by the path relevance, multiplied by an adjustable parameter. The event correlation matrix is ​​an m×m square matrix, where m is the number of event nodes. The elements in the matrix represent the relevance between corresponding pairs of event nodes. For example, if event nodes A and B have importances of 0.8 and 0.6, respectively, the shortest path between them has 1 hop count (direct connection), the path relevance is 0.5, and the adjustable parameter is 2, then the event correlation between A and B is (0.8 + 0.6) / 2×0.5×2=0.7×0.5×2=0.7.

[0060] In practical applications, this method can be applied to the monitoring and fault diagnosis of large distributed systems. For example, a certain e-commerce platform system is composed of multiple microservices, including user service, commodity service, order service, payment service, etc. The system log records the running state and abnormal conditions of each service. By extracting knowledge entities in the log, such as "user service instance 1", "commodity database", "order processing timeout", etc., the semantic similarity and business dependency relationship between entities are calculated to construct a relationship network. Based on this network, the importance of event nodes and the path correlation are calculated to generate an event correlation matrix. The matrix shows that the correlation degree between "database connection pool depletion" and "order processing timeout" is 0.85, the correlation degree between "network delay increase" and "user login failure" is 0.72, and the correlation degree between "cache service restart" and "commodity search slow" is 0.63. Using this correlation degree information, system operation and maintenance personnel can quickly locate the root cause of the fault, prioritize the processing of high-correlation event chains, and improve system reliability and user experience.

[0061] The technical solution provided by the embodiment constructs a knowledge entity relationship network through semantic analysis and business relationship in two dimensions, and calculates event correlation degree based on network topology characteristics, which can accurately reflect the internal relationship between log events in a complex system, and provide strong support for subsequent risk assessment and fault diagnosis. This method is suitable for log analysis scenarios of various information systems and has strong universality and practical value.

[0062] In an optional embodiment, the conditional dependency relationship between nodes in the relationship network is calculated by a bidirectional propagation probabilistic graph reasoning method, which includes: Statistically, the co-occurrence frequency between event nodes in the relationship network is calculated, and the information transmission probability of the event node pair is obtained by dividing the co-occurrence frequency by the total number of events. A bidirectional information transmission path is established, the prior belief value of the target node is calculated from the parent node downward, and the prior belief value is the weighted product of the parent node belief and the information transmission probability. The diagnostic information value is calculated from the child node upward, and the diagnostic information value is the cumulative product of the child node information and the information transmission probability. The posterior probability of the node is obtained by multiplying the normalized prior belief value and the diagnostic information value. The conditional mutual information between any adjacent node pairs is calculated using the posterior probability, the information transmission probability of the node is updated based on the difference between the conditional mutual information and the dependency threshold, the updated information transmission probability is substituted into the bidirectional information transmission path to generate a new posterior probability, and the conditional dependency relationship between nodes is obtained by iterative optimization.

[0063] Event co-occurrence refers to the simultaneous occurrence of two events within a specific time window. For log analysis, you can set an appropriate time window size, such as 5, 15, or 30 minutes, and count co-occurring event pairs within that window. For example, in the log data of a distributed system, it was observed that the "database connection timeout" event and the "application service response delay" event co-occurred 120 times out of 1,000 observations, while the "network traffic surge" event and the "load balancer switchover" event co-occurred 80 times out of 1,000 observations. These statistics reflect preliminary correlations between events.

[0064] Divide the co-occurrence frequency by the total number of events to obtain the information transfer probability for an event-node pair. The information transfer probability measures the strength of information flow between two event nodes. Its value ranges from 0 to 1, with larger values ​​indicating a higher probability of information transfer. For the above example, the information transfer probability between the "Database Connection Timeout" event and the "Application Service Response Delay" event is 120 / 1000 = 0.12, while the information transfer probability between the "Network Traffic Surge" event and the "Load Balancer Switchover" event is 80 / 1000 = 0.08. By calculating the information transfer probability between all event-node pairs, we can construct an information transfer probability matrix. This matrix is ​​an n×n square matrix, where n is the number of event nodes. Each element in the matrix represents the information transfer probability between the corresponding event-node pair.

[0065] A bidirectional information transfer path is established, calculating the prior belief value of the target node from the parent node downward. A parent node refers to the node in the relationship network that points to the target node. The prior belief value represents a preliminary estimate of the target node's state, without considering other evidence. The parent node's belief refers to the parent node's confidence in its own state, determined by prior knowledge or historical data, and represents the probability of the node being activated or in a specific state. For example, for the event node "High CPU Usage," a parent node belief value of 0.8 indicates an 80% probability of the event occurring or being observed. The calculation of the prior belief value involves the parent node's belief value and the probability of information transfer from the parent node to the target node. Specifically, the target node's prior belief value is equal to the sum of the weighted products of the belief values ​​of all parent nodes and the corresponding information transfer probabilities. Weighting factors can be introduced during the calculation to adjust the influence of different parent nodes. For example, for the event node "system performance degradation", its parent nodes include "high CPU usage", "insufficient memory" and "disk I / O blocking", the belief values ​​of these parent nodes are 0.8, 0.6 and 0.7 respectively, the corresponding information transmission probabilities are 0.15, 0.12 and 0.18 respectively, and the weight factors are 0.4, 0.3 and 0.3 respectively. Then the prior belief value of "system performance degradation" is 0.8×0.15×0.4+0.6×0.12×0.3+0.7×0.18×0.3=0.048+0.0216+0.0378=0.1074.

[0066] Diagnostic information values ​​are calculated from child nodes upward. A child node is a node pointed to by a target node in a relationship network. The diagnostic information value represents the impact of feedback from the child node on the target node's state estimate. Child node information refers to the evidence or observations fed back from the child node, reflecting the degree of support the child node provides for the parent node's state inference. Child node information can come from direct observation or measurement, or it can be the output of other reasoning processes. For example, an information value of 0.5 for the event node "SQL query failure" indicates that this observation provides moderate support for its parent node "database connection timeout." In practical systems, child node information can be specific events detected by monitoring tools, such as error logs or abnormal performance indicators. The calculation of the diagnostic information value involves the information value of the child node and the probability of information transfer from the target node to the child node. Specifically, the diagnostic information value of the target node is equal to the sum of the cumulative product of the information values ​​of all child nodes and the corresponding information transfer probabilities. Similar to the calculation of prior belief values, weighting factors can also be introduced to adjust the influence of different child nodes. For example, for the event node "Database connection timeout", its child nodes include "SQL query failure" and "Transaction rollback". The information values ​​of these child nodes are 0.5 and 0.4, respectively. The corresponding information transmission probabilities are 0.2 and 0.15, respectively, and the weight factors are 0.6 and 0.4, respectively. Therefore, the diagnostic information value of "Database connection timeout" is 0.5×0.2×0.6+0.4×0.15×0.4=0.06+0.024=0.084.

[0067] The prior belief value and the diagnostic information value are normalized and then multiplied to obtain the node's posterior probability. Normalization ensures that the calculated result is within the valid range and avoids computational errors caused by excessively large or small values. Normalization methods can use techniques such as maximum normalization or minimum-maximum normalization. For example, the prior belief value and the diagnostic information value can be divided by their respective maximum values ​​in the network to obtain the normalized values. Assuming that the maximum prior belief value in the network is 0.3 and the maximum diagnostic information value is 0.25, the normalized prior belief value of the "Database Connection Timeout" node is 0.1074 / 0.3≈0.358, and the normalized diagnostic information value is 0.084 / 0.25≈0.336. Its posterior probability is 0.358×0.336≈0.12. The posterior probability reflects the estimate of the node state after comprehensively considering contextual information and is the core output of the bidirectional propagation algorithm.

[0068] The conditional mutual information between any adjacent node pairs is calculated using the posterior probability. Conditional mutual information is an indicator that measures the degree of mutual dependence between two variables given a third variable. In this embodiment, conditional mutual information is used to evaluate the strength of the dependency relationship between two event nodes taking into account the influence of other nodes. When calculating conditional mutual information, it is necessary to consider the posterior probability of the node, the joint probability of the node pair, and the marginal probability of the node. For example, for event node A "high server CPU usage" and event node B "increased application response time", assuming that their posterior probabilities are 0.15 and 0.18 respectively, and the joint probability is 0.05, then their conditional mutual information can be obtained by correlation calculation, assuming that the calculation result is 0.023.

[0069] The information transfer probability of a node is updated based on the difference between the conditional mutual information (CMI) and the dependency threshold. The dependency threshold is a pre-set parameter used to determine the significance of the dependency relationship between nodes. If the CMI is greater than the dependency threshold, it indicates a significant dependency between the two nodes; otherwise, it indicates a weak dependency. The information transfer probability can be updated using a gradient descent method, proportionally adjusting the original information transfer probability based on the difference between the CMI and the dependency threshold. For example, if the dependency threshold is set to 0.02 and the learning rate is 0.1, then for event nodes A and B, the difference between their CMI and the dependency threshold is 0.023 - 0.02 = 0.003, and the update to the information transfer probability is 0.003 × 0.1 = 0.0003. If the original information transfer probability is 0.12, the updated information transfer probability is 0.12 + 0.0003 = 0.1203.

[0070] The updated information transfer probability is substituted into the bidirectional information transfer path to generate a new posterior probability. This is an iterative process, continuously adjusting the information transfer probability to ensure that the model output more accurately reflects the true dependencies between nodes. In each iteration, the prior belief values, diagnostic information values, and posterior probabilities for all nodes are recalculated. For example, using the updated information transfer probability, the prior belief value and diagnostic information value for the node "Database Connection Timeout" are recalculated to obtain a new posterior probability.

[0071] Through iterative optimization, conditional dependencies between nodes are derived. The iteration process can be terminated with specified conditions, such as reaching a maximum number of iterations (e.g., 100) or when the change in the posterior probability falls below a specific threshold (e.g., 0.0001). When the iteration terminates, the final information transfer probability matrix reflects the conditional dependencies between nodes. For example, after multiple iterations, the conditional dependency strength between the "database connection timeout" event and the "application service response delay" event is 0.35, and the conditional dependency strength between the "network traffic surge" event and the "load balancer switchover" event is 0.28.

[0072] In practical applications, this method can be applied to anomaly detection and fault diagnosis in large-scale IT infrastructures. For example, a cloud computing platform consists of hundreds of server nodes and thousands of service instances. System logs record a variety of events, such as hardware failures, software errors, and network anomalies. By constructing a relational network and applying a probabilistic graph inference method with bidirectional propagation, conditional dependencies between these events can be identified. Analysis results show that the conditional dependency strength between "slow storage node response" and "data write failure" is 0.42, the conditional dependency strength between "network partition" and "service discovery failure" is 0.38, and the conditional dependency strength between "configuration update" and "service restart" is 0.31. Based on these dependencies, system administrators can construct fault propagation models, predict failure chains, and take appropriate preventive measures.

[0073] The bidirectional propagation probability graph inference method provided in this embodiment calculates the posterior probability of nodes through bottom-up and top-down bidirectional information flow, and then derives conditional dependencies based on the posterior probabilities, which can accurately capture the dependency structure between events in complex systems. This method combines the advantages of Bayesian networks and information theory, is highly robust to incomplete or noisy data, and is suitable for log analysis and fault diagnosis scenarios in large-scale distributed systems. Through an iterative optimization mechanism, this method can continuously adjust model parameters to adapt to dynamic changes in system status and provide more accurate dependency evaluation results.

[0074] Figure 3 This is a performance comparison diagram of the bidirectional propagation probability graph reasoning method, which shows the comparative analysis results of the bidirectional propagation probability graph reasoning method of the present invention and three other traditional methods on four key performance indicators. The black, white and gray bar chart in the figure intuitively presents the performance differences of each method, where the black bar represents the method of the present invention, the dark gray bar represents the traditional Bayesian network, the light gray bar represents the Markov chain model, and the white bar represents the basic association rule method. In terms of anomaly detection accuracy, the present invention reached 92.8%, an increase of 14.4 percentage points compared to the 78.4% of the traditional Bayesian network; in terms of fault location accuracy, the present invention achieved an accuracy of 89.5%, significantly surpassing the 69.8% of the Markov chain model; in the prediction convergence speed index, the present invention leads the basic association rule with an excellent performance of 94.2% with 55.1%; in terms of computing efficiency improvement, the present invention's 87.3% also maintains a significant advantage. These data fully verify that the technical solution of the present invention, which calculates prior belief values ​​and diagnostic information values ​​through a two-way information transmission path and combines conditional mutual information to iteratively optimize conditional dependencies between nodes, has significant performance advantages, especially when processing complex log event dependencies, showing higher accuracy and efficiency, and providing reliable technical support for actual anomaly detection and fault location applications.

[0075] In an optional embodiment, combining the event correlation matrix, performing top-down event decomposition and bottom-up feature aggregation in the relationship network through a message passing algorithm, and calculating transition probabilities between adjacent node pairs, selecting an optimal log archiving strategy based on a maximum likelihood criterion includes: Calculating the information transfer weight using the product of the cosine similarity of the event correlation matrix and the inverse of the shortest path distance between the node pairs; On the message transmission link, a downlink decomposition message of each node is calculated. The downlink decomposition message is the sum of the product of the information transmission weights of all neighboring nodes of the node and their historical downlink messages. An uplink aggregate message is calculated for each node. The uplink aggregate message is the sum of the product of the information transmission weights of all neighboring nodes of the node and their historical uplink messages. The downlink decomposition message and the uplink aggregate message are added together to obtain the sum of the node's two-way messages. The posterior probability is multiplied by the information transfer weight to obtain the transition probability between the node pairs; the posterior probability and the transition probability are substituted into the log-likelihood function, the posterior probability and the transition probability are iteratively updated until the log-likelihood function converges, and the optimal archiving scheme for the log event is determined based on the updated posterior probability and the transition probability.

[0076] The information transfer weight is calculated using the cosine similarity of the event correlation matrix multiplied by the inverse of the shortest path distance between node pairs. The event correlation matrix is ​​a matrix that describes the strength of association between log events, where each element represents the degree of association between corresponding event pairs. Cosine similarity is an indicator that measures the similarity between two vectors, and its value range is between -1 and 1, with larger values ​​indicating higher similarity. In this embodiment, the row vectors corresponding to each event in the event correlation matrix can be extracted, and the cosine similarity between these vectors can be calculated. For example, for the event "server response timeout" and the event "database connection failure", their corresponding row vectors are extracted from the event correlation matrix, and the calculated cosine similarity is 0.75.

[0077] The shortest path distance between a pair of nodes refers to the length of the shortest path connecting two nodes in a relationship network. The reciprocal of the shortest path distance reflects the proximity between nodes. The shorter the distance, the larger the reciprocal value, indicating a closer connection between the nodes. The shortest path distance can be calculated using methods such as breadth-first search or the Dijkstra algorithm. For example, the shortest path distance from the "Server Response Timeout" event to the "Database Connection Failure" event is 2, and its reciprocal is 0.5.

[0078] Multiplying the cosine similarity by the inverse of the shortest path distance yields the message transfer weight. This weight reflects the strength of information flow between two event nodes and is a crucial parameter in the message passing algorithm. For example, the message transfer weight for the two events above is 0.75 × 0.5 = 0.375. For all pairs of adjacent nodes in the relationship network, the message transfer weights must be calculated to form a weight matrix. This matrix has the same structure as the adjacency matrix of the relationship network, but its elements are message transfer weights rather than simple connectivity relationships.

[0079] On the message transmission link, the downlink decomposition message of each node is calculated. Downlink decomposition messages are information transmitted from upper-level nodes to lower-level nodes, reflecting the influence of the upper-level node on the lower-level node. The calculation process involves the node's neighboring nodes, message transmission weights, and historical downlink messages. Specifically, the downlink decomposition message of a node is equal to the sum of the product of the message transmission weights of all its neighboring nodes and the corresponding historical downlink messages. Initially, the historical downlink messages of all nodes can be set to a predefined initial value, such as 1.0. For example, for the event node "Application Service Crash," assuming its neighboring nodes include "Memory Leak" and "Thread Deadlock," with message transmission weights of 0.42 and 0.38, respectively, and historical downlink messages of 1.0 and 1.0, respectively, the downlink decomposition message for "Application Service Crash" is 0.42 × 1.0 + 0.38 × 1.0 = 0.8.

[0080] At the same time, each node's upstream aggregate message is calculated. Upward aggregate messages are information transmitted from lower-level nodes to upper-level nodes, reflecting feedback from lower-level nodes to upper-level nodes. The calculation process is similar to that for downward decomposed messages, involving the node's neighbors, information transfer weights, and historical upstream messages. Specifically, a node's upstream aggregate message is equal to the sum of the product of the information transfer weights of all its neighbors and the corresponding historical upstream messages. Initially, the historical upstream messages of all nodes can be set to a predefined initial value, such as 1.0. For example, for the event node "Database Connection Pool Exhausted," assuming its neighbors include "SQL Query Timeout" and "Increased Transaction Rollbacks," with information transfer weights of 0.35 and 0.28, respectively, and historical upstream messages of 1.0 and 1.0, respectively, the upstream aggregate message for "Database Connection Pool Exhausted" is 0.35 × 1.0 + 0.28 × 1.0 = 0.63.

[0081] The sum of the node's bidirectional messages is calculated by adding the downstream decomposed messages and the upstream aggregated messages. This sum comprehensively reflects the node's importance in the network and the degree of information flow. For example, if the downstream decomposed message for the event node "Increased Network Latency" is 0.75 and the upstream aggregated message is 0.58, the sum of its bidirectional messages is 0.75 + 0.58 = 1.33. The sum of bidirectional messages can be used as an indicator of node priority and subsequently used in log archiving strategy development.

[0082] Multiplying the posterior probability by the information transfer weight yields the transition probability between node pairs. The posterior probability is calculated using the aforementioned probabilistic graph inference method for bidirectional propagation and reflects the estimated probability of a node's state. Transition probabilities represent the probability of information being transferred from one node to another and are an important basis for developing log archiving strategies. For example, if the posterior probability of the "server response timeout" event is 0.25, and the information transfer weight between it and the "database connection failure" event is 0.375, then the transition probability between them is 0.25 × 0.375 = 0.09375.

[0083] Substitute the posterior probability and the transition probability into the log-likelihood function to calculate the likelihood value under the current parameter settings. The log-likelihood function is an indicator for evaluating the degree of fit of the model parameters. The larger its value, the higher the degree of match between the model and the observed data. In this embodiment, the log-likelihood function can be constructed based on the observed log event sequence and the event transition probability predicted by the model. For example, if the observed log event sequence contains events A, B, and C, and the transition probability predicted by the model is P(A)=0.3, P(B|A)=0.4, and P(C|B)=0.5, then the log-likelihood value can be obtained by relevant calculation, assuming that the calculation result is -2.5.

[0084] The posterior and transition probabilities are iteratively updated until the log-likelihood function converges. This is an optimization process, the goal of which is to find the parameter settings that maximize the log-likelihood function. In each iteration, the parameters can be updated using gradient ascent, adjusting the parameter values ​​according to the partial derivatives of the log-likelihood function with respect to each parameter. The iterative process can be terminated with specified conditions, such as reaching a maximum number of iterations (e.g., 100) or when the change in the log-likelihood value falls below a specified threshold (e.g., 0.001). For example, after multiple iterations, as the model parameters are continuously adjusted, the log-likelihood value may gradually increase from an initial value of -2.5 to -1.8, eventually converging.

[0085] Based on the updated posterior probabilities and transition probabilities, the optimal archiving plan for log events is determined. The log archiving plan includes factors such as log classification, storage location, and retention period. Factors such as event importance (reflected by the posterior probability), correlation between events (reflected by the transition probability), storage cost, and query efficiency can be considered when determining the optimal archiving plan. For example, events can be classified into three importance levels: high, medium, and low based on their posterior probability. Events with a posterior probability greater than 0.5 are classified as high importance, with a retention period of 90 days; events with a posterior probability between 0.2 and 0.5 are classified as medium importance, with a retention period of 30 days; and events with a posterior probability less than 0.2 are classified as low importance, with a retention period of 7 days.

[0086] Furthermore, considering the transition probabilities between events, event pairs with high transition probabilities (e.g., those with a transition probability greater than 0.3) are archived into the same storage area to improve query efficiency. For example, the transition probability between the "load balancer switchover" event and the "service instance restart" event is 0.35, exceeding the threshold of 0.3. Therefore, these two types of events are archived into the same storage area. Furthermore, different indexing strategies can be set based on the sum of bidirectional messages per event. Events with high bidirectional message sums (e.g., greater than 1.5) will have finer-grained indexes established to accelerate queries.

[0087] In practical applications, this method can be applied to log management systems for large-scale internet services. For example, an e-commerce platform generates massive amounts of log data daily, including user access logs, system operation logs, and error logs. By constructing a relational network and applying a message passing algorithm, it is possible to identify the correlations and importance levels between log events. Analysis results show that the posterior probability of the event "database master-slave switch" is 0.62, the posterior probability of the event "cache service restart" is 0.48, and the posterior probability of the event "page load timeout" is 0.35. Based on this data, a log archiving strategy is automatically generated: logs related to "database master-slave switch" are classified as high importance and stored in a high-performance storage area with a retention period of 90 days; logs related to "cache service restart" are classified as medium importance and stored in a standard storage area with a retention period of 30 days; and logs related to "page load timeout" are classified as low importance. However, since their transition probability to high-importance events reaches 0.4, they are also stored in a high-performance storage area with a retention period of 30 days.

[0088] By applying this optimal archiving strategy, the platform's log management efficiency has significantly improved: storage space utilization increased by 25%, critical event query response time decreased by 40%, and fault diagnosis time was shortened by an average of 35%. This demonstrates that the technical solution provided by this invention can effectively solve the data archiving problem in large-scale log systems, improving log management efficiency and data value.

[0089] The message passing algorithm provided in this implementation combines the advantages of graph neural networks and probabilistic reasoning. Through top-down event decomposition and bottom-up feature aggregation, it comprehensively captures the complex relationships between log events. An optimization method based on the maximum likelihood criterion can find model parameters that best match the actual data distribution, providing a scientific basis for log archiving strategies. This method is applicable to various large-scale log systems and has strong versatility and practical value.

[0090] The embodiment of the present invention is based on a log processing system with dual-path processing and intelligent archiving, and the system includes: The first unit performs semantic segmentation and structural analysis on the original log data to obtain log events; The second unit calculates a causal relationship chain between the log events through a Bayesian network, identifies a potential risk propagation path based on the causal relationship chain, and shunts the log events to different processing optimization units based on the potential risk propagation path. The third unit extracts knowledge entities from the log events, establishes connection edges according to business dependency relationships and semantic similarities between the knowledge entities, forms a relationship network, performs correlation analysis on the log events based on the association relationships of the relationship network to obtain an event correlation matrix, calculates conditional dependency relationships between nodes in the relationship network through a bidirectional propagation probabilistic graph reasoning method, combines the event correlation matrix, performs top-down event decomposition and bottom-up feature aggregation in the relationship network through a message passing algorithm, simultaneously calculates transition probabilities between adjacent node pairs, and selects an optimal log archiving strategy based on a maximum likelihood criterion.

[0091] In a third aspect, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0092] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0093] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0094] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A log processing method based on dual-path processing and intelligent archiving, characterized in that: include: Perform semantic segmentation and structural analysis on the original log data to obtain log events; Calculating the causal relationship chain between the log events through a Bayesian network, identifying potential risk propagation paths based on the causal relationship chain, and diverting the log events to different processing optimization units based on the potential risk propagation paths; Extract knowledge entities from log events, establish connections based on business dependencies and semantic similarity between knowledge entities, and form a relationship network. Based on the correlation relationship of the relationship network, the log events are subjected to correlation analysis to obtain an event correlation matrix; The conditional dependencies between nodes in the relational network are calculated through a probabilistic graph reasoning method with bidirectional propagation. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relational network. At the same time, the transition probabilities between adjacent node pairs are calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

2. The method according to claim 1, characterized in that Calculating the causal relationship chain between the log events through a Bayesian network, identifying potential risk propagation paths based on the causal relationship chain, and diverting the log events to different processing optimization units based on the potential risk propagation paths includes: Log events are constructed as nodes, dependencies between the log events are identified to form an edge set, a directed acyclic graph is constructed, conditional probabilities between nodes are determined in the directed acyclic graph by using a statistical ratio of counts between nodes and their parent nodes, and conditional probability transfer relationships between nodes are sequentially calculated based on a topological sequence of the directed acyclic graph to obtain a causal chain between the log events; The continuous product of the conditional probability of each node on the causal chain and the corresponding risk weight is used as the risk propagation strength. The cumulative risk value of the node is iteratively updated and the unvisited node with the smallest cumulative risk value is searched. The cumulative risk value of the adjacent nodes is updated by adding the risk propagation strength to the cumulative risk value of the current node until the cumulative risk values ​​of all nodes converge or the maximum number of iterations is reached, thereby obtaining the potential risk propagation path. Calculate the maximum risk propagation intensity of the potential risk propagation path where the log event is located, count the number of potential risk propagation paths involved in the log event, and take the weighted sum of the maximum risk propagation intensity and the number of potential risk propagation paths to obtain a comprehensive risk score for the log event. According to the comprehensive risk score, the log event is diverted to the corresponding processing optimization unit for processing.

3. The method according to claim 2, characterized in that Determining the conditional probability between nodes in the directed acyclic graph by using the count statistics ratio of the node to the parent node, and sequentially calculating the conditional probability transfer relationship between nodes based on the topological sequence of the directed acyclic graph to obtain the causal relationship chain between the log events includes: Count the number of co-occurrences of each node with its corresponding parent node and the number of independent occurrences of the corresponding parent node, divide the co-occurrence number by the independent occurrence number, perform Laplace smoothing on the result to obtain the conditional probability, and record all conditional probabilities in a conditional probability matrix; A Kahn algorithm is used to obtain a topological sequence of the directed acyclic graph, and based on the topological sequence, a path from the direct predecessor node of each node to the source node is sequentially identified, and a conditional probability transfer relationship is obtained by calculating the continuous product of conditional probabilities between adjacent nodes on the path, and all conditional probability transfer relationships are recorded in a transfer probability matrix; Multiply each element in the conditional probability matrix by a preset direct influence weight factor, multiply each element in the transfer probability matrix by a preset indirect influence weight factor, add the two multiplied matrices together to obtain a causal strength matrix, and extract the node pairs that meet the preset strength threshold in the causal strength matrix to construct a causal relationship chain.

4. The method according to claim 1, wherein Extract knowledge entities from log events, establish connections based on business dependencies and semantic similarity between knowledge entities, and form a relationship network. Based on the association relationship of the relationship network, the event correlation matrix obtained by performing correlation analysis on the log events includes: Extract event subjects, event objects, and event attributes from log events as knowledge entities; calculate the cosine similarity of word vectors between the knowledge entities to obtain semantic similarity, obtain the business dependency between the knowledge entities based on the system call chain and event timing dependency, and perform weighted summation with the semantic similarity to obtain entity association strength; construct an entity relationship matrix based on the entity association strength, and establish connecting edges between entity pairs in the entity relationship matrix that are greater than a preset association threshold to form a relationship network; Based on the association relationships in the relationship network, the degree centrality and betweenness centrality of the event nodes in the relationship network are calculated to obtain the node importance, the number of shortest path hops between the event nodes is counted to obtain the path relevance, and based on the weighted combination of the node importance and the path relevance, an event relevance matrix is ​​generated.

5. The method according to claim 1, wherein Calculating the conditional dependency between nodes in the relationship network using a probabilistic graph reasoning method using bidirectional propagation includes: Counting the co-occurrence frequency between event nodes in the relationship network, and dividing the co-occurrence frequency by the total number of events to obtain the information transmission probability of the event node pair; Establish a bidirectional information transmission path, calculate the prior belief value of the target node from the parent node downward, the prior belief value is the weighted product of the parent node belief and the information transmission probability, calculate the diagnostic information value from the child node upward, the diagnostic information value is the cumulative product of the child node information and the information transmission probability, and multiply the normalized prior belief value and the diagnostic information value to obtain the posterior probability of the node; The conditional mutual information between any pair of adjacent nodes is calculated using the posterior probability, and the information transmission probability of the nodes is updated based on the difference between the conditional mutual information and the dependency threshold. The updated information transmission probability is substituted into the bidirectional information transmission path to generate a new posterior probability, and the conditional dependency relationship between the nodes is obtained through iterative optimization.

6. The method according to claim 1, characterized in that Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relationship network. Transition probabilities between adjacent node pairs are calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion. The strategy includes: Calculating the information transfer weight using the product of the cosine similarity of the event correlation matrix and the inverse of the shortest path distance between the node pairs; On the message transmission link, a downlink decomposition message of each node is calculated. The downlink decomposition message is the sum of the product of the information transmission weights of all neighboring nodes of the node and their historical downlink messages. An uplink aggregate message is calculated for each node. The uplink aggregate message is the sum of the product of the information transmission weights of all neighboring nodes of the node and their historical uplink messages. The downlink decomposition message and the uplink aggregate message are added together to obtain the sum of the node's two-way messages. The posterior probability is multiplied by the information transfer weight to obtain the transition probability between the node pairs; the posterior probability and the transition probability are substituted into the log-likelihood function, the posterior probability and the transition probability are iteratively updated until the log-likelihood function converges, and the optimal archiving scheme for the log event is determined based on the updated posterior probability and the transition probability.

7. A log processing system based on dual-path processing and intelligent archiving, for implementing the method according to any one of claims 1 to 6, characterized in that: include: The first unit performs semantic segmentation and structural analysis on the original log data to obtain log events; The second unit calculates the causal relationship chain between the log events through a Bayesian network, identifies potential risk propagation paths based on the causal relationship chain, and diverts the log events to different processing optimization units based on the potential risk propagation paths; The third unit extracts knowledge entities from log events and establishes connection edges based on the business dependencies and semantic similarities between knowledge entities to form a relationship network; Based on the correlation relationship of the relationship network, the log events are subjected to correlation analysis to obtain an event correlation matrix; The conditional dependencies between nodes in the relational network are calculated through a probabilistic graph reasoning method with bidirectional propagation. Combined with the event correlation matrix, a message passing algorithm is used to perform top-down event decomposition and bottom-up feature aggregation in the relational network. At the same time, the transition probabilities between adjacent node pairs are calculated, and the optimal log archiving strategy is selected based on the maximum likelihood criterion.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images