A log analysis alarm method, device, equipment and storage medium

Abstract

The application discloses a log analysis alarm method and device, equipment and storage medium, applied to data processing technical field, including obtaining a plurality of log text data of a target system and time stamp data, host identification data and process identification data corresponding to the log text data;A plurality of log text data are coded and processed, based on the obtained log vector, the same transaction positive sample and the cross-transaction negative sample are constructed, and the candidate edge is obtained;Based on the obtained candidate causal relationship data, time delay attribute data and edge existence probability, a time sequence causal graph is obtained;The time sequence causal graph is subjected to causal path anomaly detection processing, and a causal path anomaly set of the target system is obtained;Based on the causal path anomaly set, an alarm result is calculated.The method provided by the application can adaptively learn system behavior patterns and improve anomaly detection accuracy through time sequence causal graph modeling and causal path anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a log analysis alarm method, apparatus, device, and storage medium. Background Technology

[0002] With the widespread application of cloud computing, the Internet of Things, and distributed systems, system logs have gradually become an important data source for ensuring the security and stable operation of information systems. Logs record the system's operating status, the interaction process of service components, and potential signs of failures and attacks, thus playing an irreplaceable role in operation and maintenance security and intelligent monitoring. However, logs are typically characterized by their large volume, diverse formats, complex semantics, and intertwined time sequences, making effective log analysis and alerting still a significant challenge.

[0003] In existing technologies, most log analysis and alarm methods are based on rule modeling. By templated log content, the system's call patterns and structural relationships can be initially captured. These methods improve detection efficiency to some extent, but they rely heavily on manual templates. Once the system function is updated or the log structure changes, the original template often becomes invalid, requiring frequent maintenance and lacking practicality. At the same time, traditional methods mainly focus on the formal pattern of logs and cannot effectively identify semantic anomalies, thus failing to provide accurate alarms. Summary of the Invention

[0004] This invention provides a log analysis alarm method, device, equipment, and storage medium to solve the technical problems of traditional rule-based log analysis methods, which heavily rely on manual templates, are difficult to adapt to system updates and log drift, and cannot effectively mine log semantic information. By using time-series causal graph modeling and causal path anomaly detection, it adaptively learns system behavior patterns, improves the accuracy of anomaly detection, and reduces manual maintenance costs.

[0005] To address the aforementioned technical problems, this invention provides a log analysis and alarm method, the method comprising: Obtain several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; The log text data is encoded, and based on the obtained log vectors, positive samples within the same transaction and negative samples across transactions are constructed to obtain candidate edges. A temporal causal graph is obtained by constructing a temporal causal graph from the candidate edges. The temporal causal graph construction process is configured to: identify the timestamp data and process identifier data using candidate relationship recognition technology to obtain candidate causal relationship data; perform attribute processing on the timestamp data and host identifier data to obtain latency attribute data; perform existence probability modeling on the candidate edges to obtain edge existence probabilities; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probabilities. The time-series cause-effect graph is subjected to cause-effect path anomaly detection processing to obtain the cause-effect path anomaly set of the target system; Based on the set of abnormal causal paths, the alarm result is calculated.

[0006] As one preferred embodiment, the process of obtaining the log vector includes: Using a pre-trained language model, several log text data are semantically encoded to obtain several semantic vectors; The log vector is obtained by mapping and concatenating several semantic vectors with the corresponding timestamp data, host identifier data and process identifier data.

[0007] As one preferred embodiment, the process of identifying and processing the timestamp data and the process identifier data using candidate relationship recognition technology to obtain candidate causal relationship data includes: Constraints are applied to the timestamp data and the process identifier data to obtain a set of candidate log pairs; The candidate log set is processed by combining transaction identifiers with verification using candidate relationship identification technology to obtain the candidate causal relationship data.

[0008] As one preferred embodiment, the step of performing attribute processing on the timestamp data and the host identifier data to obtain latency attribute data includes: The timestamp data and the host identifier data are processed using candidate relationship recognition technology to obtain a candidate edge set. The candidate edge set is sequentially subjected to delay calculation and threshold filtering to obtain the delay attribute data.

[0009] As one preferred embodiment, the existence probability modeling process for the candidate edges to obtain the edge existence probability includes: The candidate edges are processed by feature concatenation using conditional probability model techniques to obtain a concatenated feature vector; The concatenated feature vector is subjected to function activation processing to obtain the probability of the existence of the edge.

[0010] As one preferred embodiment, obtaining the temporal causal graph based on the candidate causal relationship data, the time delay attribute data, and the edge existence probability includes: The candidate causal relationship data, the time delay attribute data, and the edge existence probability are input into the edge feature integration model constructed from the log vector to obtain the edge feature set; The edge feature set is integrated to obtain the temporal causal graph.

[0011] As one preferred embodiment, the causal path anomaly detection processing performed on the time-series causal graph to obtain the causal path anomaly set of the target system includes: The time-series cause-effect graph is subjected to path search processing to obtain a path set; The path set is input into the constructed path scoring function for processing to obtain the scores of several paths in the path set; The score is compared with a preset threshold to obtain the set of causal path anomalies of the target system.

[0012] The present invention also provides a log analysis alarm device, comprising: The acquisition module is used to acquire several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; The encoding module is used to encode several log text data, and based on the obtained log vector, construct positive samples of the same transaction and negative samples of cross transactions to obtain candidate edges; A construction module is configured to perform temporal causal graph construction processing on the candidate edges to obtain a temporal causal graph. The temporal causal graph construction process is configured to: use candidate relationship recognition technology to identify the timestamp data and process identifier data to obtain candidate causal relationship data; perform attribute processing on the timestamp data and host identifier data to obtain latency attribute data; perform existence probability modeling processing on the candidate edges to obtain edge existence probabilities; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probabilities. The detection module is used to perform causal path anomaly detection processing on the time-series causal graph to obtain the causal path anomaly set of the target system; The processing module is used to calculate alarm results based on the set of causal path anomalies.

[0013] The present invention also provides a log analysis alarm device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the log analysis alarm method as described above.

[0014] The present invention further provides a computer-readable storage medium storing a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the log analysis and alarm method described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are at least one of the following: This invention acquires several log text data and corresponding timestamp data, host identifier data, and process identifier data of a target system; encodes the log text data, constructs positive samples within the same transaction and negative samples across transactions based on the obtained log vectors, and obtains candidate edges; constructs a time-series causal graph based on the candidate edges, wherein the time-series causal graph construction process is configured to use candidate relationship recognition technology to identify the timestamp data and process identifier data to obtain candidate causal relationship data; performs attribute processing on the timestamp data and host identifier data to obtain latency attribute data; performs existence probability modeling on the candidate edges to obtain edge existence probability; obtains the time-series causal graph based on the candidate causal relationship data, latency attribute data, and edge existence probability; performs causal path anomaly detection processing on the time-series causal graph to obtain a causal path anomaly set of the target system; and calculates alarm results based on the causal path anomaly set.

[0016] Compared with existing technologies, this invention first performs semantic encoding on the log text to generate log vectors, automatically capturing the deep semantics of the text and avoiding the limitations of manually defined templates. Second, it constructs a time-series causal graph based on timestamps, host and process identifiers, and dynamically learns the time-series and causal relationships between logs through candidate relationship identification, latency attribute analysis and edge existence probability modeling, thereby adapting to system updates or log drift without frequent maintenance. Finally, it performs path anomaly detection on the causal graph, identifying semantic and structural deviations from the overall behavior pattern and improving alarm accuracy. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the log analysis and alarm method in one embodiment of the present invention; Figure 2 This is a schematic diagram of the log analysis alarm device in one embodiment of the present invention; Figure 3 This is a schematic diagram of the log analysis alarm device in one embodiment of the present invention; Figure label: Among them, 11 is the acquisition module; 12 is the encoding module; 13 is the construction module; 14 is the detection module; 15 is the processing module; 21 is the processor; and 22 is the memory. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] One embodiment of the present invention provides a log analysis and alarm method. For details, please refer to [link / reference]. Figure 1 , Figure 1 The diagram shown is a flowchart of a log analysis alarm method according to one embodiment of the present invention. The method includes: S1: Obtain several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; S2: Encode several log text data, and based on the obtained log vectors, construct positive samples within the same transaction and negative samples across transactions to obtain candidate edges; S3: A temporal causal graph is obtained by constructing a temporal causal graph from the candidate edges. The temporal causal graph construction process is configured to: use candidate relationship recognition technology to identify the timestamp data and process identifier data to obtain candidate causal relationship data; perform attribute processing on the timestamp data and host identifier data to obtain latency attribute data; perform existence probability modeling on the candidate edges to obtain edge existence probabilities; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probabilities. S4: Perform causal path anomaly detection processing on the time-series causal graph to obtain the causal path anomaly set of the target system; S5: Calculate the alarm result based on the set of abnormal causal paths.

[0021] Specifically, the original log set is first preprocessed uniformly.

[0022] The process of obtaining the log vector includes: semantically encoding several log text data using a pre-trained language model to obtain several semantic vectors; mapping and concatenating the several semantic vectors with the corresponding timestamp data, host identifier data and process identifier data to obtain the log vector.

[0023] The original log set is denoted as ,in The total number of log entries for each log entry. The timestamp, host identifier, process identifier, and log text content are extracted and standardized sequentially. The timestamp is uniformly converted to UTC format and denoted as [format not specified]. Host identifiers are normalized and mapped to one-hot vectors. ,in The number of hosts or containers; process identifiers converted to integer indices. This is for subsequent embedding. The log text is segmented to obtain a word sequence. ,in This represents the length of the log text. Through the above processing, the original unstructured log is converted into a structured multimodal feature representation.

[0024] After preprocessing, a pre-trained language model is used to semantically encode the log text, and the encoding function is defined as follows: ,in For model parameters, This is a semantic vector representation of the log text. For vector dimensions.

[0025] Based on this, semantic vectors Host vector Process Index and timestamp Concatenate to form a complete log vector representation. This vector preserves semantic features and system context information, providing an input basis for subsequent causal graph modeling.

[0026] Based on the obtained log vectors, construct positive samples within the same transaction and negative samples across transactions to obtain candidate edges.

[0027] Positive samples within the same transaction refer to log pairs belonging to the same transaction, while negative samples across transactions refer to log pairs belonging to different transactions. The negative sample set is dynamically generated from the batch processing, such as the comparison set within the same batch as sample i.

[0028] Let the transaction set be Each transaction Corresponding log subset .

[0029] Using a contrastive learning method, positive sample pairs are defined as log pairs within the same transaction. Negative sample pairs are log pairs across transactions. .

[0030] The loss function is in For cosine similarity, This is the temperature coefficient.

[0031] By minimizing this loss function, the semantic vectors of the same transaction log can be more closely distributed, thereby enhancing semantic distinguishability.

[0032] In multi-source log scenarios, to avoid semantic drift and lexical ambiguity between different systems, a unified vocabulary is constructed. And establish cross-system mapping relationships Where M represents a mapping function from a local vocabulary to a unified vocabulary, used to map synonyms, abbreviations, or system-specific expressions to the same standard term; It is a globally unified vocabulary used to represent a standardized set of terms shared across systems; This provides a local thesaurus for each individual system. By mapping synonyms and abbreviations, consistency in the semantic space is ensured for synonym logs across different systems.

[0033] Further define the alignment loss function By training, the semantic vector distribution across systems tends to be consistent, thereby achieving unified modeling.

[0034] in Represents a pair through a mapping relationship Related terms or log text units, A local representation from a certain system, This is its corresponding representation in the unified vocabulary space. For semantic encoding functions, it represents the expression of parameters. A controlled semantic encoding model for mapping input terms or log text into continuous semantic vectors of fixed dimensions. It is the Euclidean norm. Used to measure the distance between mapping word pairs in the semantic vector space.

[0035] Similarly, sample construction and contrastive learning do not directly obtain candidate edges, but indirectly influence the quality of candidate edges by optimizing log vectors. A sliding window mechanism is used for small-batch updates to adapt to changes in the log stream and ensure the long-term effectiveness of the vectors. The optimized log vectors are better able to capture the semantic relationships between logs, providing more accurate input for candidate edge identification.

[0036] In the process of obtaining candidate edges, the log vector set is traversed. For each pair of log vectors, the timestamp is checked, and only if the time order condition is met is it considered as a candidate edge pair.

[0037] In this embodiment, potential causal relationship candidates are first identified based on the time sequence of logs, host process context, and transaction identification information.

[0038] Set up two logs The timestamps are respectively If satisfied And if both belong to the same transaction or the same call chain, then... As candidate edge pairs. Candidate edges are denoted as a set. ,in This is a context constraint function, which takes a value of 1 when the log satisfies process inheritance, host continuity, or transaction consistency.

[0039] Furthermore, a temporal causal graph is obtained by constructing a temporal causal graph from the candidate edges. This temporal causal graph construction process is configured to: use candidate relationship recognition technology to identify the timestamp data and process identifier data to obtain candidate causal relationship data; perform attribute processing on the timestamp data and host identifier data to obtain latency attribute data; perform existence probability modeling on the candidate edges to obtain edge existence probabilities; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probabilities.

[0040] Specifically, candidate causal relationship data is obtained by using candidate relationship recognition technology to identify and process the timestamp data and the process identifier data, including: constraining the timestamp data and the process identifier data to obtain a set of candidate log pairs; and using candidate relationship recognition technology to perform transaction identifier combination verification processing on the set of candidate log pairs to obtain the candidate causal relationship data.

[0041] Based on preset time window thresholds and process ownership consistency conditions, the acquired timestamp data and process identifier data are filtered and constrained.

[0042] For each log data, a time range is defined based on its timestamp. Other log data with the same process identifier as this log within this time range are filtered out. Two logs that meet the requirements of time sequence and process consistency are paired up to form multiple log pairing combinations. Finally, these combinations are integrated to obtain a candidate log pair set.

[0043] Association rules related to transaction identifiers are introduced to verify and analyze each log pair in the candidate log pair set. Combined with the transaction logic of the system operation, it is determined whether each candidate log pair belongs to the same transaction process or whether there is a triggering and being triggered relationship at the transaction level. For example, it is determined whether the operation corresponding to the previous log will trigger the system response corresponding to the next log. Log pair combinations that conform to the transaction operation logic are filtered out. These verified log pairs and their corresponding association characteristics are integrated to obtain candidate causal relationship data.

[0044] By relying on transaction identifiers combined with verification, log pairs that only meet time and process constraints but lack actual business logic connections can be filtered out. This improves the reliability and effectiveness of candidate causal relationship data, ensuring that the causal relationships contained in the subsequently constructed time-series causal graph conform to the actual operating rules of the system, and provides high-quality causal relationship foundational data for accurate anomaly detection.

[0045] It should be noted that in the process of verifying the causal relationship between candidate logs based on transaction identifiers, the association rules related to transaction identifiers mainly include four categories. The specific operation methods and functions of each type of rule are as follows: 1. Transaction consistency rules. Specifically, based on the system's preset transaction identifier generation logic, the transaction identifier information hidden in each log is extracted through process call chain tracing, session ID matching, etc., to determine whether two logs in a candidate log pair carry the same transaction identifier, or whether they belong to the same sub-process identifier of the same transaction.

[0046] 2. Transaction timing triggering rules. Specifically, this involves combining the timestamp data of the logs to verify the time sequence of log pairs under the same transaction identifier, setting a threshold for the reasonableness of the timing of operations within the transaction, and determining whether the operation corresponding to the previous log is earlier than the operation corresponding to the next log in time, and whether the time interval is within the normal latency range of operations within the transaction.

[0047] 3. Transaction operation dependency rules. Specifically, this involves outlining the standard operation flow of system transactions and defining the dependency relationships between operations within a transaction. For example, transaction initialization operations must precede transaction processing operations, and transaction processing operations must precede transaction closing operations. For candidate log pairs, it verifies whether their corresponding operations conform to this dependency relationship. At the same time, it combines the semantic vector of the log text to determine whether the semantic logic of the operations matches the dependency relationship.

[0048] 4. Transaction boundary constraint rules. Specifically, the start and end logs of a transaction are identified as transaction boundary markers. It is determined whether candidate log pairs are between the start and end markers of the same transaction, while excluding log pairs that cross multiple transaction boundaries. For candidate pairs that contain transaction boundary logs, their association logic with boundary operations is verified separately.

[0049] The time stamp data and the host identifier data are processed to obtain latency attribute data, including: using candidate relationship recognition technology to process the attributes of the time stamp data and the host identifier data to obtain a candidate edge set; and performing latency calculation and threshold filtering on the candidate edge set in sequence to obtain the latency attribute data.

[0050] Specifically, based on the association logic of candidate relationship identification technology, the log data is bidirectionally matched and filtered using the chronological order of timestamp data as the core temporal basis and the attribution information of host identifier data as the node association condition. For each log, its timestamp is first extracted to determine its temporal position, and then it is matched with potentially related target logs. The target logs must satisfy the following conditions: they must have a temporal correlation with the source logs, and their host identifiers must belong to the same host or meet the preset rules for cross-host interaction. Each successfully matched source log and target log is mapped to an edge, and all edges that meet the conditions are integrated to form a candidate edge set.

[0051] Next, a latency calculation operation is performed. For each edge in the candidate edge set, the timestamp data corresponding to its source log and target log are extracted, and the difference between the two timestamps is calculated to obtain the latency value corresponding to each candidate edge. Then, a threshold filtering operation is performed. Based on the historical operating data of the target system or a preset normal interaction latency range, a reasonable latency threshold range is set. Edges with latency values ​​exceeding this range are removed, and candidate edges with latency values ​​within the normal range are retained. Finally, these valid candidate edges and their corresponding latency values ​​are integrated to form structured latency attribute data.

[0052] The existence probability modeling process for the candidate edges is performed to obtain the edge existence probability, including: using conditional probability modeling techniques to perform feature concatenation on the candidate edges to obtain a concatenated feature vector; and performing function activation on the concatenated feature vector to obtain the edge existence probability.

[0053] Specifically, multi-dimensional features are extracted for each candidate edge, including log semantic vectors of the source and target nodes, correlation features in the candidate causal relationship data, latency numerical features in the latency attribute data, and attribution correlation features corresponding to host and process identifiers. Based on the dimensionality requirements of the conditional probability model for input features, these features from different sources and with different dimensions are systematically integrated in a preset order to form a high-dimensional feature vector with a unified structure, i.e., a concatenated feature vector.

[0054] The concatenated feature vectors are input into the computational framework of the conditional probability model. First, a weighted summation operation is performed on the concatenated feature vectors through the model's linear transformation layer to obtain a linear combination result. Then, an appropriate activation function is selected to map the linear combination result to a numerical range of 0 to 1. The output value within this range represents the probability of the corresponding candidate edge's existence. The closer the value is to 1, the higher the probability that the causal relationship corresponding to the candidate edge truly exists; the closer the value is to 0, the lower the probability.

[0055] The process of obtaining the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probability includes: inputting the candidate causal relationship data, the latency attribute data, and the edge existence probability into an edge feature integration model constructed from the log vector to obtain an edge feature set; and integrating the edge feature set to obtain the temporal causal graph.

[0056] Specifically, the input dimensions and feature adaptation rules of the edge feature integration model are first determined. This model is pre-trained based on log vectors and is compatible with the fusion processing of heterogeneous feature data. The log pairs carrying the candidate causal relationship data (causal association logical features), the interaction latency numerical features contained in the latency attribute data, and the causal association probability quantification features corresponding to the edge existence probability are associated and mapped with the node semantic features represented by the log vectors. Following the model's preset feature input order, these three types of data are simultaneously input into the edge feature integration model. Through feature alignment, dimensional unification, and weighted fusion operations, the model transforms the multi-dimensional features corresponding to each candidate edge into a structured comprehensive feature vector. The comprehensive feature vectors of all candidate edges are then integrated to form an edge feature set.

[0057] Using the system events or operations corresponding to the log vectors as nodes in the time-series causal graph, a probability screening threshold is set based on the edge existence probability value of each comprehensive feature vector in the edge feature set. False correlation edge features with probability values ​​lower than the threshold are eliminated, and valid edge features with the possibility of real causal correlation are retained.

[0058] Then, each valid edge feature is assigned a corresponding attribute label: candidate causal relationship data is labeled as the causal association attribute of the edge, time delay attribute data is labeled as the temporal delay attribute of the edge, and the edge existence probability is labeled as the reliability attribute of the edge.

[0059] Finally, according to the order of log timestamps, the nodes are topologically connected with the valid edges with attribute labels to construct a graph structure in which nodes represent system events and edges represent the temporal causal relationships between events, thus obtaining the temporal causal graph.

[0060] Specifically, step 1: Candidate relation identification: After obtaining the log vector set output from step 1... Next, based on the chronological order of the logs, the host process context, and transaction identification information, potential causal relationship candidates are identified. Assume two log entries... The timestamps are respectively If satisfied And if both belong to the same transaction or the same call chain, then... As candidate edge pairs. Candidate edges are denoted as a set. ,in This is a context constraint function, which takes a value of 1 when the log satisfies process inheritance, host continuity, or transaction consistency.

[0061] Step 2: Delay attribute calculation: For each candidate edge Calculate its corresponding delay attribute To avoid interference from extreme values, upper and lower thresholds are set. ,when When the edge exceeds this range, the candidate edge is discarded, and the remaining delay attribute is used as the first type of feature of the edge for the identification of delay anomalies in subsequent anomaly detection.

[0062] Step 3: Semantic Similarity Weighting: Using the log semantic vector obtained in Step 1, calculate the semantic similarity between the logs at both ends of the edge: when Higher than the set threshold When the edge is considered to have strong semantic relevance, semantic weights are added to the edge's feature vector. This ensures that the graph structure reflects the causal order while also introducing semantic information from the log content.

[0063] Step 4: Edge Existence Probability Estimation: To avoid noisy edges being incorrectly included in the causal graph, an existence probability model is performed for each candidate edge. A conditional probability model based on trigger strength is adopted: in For the sigmoid function, This is the concatenation of two log vectors. These are learnable parameters. The model is trained using historical log sequences to assign probability scores to the rationality of edges, which are then used as the basis for the credibility of the causal graph construction.

[0064] Step 5: Temporal Cause-Effect Graph Generation: Combining the results of Steps 2, 3, and 4, define a complete feature vector for each candidate edge: Finally, a time-series causal graph was constructed. in For a set of nodes, corresponding to a log vector; The set of edges to be retained; This graph represents the set of edge features. Structurally, it preserves the temporal dependencies of the logs; semantically, it retains the log content information; and numerically, it incorporates latency and probability weights.

[0065] Through the above sub-steps, a semantically enhanced temporal causal graph is finally obtained. Each node represents a semantic vector of a log entry, and each edge represents a causal relationship between log entries, with three attributes: latency, semantic similarity, and probability of existence.

[0066] The causal path anomaly detection process is performed on the time-series causal graph to obtain the causal path anomaly set of the target system, including: performing path search processing on the time-series causal graph to obtain a path set; inputting the path set into a constructed path scoring function for processing to obtain the scores of several paths in the path set; and comparing the scores with a preset threshold to obtain the causal path anomaly set of the target system.

[0067] First, determine the core rules for path search. These rules need to combine the temporal attributes of the nodes and the causal association attributes of the edges in the temporal cause-effect graph to clarify the range of the starting and ending nodes of the search. The starting node is usually the node corresponding to the triggering operation log of the system transaction, and the ending node is usually the node corresponding to the closing operation log of the transaction.

[0068] Then, a graph traversal algorithm, such as depth-first search or breadth-first search, is used to perform a full traversal of the temporal causal graph, mining all possible paths from the starting node to the ending node, including process interaction paths within a single host and transaction collaboration paths across hosts. At the same time, the node sequence, edge existence probability attributes, and latency attributes contained in each path are recorded, and all traversed paths are integrated to form a structured path set.

[0069] Next, we will clarify the construction logic of the path scoring function. This function is trained based on the path data of the system's historical normal operation. The input dimensions of the function include core features such as the number of nodes in the path, the average probability of edge existence, the fluctuation range of the latency attribute, and the consistency of the causal relationship attribute.

[0070] Then, the aforementioned feature parameters corresponding to each path in the path set are extracted, and the parameters are standardized according to the input format requirements of the scoring function. Finally, the standardized parameters are input into the path scoring function.

[0071] The function uses a pre-defined weighted calculation logic to quantitatively evaluate the normality of each path and outputs a corresponding score. The higher the score, the more the path conforms to the normal behavior pattern of the system, and the lower the score, the greater the possibility of deviating from the normal pattern.

[0072] Next, based on the score distribution characteristics of the system's historical normal paths and the accuracy requirements of anomaly detection, a reasonable anomaly judgment threshold is set. This threshold can be determined by statistically analyzing the mean and standard deviation of historical data, or it can be configured as an adaptive threshold that is dynamically adjusted according to the system's operating status.

[0073] Then, the score of each path in the path set is compared with the preset threshold one by one, and paths with scores lower than the threshold are filtered out. These paths represent abnormal paths that deviate from the normal behavior pattern of the system.

[0074] Finally, the node sequence, abnormal characteristics, and degree of deviation of each abnormal path are recorded, and all abnormal paths are integrated to form a causal path anomaly set of the target system.

[0075] The causal path anomaly set includes at least a set of anomalous events, each accompanied by a comprehensive anomaly score and a corresponding causal path explanation.

[0076] In a specific embodiment, there are 5 steps.

[0077] Step 1: Graph Structure Prediction and Structural Anomaly Identification: The time-series causal graph constructed based on the above steps. First, predictive modeling is performed on the graph structure, defining the prediction model. Its input is a subgraph within the time window. The output is the predicted probability distribution of the existence of the edge in the next step. For any real edge ,like If the value is significantly lower than the set threshold, it indicates that the edge should not appear in a normal structure, and is therefore judged as a structural anomaly.

[0078] To quantify the degree of anomaly, a structural anomaly score is defined: when Above the threshold At that time, a structural anomaly alarm is triggered.

[0079] Step 2: Delay Distribution Modeling and Delay Anomaly Identification: In the graph edge features, each edge contains a delay attribute. The normal time delay distribution of the edge is fitted using a probability model based on a Gaussian mixture model or an exponential distribution, denoted as . When the observed time delay If the value falls within the low-probability range, it indicates that an abnormal delay has occurred on that edge. The delay anomaly score is defined as: like If the latency is abnormal, it is considered an anomaly. This mechanism ensures the system's sensitivity to performance degradation and potential attacks in the time dimension.

[0080] Step 3: Semantic Reconstruction and Semantic Anomaly Identification: To capture anomalous semantics in log content, a semantic reconstructor is introduced. The model takes a log node vector as input. Output its reconstruction result The semantic anomaly score is defined as follows: That is, the Euclidean distance between the original semantic vector and the reconstructed semantic vector. When this score exceeds a threshold... When the log content deviates from the normal semantic pattern, a semantic exception is triggered.

[0081] Step 4: Anomaly Score Fusion and Event-Level Anomaly Detection: For the same log event or edge, fuse the structural anomaly scores. Delay anomaly score With semantic anomaly score This yields a uniform abnormal score: in These are the weighting coefficients, which are dynamically optimized using the reinforcement learning method in step 5. If... Exceeding the global threshold Then the event It was identified as an abnormal event.

[0082] Step 5: Causal Path Reasoning and Alarm Chain Generation: After identifying the abnormal event, from the graph... The causal path is searched in reverse. Let the abnormal event be a node. Select with k forward high-confidence paths leading to the destination Calculate the path score for each path: in The time delay weighting parameter is used for adjustment. The path with the highest score is ultimately selected as the causal chain for anomaly explanation.

[0083] The above sub-steps yield a set of abnormal events. Each abnormal event is accompanied by a comprehensive abnormality score. and the corresponding causal path .

[0084] Based on the set of abnormal causal paths, the alarm result is calculated.

[0085] Specifically, in the obtained set of abnormal events and its causal path set Based on this, we first collect multidimensional evidence corresponding to each anomalous event. The evidence includes the original log text of the anomalous node, semantic vectors, etc. Node context information (host) ,process timestamp ), and the time delay attributes of each side in the causal path. semantic similarity With probability of existence In addition, similar cases were retrieved from the historical anomaly database as supplementary evidence. All evidence was organized into an evidence package. This provides data support for interpreting the generation process.

[0086] Next, for different user roles, a multi-level interpretation template set is predefined. When generating the explanation, the corresponding template level is first selected based on the user's identity. Then fill the template slot with information from... The key evidentiary information is used to ensure that the interpretation is both standardized and contains sufficient detail.

[0087] To improve the readability and flexibility of the explanations, a natural language generation model is introduced on top of template filling. The model uses evidence packages. Using template filling results as input, generate natural language descriptions. .

[0088] Through the above sub-steps, the final output is a set of alarm explanations with natural language descriptions. Each explanatory text is associated with an anomaly event. It corresponds one-to-one and includes quantitative evidence, causal path information, and treatment recommendations.

[0089] After the alarm interpretation is generated, the system first models the state of the current operating environment. (Set time...) The system status is Its content includes the distribution of alarm quantity and type within a recent time window, quality assessment indicators such as false alarm rate and false negative rate, system load characteristics such as CPU and memory utilization, and a set of threshold and weight parameters used for anomaly detection. Among them, the global threshold Used to control the overall abnormal score Alarm trigger threshold; structural anomaly threshold Delay anomaly threshold With semantic anomaly threshold These correspond to the three types of anomaly scores defined in step 3, and are used to define the judgment boundaries for different dimensions; weighting coefficients This determines the relative contribution of the three types of outlier scores to the calculation of the overall outlier score. This is achieved by constructing a state vector that includes the above information. The system can comprehensively reflect the current alarm detection performance and operating environment status, providing complete input for subsequent threshold optimization and strategy learning.

[0090] Based on state vector The system selects an adjustment action at each moment. The action set includes adjusting the global anomaly threshold. Various abnormal thresholds and fusion weight For example, if an increase in the false alarm rate is detected, the system may raise the semantic anomaly threshold. Actions may be taken to lower the global threshold if the false negative rate increases. The action.

[0091] By selecting actions, the system can optimize alarm sensitivity and stability in real time.

[0092] In performing the action Then, the system calculates the reward value based on the alarm results and feedback from maintenance personnel. Define the reward function as follows: in They are time points The number of true positives, false positives, and false negatives. λ is the weighting coefficient, and λ is the penalty factor for motion changes. This reward function can simultaneously measure detection accuracy and the stability of motion adjustment.

[0093] The system uses reinforcement learning to train the threshold adjustment strategy.

[0094] Let the policy function be , indicating the state Select action The probability distribution; the value function is , indicating the state The long-term expected return is determined by updating parameters using the policy gradient method. Parameters are updated using the time-difference method. Iterate and optimize until convergence.

[0095] During the training phase, a centralized approach is used to collect feedback from multiple subsystems to ensure the model has cross-scenario generalization ability; during the inference phase, a decentralized approach is adopted, with each subsystem independently executing the strategy. This enables distributed dynamic threshold adjustment.

[0096] Another embodiment of the present invention provides a log analysis alarm device; for details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 The diagram shown is a structural schematic of a log analysis alarm device according to one embodiment of the present invention. The device includes: The acquisition module 11 is used to acquire several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; Encoding module 12 is used to encode several log text data, and based on the obtained log vector, construct positive samples of the same transaction and negative samples of cross transactions to obtain candidate edges; Construction module 13 is configured to obtain a temporal causal graph by performing temporal causal graph construction processing on the candidate edges, wherein the temporal causal graph construction processing is configured to use candidate relationship recognition technology to identify and process the timestamp data and the process identifier data to obtain candidate causal relationship data; perform attribute processing on the timestamp data and the host identifier data to obtain latency attribute data; perform existence probability modeling processing on the candidate edges to obtain edge existence probability; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probability. Detection module 14 is used to perform causal path anomaly detection processing on the time-series causal graph to obtain the causal path anomaly set of the target system; Processing module 15 is used to calculate alarm results based on the set of causal path anomalies.

[0097] See Figure 3 This is a schematic diagram of the structure of a log analysis alarm device provided in an embodiment of the present invention. The log analysis alarm device provided in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, it implements the steps described in the above-described log analysis alarm method embodiment, for example... Figure 1 The steps S1 to S5 described above; or, when the processor 21 executes the computer program, it implements the functions of each module in the above-described device embodiments, such as the acquisition module 11.

[0098] For example, the computer program can be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the log analysis alarm device. For example, the computer program can be divided into an acquisition module 11, an encoding module 12, a construction module 13, etc., with the specific functions of each module as follows: The acquisition module 11 is used to acquire several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; Encoding module 12 is used to encode several log text data, and based on the obtained log vector, construct positive samples of the same transaction and negative samples of cross transactions to obtain candidate edges; Construction module 13 is configured to obtain a temporal causal graph by performing temporal causal graph construction processing on the candidate edges, wherein the temporal causal graph construction processing is configured to use candidate relationship recognition technology to identify and process the timestamp data and the process identifier data to obtain candidate causal relationship data; perform attribute processing on the timestamp data and the host identifier data to obtain latency attribute data; perform existence probability modeling processing on the candidate edges to obtain edge existence probability; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probability. Detection module 14 is used to perform causal path anomaly detection processing on the time-series causal graph to obtain the causal path anomaly set of the target system; Processing module 15 is used to calculate alarm results based on the set of causal path anomalies.

[0099] The log analysis alarm device may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a log analysis alarm device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the log analysis alarm device may also include input / output devices, network access devices, buses, etc.

[0100] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the log analysis and alarm device, connecting all parts of the device via various interfaces and lines.

[0101] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the log analysis alarm device by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0102] If the integrated module of the log analysis alarm device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0104] Accordingly, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform steps in the log analysis alarm method of the above embodiments, for example... Figure 1 Steps S1 to S5 as described above.

[0105] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A log analysis and alarm method, characterized in that, include: Obtain several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; The log text data is encoded, and based on the obtained log vectors, positive samples within the same transaction and negative samples across transactions are constructed to obtain candidate edges. A temporal causal graph is obtained by constructing a temporal causal graph from the candidate edges. The temporal causal graph construction process is configured to: identify the timestamp data and process identifier data using candidate relationship recognition technology to obtain candidate causal relationship data; perform attribute processing on the timestamp data and host identifier data to obtain latency attribute data; perform existence probability modeling on the candidate edges to obtain edge existence probabilities; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probabilities. The time-series cause-effect graph is subjected to cause-effect path anomaly detection processing to obtain the cause-effect path anomaly set of the target system; Based on the set of abnormal causal paths, the alarm result is calculated.

2. The log analysis and alarm method as described in claim 1, characterized in that, The process of obtaining the log vector includes: Using a pre-trained language model, several log text data are semantically encoded to obtain several semantic vectors; The log vector is obtained by mapping and concatenating several semantic vectors with the corresponding timestamp data, host identifier data and process identifier data.

3. The log analysis and alarm method as described in claim 1, characterized in that, The process of identifying and processing the timestamp data and process identifier data using candidate relationship recognition technology to obtain candidate causal relationship data includes: Constraints are applied to the timestamp data and the process identifier data to obtain a set of candidate log pairs; The candidate log set is processed by combining transaction identifiers with verification using candidate relationship identification technology to obtain the candidate causal relationship data.

4. The log analysis and alarm method as described in claim 1, characterized in that, The step of performing attribute processing on the timestamp data and the host identifier data to obtain latency attribute data includes: The timestamp data and the host identifier data are processed using candidate relationship recognition technology to obtain a candidate edge set. The candidate edge set is sequentially subjected to delay calculation and threshold filtering to obtain the delay attribute data.

5. The log analysis and alarm method as described in claim 1, characterized in that, The existence probability modeling process for the candidate edges to obtain the edge existence probability includes: The candidate edges are processed by feature concatenation using conditional probability model techniques to obtain a concatenated feature vector; The concatenated feature vector is subjected to function activation processing to obtain the probability of the existence of the edge.

6. The log analysis and alarm method as described in claim 1, characterized in that, The process of obtaining the temporal causal graph based on the candidate causal relationship data, the time delay attribute data, and the edge existence probability includes: The candidate causal relationship data, the time delay attribute data, and the edge existence probability are input into the edge feature integration model constructed from the log vector to obtain the edge feature set; The edge feature set is integrated to obtain the temporal causal graph.

7. The log analysis and alarm method as described in claim 1, characterized in that, The causal path anomaly detection process performed on the time-series causal graph yields a set of causal path anomalies for the target system, including: The time-series cause-effect graph is subjected to path search processing to obtain a path set; The path set is input into the constructed path scoring function for processing to obtain the scores of several paths in the path set; The score is compared with a preset threshold to obtain the set of causal path anomalies of the target system.

8. A log analysis alarm device, characterized in that, include: The acquisition module is used to acquire several log text data of the target system and timestamp data, host identifier data and process identifier data corresponding to the log text data; The encoding module is used to encode several log text data, and based on the obtained log vector, construct positive samples of the same transaction and negative samples of cross transactions to obtain candidate edges; A construction module is configured to perform temporal causal graph construction processing on the candidate edges to obtain a temporal causal graph. The temporal causal graph construction process is configured to: use candidate relationship recognition technology to identify the timestamp data and process identifier data to obtain candidate causal relationship data; perform attribute processing on the timestamp data and host identifier data to obtain latency attribute data; perform existence probability modeling processing on the candidate edges to obtain edge existence probabilities; and obtain the temporal causal graph based on the candidate causal relationship data, the latency attribute data, and the edge existence probabilities. The detection module is used to perform causal path anomaly detection processing on the time-series causal graph to obtain the causal path anomaly set of the target system; The processing module is used to calculate alarm results based on the set of causal path anomalies.

9. A log analysis alarm device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the log analysis alarm method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the log analysis alarm method as described in any one of claims 1 to 7.

Images