Abnormal link self-healing method based on log mode evolution learning

Abstract

The invention relates to the field of data processing and intelligent operation and maintenance, in particular to an abnormal link self-healing method based on log mode evolution learning. According to the abnormal link self-healing method based on log mode evolution learning, abnormal logs are recognized based on a log mode evolution library T *, and a self-healing strategy is matched. Matching the current log with the log mode evolution library T * to obtain a probability P of the current log appearing in a template of the log mode evolution library T *, obtaining an abnormal score, and judging whether the current log is an abnormal log or not if the abnormal score is higher than a second threshold value; and clustering the abnormal logs into different abnormal categories by adopting a mode of classifying according to error codes, keywords and templates, and substituting the abnormal categories and corresponding components into a strategy knowledge base to intelligently match corresponding self-healing actions, thereby realizing accurate identification of the abnormal logs and intelligent self-healing. Therefore, the technical problem that an existing abnormal link self-healing method is difficult to accurately recognize abnormal logs of a complex link for intelligent self-healing is solved.

Description

Technical Field

[0001] This invention relates to the fields of data processing and intelligent operation and maintenance, and in particular to an abnormal link self-healing method based on log pattern evolution learning. Background Technology

[0002] With the increasing complexity of information systems and the widespread adoption of data platform architectures, the operation of these systems involves multiple levels (also known as multiple components) of data collection, storage, processing, and service, resulting in numerous links and complex dependencies. Once issues such as data loss, delays, or computational anomalies occur, they often lead to widespread disruption of upstream and downstream links, impacting business continuity and reliability.

[0003] Existing anomaly handling methods primarily rely on manual operations and maintenance or automated scripts based on fixed rules. These methods have the following shortcomings: 1. Fixed log patterns: Traditional log analysis methods depend on preset rules and templates, making it difficult to adapt to the evolving log formats that change with version iterations and runtime environments. 2. Delayed anomaly localization: Relying on manual log filtering or fixed keyword matching fails to capture new or mutated anomaly patterns in a timely manner, easily leading to diagnostic delays. 3. Lack of intelligent self-healing mechanisms: Most existing automated scripts are static processes, unable to dynamically adapt to different component types and anomaly scenarios, making it difficult to meet the intelligent self-healing requirements of complex processes. Summary of the Invention

[0004] To address the technical problem that existing abnormal link self-healing methods struggle to accurately identify abnormal logs in complex links for intelligent self-healing, this invention provides an abnormal link self-healing method based on log pattern evolution learning.

[0005] This invention discloses an anomaly link self-healing method based on log pattern evolution learning. The method identifies anomaly logs and matches them with a self-healing strategy using the log pattern evolution library T*. The method includes: Anomaly detection: Match the current log with the log pattern evolution library T* and calculate the probability P of the current log appearing in the template of the log pattern evolution library T*. Take the logarithm of the probability P and the log pattern evolution library T* to obtain the anomaly score. If the anomaly score is higher than the threshold of two, the current log is determined to be an abnormal log. Self-healing strategy matching: Cluster the abnormal logs into different abnormal categories according to error code, keywords, and templates; substitute the abnormal categories and their corresponding components into the strategy knowledge base to match the corresponding self-healing actions; the strategy knowledge base has preset self-healing actions for different components under different abnormal scenarios. The construction method of the log pattern evolution library T* is as follows: real-time logs from multiple components in the data middleware are collected and preprocessed to obtain a log set L containing multiple standard logs; based on the Drain algorithm, multiple standard logs in the log set L are transformed into multiple templates to obtain a template set T; when the similarity between subsequent new logs and template set T is lower than a threshold, a new template is created; the template set T is continuously updated to construct the log pattern evolution library T*.

[0006] Furthermore, the method for obtaining anomaly scores by taking the logarithmic value of probability P and log pattern evolution library T* includes substituting probability P and log pattern evolution library T* into the anomaly scoring equation to obtain the anomaly score, as shown in the following anomaly scoring equation: ; in, li This indicates the log instance to be analyzed. These are the outlier scores in this analysis.

[0007] Furthermore, the second threshold is 3.

[0008] Furthermore, when matching self-healing actions, priority can be used to calculate the corresponding self-healing action. The method for calculating the corresponding self-healing action based on priority includes: using a multi-index scoring function to score the historical success rate, average time consumption, and the degree of matching between the self-healing action and the current scene, and then executing the self-healing action based on the strategy with the highest score. The multi-index scoring function is as follows: ; in, Indicates historical success rate; Indicates the average execution time; Indicates the degree of matching with the current scene; These are weight parameters; This indicates the final score.

[0009] Furthermore, the preprocessing process after log collection includes: timestamp alignment: uniformly converting to UTC time; regularization: using regular expressions to extract key fields containing IP, port, SQL statement, and error code; noise removal: removing noise from debug logs and heartbeat logs.

[0010] Furthermore, the expression for the log set L is: ; in, This represents the nth standardized log entry.

[0011] Furthermore, the method for transforming multiple standard logs in log set L into multiple templates to obtain template set T based on the Drain algorithm includes: vectorizing the logs to list multiple words; substituting several vectorized logs containing multiple words into the tree-structure-based Drain algorithm to obtain template set T; the log vectorization representation is as follows: ; Where t represents the m-th word in the log, if it is a constant it is retained, if it is a variable it is replaced with the wildcard "*"; The template set T is represented as follows: ; in, Ck This indicates the first [item] in the log template. k The template corresponding to each word element position; word is a specific lexical value constant, and * is a wildcard.

[0012] Furthermore, the threshold is 0.8.

[0013] Furthermore, similarity is defined as follows: ; in, This indicates the number of terms in the log that match the template. This indicates a new log entry.

[0014] Furthermore, the log pattern evolution learning-based anomaly link self-healing method also includes an automatic execution and feedback learning mechanism and a policy knowledge base update mechanism. The automatic execution and feedback learning mechanism includes: executing the self-healing strategy and monitoring the link recovery status in real time; if the link recovers normally, marking the handling as successful and updating the success rate of the self-healing strategy in the policy knowledge base; if the handling fails, triggering a manual alarm and storing the failure sample in the policy knowledge base; the policy knowledge base update mechanism includes: recording new anomaly patterns and handling results in the policy knowledge base; the policy knowledge base is optimized using reinforcement learning, using Q-learning to adjust the priority of different strategies, and automatically updating the policy knowledge base through continuous learning.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Based on the matching of the current log with the log pattern evolution library T*, the probability P of the current log appearing in the template of the log pattern evolution library T* is obtained. The anomaly score is obtained by taking the logarithm of the probability P and the log pattern evolution library T*. If the anomaly score is higher than the threshold, the current log is determined to be an anomalous log, indicating that the problematic anomalous log has been accurately found. Then, the anomalous logs are clustered into different anomaly categories by error code, keyword, and template classification. The anomaly categories and their corresponding components are used to intelligently match the corresponding self-healing actions in the strategy knowledge base, thereby achieving accurate identification of anomalous logs and intelligent self-healing. This solves the technical problem that existing anomaly link self-healing methods are unable to accurately identify anomalous logs in complex links for intelligent self-healing.

[0016] 2. The strategy knowledge base can be continuously and automatically updated to adapt to new types of abnormal logs, thereby meeting the intelligent self-healing requirements of complex processes.

[0017] 3. Priority-based calculation can improve the accuracy of matching anomaly logs with self-healing strategies.

[0018] 4. Typical handling steps are transformed into reusable action templates, and reinforcement learning algorithms are used to provide feedback optimization of the self-healing strategy, enabling the evolution and updating of the self-healing strategy. Compared with static scripts, this invention can cover a wider range of anomaly types, and the mean time to recovery (MTTR) can be reduced by more than 30%. Attached Figure Description

[0019] Figure 1 Flowchart of an anomaly self-healing method based on log pattern evolution learning; Figure 2 This is a system architecture diagram for an anomaly link self-healing method based on log pattern evolution learning. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Embodiments of this invention provide an anomaly link self-healing method based on log pattern evolution learning, which identifies anomaly logs and matches them with self-healing strategies based on the log pattern evolution library T*. For example... Figure 1As shown, the method includes two steps: anomaly detection and diagnosis, and self-healing strategy matching. The anomaly detection and diagnosis step accurately identifies abnormal logs in the logs, and the self-healing strategy matching step matches the abnormal logs with the corresponding self-healing strategies, so as to accurately identify abnormal logs in complex links and perform intelligent self-healing.

[0022] Anomaly detection and diagnosis: 1. By inputting the current logs corresponding to multiple components, these current logs are matched with the log pattern evolution library T*.

[0023] The construction method of the log pattern evolution library T* is as follows: real-time logs from multiple components in the data platform (such as the collection layer, access layer, storage layer, computing layer, service layer, etc.) are collected and preprocessed to obtain a log set L containing multiple standard logs; based on the Drain algorithm, the multiple standard logs in the log set L are transformed into multiple templates to obtain a template set T; when the similarity between a subsequent new log and the template set T is lower than a threshold, a new template is created; the template set T is continuously updated to construct the log pattern evolution library T*.

[0024] 1.1 The preprocessing steps are as follows: timestamp alignment, uniformly converting to UTC time; regularization processing, using regular expressions to extract key fields containing IP, port, SQL statements, and error codes; noise removal, removing noise (low-value content) including debug logs and heartbeat logs. After preprocessing, a log set L is obtained: ; in, This represents the nth standardized log entry.

[0025] 1.2 When constructing the template set T, the logs are vectorized to list multiple terms; several vectorized logs containing multiple terms are then substituted into the tree-based Drain algorithm to obtain the template set T (the tree-based Drain algorithm transforms logs into templates). The log vectorization representation in this step is as follows: ; Here, t represents the m-th term in the log; if it is a constant, it is retained; if it is a variable, it is replaced with the wildcard "*". By statistically abstracting the terms at each term position in multiple logs, stable terms are retained as template elements or wildcards, forming a template set T, which is represented as follows: ; in, Ck This indicates the first [item] in the log template. k The template corresponding to each word element position; word is a specific lexical value constant, and * is a wildcard.

[0026] 1.3 When evolving templates, a threshold of 0.8 is used as the standard for generating new templates. When a new log... If the similarity to template set T is below a threshold of 0.8, the log is considered to not match any template, and a new template needs to be created. The similarity is defined as follows: ; in, This indicates the number of terms in the log that match the template. This indicates a new log entry.

[0027] By continuously updating the template set T, a log pattern evolution library T* is built to dynamically reflect the evolution of log formats and operating environments.

[0028] 2. Calculate the frequency of occurrence of the log to be analyzed (the current log) in the templates of the log pattern evolution library T*, and estimate the probability P of the current log appearing in the templates of the log pattern evolution library T* based on this frequency. Take the logarithm of the probability P and the log pattern evolution library T* to obtain an anomaly score. If the anomaly score is higher than a threshold of 2, the current log is determined to be an anomalous log. The threshold of 2 is 3.0 or another configurable threshold. Substitute the probability P and the log pattern evolution library T* into the anomaly score equation to obtain the anomaly score, as follows: ; in, li This indicates the log instance to be analyzed. These are the outlier scores in this analysis.

[0029] Self-healing strategy matching: 3. Cluster exception logs into different exception categories based on error codes, keywords, and templates. Substitute the exception categories and their corresponding components into the strategy knowledge base to match the corresponding self-healing actions.

[0030] The strategy knowledge base pre-defines self-healing actions for different components under various abnormal scenarios, such as "restarting the process," "switching backup nodes," "clearing the cache," and "triggering data re-collection." When selecting a strategy, since a single component may have multiple abnormal types (e.g., server services may experience dozens of types such as insufficient CPU resources, memory overflow, disk full, and network latency), it is necessary to search the candidate strategy set based on the abnormality category and component type to select the appropriate self-healing strategy. This invention further proposes a priority calculation method to improve the matching accuracy of the self-healing strategy. The priority calculation method is as follows: A multi-metric scoring function is used to score the historical success rate, average time consumption, and matching degree between the self-healing action and the current scene, and the strategy with the highest score is used to execute the self-healing action; the multi-metric scoring function is as follows: ; in, Indicates historical success rate; Indicates the average execution time; Indicates the degree of matching with the current scene; These are weight parameters; This indicates the final score.

[0031] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0032] In summary, the probability P of the current log appearing in the template of the log pattern evolution library T* is obtained by matching the current log with the log pattern evolution library T*. The anomaly score is obtained by taking the logarithm of the probability P and the log pattern evolution library T*. Whether the anomaly score is higher than the threshold value is used to determine whether the current log is an anomalous log. If so, it means that the problematic anomalous log has been accurately found. Then, the anomalous logs are clustered into different anomaly categories by error code, keyword, and template classification. The anomaly categories and their corresponding components are used to intelligently match the corresponding self-healing actions in the strategy knowledge base, thereby achieving accurate identification of anomalous logs and intelligent self-healing. This solves the technical problem that existing anomaly link self-healing methods are unable to accurately identify anomalous logs in complex links for intelligent self-healing.

[0033] In addition, see Figure 2 , Figure 2 This paper presents a system architecture diagram for an anomaly self-healing method based on log pattern evolutionary learning. The diagram illustrates a closed-loop mechanism encompassing log pattern evolutionary learning, anomaly detection and diagnosis, self-healing strategy matching, automatic execution and feedback learning, and updating the self-healing database. This closed-loop mechanism enables dynamic learning of log patterns and accumulation of anomaly handling knowledge, achieving rapid identification and automatic repair of link anomalies, thereby improving the stability and operational intelligence of the data platform system.

[0034] Initial log collection and preprocessing: Logs are collected from various components of the data platform and then cleaned, segmented, and standardized.

[0035] Log Pattern Evolutionary Learning: This approach dynamically identifies changing trends in log patterns through log template mining and evolutionary learning algorithms, constructing a log pattern evolution library to capture new patterns that emerge with system upgrades and changes in the operating environment. It identifies changes in log patterns over time and with the environment through template mining and evolutionary learning algorithms. Although system logs are voluminous and varied in format, they exhibit recurring patterns at the underlying level (e.g., identical SQL error templates, timeout error templates). First, it distinguishes between invariant fields (fixed phrases) and variable fields (parameters, values), thus forming generalizable patterns. Second, when the system version is upgraded or the operating environment changes, old patterns cannot cover new log formats (templates), therefore an evolutionary mechanism is needed. This involves continuously comparing new samples with existing clusters; if the similarity is insufficient, a new cluster is generated. This maintains the inheritance of old knowledge while quickly adapting to new knowledge. This mechanism dynamically discovers, extracts, and updates abnormal patterns in the log stream, solving the problem that traditional static rules cannot adapt to new anomalies. This mechanism can also be incorporated into an online log pattern evolutionary learning module as hardware. The log pattern evolution learning algorithm achieves dynamic learning of log patterns and anomaly pattern recognition through a clustering + pattern update mechanism. Existing technologies mostly rely on fixed rules or static models, requiring manual rule updates when log patterns change, leading to response delays. This invention, by introducing a log pattern evolution learning mechanism, can discover new anomaly patterns in the log stream in real time and dynamically update the pattern library (log pattern evolution library T*), enabling the system to have continuous adaptive capabilities and significantly improving the recognition rate of unknown anomalies.

[0036] Anomaly Detection and Diagnosis: The frequency and distribution of log patterns are relatively stable during normal operation. However, once an anomaly occurs, the distribution of log patterns will change abruptly or shift. Based on probabilistic modeling and distribution deviation: If the probability of a certain pattern occurring is too low or its deviation exceeds a set threshold (i.e., the anomaly score is higher than threshold two), it can be considered an anomaly. In complex chains, there are dependencies between components, and anomalies often propagate along the chain. Traditional methods mostly rely on single-point log alarms, making it difficult to accurately locate the root cause in the chain. This invention obtains an anomaly score by taking the logarithmic value of the probability P and the log pattern evolution library T*. If the anomaly score is higher than threshold two, it is determined whether the current log is an anomaly log, thus tracing the source of the anomaly. Compared with existing location methods that rely on human experience, this invention can improve the identification accuracy by more than 20%.

[0037] Self-healing strategy matching: Based on component type, anomaly category, and historical handling experience, the optimal self-healing solution is selected. When a system anomaly occurs, not all handling methods are effective or efficient. Traditional operations and maintenance rely on manual experience, while the core of intelligent self-healing is a sample database (strategy knowledge base) and strategy optimization. The strategy knowledge base is similar to a set of actions, storing possible handling methods for different component anomalies. The principle of strategy matching is multi-dimensional optimization selection: comprehensively considering historical success rate, execution cost, and environmental matching degree, candidate strategies are scored to achieve the selection of the "best action." Existing operations and maintenance self-healing solutions are mostly based on static scripts, which have poor adaptability and often only cover a few common anomalies. This invention transforms typical handling steps into reusable action templates and combines reinforcement learning algorithms to optimize the strategy effect through feedback, realizing the evolution and updating of the self-healing strategy. Compared with static scripts, this invention can cover a wider range of anomaly types, and the mean time to recovery (MTTR) can be shortened by more than 30%. It abstracts historical manual handling steps into templates and continuously optimizes them through feedback mechanisms such as reinforcement learning, realizing the dynamic evolution of the self-healing process. The feedback optimization mechanism adjusts the strategy priority and applicable scenarios.

[0038] Automated Execution and Feedback Learning: The selected self-healing strategy is automatically applied to the target link, and the execution results are monitored. If the handling is successful, the self-healing knowledge base is updated; if it fails, manual intervention is triggered, and the strategy base is optimized. 1. Execute the strategy and monitor the link recovery status in real time; 2. If the link recovers normally, mark the handling as successful and update the success rate of the self-healing strategy in the strategy knowledge base; 3. If the handling fails, trigger a manual alarm, store the failed sample in the strategy knowledge base, mark it as a failure, and optimize the process after manual investigation. If self-healing handling is indeed impossible, an alarm will be triggered directly in the next matching. There are also some extreme cases, such as some components of the data platform not providing recovery interfaces, requiring manual handling. Such cases are extremely rare, accounting for less than 0.1% of all tasks. Failed samples are used to optimize the model, which refers to setting up a reinforcement learning model to automatically update the strategy knowledge base. After selecting a strategy, the system needs to execute it automatically. This process is similar to the closed-loop feedback of the control system. Its principle is based on the action-feedback mechanism, that is, execute the action (self-healing operation), monitor the result (whether the link is restored), and adjust the strategy knowledge base according to the result. If the feedback is successful, the strategy is strengthened; if it fails, the strategy's weight is weakened.

[0039] Update the self-healing database: This includes a policy knowledge base. New anomaly categories and their handling results are recorded in the policy knowledge base. The policy knowledge base is optimized using reinforcement learning, leveraging Q-learning to adjust the priority of different policies. Through continuous learning, the database achieves self-updating and evolution.

[0040] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An anomaly link self-healing method based on log pattern evolution learning, characterized in that, Based on the log pattern evolution library T*, abnormal logs are identified and matched with self-healing strategies. The abnormal link self-healing methods include: Anomaly detection: Match the current log with the log pattern evolution library T* and calculate the probability P of the current log appearing in the template of the log pattern evolution library T*. Take the logarithm of the probability P and the log pattern evolution library T* to obtain the anomaly score. If the anomaly score is higher than the threshold of two, the current log is determined to be an abnormal log. Self-healing strategy matching: Cluster the abnormal logs into different abnormal categories according to error code, keywords, and templates; substitute the abnormal categories and their corresponding components into the strategy knowledge base to match the corresponding self-healing actions; the strategy knowledge base has preset self-healing actions for different components under different abnormal scenarios. The construction method of the log pattern evolution library T* is as follows: real-time logs from multiple components in the data middleware are collected and preprocessed to obtain a log set L containing multiple standard logs; based on the Drain algorithm, multiple standard logs in the log set L are transformed into multiple templates to obtain a template set T; when the similarity between subsequent new logs and template set T is lower than a threshold, a new template is created; the template set T is continuously updated to construct the log pattern evolution library T*.

2. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, The method for obtaining anomaly scores by taking the logarithmic value of probability P and log pattern evolution library T* includes substituting probability P and log pattern evolution library T* into the anomaly scoring equation to obtain the anomaly score, as shown in the following equation: ； in, li This indicates the log instance to be analyzed. These are the outlier scores in this analysis.

3. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, Threshold 2 is 3.

4. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, When matching self-healing actions, the corresponding self-healing action can also be calculated based on priority. The priority-based method for calculating and matching corresponding self-healing actions includes: using a multi-index scoring function to score the historical success rate, average time consumption, and matching degree between the self-healing action and the current scene, and executing the self-healing action based on the strategy with the highest score; the multi-index scoring function is as follows: ； in, Indicates historical success rate; Indicates the average execution time; Indicates the degree of matching with the current scene; These are weight parameters; This indicates the final score.

5. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, The preprocessing process after log collection includes: Timestamp alignment: uniformly convert to UTC time; Regularization processing: Use regular expressions to extract key fields containing IP address, port number, SQL statement, and error code; Denoising: Remove noise including debug logs and heartbeat logs.

6. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, The expression for the log set L is: ； in, This represents the nth standardized log entry.

7. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, The method for transforming multiple standard logs in log set L into multiple templates to obtain template set T based on the Drain algorithm includes: vectorizing the logs to list multiple words; substituting several vectorized logs containing multiple words into the tree-structure-based Drain algorithm to obtain template set T; the log vectorization representation is as follows: ； Where t represents the m-th word in the log, if it is a constant it is retained, if it is a variable it is replaced with the wildcard "*"; The template set T is represented as follows: ； in, Ck This indicates the first [item] in the log template. k The template corresponding to each word element position; word is a specific lexical value constant, and * is a wildcard.

8. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, The threshold is 0.

8.

9. The abnormal link self-healing method based on log pattern evolution learning according to claim 1, characterized in that, Similarity is defined as follows: ； in, This indicates the number of terms in the log that match the template. This indicates a new log entry.

10. The abnormal link self-healing method based on log pattern evolutionary learning according to claim 1, characterized in that, The log pattern evolution learning-based anomaly link self-healing method also includes automatic execution and feedback learning mechanisms, and policy knowledge base update mechanisms; The automatic execution and feedback learning mechanism includes: executing self-healing strategies and monitoring the link recovery status in real time; if the link recovers normally, marking the handling as successful and updating the success rate of the self-healing strategy in the strategy knowledge base; if the handling fails, triggering a manual alarm and storing the failure samples in the strategy knowledge base. The strategy knowledge base update mechanism includes: recording new anomaly patterns and their handling results into the strategy knowledge base; optimizing the strategy knowledge base using reinforcement learning, adjusting the priority of different strategies using Q-learning, and automatically updating the strategy knowledge base through continuous learning.

Images