Log classification method and device, electronic equipment and storage medium

By using a BERT-based log defect classification model, combined with the Drain algorithm and model pruning techniques, the problems of inaccurate log classification and high resource consumption are solved, achieving efficient log classification on terminal devices.

CN116432099BActive Publication Date: 2026-04-24YINGCHE XINGCHUANG INTELLIGENT TECH (SHANGHAI) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
YINGCHE XINGCHUANG INTELLIGENT TECH (SHANGHAI) CO LTD
Filing Date
2023-03-16
Publication Date
2026-04-24

AI Technical Summary

Technical Problem

Traditional log classification methods suffer from inaccurate results due to large log text volumes, inconsistent log output formats, and unclear contextual semantics. Furthermore, the analysis and statistical models incur significant memory overhead, time sensitivity, and resource consumption, making them difficult to apply to terminal devices with limited memory and storage capacity.

Method used

A BERT-based log defect classification model is adopted. Logs are parsed using regular expressions, log token data is extracted using the Drain algorithm, and log token data with importance scores is filtered through a multi-layer network structure. The precision of Query, Key, and Value values ​​is adjusted to reduce memory overhead and resource consumption.

Benefits of technology

It improves the accuracy of log classification, reduces memory overhead and resource consumption, has a certain degree of generalization ability, and is suitable for terminal devices with small memory capacity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116432099B_ABST
    Figure CN116432099B_ABST
Patent Text Reader

Abstract

The application provides a log classification method and device, electronic equipment and a storage medium. The log classification method comprises the following steps: collecting a log to be classified, analyzing the log to be classified, and obtaining a log template sequence to be classified; inputting the log template sequence to be classified into a pruned log defect classification model to output a log classification result, wherein the log defect classification model is trained according to a log template sequence training set, and the model inference on the log to be classified can automatically classify according to log semantics, thereby ensuring the accuracy of the classification. Moreover, the pruned log defect classification model can reduce memory overhead, time efficiency and resource consumption, and has a certain generalization ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of log classification technology, and in particular to a log classification method, apparatus, electronic device and storage medium. Background Technology

[0002] The runtime state of software is crucial for software quality assessment. Logging records of runtime anomalies allows for effective analysis of problems encountered during software operation, facilitating system optimization. In current software development and operation, using logs to detect, classify, statistically analyze, and locate runtime issues is a common technique used by developers and maintenance personnel. Traditional log classification methods utilize Aho-Corasick automatons (AC), a type of finite-state automaton used for multi-pattern string matching. However, due to the large volume of log text, inconsistent log output formats, and unclear contextual semantics, AC automaton classification results are inaccurate, hindering subsequent anomaly localization. Related technologies employ analytical and statistical modeling techniques for real-time automated log analysis, such as traditional decision tree models and transformer models for log text analysis and anomaly classification. However, these models are computationally intensive, resulting in significant memory overhead, time constraints, and resource consumption, making them unsuitable for terminal devices with limited memory and storage, such as in-vehicle terminals. Summary of the Invention

[0003] This invention provides a log classification method, apparatus, electronic device, and storage medium to address the shortcomings of traditional AC automata, which suffer from inaccurate log classification results due to large log text volume, inconsistent log output formats, and unclear contextual semantics, as well as the high memory overhead, timeliness, and resource consumption required for analysis and statistical models.

[0004] This invention provides a log classification method, comprising:

[0005] Collect logs to be categorized;

[0006] The logs to be classified are parsed to obtain the log template sequence.

[0007] The log template sequence to be classified is input into the pruned log defect classification model to output the log classification result, wherein the log defect classification model is trained based on the log template sequence training set.

[0008] According to a log classification method provided by the present invention, the step of parsing the log to be classified to obtain a log template sequence includes:

[0009] The node identifiers in the logs to be classified are found using regular expressions, and the log token data of the logs to be classified is extracted using the node identifiers as the dividing points.

[0010] The field length of the log token data is matched with the log length of the first-level node in the predetermined parse tree. After a node with the same length is matched, the matching check of the downstream child nodes of this node is continued.

[0011] The complete words of the log token data are matched with the complete words in the corresponding template branches of the predetermined parse tree. After a node with the same complete word is matched, the remaining log token data is matched with the downstream sub-branches of this node until the matching degree with a certain log template in the predetermined parse tree exceeds a preset threshold. Then the log template to which the log belongs is obtained.

[0012] According to a log classification method provided by the present invention, the log defect classification model is a BERT-based log defect classification model, and the training method of the log defect classification model includes:

[0013] The historical log data is parsed to obtain the training log template sequence;

[0014] A log template sequence training set is generated based on the training log template sequence and its corresponding labeled classification results;

[0015] The training log template sequence is input into the BERT-based log defect classification model to obtain the predicted classification result;

[0016] The loss function is obtained based on the predicted classification result and the labeled classification result corresponding to the training log template sequence in the log template sequence training set.

[0017] The parameters in the BERT-based log defect classification model are optimized based on the results of the loss function.

[0018] According to a log classification method provided by the present invention, the BERT-based log defect classification model includes multiple network structure layers, each network structure layer corresponding to a computation module, and the pruning method of the log defect classification model includes:

[0019] In each structural layer, the word embedding tensor of the training log template sequence is converted into a Query matrix, a Key matrix, and a Value matrix;

[0020] The attention score of each log token data obtained in the calculation module corresponding to the previous structural layer is used to calculate the importance score of each log token data.

[0021] Log token data corresponding to multiple important threshold ranges are filtered out based on the importance score;

[0022] The precision of the Query, Key, and Value values ​​corresponding to each log token data is adjusted according to different importance thresholds in order to prune the BERT-based log defect classification model.

[0023] According to a log classification method provided by the present invention, the calculation of the importance score of each log token data includes:

[0024] For each log token in the current network structure layer, obtain the attention score matrix of the current structure layer by taking the Query value, Key value, and Value value corresponding to each log token data.

[0025] The sum of each column in the attention score matrix of the current network structure layer is used as the current structure layer importance score of the corresponding log token data;

[0026] The importance score of the current structural layer is added to the importance score of the previous structural layer, and the result is used as the importance score of each log token data in the next layer.

[0027] According to a log classification method provided by the present invention, the step of filtering log token data corresponding to multiple importance threshold ranges based on the importance score includes:

[0028] The log token data is divided into important log token data and unimportant log token data based on a first importance threshold.

[0029] The important log token data is divided into the most important log token data and the medium important log token data based on the second importance threshold.

[0030] According to a log classification method provided by the present invention, adjusting the precision of the Query value, Key value, and Value value corresponding to each log token data according to different importance thresholds includes:

[0031] Adjust the precision of the Query, Key, and Value values ​​corresponding to the most important log token data to the first precision value;

[0032] Adjust the precision of the Query, Key, and Value values ​​corresponding to the moderately important log token data to the second precision value;

[0033] Adjust the precision of the Query, Key, and Value values ​​corresponding to unimportant log token data to the third precision value;

[0034] The first precision value is greater than the second precision value, and the second precision value is greater than the third precision value.

[0035] The log classification method provided by the present invention further includes:

[0036] Record the classification information for each log to be classified, including the cause of the log anomaly, the anomaly classification, and the confidence information of the classification result;

[0037] Real-time anomaly alerts and / or offline analysis are performed based on the classification information.

[0038] The present invention also provides a log classification device, comprising:

[0039] The collection module is used to collect logs to be categorized.

[0040] The parsing module is used to parse the logs to be classified and obtain the log template sequence to be classified.

[0041] The classification module is used to input the log template sequence to be classified into the pruned log defect classification model to output the log classification result, wherein the log defect classification model is trained based on the log template sequence training set.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the log classification method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log classification method as described above.

[0044] The log classification method, apparatus, electronic device, and storage medium provided by this invention collect logs to be classified; parse the logs to be classified to obtain a log template sequence; input the log template sequence into a pruned log defect classification model to output the log classification result. The log defect classification model is trained based on a log template sequence training set. Since parsing the logs to be classified can automatically classify them according to log semantics, the accuracy of classification is guaranteed. Furthermore, the pruned log defect classification model can reduce memory overhead, reduce time and resource consumption, and has a certain generalization ability. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is one of the flowcharts illustrating the log classification method provided by the present invention;

[0047] Figure 2 This is the second flowchart of the log classification method provided by the present invention;

[0048] Figure 3 This is the third flowchart of the log classification method provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the log defect classification model structure based on BERT provided by the present invention;

[0050] Figure 5 This is the fourth flowchart of the log classification method provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the attention score matrix provided by the present invention;

[0052] Figure 7 This is the fifth flowchart illustrating the log classification method provided by the present invention;

[0053] Figure 8 This is a schematic diagram of the log classification device provided by the present invention;

[0054] Figure 9 This is an application architecture diagram of the log classification device provided by the present invention;

[0055] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] Figure 1 A flowchart of the log classification method provided in the embodiments of the present invention is shown below. Figure 1As shown, the log classification method provided in this embodiment of the invention includes:

[0058] Step 101: Collect logs to be categorized;

[0059] Step 102: Parse the logs to be classified to obtain the log template sequence;

[0060] Step 103: Input the log template sequence to be classified into the pruned log defect classification model to output the log classification result. The log defect classification model is trained based on the log template sequence training set.

[0061] Traditional log classification methods rely on Aho-Corasick automata (AC automata) for automatic log categorization. However, the large volume of log text, inconsistent output formats, and unclear contextual semantics lead to inaccurate classification results from AC automata, hindering subsequent software anomaly localization. Related technologies employ analytical and statistical modeling techniques for real-time automated log analysis, such as traditional decision tree models and transformer models for log text analysis and anomaly classification. However, these models are computationally intensive, resulting in significant memory overhead, time constraints, and resource consumption, making them unsuitable for terminal devices with limited memory and storage, such as in-vehicle terminals.

[0062] The log classification method provided in this embodiment of the invention collects logs to be classified, parses the logs to be classified, and obtains a log template sequence to be classified. The log template sequence is then input into a pruned log defect classification model to output the log classification result. The log defect classification model is trained based on the log template sequence training set. Since parsing the logs to be classified can automatically classify them according to the log semantics, the accuracy of the classification is guaranteed. Furthermore, the pruned log defect classification model can reduce memory overhead, reduce time and resource consumption, and has a certain generalization ability.

[0063] like Figure 2 As shown, in this embodiment of the invention, the logs to be classified are parsed, and the log template sequence to be classified includes:

[0064] Step 201: Find the node identifier in the log to be classified using regular expressions, and extract the log token data of the log to be classified using the node identifier as the split point.

[0065] In this embodiment of the invention, node identifiers include, but are not limited to, unstructured text data such as spaces, commas, periods, and colons.

[0066] For example, if the log to be categorized includes "Li Bai was famous in the Tang dynasty" and the node identifier is a space, then the log token data can be extracted as Li, Bai, was, famous, in, Tang, and dynasty.

[0067] Step 202: Match the field length of the log token data with the log length of the first-level node in the predefined parse tree. After matching a node with the same length, continue to check the matching of the downstream child nodes of this node.

[0068] In this embodiment of the invention, the Drain algorithm is used to match the field length of the log token data with the log length of the first-level node in the parse tree of the Drain algorithm.

[0069] Step 203: Match the complete words of the log token data with the complete words in the corresponding template branches of the predetermined parse tree. After matching a node with the same complete word, continue to match the remaining log token data with the downstream sub-branches of this node until the matching degree with a certain log template in the predetermined parse tree exceeds a preset threshold. Then, the log template to which the log belongs is obtained.

[0070] In this embodiment of the invention, the Drain algorithm is used to match the complete words in the log token data with the complete words in the template branch corresponding to the parse tree in the Drain algorithm.

[0071] In this embodiment of the invention, Drain is used to parse log data, replacing traditional classification methods such as the AC automaton. This allows for more flexible identification and classification of anomaly types by understanding the semantics of the logs, thereby improving the accuracy of the classification results.

[0072] Based on any of the above embodiments, this log defect classification model is a log defect classification model based on the BERT (Bidirectional Encoder Representation from Transformers) model, such as... Figure 3 As shown, the training method for this log defect classification model includes:

[0073] Step 301: Use the Drain algorithm to parse the historical log data to obtain the training log template sequence;

[0074] Step 302: Generate a log template sequence training set based on the training log template sequence and its corresponding labeled classification results;

[0075] Step 303: Input the training log template sequence into the BERT-based log defect classification model to obtain the predicted classification results;

[0076] Step 304: Obtain the loss function result based on the predicted classification result and the labeled classification result corresponding to the training log template sequence in the log template sequence training set;

[0077] Step 305: Optimize the parameters in the BERT-based log defect classification model based on the results of the loss function.

[0078] In this embodiment of the invention, since the log defect classification model is trained based on the log template sequence, and the token data in the log template sequence has a uniform format and does not contain unstructured data, BERT utilizes all the contextual information of each word during the encoding process of the input text. Compared with the unidirectional encoder that can only use pre-sequence information to extract semantics, BERT has a stronger semantic information extraction capability. Therefore, training the BERT-based log defect classification model based on the log template sequence training set can further improve the accuracy of the classification results of the BERT-based log defect classification model.

[0079] Based on any of the above embodiments, the structure of the BERT-based log defect classification model is as follows: Figure 4 As shown, it includes multiple network structure layers, each corresponding to a computing module.

[0080] like Figure 5 As shown, the pruning method for this log defect classification model includes:

[0081] Step 501: In each structural layer, convert the token embeddings of the training log template sequence into a Query matrix, a Key matrix, and a Value matrix;

[0082] Step 502: Calculate the importance score of each log token data obtained in the calculation module corresponding to the previous structural layer;

[0083] In this embodiment of the invention, calculating the importance score for each log token data includes:

[0084] Step 5021: Obtain the attention score matrix of the current structure layer based on the Query value, Key value, and Value value corresponding to each log token data in the current structure layer;

[0085] In this embodiment of the invention, the attention score matrix is ​​as follows: Figure 6 As shown;

[0086] Step 5022: Use the sum of each column in the attention score matrix of the current network structure layer as the importance score of the current structure layer for the corresponding log token data;

[0087] Step 5023: Add the importance score of the current network structure layer to the importance score of the previous structure layer, and use the sum as the importance score of each log token data in the next layer.

[0088] Step 503: Filter log token data corresponding to multiple important threshold ranges based on importance scores;

[0089] In this embodiment of the invention, log token data corresponding to multiple importance threshold ranges are filtered out based on importance scores, including:

[0090] The log token data is divided into important log token data and unimportant log token data based on a first importance threshold.

[0091] The important log token data is divided into the most important log token data and the medium important log token data based on the second importance threshold.

[0092] Step 504: Adjust the precision of the Query, Key, and Value values ​​corresponding to each log token data according to different importance thresholds in order to prune the BERT-based log defect classification model.

[0093] In this embodiment of the invention, adjusting the precision of the Query value, Key value, and Value value corresponding to each log token data according to different importance thresholds includes:

[0094] Adjust the precision of the Query, Key, and Value values ​​corresponding to the most important log token data to the first precision value;

[0095] Adjust the precision of the Query, Key, and Value values ​​corresponding to the moderately important log token data to the second precision value;

[0096] Adjust the precision of the Query, Key, and Value values ​​corresponding to unimportant log token data to the third precision value;

[0097] The first precision value is greater than the second precision value, and the second precision value is greater than the third precision value.

[0098] The first precision value is, for example, 8 bits, the second precision value is, for example, 4 bits, and the third precision value is, for example, 0 bits.

[0099] The first importance threshold is, for example, 85% of the log token data in terms of importance, and the second importance threshold is, for example, 70% of the log token data in terms of importance.

[0100] In some embodiments of the present invention, the log classification method further includes:

[0101] Record the classification information for each log to be classified, including the cause of the log anomaly, the anomaly classification, and the confidence information of the classification result;

[0102] Real-time anomaly alerts and / or offline analysis are performed based on classification information.

[0103] In this embodiment of the invention, real-time anomaly alarms and / or offline analysis can promptly alert users to abnormal situations, making it easier for users to locate problems and continuously improve the system.

[0104] like Figure 7 As shown, in some embodiments, the log classification method includes a training phase and a runtime phase. During the training phase, the method includes the following steps:

[0105] Step 1: Real-time log collection;

[0106] Step 2: Extract key data, preprocess logs, use Drain to parse log data, and build a training dataset;

[0107] Step 3: Build and train the original defect classification and analysis model based on BERT;

[0108] Step 4: Perform dynamic quantization pruning on the original defect classification model based on the token to obtain the optimized defect classification model.

[0109] During the runtime phase, the method includes the following steps:

[0110] Step 5: Deploy the defect classification model obtained in Step 4 to the log analysis service;

[0111] Step 6: During program execution, collect runtime logs, use the Drain algorithm to parse the log data, and use the accelerated defect classification model to identify and classify anomalies in the parsed logs, saving analysis records such as anomaly causes, anomaly classifications, and confidence levels.

[0112] Step 7: Based on the analysis records obtained in Step 6, perform real-time anomaly alerts and offline aggregate analysis.

[0113] In this embodiment of the invention, the DTQ (Dynamic token-based quantization) algorithm architecture is used to lightweight the BERT model, reducing computational load and accelerating prediction speed. Since different tokens exhibit varying degrees of tolerance to noise, the importance score of log token data in each computation module is calculated using attention probabilities. Based on the importance score, the distribution of log token data is adjusted to a score range of 15% important, 70% moderately important, and 15% unimportant. Unimportant log token data is quantized to 0 bits for model compression, thereby accelerating classification.

[0114] The log classification device provided by the present invention is described below. The log classification device described below and the log classification method described above can be referred to in correspondence.

[0115] Figure 8 This is a schematic diagram of a log classification device provided in an embodiment of the present invention, such as... Figure 8 As shown, the log classification device provided in this embodiment of the invention includes:

[0116] Collection module 801 is used to collect logs to be classified;

[0117] Parsing module 802 is used to parse the logs to be classified and obtain the log template sequence to be classified;

[0118] The classification module 803 is used to input the log template sequence to be classified into the pruned log defect classification model to output the log classification result. The log defect classification model is trained based on the log template sequence training set.

[0119] like Figure 9 As shown, a log classification device collects log data from each terminal's app. In some embodiments of this invention, the log classification device includes a log parser, a model training module, a model compression module, an inference classification module, a log storage module, an anomaly alarm module, and a statistical display module. This log classification device interfaces with a data storage layer, which includes a log repository and a database.

[0120] The log classification device provided in this embodiment of the invention collects logs to be classified; parses the logs to be classified to obtain a log template sequence; and inputs the log template sequence into a pruned log defect classification model to output the log classification result. The log defect classification model is trained based on the log template sequence training set. Since parsing the logs to be classified can automatically classify them according to the log semantics, the accuracy of the classification is guaranteed. Furthermore, the pruned log defect classification model can reduce memory overhead, reduce time and resource consumption, and has a certain generalization ability.

[0121] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a log classification method. This method includes: collecting logs to be classified; parsing the logs to be classified to obtain a log template sequence; inputting the log template sequence into a pruned log defect classification model to output a log classification result. The log defect classification model is trained based on a log template sequence training set.

[0122] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the log classification method provided by the above methods. The method includes: collecting logs to be classified; parsing the logs to be classified to obtain a log template sequence; and inputting the log template sequence into a pruned log defect classification model to output a log classification result, wherein the log defect classification model is trained based on a log template sequence training set.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log classification method, characterized in that, include: Collect logs to be categorized; The logs to be classified are parsed to obtain the log template sequence. The log template sequence to be classified is input into the pruned log defect classification model to output the log classification result. The log defect classification model is trained based on the log template sequence training set and the historical log data is parsed to obtain the training log template sequence. The log defect classification model is a BERT-based log defect classification model; the BERT-based log defect classification model includes multiple network structure layers, each network structure layer corresponding to a computation module, and the pruning method of the log defect classification model includes: In each structural layer, the word embedding tensor of the training log template sequence is converted into a Query matrix, a Key matrix, and a Value matrix; For each log token data obtained in the calculation module corresponding to the previous structural layer, the importance score of each log token data is calculated. This includes calculating the attention score matrix of the current structural layer based on the Query value, Key value, and Value value corresponding to each log token data in the current structural layer; using the sum of each column in the attention score matrix of the current structural layer as the importance score of the corresponding log token data in the current structural layer; and adding the importance score of the current structural layer to the importance score of the previous structural layer, using the accumulated result as the importance score of each log token data in the next layer. Log token data corresponding to multiple importance threshold ranges are filtered based on the importance score, including dividing the log token data into important log token data and unimportant log token data according to a first importance threshold; and dividing the important log token data into most important log token data and moderately important log token data according to a second importance threshold. The precision of the Query, Key, and Value values ​​corresponding to each log token is adjusted according to different importance thresholds. This includes adjusting the precision of the Query, Key, and Value values ​​corresponding to the most important log token data to a first precision value; adjusting the precision of the Query, Key, and Value values ​​corresponding to the medium-important log token data to a second precision value; and adjusting the precision of the Query, Key, and Value values ​​corresponding to the unimportant log token data to a third precision value. The first precision value is greater than the second precision value, and the second precision value is greater than the third precision value, in order to prune the BERT-based log defect classification model.

2. The log classification method according to claim 1, characterized in that, The step of parsing the logs to be classified to obtain the log template sequence includes: The node identifiers in the logs to be classified are found using regular expressions, and the log token data of the logs to be classified is extracted using the node identifiers as the dividing points. The field length of the log token data is matched with the log length of the first-level node in the predetermined parse tree. After a node with the same length is matched, the matching check of the downstream child nodes of this node is continued. The complete words of the log token data are matched with the complete words in the corresponding template branches of the predetermined parse tree. After a node with the same complete word is matched, the remaining log token data is matched with the downstream sub-branches of this node until the matching degree with a certain log template in the predetermined parse tree exceeds a preset threshold. Then the log template to which the log belongs is obtained.

3. The log classification method according to claim 1, characterized in that, The training method for the log defect classification model includes: A log template sequence training set is generated based on the training log template sequence and its corresponding labeled classification results. The training log template sequence from the log template sequence training set is input into the BERT-based log defect classification model to obtain the predicted classification result; The loss function is obtained based on the predicted classification result and the labeled classification result corresponding to the training log template sequence. The parameters in the BERT-based log defect classification model are optimized based on the results of the loss function.

4. The log classification method according to claim 1, characterized in that, Also includes: Record the classification information for each log to be classified, including the cause of the log anomaly, the anomaly classification, and the confidence information of the classification result; Real-time anomaly alerts and / or offline analysis are performed based on the classification information.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the log classification method as described in any one of claims 1 to 4.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the log classification method as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • BERT anomaly detection method and equipment based on template sequence or word sequence

    CN113111908A

  • Log sequence anomaly detection method based on out-of-stream regularization

    CN114416479A

  • System abnormal log detection method and system based on log semantic encoder

    CN115794480A