A method and apparatus for discovering network device log anomalies

Abstract

The application provides a network device log abnormality discovery method and device, the method comprising: obtaining online logs of a network device; matching the online logs with a template library, the template library comprising normal templates, abnormal templates and unclassified templates; if the online logs match the abnormal templates, determining that the network has an abnormality; if the online logs match the unclassified templates, generating a first time series index according to the number of online logs matching each unclassified template in each time period, and monitoring the abnormality of the first time series index; and if the online logs do not match the template library, generating a second time series index according to the number of online logs not matching the template library in each time period, and monitoring the abnormality of the second time series index. The method uses log templates for monitoring, discovers network abnormality problems in advance, and greatly shortens troubleshooting time.

Description

Technical Field

[0001] This invention relates to the field of computer network operation and maintenance, and in particular to a method and apparatus for detecting network device log anomalies. Background Technology

[0002] The importance of networks to modern enterprise production and residents' consumption is increasing. The operation and management of production equipment, internal communication, and external presentation all require smooth network connectivity. However, physical and logical failures are difficult to avoid. Moreover, as network structures become more complex, the number of possible failure paths increases non-linearly with the number of devices, making troubleshooting increasingly difficult.

[0003] After a network failure occurs, the usual approach is to check the logs of related network devices one by one using rule-based and experience-based methods. This method is labor-intensive and reactive. With 40GB of network device logs per day, it is difficult to perform a full manual analysis. There is a lack of correlation analysis between adjacent logs and intelligent aggregation analysis of operation and maintenance logs, resulting in long troubleshooting cycles and difficulties in identifying and diagnosing faults. Summary of the Invention

[0004] This invention provides a method and apparatus for detecting network device log anomalies. By aggregating and analyzing massive amounts of network device logs and using log template monitoring, problems can be detected in advance, solving the problems of slow troubleshooting time and cumbersome troubleshooting process when network faults occur.

[0005] This invention provides a method for detecting network device log anomalies, comprising:

[0006] Obtain online logs from network devices;

[0007] The online logs are matched with a template library, which includes normal templates, abnormal templates, and templates with no clear classification.

[0008] If the online log matches the anomaly template, it is determined that a network anomaly has occurred;

[0009] If the online log matches the undefined classification template, a first time-series indicator is generated based on the number of online logs that match each undefined classification template in each time period, and anomalies in the first time-series indicator are monitored.

[0010] If the online logs do not match the template library, a second time-series indicator is generated based on the number of online logs that do not match the template library in each time period, and anomalies in the second time-series indicator are monitored.

[0011] According to a method for detecting network device log anomalies provided by the present invention, monitoring anomalies in the first time-series index includes:

[0012] The growth rate of the number of online logs matching the undefined classification template in the current time period relative to the number of online logs matching the undefined classification template in the previous time period is greater than a threshold;

[0013] Monitor for anomalies in the second time-series metric, including:

[0014] The growth rate of the number of online logs that did not match the template library in the current time period relative to the number of online logs that did not match the template library in the previous time period is greater than a threshold.

[0015] According to a method for detecting network device log anomalies provided by the present invention, if the online log does not match the template library, the method further includes:

[0016] Logs that do not match the template library are clustered, and auxiliary anomaly detection is performed based on the clustering results.

[0017] According to a method for detecting network device log anomalies provided by the present invention, generating the normal template, the abnormal template, and the unclassified template includes:

[0018] Get the original log collection;

[0019] The original log set is classified to obtain categorized logs;

[0020] Based on the categorized logs, a template tree is generated for each log category, wherein the template tree includes multiple log templates;

[0021] The log template is input into a pre-trained labeling model to generate a labeling category for the log template, wherein the labeling category includes normal templates, abnormal templates, and unclassified templates.

[0022] According to a method for detecting network device log anomalies provided by the present invention, the classification of the original log set to obtain classified logs includes:

[0023] The edit distance between the original logs is calculated based on regular expression matching of the original log set.

[0024] Original logs with an edit distance less than a set threshold are classified as logs of the same type.

[0025] According to a method for detecting network device log anomalies provided by the present invention, based on the categorized logs, a template tree corresponding to each type of log is generated, including:

[0026] The classified logs are segmented into word units;

[0027] Calculate the frequency of the word unit in the original log set;

[0028] Based on the frequency, the word units of each type of log are stored in a tree structure to obtain an initial template tree; wherein, the log type is used as the root node of the initial template tree, and the word units of each type of log are used as child nodes;

[0029] The initial template tree is pruned according to the preset pruning parameter thresholds;

[0030] The order of child nodes in the initial template tree is adjusted according to the semantics of the word unit to obtain an updated template tree. The updated template tree includes multiple paths, each path being generated by the root node and the adjusted child nodes.

[0031] According to the present invention, a method for detecting network device log anomalies includes, in which the initial template tree is obtained by storing word units of each type of log in a tree structure based on the frequency, comprising:

[0032] For the first log entry:

[0033] Arrange the word units of the first log in descending order of frequency, and use each word unit in descending order as a child node. Generate the first path based on the root node and the child nodes.

[0034] For the other log entries:

[0035] The word units of the current log entry are sorted in descending order according to the frequency, and each word unit in the descending order is compared with the child nodes in the initial template tree in turn.

[0036] If they are the same, the comparison will continue to the next word unit of the current log entry;

[0037] If they are different, the corresponding child node is generated based on the word unit of the current log and added to the initial template tree.

[0038] The present invention also provides a device for detecting network device log anomalies, comprising:

[0039] The log acquisition module is used to acquire online logs from network devices;

[0040] The template matching and exception handling module is used to match the online logs with a template library, wherein the template library includes normal templates, exception templates, and templates with no clear classification.

[0041] If the online log matches the anomaly template, it is determined that a network anomaly has occurred;

[0042] If the online log matches the undefined classification template, a first time-series indicator is generated based on the number of online logs that match each undefined classification template in each time period, and anomalies in the first time-series indicator are monitored.

[0043] If the online logs do not match the template library, a second time-series indicator is generated based on the number of online logs that do not match the template library in each time period, and anomalies in the second time-series indicator are monitored.

[0044] According to the network device log anomaly detection device provided by the present invention, the template matching and anomaly handling module is further configured to monitor the anomalies of the first time-series indicator and the second time-series indicator, specifically:

[0045] The growth rate of the number of online logs matching the undefined classification template in the current time period relative to the number of online logs matching the undefined classification template in the previous time period is greater than a threshold;

[0046] The growth rate of the number of online logs that did not match the template library in the current time period relative to the number of online logs that did not match the template library in the previous time period is greater than a threshold.

[0047] According to the network device log anomaly detection device provided by the present invention, the template matching and anomaly handling module is further configured to:

[0048] Logs that do not match the template library are clustered, and auxiliary anomaly detection is performed based on the clustering results.

[0049] According to the network device log anomaly detection device provided by the present invention, the template matching and anomaly handling module is further configured to:

[0050] Get the original log collection;

[0051] The original log set is classified to obtain categorized logs;

[0052] Based on the categorized logs, a template tree is generated for each log category, wherein the template tree includes multiple log templates;

[0053] The log template is input into a pre-trained labeling model to generate a labeling category for the log template, wherein the labeling category includes normal templates, abnormal templates, and unclassified templates.

[0054] According to the network device log anomaly detection device provided by the present invention, the template matching and anomaly handling module is further configured to:

[0055] The edit distance between the original logs is calculated based on regular expression matching of the original log set.

[0056] Original logs with an edit distance less than a set threshold are classified as logs of the same type.

[0057] According to the network device log anomaly detection device provided by the present invention, the template matching and anomaly handling module is further configured to:

[0058] The classified logs are segmented into word units;

[0059] Calculate the frequency of the word unit in the original log set;

[0060] Based on the frequency, the word units of each type of log are stored in a tree structure to obtain an initial template tree; wherein, the log type is used as the root node of the initial template tree, and the word units of each type of log are used as child nodes;

[0061] The initial template tree is pruned according to the preset pruning parameter thresholds;

[0062] The order of child nodes in the initial template tree is adjusted according to the semantics of the word unit to obtain an updated template tree. The updated template tree includes multiple paths, each path being generated by the root node and the adjusted child nodes.

[0063] According to the network device log anomaly detection device provided by the present invention, the template matching and anomaly handling module is further configured to:

[0064] For the first log entry:

[0065] Arrange the word units of the first log in descending order of frequency, and use each word unit in descending order as a child node. Generate the first path based on the root node and the child nodes.

[0066] For the other log entries:

[0067] The word units of the current log entry are sorted in descending order according to the frequency, and each word unit in the descending order is compared with the child nodes in the initial template tree in turn.

[0068] If they are the same, the comparison will continue to the next word unit of the current log entry;

[0069] If they are different, the corresponding child node is generated based on the word unit of the current log and added to the initial template tree.

[0070] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the network device log anomaly detection method as described above.

[0071] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the network device log anomaly detection method as described above.

[0072] The present invention provides a method and apparatus for detecting network device log anomalies. By acquiring online logs of network devices and matching them with normal templates, abnormal templates, and unclassified templates in a template library, the method determines that a network anomaly has occurred if an online log matches an abnormal template. If an online log matches an unclassified template, a first time-series indicator is generated based on the number of online logs matching each unclassified template within each time period, and anomalies in the first time-series indicator are monitored. If an online log does not match the template library, a second time-series indicator is generated based on the number of online logs not matching the template library within each time period, and anomalies in the second time-series indicator are monitored. This method utilizes log template monitoring to detect network anomalies in advance, significantly shortening troubleshooting time. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0074] Figure 1 This is one of the flowcharts illustrating the method for discovering network device log anomalies provided by the present invention;

[0075] Figure 2 This is the second flowchart of the method for discovering network device log anomalies provided by the present invention;

[0076] Figure 3 This is the third flowchart of the method for discovering network device log anomalies provided by the present invention;

[0077] Figure 4 This is the fourth flowchart of the method for discovering network device log anomalies provided by the present invention;

[0078] Figure 5 This is the fifth flowchart of the method for discovering network device log anomalies provided by the present invention;

[0079] Figure 6 This is the sixth flowchart of the method for discovering network device log anomalies provided by the present invention;

[0080] Figure 7 This is a schematic diagram of the network device log anomaly detection device provided by the present invention;

[0081] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0083] Figure 1 The flowchart illustrates the method for detecting network device log anomalies provided by this invention, as follows: Figure 1 As shown, the method includes:

[0084] Step 110: Obtain the online logs of the network device.

[0085] Logs are used to record information such as user operations and system running status. They can be used for post-event diagnosis of abnormal events. Compared with the traditional method of anomaly detection by monitoring KPI curves, log messages contain events related to services or devices, and can reflect anomalies of neighboring nodes, changes in port status, etc. They can describe some events that KPI curves cannot reflect, thus achieving better anomaly detection results.

[0086] Step 120: Match the online logs with the template library, which includes normal templates, abnormal templates, and templates with no clear category. If the online logs match the abnormal templates, proceed to step 130; if the online logs match the templates with no clear category, proceed to step 140; if the online logs do not match the template library, proceed to step 150.

[0087] Based on the template content, templates that can determine that the information recorded in the corresponding log indicates that the current network has not experienced a failure are defined as normal templates; templates that can determine that the current log indicates an abnormal network status and can clearly define the type of abnormality are defined as abnormal templates; and other templates that cannot determine the problem in the corresponding log based on the template content are defined as unclassified templates.

[0088] Step 130: Determine if a network anomaly has occurred.

[0089] Step 140: Generate a first time-series indicator based on the number of online logs that match each of the undefined classification templates within each time period, and monitor any abnormalities in the first time-series indicator.

[0090] Step 150: Based on the number of online logs that did not match the template library in each time period, generate a second time-series indicator and monitor any abnormalities in the second time-series indicator.

[0091] Online logs that do not match the template library are defined as newly generated logs. These logs may be the root cause of the problem and are relatively important. They need to be further statistically analyzed in order to detect the problem early.

[0092] According to one embodiment of the present invention, monitoring anomalies in the first time-series indicator includes: the growth rate of the number of online logs matching the undefined classification template in the current time period relative to the number of online logs matching the undefined classification template in the previous time period is greater than a threshold.

[0093] Monitor abnormal situations of the second time-series indicator, including: the growth rate of the number of online logs that do not match the template library in the current time period relative to the number of online logs that do not match the template library in the previous time period is greater than a threshold.

[0094] According to one embodiment of the present invention, if the online logs do not match the template library, the method further includes: clustering the logs that do not match the template library, and performing auxiliary anomaly detection based on the clustering results.

[0095] According to one embodiment of the present invention, Figure 2 A flowchart illustrating the process of generating normal templates, abnormal templates, and templates with undefined categories is shown below. Figure 2 As shown, it includes:

[0096] Step 210: Obtain the original log set.

[0097] Step 220: Classify the original log set to obtain classified logs.

[0098] Step 230: Based on the classified logs, generate a template tree corresponding to each type of log, wherein the template tree includes multiple log templates.

[0099] Step 240: Input the log template into a pre-trained labeling model to generate the labeling category of the log template, wherein the labeling category includes normal template, abnormal template and unclassified template.

[0100] First, manual annotation of the generated log templates is performed to identify the corresponding issues. Then, based on the annotated template tags, a named entity recognition model is trained to categorize the log templates.

[0101] According to one embodiment of the present invention, Figure 3 To categorize the original log collection, a flowchart illustrating the process of categorizing logs is provided, as follows: Figure 3 As shown, it includes:

[0102] Step 310: Calculate the edit distance between the original logs by matching the original log set using regular expressions.

[0103] Regular expressions are text patterns composed of ordinary and special characters, used in the string matching process. Edit distance is a metric used to measure the similarity between two sequences; it is the minimum number of single-character edits required to transform one string into another.

[0104] Step 320: The original logs with an edit distance less than a set threshold are classified as logs of the same type.

[0105] According to one embodiment of the present invention, Figure 4 This is a flowchart illustrating the process of generating a template tree for each type of log based on categorized logs, as shown below. Figure 4 As shown, it includes:

[0106] Step 410: Segment the classified logs into word units.

[0107] Step 420: Calculate the frequency of the word unit in the original log set.

[0108] Step 430: Based on the frequency, store the word units of each type of log in a tree structure to obtain an initial template tree; wherein the log type is used as the root node of the initial template tree, and the word units of each type of log are used as child nodes.

[0109] Specifically, the word units are classified into constants and variables, with constants serving as template words and variables serving as parameter words.

[0110] Step 440: Prune the initial template tree according to a preset pruning parameter threshold. Optionally, the number of nodes in the path is greater than or equal to 5 to convey more template information.

[0111] Step 450: Adjust the order of child nodes in the initial template tree according to the semantics of the word unit to obtain an updated template tree. The updated template tree includes multiple paths, each path being generated by the root node and the adjusted child nodes.

[0112] Specifically, the variables in the path are replaced with *, which is used as the final template.

[0113] According to one embodiment of the present invention, the step of storing word units of each type of log in a tree structure according to the frequency to obtain an initial template tree includes:

[0114] For the first log entry:

[0115] The word units in the first log entry are sorted in descending order of frequency. Each word unit in the descending order is used as a child node. A first path is generated based on the root node and the child nodes. Optionally, word units with excessively low conditional probabilities are filtered out.

[0116] For the other log entries Figure 5 This is a flowchart illustrating the process of adding word units from the log to the template tree, such as... Figure 5 As shown, it includes:

[0117] Step 510: Sort the word units of the current log entry in descending order of frequency, and compare each word unit in the descending order with the child nodes in the initial template tree in turn. Optionally, word units with too low conditional probabilities are filtered out. If they are the same, proceed to step 520; if they are different, proceed to step 530.

[0118] Step 520: Continue comparing the next word unit of the current log entry.

[0119] Step 530: Generate corresponding child nodes based on the word units of the current log entry and add them to the initial template tree.

[0120] When the template tree branches and has many child nodes, use * to replace the variable.

[0121] The following example illustrates the specific process of generating the initial template tree. Table 1 shows four log entries of type A. Figure 6 This is the process of generating the corresponding initial template tree.

[0122] Table 1

[0123] Log 1 Vocabulary Unit 1 Vocabulary Unit 2 Vocabulary Unit 3 Vocabulary Unit 4 Log 2 Vocabulary Unit 2 Vocabulary Unit 6 Vocabulary Unit 4 Log 3 Vocabulary Unit 1 Vocabulary Unit 2 Vocabulary Unit 6 Log 4 Vocabulary Unit 2 Vocabulary Unit 3 Vocabulary Unit 6 Vocabulary Unit 7

[0124] The method for detecting network device log anomalies provided by this invention obtains the online logs of the network device and matches them with normal templates, abnormal templates, and unclassified templates in a template library. If the online log matches an abnormal template, a network anomaly is determined. If the online log matches an unclassified template, a first time-series indicator is generated based on the number of online logs matching each unclassified template in each time period, and anomalies in the first time-series indicator are monitored. If the online log does not match the template library, a second time-series indicator of the network device is generated based on the number of online logs not matching the template library in each time period, and anomalies in the second time-series indicator are monitored. This method enables early detection of network anomalies using log template monitoring, significantly shortening troubleshooting time.

[0125] The following describes the network device log anomaly detection device provided by the present invention. The network device log anomaly detection device described below and the network device log anomaly detection method described above can be referred to in correspondence.

[0126] Figure 7 This is a schematic diagram of the network device log anomaly detection device provided by the present invention, as shown below. Figure 7 As shown, the device includes:

[0127] The log acquisition module 710 is used to acquire online logs of network devices;

[0128] The template matching and exception handling module 720 is used to match the online logs with a template library, wherein the template library includes normal templates, exception templates and templates with no clear classification.

[0129] If the online log matches the anomaly template, it is determined that a network anomaly has occurred;

[0130] If the online log matches the undefined classification template, a first time-series indicator is generated based on the number of online logs that match each undefined classification template in each time period, and anomalies in the first time-series indicator are monitored.

[0131] If the online logs do not match the template library, a second time-series indicator is generated based on the number of online logs that do not match the template library in each time period, and anomalies in the second time-series indicator are monitored.

[0132] According to an embodiment of the present invention, the template matching and anomaly handling module 720 is further configured to monitor anomalies in the first time-series index and anomalies in the second time-series index, specifically:

[0133] The growth rate of the number of online logs matching the undefined classification template in the current time period relative to the number of online logs matching the undefined classification template in the previous time period is greater than a threshold;

[0134] The growth rate of the number of online logs that did not match the template library in the current time period relative to the number of online logs that did not match the template library in the previous time period is greater than a threshold.

[0135] According to an embodiment of the present invention, the template matching and exception handling module 720 is further configured to:

[0136] Logs that do not match the template library are clustered, and auxiliary anomaly detection is performed based on the clustering results.

[0137] According to an embodiment of the present invention, the template matching and exception handling module 720 is further configured to:

[0138] Get the original log collection;

[0139] The original log set is classified to obtain categorized logs;

[0140] Based on the categorized logs, a template tree is generated for each log category, wherein the template tree includes multiple log templates;

[0141] The log template is input into a pre-trained labeling model to generate a labeling category for the log template, wherein the labeling category includes normal templates, abnormal templates, and unclassified templates.

[0142] According to an embodiment of the present invention, the template matching and exception handling module 720 is further configured to:

[0143] The edit distance between the original logs is calculated based on regular expression matching of the original log set.

[0144] Original logs with an edit distance less than a set threshold are classified as logs of the same type.

[0145] According to one embodiment of the present invention, the template matching and exception handling module 720 is further configured to:

[0146] The classified logs are segmented into word units;

[0147] Calculate the frequency of the word unit in the original log set;

[0148] Based on the frequency, the word units of each type of log are stored in a tree structure to obtain an initial template tree; wherein, the log type is used as the root node of the initial template tree, and the word units of each type of log are used as child nodes;

[0149] The initial template tree is pruned according to the preset pruning parameter thresholds;

[0150] The order of child nodes in the initial template tree is adjusted according to the semantics of the word unit to obtain an updated template tree. The updated template tree includes multiple paths, each path being generated by the root node and the adjusted child nodes.

[0151] According to one embodiment of the present invention, the template matching and exception handling module 720 is further configured to:

[0152] For the first log entry:

[0153] Arrange the word units of the first log in descending order of frequency, and use each word unit in descending order as a child node. Generate the first path based on the root node and the child nodes.

[0154] For the other log entries:

[0155] The word units of the current log entry are sorted in descending order according to the frequency, and each word unit in the descending order is compared with the child nodes in the initial template tree in turn.

[0156] If they are the same, the comparison will continue to the next word unit of the current log entry;

[0157] If they are different, the corresponding child node is generated based on the word unit of the current log and added to the initial template tree.

[0158] The network device log anomaly detection device provided by this invention acquires online logs of network devices and matches them with normal templates, abnormal templates, and unclassified templates in a template library. If an online log matches an abnormal template, a network anomaly is determined. If an online log matches an unclassified template, a first time-series indicator is generated based on the number of online logs matching each unclassified template within each time period, and anomalies in the first time-series indicator are monitored. If an online log does not match the template library, a second time-series indicator of the network device is generated based on the number of online logs not matching the template library within each time period, and anomalies in the second time-series indicator are monitored. This allows for early detection of network anomalies using log template monitoring, significantly shortening troubleshooting time.

[0159] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method for detecting network device log anomalies. This method includes: acquiring online logs of the network device; matching the online logs with a template library, wherein the template library includes normal templates, abnormal templates, and unclassified templates; if the online logs match the abnormal templates, determining that a network anomaly has occurred; if the online logs match the unclassified templates, generating a first time-series indicator based on the number of online logs matching each unclassified template in each time period, and monitoring the anomalies of the first time-series indicator; if the online logs do not match the template library, generating a second time-series indicator based on the number of online logs not matching the template library in each time period, and monitoring the anomalies of the second time-series indicator.

[0160] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the network device log anomaly detection algorithm provided by the above methods, the method comprising: acquiring online logs of network devices; matching the online logs with a template library, wherein the template library includes normal templates, abnormal templates, and unclassified templates; if the online logs match the abnormal templates, determining that a network anomaly has occurred; if the online logs match the unclassified templates, generating a first time-series indicator based on the number of online logs matching each unclassified template in each time period, and monitoring the anomaly of the first time-series indicator; if the online logs do not match the template library, generating a second time-series indicator based on the number of online logs not matching the template library in each time period, and monitoring the anomaly of the second time-series indicator.

[0162] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an algorithm for detecting network device log anomalies provided by the methods described above. The method includes: acquiring online logs of a network device; matching the online logs with a template library, wherein the template library includes normal templates, abnormal templates, and unclassified templates; if the online logs match the abnormal templates, determining that a network anomaly has occurred; if the online logs match the unclassified templates, generating a first time-series indicator based on the number of online logs matching each unclassified template in each time period, and monitoring anomalies in the first time-series indicator; if the online logs do not match the template library, generating a second time-series indicator based on the number of online logs not matching the template library in each time period, and monitoring anomalies in the second time-series indicator.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting anomalies in network device logs, characterized in that, include: Obtain online logs from network devices; The online logs are matched with a template library, which includes normal templates, abnormal templates, and templates with no clear classification. If the online log matches the anomaly template, it is determined that a network anomaly has occurred; If the online log matches the undefined classification template, a first time-series indicator is generated based on the number of online logs that match each undefined classification template in each time period, and anomalies in the first time-series indicator are monitored. Monitoring abnormal situations of the first time-series indicator includes: the growth rate of the number of online logs matching the undefined classification template in the current time period relative to the number of online logs matching the undefined classification template in the previous time period is greater than a threshold; If the online logs do not match the template library, a second time-series indicator is generated based on the number of online logs that do not match the template library in each time period, and anomalies in the second time-series indicator are monitored. Anomalies in the second time-series indicator include: the growth rate of the number of online logs that do not match the template library in the current time period relative to the number of online logs that do not match the template library in the previous time period is greater than a threshold.

2. The method for detecting network device log anomalies according to claim 1, characterized in that, If the online logs do not match the template library, the method further includes: Logs that do not match the template library are clustered, and auxiliary anomaly detection is performed based on the clustering results.

3. The method for detecting network device log anomalies according to claim 1, characterized in that, Generating the normal template, abnormal template, and unclassified template includes: Get the original log collection; The original log set is classified to obtain categorized logs; Based on the categorized logs, a template tree is generated for each log category, wherein the template tree includes multiple log templates; The log template is input into a pre-trained labeling model to generate a labeling category for the log template, wherein the labeling category includes normal templates, abnormal templates, and unclassified templates.

4. The method for detecting network device log anomalies according to claim 3, characterized in that, The process of classifying the original log set to obtain categorized logs includes: The edit distance between the original logs is calculated based on regular expression matching of the original log set. Original logs with an edit distance less than a set threshold are classified as logs of the same type.

5. The method for detecting network device log anomalies according to claim 3, characterized in that, Based on the categorized logs, a template tree corresponding to each log category is generated, including: The classified logs are segmented into word units; Calculate the frequency of the word unit in the original log set; Based on the frequency, the word units of each type of log are stored in a tree structure to obtain an initial template tree; wherein, the log type is used as the root node of the initial template tree, and the word units of each type of log are used as child nodes; The initial template tree is pruned according to the preset pruning parameter thresholds; The order of child nodes in the initial template tree is adjusted according to the semantics of the word unit to obtain an updated template tree. The updated template tree includes multiple paths, each path being generated by the root node and the adjusted child nodes.

6. The method for detecting network device log anomalies according to claim 5, characterized in that, The step of storing word units for each type of log in a tree structure according to the frequency to obtain the initial template tree includes: For the first log entry: Arrange the word units of the first log in descending order of frequency, and use each word unit in descending order as a child node. Generate the first path based on the root node and the child nodes. For the other log entries: The word units of the current log entry are sorted in descending order according to the frequency, and each word unit in the descending order is compared with the child nodes in the initial template tree in turn. If they are the same, the comparison will continue to the next word unit of the current log entry; If they are different, the corresponding child node is generated based on the word unit of the current log and added to the initial template tree.

7. A device for detecting network device log anomalies, characterized in that, include: The log acquisition module is used to acquire online logs from network devices; The template matching and exception handling module is used to match the online logs with a template library, wherein the template library includes normal templates, exception templates, and templates with no clear classification. If the online log matches the anomaly template, it is determined that a network anomaly has occurred; If the online log matches the undefined classification template, a first time-series indicator is generated based on the number of online logs that match each undefined classification template in each time period, and anomalies in the first time-series indicator are monitored. If the online logs do not match the template library, a second time-series indicator is generated based on the number of online logs that do not match the template library in each time period, and abnormal situations of the second time-series indicator are monitored. The template matching and anomaly handling module is also used to monitor the anomalies of the first time series indicator and the second time series indicator, specifically: the growth rate of the number of online logs that match the unclassified template in the current time period relative to the number of online logs that match the unclassified template in the previous time period is greater than a threshold. The growth rate of the number of online logs that did not match the template library in the current time period relative to the number of online logs that did not match the template library in the previous time period is greater than a threshold.

8. The network device log anomaly detection device according to claim 7, characterized in that, The template matching and exception handling module is also used for: Logs that do not match the template library are clustered, and auxiliary anomaly detection is performed based on the clustering results.

9. The network device log anomaly detection device according to claim 7, characterized in that, The template matching and exception handling module is also used for: Get the original log collection; The original log set is classified to obtain categorized logs; Based on the categorized logs, a template tree is generated for each log category, wherein the template tree includes multiple log templates; The log template is input into a pre-trained labeling model to generate a labeling category for the log template, wherein the labeling category includes normal templates, abnormal templates, and unclassified templates.

10. The network device log anomaly detection device according to claim 9, characterized in that, The template matching and exception handling module is also used for: The edit distance between the original logs is calculated based on regular expression matching of the original log set. Original logs with an edit distance less than a set threshold are classified as logs of the same type.

11. The network device log anomaly detection device according to claim 9, characterized in that, The template matching and exception handling module is also used for: The classified logs are segmented into word units; Calculate the frequency of the word unit in the original log set; Based on the frequency, the word units of each type of log are stored in a tree structure to obtain an initial template tree; wherein, the log type is used as the root node of the initial template tree, and the word units of each type of log are used as child nodes; The initial template tree is pruned according to the preset pruning parameter thresholds; The order of child nodes in the initial template tree is adjusted according to the semantics of the word unit to obtain an updated template tree. The updated template tree includes multiple paths, each path being generated by the root node and the adjusted child nodes.

12. The network device log anomaly detection device according to claim 11, characterized in that, The template matching and exception handling module is also used for: For the first log entry: Arrange the word units of the first log in descending order of frequency, and use each word unit in descending order as a child node. Generate the first path based on the root node and the child nodes. For the other log entries: The word units of the current log entry are sorted in descending order according to the frequency, and each word unit in the descending order is compared with the child nodes in the initial template tree in turn. If they are the same, the comparison will continue to the next word unit of the current log entry; If they are different, the corresponding child node is generated based on the word unit of the current log and added to the initial template tree.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for detecting network device log anomalies as described in any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting network device log anomalies as described in any one of claims 1 to 6.

Images