Log anomaly detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to a log anomaly detection method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a log data file; the log data file is labeled according to the full-amount log template, the labeled log data file is obtained, the full-amount log template comprises a routing inspection log template corresponding to the network element routing inspection task list, and in the process of labeling the log data file, the routing inspection log template corresponds to the network element routing inspection task list; if a first log stream meeting a first matching requirement with the inspection log template exists in the log data file, marking the first log stream as the inspection log template; and performing log anomaly detection on the labeled log data file. By the adoption of the scheme, the problem that anomaly detection results are relatively concentrated can be solved, and the accuracy of log anomaly detection is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a log anomaly detection method, device, electronic device and storage medium. Background Art

[0002] In the related technology, the application of log information is usually to obtain log content or log stream from the log source, and establish a log anomaly template or log anomaly recognition model through clustering and association processing / analysis of log information, and then associate the log anomaly with the corresponding network element and other equipment to achieve fault diagnosis and processing of network operation. This log anomaly detection method has a strong dependence on cluster analysis. Although it can solve the detection problem of most conventional log anomalies, there is a certain amount of information loss in clustering, resulting in relatively concentrated anomaly detection results, and only has a good detection effect on logs with significant anomalies in the training set.

[0003] However, for the log data generated by network element devices in response to network element inspection, since network element inspection is often performed concurrently with multiple inspection tasks, the network element devices involved in each inspection task are overlapped, and in the same network element inspection task executed periodically, some network element devices do not need to execute task instructions in certain cycles, resulting in the log data generated by network element devices in response to network element inspection showing no significant difference in log content and log sequence. In this case, still using the above-mentioned log anomaly detection method to process the log data generated by network element inspection together with other log data will have an adverse effect on the accuracy of fault diagnosis results. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the first purpose of the present disclosure is to propose a log anomaly detection method to improve the problem of relatively concentrated anomaly detection results and improve the accuracy of log anomaly detection.

[0006] The second objective of the present disclosure is to provide a log anomaly detection device.

[0007] A third objective of the present disclosure is to provide an electronic device.

[0008] A fourth objective of the present disclosure is to provide a computer-readable storage medium.

[0009] A fifth object of the present disclosure is to provide a computer program product.

[0010] To achieve the above objectives, the first embodiment of the present disclosure proposes a log anomaly detection method, including:

[0011] Get the log data file;

[0012] The log data file is annotated according to the full log template to obtain the annotated log data file, wherein the full log template includes the inspection log template corresponding to the network element inspection task list, and in the process of annotating the log data file, if there is a first log stream in the log data file that meets the first matching requirement with the inspection log template, the first log stream is annotated as the inspection log template;

[0013] Perform log anomaly detection on the annotated log data files.

[0014] Optionally, in the process of marking the log data file, the method further includes:

[0015] If there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, then obtaining a target log in the second log stream whose matching degree with the inspection log template is less than the first matching degree threshold;

[0016] If the target log meets the third matching requirement, the second log stream is determined to be the first log stream.

[0017] Optionally, if there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, before obtaining a target log in the second log stream whose matching degree with the inspection log template is less than a matching degree threshold, the method further includes:

[0018] Determine the network elements and task sequence corresponding to the network element inspection task list;

[0019] If there is a third log stream in the log data file that matches both the network element and the task sequence, and the log content of the third log stream meets the fourth matching requirement, the third log stream is determined to be the second log stream.

[0020] Optionally, the log content of the third log stream meets the fourth matching requirement, including:

[0021] The matching degree between the log content of the third log stream and the log content of the inspection log template is lower than the second matching degree threshold; and / or,

[0022] The aggregatability degree of the log content of the third log stream is greater than the aggregatability degree threshold.

[0023] Optionally, after obtaining a target log in the second log stream whose matching degree with the inspection log template is less than a first matching degree threshold, the method further includes:

[0024] Get the calling time of the service interface of the network element corresponding to the target log;

[0025] If the call duration is less than the call duration threshold, it is determined that the target log meets the third matching requirement.

[0026] Optionally, the log data file is annotated according to the full log template to obtain the annotated log data file, including:

[0027] Compare the full log template with the log data file;

[0028] If the log data file contains a fourth log stream that meets the fifth matching requirement with the full log template, the fourth log stream is marked according to the full log template;

[0029] A cluster analysis is performed on the fifth log stream except the fourth log stream in the log data file to obtain a new log template, and the new log template is incorporated into the full log template to re-compare the log data file until the log data file meets the labeling requirements to obtain a labeled log data file.

[0030] Optionally, log anomaly detection is performed on the annotated log data file, including:

[0031] Performing log file segmentation on the marked log data file to obtain a segmented log data file;

[0032] The cut log data file is input into the anomaly detection model to perform anomaly detection and obtain anomaly detection results;

[0033] If the abnormality detection result indicates that there is abnormal log data in the cut log data file, a fault diagnosis is performed on the abnormal log data.

[0034] To achieve the above-mentioned purpose, a second aspect of the present disclosure provides a log anomaly detection device, including:

[0035] A log acquisition unit, used for acquiring log data files;

[0036] A log marking unit is used to mark the log data file according to the full log template to obtain the marked log data file, wherein the full log template includes the inspection log template corresponding to the network element inspection task list, and in the process of marking the log data file, if there is a first log stream in the log data file that meets the first matching requirement with the inspection log template, then the first log stream is marked as the inspection log template;

[0037] The anomaly detection unit is used to perform log anomaly detection on the annotated log data files.

[0038] To achieve the above-mentioned purpose, a third aspect of the present disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0039] Memory stores computer-executable instructions;

[0040] The processor executes the computer-executable instructions stored in the memory to implement the method shown in any one of the first aspects above.

[0041] To achieve the above-mentioned objectives, an embodiment of the fourth aspect of the present disclosure proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method shown in any one of the above-mentioned first aspects.

[0042] To achieve the above-mentioned objective, a fifth aspect of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the method shown in any one of the above-mentioned first aspects.

[0043] In summary, the method, device, equipment and storage medium provided by the present invention can solve the problem of network element inspection log template mining caused by unclear significance by introducing network element inspection related content in the process of annotating log data files, improve the problem of relatively concentrated anomaly detection results, and improve the accuracy of log anomaly detection.

[0044] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 A schematic diagram of a log system monitoring structure provided by an embodiment of the present disclosure;

[0047] Figure 2 A flow chart of a log anomaly detection method provided by an embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram of the architecture of a log system provided by an embodiment of the present disclosure;

[0049] Figure 4 A flow chart of a log anomaly detection method provided by another embodiment of the present disclosure;

[0050] Figure 5 A schematic diagram of the structure of a log anomaly detection device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0052] In the communications industry, log system monitoring is a necessary means of monitoring and operating 2 / 3 / 4G traditional core network elements and 5G / Network Functions Virtualization (NFV) regional cloud. Figure 1 This is a schematic diagram of a log system monitoring structure provided by an embodiment of the present disclosure. Figure 1 As shown in the figure, the maintenance of traditional core network telecommunication equipment relies on alarms, and the maintenance of the Internet (IT) relies on logs; 5G / NFV cloud networks are IT-based telecommunication equipment. In the process of integration and transformation from communication technology (CT) to information communication technology (ICT), log monitoring shoulders the heavy responsibility of transforming the monitoring capabilities of network operators. By analyzing the process event record data generated by the network communication system recorded in the log data, network operators can analyze the behavior of network equipment and users in the historical time period, and thus mine log templates for functions such as fault diagnosis. Therefore, how to effectively apply log information to important monitoring scenarios such as emergency fault handling, daily monitoring, and network engineering cutover control is an important issue in the operation and maintenance work of network operators today.

[0053] In the related technology, the application of log information is usually to obtain log content or log stream from the log source, and establish a log anomaly template or log anomaly recognition model through clustering and association processing / analysis of log information, and then associate the log anomaly with the corresponding network element and other equipment to achieve fault diagnosis and processing of network operation. This log anomaly detection method has a strong dependence on cluster analysis. Although it can solve the detection problem of most conventional log anomalies, there is a certain amount of information loss in clustering, resulting in relatively concentrated anomaly detection results, and only has a good detection effect on logs with significant anomalies in the training set.

[0054] Secondly, network element inspection is an important part of the network operator's maintenance work. The basic process of network element inspection is: establish network element inspection tasks through the network element automatic inspection system. The network element inspection tasks include a list of network elements to be inspected (including network element name, network element type), task execution plan, number of concurrent inspections, task level, task description, etc.; the network element automatic inspection system connects to each network element to be inspected in turn, sends relevant inspection instructions to the network element to be inspected, and the network element to be inspected returns the corresponding instruction execution report (feedback data) after receiving the instruction. Since there are network element devices in the network element inspection that change slightly within a certain time interval, the changes in their feedback data for multiple periodic inspection tasks can be ignored. Therefore, there is a data extension scheme in the network element automatic inspection system. That is, for these network element devices, an extension storage time is set for their last inspection feedback data in the network element automatic inspection system. If the next inspection task of the network element automatic inspection system for the network element device is within the extended storage time range, the network element automatic inspection system does not require the network element device to execute the inspection instruction after connecting to the network element during the inspection task. Instead, the last inspection feedback data within the extended storage time is used as the feedback data for the current inspection task, thereby reducing the impact of the inspection task on the operation of the network element device.

[0055] In other words, the log data generated by the network element inspection of the network element equipment is not significant in terms of log content and log sequence.

[0056] However, for the log data generated by network element devices in response to network element inspection, since network element inspection is often performed concurrently with multiple inspection tasks, the network element devices involved in each inspection task are overlapped, and in the same network element inspection task executed periodically, there are cases where some network element devices do not need to execute task instructions in certain cycles, resulting in the log data generated by network element devices in response to network element inspection showing no significant difference in log content and log sequence. In this case, the existing log anomaly detection method does not provide a better adaptive adjustment solution, and still uses the above-mentioned log anomaly detection method to process the log data generated by network element inspection together with other log data, which will have an adverse effect on the accuracy of fault diagnosis results.

[0057] The present disclosure is described in detail below with reference to specific embodiments.

[0058] In the first embodiment, if Figure 2 As shown, Figure 2 The present invention provides a flow chart of a log anomaly detection method provided by an embodiment of the present invention. The method can be implemented by a computer program and can be run on a device for log anomaly detection. The computer program can be integrated into an application or run as an independent tool application.

[0059] The log anomaly detection device may be an electronic device with a log anomaly detection function.

[0060] Among them, the log anomaly detection method can be executed by an electronic device.

[0061] For example, the log anomaly detection method includes the following steps:

[0062] S101, obtaining a log data file;

[0063] According to some embodiments, a log data file refers to a file consisting of log data obtained from a log source.

[0064] S102, annotating the log data file according to the full log template to obtain the annotated log data file, wherein the full log template includes the inspection log template corresponding to the network element inspection task list, and in the process of annotating the log data file, if there is a first log stream in the log data file that meets the first matching requirement with the inspection log template, the first log stream is annotated as the inspection log template;

[0065] According to some embodiments, the full log template refers to a set consisting of a large number of standard log templates.

[0066] In some embodiments, the network element inspection task list refers to a list consisting of at least one network element inspection task.

[0067] According to some embodiments, the inspection log template refers to a log template corresponding to the network element inspection task list. The inspection log template does not specifically refer to a fixed template. For example, when the network element inspection task list changes, the inspection log template may also change.

[0068] In some embodiments, the first log stream refers to a log stream that meets a first matching requirement with the inspection log template. The first matching requirement refers to a requirement used when determining whether the matching between the log stream and the inspection log template reaches a preset degree.

[0069] S103: Perform log anomaly detection on the marked log data file.

[0070] According to some embodiments, when the electronic device obtains the annotated log data file, the electronic device may perform log anomaly detection on the annotated log data file.

[0071] In summary, the method provided in this embodiment can solve the problem of network element inspection log template mining caused by unclear significance by introducing network element inspection related content in the process of marking log data files, improve the problem of relatively concentrated anomaly detection results, and improve the accuracy of log anomaly detection.

[0072] This embodiment also provides another log anomaly detection method, which can be executed by an electronic device.

[0073] For example, the log anomaly detection method may include the following steps:

[0074] S201, obtaining a log data file;

[0075] Take a scenario as an example. Figure 3 This is a schematic diagram of the architecture of a log system provided by an embodiment of the present disclosure. Figure 3 As shown, a log system can be built in an electronic device. The log information obtained by the log system from the log source is collected and cleaned by the bottom layer and processed by the middle layer, and then stored in a distributed database in the form of a log stream. The log content of each log in the log stream can be called through an interface to cluster and associate the log information / analyze it, so as to establish aggregation rules and association rules for log abnormal data and store them in the log upper-layer application, thereby constructing a log abnormality template or log abnormality rule, and then obtaining application models such as log abnormality warning and associated fault diagnosis.

[0076] The log sources include but are not limited to cs domain, ps domain, IMS domain, data domain, 4 / 5G VNF domain, etc.

[0077] Among them, the underlying collection and cleaning processing includes but is not limited to manufacturer gateway, Sftp / ftp, syslog, ftps, log acquisition, log translation, log filtering, log forwarding, etc.;

[0078] Among them, middle-level processing includes but is not limited to real-time warning, message encapsulation, quality management, hierarchical association, rule processing, interface adaptation, data analysis, interface management, task scheduling, task distribution, etc.

[0079] Among them, the upper-layer applications of logs include but are not limited to business chain analysis, early warning statistics, data quality, statistical analysis, log query, early warning interface, word segmentation management, parsing rules, association rules, aggregation rules, cross-layer rules, SMS interface, etc.

[0080] In some embodiments, the electronic device can control the log system to collect cleaned log data files from the bottom layer at a set time, such as every day; the log extraction time can be confirmed based on the log traffic processed by the system, and is usually set to be periodic.

[0081] S202 compares the full log template with the log data file;

[0082] According to some embodiments, the comparison method includes but is not limited to cosine similarity, Euclidean distance, etc.

[0083] In some embodiments, in the initial stage of establishing the log system, a large number of standard log templates obtained from previous research can be stored in the log system, recorded as full log templates, and a log template clustering strategy can be written. Subsequently, the full log template can be expanded by performing template mining on the log data file.

[0084] S203: If there is a fourth log stream in the log data file that meets the fifth matching requirement with the full log template, mark the fourth log stream according to the full log template;

[0085] According to some embodiments, the fourth log stream refers to a log stream that satisfies a fifth matching requirement with the full log template. The fifth matching requirement refers to a requirement used when determining whether the matching between the log stream and a log template in the full log template reaches a preset degree.

[0086] In some embodiments, the preset matching degrees corresponding to different log templates may be different. For example, the preset matching degree corresponding to the inspection log template may be 30%, and the preset matching degree corresponding to other log templates in the full log template except the inspection log template may be 90%.

[0087] According to some embodiments, for the situation where the network element device feedbacks inspection data but does not generate an inspection log due to the existence of an extended storage time, an inspection log template mining rule can be introduced in the log template, and a corresponding inspection log template can be generated according to the network element inspection task list. Then, in the process of marking the log data file, if there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, a target log in the second log stream whose matching degree with the inspection log template is less than the first matching degree threshold is obtained; if the target log meets the third matching requirement, the second log stream is determined to be the first log stream; if there is a first log stream in the log data file that meets the first matching requirement with the inspection log template, the first log stream is marked as the inspection log template.

[0088] In some embodiments, the second log stream refers to a log stream that meets a second matching requirement with the inspection log template. The second matching requirement refers to a requirement used when determining whether the matching between the log stream and a log template in the inspection log template reaches a preset degree.

[0089] According to some embodiments, the network elements and task sequence corresponding to the network element inspection task list can be determined; if there is a third log stream in the log data file that matches both the network elements and the task sequence, and the log content of the third log stream meets the fourth matching requirement, the third log stream is determined to be the second log stream.

[0090] In some embodiments, when the log content of the third log stream meets the fourth matching requirement, at least one of the following conditions must be met:

[0091] The matching degree between the log content of the third log stream and the log content of the inspection log template is lower than the second matching degree threshold;

[0092] The aggregatability degree of the log content of the third log stream is greater than the aggregatability degree threshold.

[0093] Taking a scenario as an example, in the process of marking log data files, if a log stream is identified that matches the inspection log template to a preset degree, such as the network elements involved in the log stream are consistent with the network elements in the network element inspection task list and the tasks are consistent with the log order, but the log content is inconsistent or the degree of aggregation of the log content reaches 30%, then it is considered that the real-time log stream can be marked as the inspection log template, so that the real-time log stream can be used as the second log stream that meets the second matching requirement.

[0094] According to some embodiments, the calling duration of the service interface of the network element corresponding to the target log may be obtained; if the calling duration is less than a calling duration threshold, it is determined that the target log meets the third matching requirement.

[0095] In some embodiments, the match degree between the target log and the inspection log template is less than the first match degree threshold, indicating that the target log does not match the inspection log template. In this case, the call duration of the service interface of the associated network element in the target log is read. If the call duration is lower than the call duration threshold, it means that the network element only responded to the inspection task but did not execute the inspection instruction. Then, it can be considered that the log stream matches the inspection log template, and can be used as the first log stream and marked as the inspection log template.

[0096] In some embodiments, the marked first log stream can also be incorporated into the full log template as a replaceable template of the inspection log template in the full log template. If the subsequent log stream matches the marked first log stream, the call duration of the service interface of the associated network element can be directly read to make a judgment, thereby improving the matching efficiency.

[0097] It is easy to understand that by analyzing the situation in which some network elements do not need to execute inspection instructions during network element inspection, additional inspection log template mining rules are added, and network element inspection-related content is introduced in log extraction and template mining, so as to screen matching log streams for them with a lower matching degree; then, the low calling time of the network element's service interface is used to characterize that the network element has only responded to the inspection task but has not executed the inspection instruction. Therefore, it can be judged that although its log template has a low matching degree with the inspection log template, it is actually a task process log consistent with the inspection log template, and supplementary mining of the inspection log template is realized. This can improve the problem that the existing log template mining technology relies too much on clustering methods, resulting in over-concentration of mining conclusions, and solves the problem that the log stream generated by the network element in response to the network element inspection is not obvious, resulting in the failure of log template mining. This can improve the problem of relatively concentrated anomaly detection results and improve the accuracy of log anomaly detection.

[0098] S204, performing cluster analysis on the fifth log stream except the fourth log stream in the log data file to obtain a newly added log template, and incorporating the newly added log template into the full log template to re-compare the log data file until the log data file meets the labeling requirements, thereby obtaining a labeled log data file;

[0099] According to some embodiments, the fifth log stream refers to a log stream that does not match any log template in the full log template.

[0100] In some embodiments, a cluster analysis can be performed on the fifth log stream except the fourth log stream in the log data file to obtain new log templates, and these new log templates are marked in the log data file, and the marked new log templates are incorporated into the full log templates so that the log data file can be re-marked later, so that all log streams in the log data file can be marked, so that the log data file meets the marking requirements, and the marked log data file is obtained.

[0101] That is to say, steps S202 to S204 have both the log data file marking function and the log template mining function. Figure 4 The following is a flow chart of a log anomaly detection method provided by an embodiment of the present disclosure. Figure 4 As shown, in the process of log template mining, the collected massive log data can be filtered by log templates through the full log template library (i.e., the first annotation of log templates), the unfiltered log data can be clustered and analyzed to obtain new log templates, the new log templates are incorporated into the full log template library and filtered again (i.e., the first annotation of log templates), and finally the log templates are annotated for the second time to obtain and output the annotated log data file.

[0102] S205, performing log file segmentation on the marked log data file to obtain a segmented log data file;

[0103] According to some embodiments, when performing log file segmentation on the annotated log data file, the log file segmentation may be performed according to a preset granularity. The preset granularity may be adjusted according to actual application scenarios. The preset granularity may be, for example, 5 minutes.

[0104] S206, inputting the cut log data file into an anomaly detection model to perform anomaly detection and obtain an anomaly detection result;

[0105] According to some embodiments, the anomaly detection model may be, for example, a squared prediction error (SPE) anomaly detection model.

[0106] In some embodiments, Figure 4 As shown, the annotated log data file can be used as training data to train the SPE anomaly detection model to obtain the SPE anomaly detection model file; then, the SPE anomaly detection can be performed on the cut log data file by calling the SPE anomaly detection model file to obtain the SPE value returned by the SPE anomaly detection model, thereby obtaining and saving the anomaly detection result.

[0107] S207: If the abnormality detection result indicates that there is abnormal log data in the cut log data file, a fault diagnosis is performed on the abnormal log data.

[0108] According to some embodiments, Figure 4 As shown, the abnormal log data can be, for example, log data of abnormal SPE values, and the log data of abnormal SPE values ​​can be defined as a fault warning log to be analyzed; the log content of the fault warning log to be analyzed is read, and it is associated with the fault event in the fault library through the fault diagnosis model. If there is a correlation, the network element device associated with the log abnormality is located, and the corresponding fault diagnosis is output.

[0109] In some embodiments, the log data with abnormal SPE values ​​can be used to train the fault diagnosis model to obtain a fault diagnosis model file, and the fault diagnosis model file can be subsequently called to perform fault diagnosis on the log data with abnormal SPE values, thereby improving the accuracy of fault diagnosis.

[0110] In summary, the method provided in this embodiment, through the analysis of the situation in which some network elements in the network element inspection do not need to execute the inspection instructions, introduces the network element inspection related content into the log extraction and template mining on the basis of the log anomaly detection cluster analysis, so as to realize the supplementary mining of the inspection log template, improve the problem that the existing log template mining technology over-relies on the clustering method and causes the mining conclusions to be too concentrated, solve the problem of network element inspection log template mining caused by unclear significance, improve the problem of relatively concentrated anomaly detection results, and improve the accuracy of log anomaly detection.

[0111] In order to implement the above embodiment, the present disclosure also proposes a log anomaly detection device.

[0112] like Figure 5 As shown, the log anomaly detection device 500 includes:

[0113] The log acquisition unit 501 is used to acquire the log data file;

[0114] The log marking unit 502 is used to mark the log data file according to the full log template to obtain the marked log data file, wherein the full log template includes the inspection log template corresponding to the network element inspection task list. In the process of marking the log data file, if there is a first log stream in the log data file that meets the first matching requirement with the inspection log template, the first log stream is marked as the inspection log template;

[0115] The anomaly detection unit 503 is used to perform log anomaly detection on the marked log data file.

[0116] Optionally, in the process of marking the log data file, the method further includes:

[0117] If there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, then obtaining a target log in the second log stream whose matching degree with the inspection log template is less than the first matching degree threshold;

[0118] If the target log meets the third matching requirement, the second log stream is determined to be the first log stream.

[0119] Optionally, the log marking unit 502 is used for, if there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, before obtaining a target log in the second log stream whose matching degree with the inspection log template is less than a matching degree threshold, further used for:

[0120] Determine the network elements and task sequence corresponding to the network element inspection task list;

[0121] If there is a third log stream in the log data file that matches both the network element and the task sequence, and the log content of the third log stream meets the fourth matching requirement, the third log stream is determined to be the second log stream.

[0122] Optionally, the log content of the third log stream meets the fourth matching requirement, including:

[0123] The matching degree between the log content of the third log stream and the log content of the inspection log template is lower than the second matching degree threshold; and / or,

[0124] The aggregatability degree of the log content of the third log stream is greater than the aggregatability degree threshold.

[0125] Optionally, the log marking unit 502 is used to, after acquiring a target log in the second log stream whose matching degree with the inspection log template is less than a first matching degree threshold, further:

[0126] Get the calling time of the service interface of the network element corresponding to the target log;

[0127] If the call duration is less than the call duration threshold, it is determined that the target log meets the third matching requirement.

[0128] Optionally, the log marking unit 502 is used to mark the log data file according to the full log template, and when the marked log data file is obtained, it is specifically used to:

[0129] Compare the full log template with the log data file;

[0130] If the log data file contains a fourth log stream that meets the fifth matching requirement with the full log template, the fourth log stream is marked according to the full log template;

[0131] A cluster analysis is performed on the fifth log stream except the fourth log stream in the log data file to obtain a new log template, and the new log template is incorporated into the full log template to re-compare the log data file until the log data file meets the labeling requirements to obtain a labeled log data file.

[0132] Optionally, when the anomaly detection unit 503 is used to perform log anomaly detection on the annotated log data file, it is specifically used to:

[0133] Performing log file segmentation on the marked log data file to obtain a segmented log data file;

[0134] The cut log data file is input into the anomaly detection model to perform anomaly detection and obtain anomaly detection results;

[0135] If the abnormality detection result indicates that there is abnormal log data in the cut log data file, a fault diagnosis is performed on the abnormal log data.

[0136] It should be noted that the above explanation of the log anomaly detection method embodiment is also applicable to the log anomaly detection device of this embodiment, and will not be repeated here.

[0137] In summary, the device provided by the embodiment of the present disclosure can solve the problem of network element inspection log template mining caused by unclear significance by introducing network element inspection related content in the process of marking log data files, improve the problem of relatively concentrated anomaly detection results, and improve the accuracy of log anomaly detection.

[0138] In order to implement the above embodiments, the present disclosure also proposes an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0139] In order to implement the above embodiments, the present disclosure further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0140] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0141] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this disclosure shall comply with the relevant laws and regulations and shall not violate public order and good morals.

[0142] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0143] The present disclosure anticipates providing implementation schemes for users to selectively block the use or access of personal information data. That is, the present disclosure anticipates providing hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0144] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present disclosure belong.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0148] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0150] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present disclosure. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present disclosure.

Claims

1. A log anomaly detection method, characterized in that: include: Get the log data file; The log data file is annotated according to the full log template to obtain the annotated log data file, wherein the full log template includes an inspection log template corresponding to the network element inspection task list, and in the process of annotating the log data file, if there is a first log stream in the log data file that meets a first matching requirement with the inspection log template, the first log stream is annotated as the inspection log template; Perform log anomaly detection on the annotated log data file.

2. The method according to claim 1, characterized in that: In the process of marking the log data file, the method further includes: If there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, obtaining a target log in the second log stream whose matching degree with the inspection log template is less than a first matching degree threshold; If the target log meets the third matching requirement, the second log stream is determined to be the first log stream.

3. The method according to claim 2, characterized in that Before obtaining a target log in the second log stream whose matching degree with the inspection log template is less than a matching degree threshold if there is a second log stream in the log data file that meets the second matching requirement with the inspection log template, the method further includes: Determine the network elements and task sequence corresponding to the network element inspection task list; If there is a third log stream in the log data file that matches both the network element and the task sequence, and the log content of the third log stream meets the fourth matching requirement, the third log stream is determined to be the second log stream.

4. The method according to claim 3, characterized in that The log content of the third log stream meets the fourth matching requirement, including: The matching degree between the log content of the third log stream and the log content of the inspection log template is lower than a second matching degree threshold; and / or, The aggregatability of the log content of the third log stream is greater than an aggregatability threshold.

5. The method according to claim 2, characterized in that: After acquiring the target log in the second log stream whose matching degree with the inspection log template is less than the first matching degree threshold, the method further includes: Obtaining the calling duration of the service interface of the network element corresponding to the target log; If the calling duration is less than the calling duration threshold, it is determined that the target log meets the third matching requirement.

6. The method according to claim 1, characterized in that The step of labeling the log data file according to the full log template to obtain the labeled log data file includes: Comparing the log data file with the full log template; If there is a fourth log stream in the log data file that meets the fifth matching requirement with the full log template, marking the fourth log stream according to the full log template; A cluster analysis is performed on the fifth log stream except the fourth log stream in the log data file to obtain a newly added log template, and the newly added log template is incorporated into the full log template to re-compare the log data file until the log data file meets the labeling requirements to obtain a labeled log data file.

7. The method according to claim 1, characterized in that The performing log anomaly detection on the annotated log data file includes: Performing log file segmentation on the marked log data file to obtain a segmented log data file; Inputting the cut log data file into an anomaly detection model to perform anomaly detection and obtain an anomaly detection result; If the abnormality detection result indicates that abnormal log data exists in the cut log data file, a fault diagnosis is performed on the abnormal log data.

8. A log anomaly detection device, characterized in that: include: A log acquisition unit, used for acquiring log data files; A log marking unit, used for marking the log data file according to the full log template to obtain the marked log data file, wherein the full log template includes the inspection log template corresponding to the network element inspection task list, and in the process of marking the log data file, if there is a first log stream in the log data file that meets the first matching requirement with the inspection log template, then the first log stream is marked as the inspection log template; The anomaly detection unit is used to perform log anomaly detection on the annotated log data file.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images