Log analysis method, electronic equipment, storage medium and program product

Through the document understanding ability of log clustering and large language model, the log parsing plug-ins and configuration are automatically generated, which solves the problem of low log parsing efficiency in the existing technology, and realizes an efficient log parsing process.

CN119045915AActive Publication Date: 2024-11-29BEIJING TOPSEC NETWORK SECURITY TECH +2

Patent Information

Application Number
CN202411213158.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-29
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The prior art requires a large number of manual customization of parsing plug-ins and configuration files during log parsing, resulting in inefficiency, especially when facing complex logs, the customization process is time-consuming and labor-intensive.

Method used

Through the document understanding ability of log clustering and large language models, log data is automatically classified, parsing plug-ins and configurations are automatically generated, syntax details are blocked, and parsing personnel can focus on demand analysis and improve parsing efficiency.

Benefits of technology

It reduces the difficulty and manual workload of writing log parsing plug-ins, improves log parsing efficiency, simplifies the plug-in customization process, and reduces the complexity of manual analysis and debugging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045915A_ABST
    Figure CN119045915A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a log analysis method, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining to-be-analyzed log data; according to the to-be-analyzed log data, determining a target log type of the to-be-analyzed log data; and analyzing the to-be-analyzed log data according to a pre-stored log analysis plug-in corresponding to the target log type to obtain an analysis result of the to-be-analyzed log data, realizing automatic classification of logs and document understanding ability of a large language model through log clustering, and summarizing a log analysis requirement specification. In combination with the analysis result of the log sample, the analysis plug-in and related configuration are automatically generated, the log analysis plug-in is generated, and the log data to be analyzed are analyzed by adopting the log plug-in to obtain the analysis result, so that the writing difficulty of the log analysis plug-in is reduced, the grammar details of the log analysis system are shielded, and the analysis efficiency is improved. Analyzers are enabled to concentrate on analysis and understanding of analysis requirements, and log analysis efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log parsing method, electronic device, storage medium and program product. Background Art

[0002] Various computer devices and software systems generate a large number of logs during operation. These logs record a variety of information such as the device's operating status, operation process, abnormal information, and problem records. By analyzing these logs, you can understand the operation of the device; when a device fails, you can use the log to trace back the sequence of operations performed by the device, find the scene where the problem occurred, and locate the specific fault location, which helps analyze the cause of the problem. In the network security industry, security personnel need to discover potential security threats through the operation logs of various devices; when an attack occurs in the network environment, they also need to review the device logs to complete event tracing and investigation and evidence collection. It can be seen that logs have great business value.

[0003] The formats of logs generated by these devices and software systems developed by different manufacturers are also very different. In order to facilitate subsequent retrieval and analysis, it is usually necessary to parse the logs and normalize them into a standard format. The content of log records of devices or systems in different fields is also very different, and even contains some professional vocabulary. In order to facilitate user viewing, after log parsing, the fields are mapped and enriched as needed, and displayed to users in a more friendly way to lower the threshold for use. At present, most of these log parsing and field processing are completed by log parsing systems. Mature log parsing systems usually have to process logs of hundreds or thousands of devices and need to be scalable. Therefore, they are usually designed to parse logs through log parsing plug-ins or rules, and complete field mapping through configuration files. These log parsing plug-ins and parsing rules generally complete the normalization of logs through a set of agreed grammatical rules and regular expressions.

[0004] For a new device, a series of plug-ins and configuration files such as parsing files need to be customized. Currently, starting from the analysis of logs, the development and configuration of parsing templates for each sub-type of logs are all done manually. When faced with the parsing of complex logs, the entire process of log parsing customization is very time-consuming. How to improve the efficiency of log parsing and reduce the workload of manual work is an urgent problem to be solved. Summary of the invention

[0005] The purpose of some embodiments of the present application is to provide a log parsing method, electronic device, storage medium and program product. Through the technical solution of the embodiment of the present application, by obtaining the log data to be parsed; according to the log data to be parsed, determining the target log type of the log data to be parsed; parsing the log data to be parsed according to the pre-stored log parsing plug-in corresponding to the target log type, obtaining the parsing result of the log data to be parsed, wherein the log parsing plug-in adopts the log clustering method to classify the sample log data to obtain the classification result, and according to the large language model, determining the log parsing specification corresponding to the classification result, and according to the classification result and the log parsing specification, the automatic classification of the log is realized by log clustering in the embodiment of the present application, and the document understanding ability of the large language model is summarized to summarize the log parsing requirement specifications. Combined with the analysis results of the log sample, the parsing plug-in and related configuration are automatically generated, the log parsing plug-in is generated, and the log plug-in is used to parse the log data to be parsed to obtain the parsing result, so as to reduce the difficulty of writing the log parsing plug-in. The grammatical details of the log parsing system are shielded, so that the parsing personnel can focus on the analysis and understanding of the parsing requirements and improve the efficiency of log parsing.

[0006] In a first aspect, some embodiments of the present application provide a log parsing method, including:

[0007] Get the log data to be parsed;

[0008] Determine, according to the log data to be parsed, a target log type corresponding to the log data to be parsed;

[0009] According to a pre-stored log parsing plug-in corresponding to the target log type, the log data to be parsed is parsed to obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and according to a large language model, a log parsing specification corresponding to the classification result is determined, and the parsing result is determined based on the classification result and the log parsing specification.

[0010] Some embodiments of the present application realize automatic classification of logs through log clustering, document understanding ability of large language models, summarize log parsing requirements and specifications, automatically generate parsing plug-ins and related configurations based on the analysis results of log samples, generate log parsing plug-ins, and use the log plug-ins to parse the log data to be parsed to obtain parsing results, thus reducing the difficulty of writing log parsing plug-ins. The grammatical details of the log parsing system are shielded, allowing parsers to focus on the analysis and understanding of parsing requirements, thereby improving log parsing efficiency.

[0011] Optionally, the log parsing plug-in is obtained in the following manner:

[0012] Get sample log data;

[0013] Classifying the sample log data by log clustering to obtain a log type corresponding to the sample log data;

[0014] Determining a log parsing specification corresponding to the sample log data according to the large language module and the sample log data;

[0015] According to the log parsing specification and the log type, a log parsing plug-in corresponding to the log type is determined.

[0016] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated in combination with the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, the difficulty of plug-in customization is reduced, and the overall efficiency of parsing customization work is improved.

[0017] Optionally, the adopting log clustering method to classify the sample log data to obtain a log type corresponding to the sample log data includes:

[0018] Preprocessing the sample log data to obtain processed sample log data;

[0019] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0020] A preset clustering algorithm is used to perform string clustering on the segmentation result or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0021] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization.

[0022] Optionally, the segmenting the processed sample log data to obtain a segmentation result corresponding to the processed sample log data includes:

[0023] If the processed sample log data is described in natural language, regular segmentation is used to obtain a regular expression corresponding to the processed sample log data;

[0024] If the processed sample log data is in a preset format, a preset segmentation function is used to perform segmentation to obtain the segmentation result.

[0025] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve the accuracy of segmentation.

[0026] Optionally, determining a log parsing specification corresponding to the sample log data according to the large language module and the sample log data includes:

[0027] The large language model is used to parse the sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data, wherein the log parsing requirement specification includes at least field names, field meanings, data types, value examples and log samples corresponding to different log types.

[0028] Some embodiments of the present application process log format description documents with the help of the document understanding ability of a large language model. At the same time, they summarize the log parsing requirement specifications through norms, and automatically generate parsing plug-ins and related configurations in combination with the analysis results of log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0029] Optionally, determining a log parsing plug-in corresponding to the log type according to the log parsing specification and the log type includes:

[0030] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

[0031] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0032] Optionally, the method further comprises:

[0033] The log parsing plug-in is verified to obtain a verification result.

[0034] In a second aspect, some embodiments of the present application provide a log parsing device, including:

[0035] The acquisition module is used to obtain the log data to be parsed;

[0036] A determination module, used to determine a target log type corresponding to the log data to be parsed according to the log data to be parsed;

[0037] A parsing module is used to parse the log data to be parsed according to a pre-stored log parsing plug-in corresponding to the target log type, and obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and determine the log parsing specification corresponding to the classification result according to a large language model, and determine it according to the classification result and the log parsing specification.

[0038] Some embodiments of the present application realize automatic classification of logs through log clustering, document understanding ability of large language models, summarize log parsing requirements and specifications, automatically generate parsing plug-ins and related configurations based on the analysis results of log samples, generate log parsing plug-ins, and use the log plug-ins to parse the log data to be parsed to obtain parsing results, thus reducing the difficulty of writing log parsing plug-ins. The grammatical details of the log parsing system are shielded, allowing parsers to focus on the analysis and understanding of parsing requirements, thereby improving log parsing efficiency.

[0039] Optionally, the device further comprises an establishing module, wherein the establishing module is configured to:

[0040] Get sample log data;

[0041] Classifying the sample log data by log clustering to obtain a log type corresponding to the sample log data;

[0042] Determining a log parsing specification corresponding to the sample log data according to the large language module and the sample log data;

[0043] According to the log parsing specification and the log type, a log parsing plug-in corresponding to the log type is determined.

[0044] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated in combination with the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, the difficulty of plug-in customization is reduced, and the overall efficiency of parsing customization work is improved.

[0045] Optionally, the establishing module is used to:

[0046] Preprocessing the sample log data to obtain processed sample log data;

[0047] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0048] A preset clustering algorithm is used to perform string clustering on the segmentation result or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0049] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization.

[0050] Optionally, the establishment module is used to

[0051] If the processed sample log data is described in natural language, regular segmentation is used to obtain a regular expression corresponding to the processed sample log data;

[0052] If the processed sample log data is in a preset format, a preset segmentation function is used to perform segmentation to obtain the segmentation result.

[0053] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve the accuracy of segmentation.

[0054] Optionally, the establishing module is used to:

[0055] The large language model is used to parse the sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data, wherein the log parsing requirement specification includes at least field names, field meanings, data types, value examples and log samples corresponding to different log types.

[0056] Some embodiments of the present application use the document understanding ability of a large language model to summarize the log parsing requirement specifications, and automatically generate a parsing plug-in and related configurations based on the analysis results of the log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0057] Optionally, the establishing module is used to:

[0058] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

[0059] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0060] Optionally, the establishing module is used to:

[0061] The log parsing plug-in is verified to obtain a verification result.

[0062] In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the log parsing method as described in any embodiment of the first aspect can be implemented.

[0063] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the log parsing method as described in any embodiment of the first aspect.

[0064] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, it can implement the log parsing method as described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A flow chart of a log parsing method provided in an embodiment of the present application;

[0067] Figure 2 A flowchart of another log parsing method provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the structure of a log parsing device provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0071] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0072] Various computer devices and software systems generate a large number of logs during operation. These logs record a variety of information such as the device's operating status, operation process, abnormal information, and problem records. By analyzing these logs, you can understand the operation of the device; when a device fails, you can use the log to trace back the sequence of operations performed by the device, find the scene where the problem occurred, and locate the specific fault location, which helps analyze the cause of the problem. In the network security industry, security personnel need to discover potential security threats through the operation logs of various devices; when an attack occurs in the network environment, they also need to review the device logs to complete event tracing and investigation and evidence collection. It can be seen that logs have great business value.

[0073] The formats of logs generated by these devices and software systems developed by different manufacturers are also very different. In order to facilitate subsequent retrieval and analysis, it is usually necessary to parse the logs and normalize them into a standard format. The content of log records of devices or systems in different fields is also very different, and even contains some professional vocabulary. In order to facilitate user viewing, after log parsing, the fields need to be mapped and enriched, and displayed to users in a more friendly way to lower the threshold of use. At present, most of these log parsing and field processing are completed by log parsing systems. Mature log parsing systems usually have to process logs of hundreds or thousands of devices and need to be scalable. Therefore, they are usually designed to parse logs through log parsing plug-ins or rules, and complete field mapping through configuration files. These log parsing plug-ins and parsing rules generally complete the normalization of logs through a set of agreed grammatical rules and regular expressions.

[0074] For a new device, a series of plug-ins and configuration files such as parsing files need to be customized. Currently, starting from the analysis of logs, the development and configuration of parsing templates for each sub-type log are all done manually. When facing the parsing of complex logs, the whole process of log parsing customization is very time-consuming. How to improve the efficiency of log parsing and reduce the workload of manual work is an urgent problem to be solved. In view of this, some embodiments of the present application provide a log parsing method, which includes obtaining log data to be parsed; determining the target log type of the log data to be parsed according to the log data to be parsed; parsing the log data to be parsed according to the pre-stored log parsing plug-in corresponding to the target log type, and obtaining the parsing result of the log data to be parsed, wherein the log parsing plug-in adopts the log clustering method to classify and process the sample log data to obtain the classification result, and determines the log parsing specification corresponding to the classification result according to the large language model, and determines the log parsing specification according to the classification result and the log parsing specification. In the embodiment of the present application, the automatic classification of logs is realized by log clustering, and the document understanding ability of the large language model is used to summarize the log parsing requirement specifications. Combined with the analysis results of the log samples, the parsing plug-in and related configurations are automatically generated, and the log parsing plug-in is generated. The log plug-in is used to parse the log data to be parsed and obtain the parsing results. In this way, the difficulty of writing the log parsing plug-in is reduced. The grammatical details of the log parsing system are shielded, so that the parser can focus on the analysis and understanding of the parsing requirements and improve the efficiency of log parsing.

[0075] like Figure 1 As shown, an embodiment of the present application provides a log parsing method, the method comprising:

[0076] S101, obtaining log data to be parsed;

[0077] S102, determining a target log type of the log data to be parsed according to the log data to be parsed;

[0078] Specifically, the terminal device obtains the log data to be parsed, which can be sent by the user's device. The device (such as a firewall) is configured to send logs, specify the receiving server IP and port, and then send the logs to the log audit server, or download them from the network, parse the log data to be parsed, and determine the target log type of the log data to be parsed, for example, the log type can be natural language, etc.

[0079] S103. According to a pre-stored log parsing plug-in corresponding to the target log type, the log data to be parsed is parsed to obtain a parsing result corresponding to the log data to be parsed, wherein the log parsing plug-in classifies the sample log data by log clustering to obtain a classification result, and determines the log parsing specification corresponding to the classification result according to the large language model, and determines the parsing result according to the classification result and the log parsing specification.

[0080] Specifically, log parsing is to extract meaningful information fragments from texts described in natural language or in a certain format, name each information fragment, and form an information object list in the form of key:value to display to the user. Here, the information object list in the form of key:value is called a Token object list, where key represents the name of the Token and value represents the value of the Token.

[0081] The embodiment of the present application classifies the original logs through clustering, analyzes the log fields by interpreting the log format document, generates a log parsing requirement specification, guides the program to generate a parsing plug-in that meets the grammatical requirements of the log parsing system according to the requirement specification, and finally uses the classified original logs and the log parsing requirement specification to verify the actual parsing effect of the plug-in. The above method solves the problem of complex process and low efficiency when manually customizing new types of parsing work.

[0082] Some embodiments of the present application realize automatic classification of logs through log clustering, document understanding ability of large language models, summarize log parsing requirements and specifications, automatically generate parsing plug-ins and related configurations based on the analysis results of log samples, generate log parsing plug-ins, and use the log plug-ins to parse the log data to be parsed to obtain parsing results, thus reducing the difficulty of writing log parsing plug-ins. The grammatical details of the log parsing system are shielded, allowing parsers to focus on the analysis and understanding of parsing requirements, thereby improving log parsing efficiency.

[0083] Another embodiment of the present application further supplements the log parsing method provided in the above embodiment.

[0084] like Figure 2 As shown, the embodiment of the present application provides an automatic log parsing method, which includes the following 4 steps:

[0085] Step (1) clustering of original logs;

[0086] Step (2) analyzing the log format description document and generating log parsing requirement specifications;

[0087] Step (3) generating a parsing plug-in according to the log parsing requirement specification;

[0088] Step (4) parses the verification of the plug-in.

[0089] Optionally, the log parsing plugin is obtained as follows:

[0090] Get sample log data;

[0091] The sample log data is classified by log clustering to obtain the log type corresponding to the sample log data;

[0092] Determine the log parsing specification corresponding to the sample log data according to the large language module and the sample log data;

[0093] According to the log parsing specification and log type, determine the log parsing plug-in corresponding to the log type.

[0094] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated in combination with the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, the difficulty of plug-in customization is reduced, and the overall efficiency of parsing customization work is improved.

[0095] Optionally, the sample log data is classified by log clustering to obtain log types corresponding to the sample log data, including:

[0096] Preprocessing the sample log data to obtain processed sample log data;

[0097] Specifically, the specific steps of the "preprocessing" in step (1) log clustering are: replace common dynamic fields in the log (such as syslog priority, time, ip, port, mac, etc.) with specific identifiers through regular matching. Function: 1. Facilitate the subsequent token segmentation; 2. Reduce the interference of these dynamic data during clustering, making the clustering results more accurate. Preset some common regular expressions, and support users to specify new patterns and content.

[0098] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0099] A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0100] Specifically, the specific steps of "log clustering" in step (1) log clustering are: there are three clustering methods: clustering according to the string consisting of part or all of the field names after segmentation; clustering using one or some field values ​​after segmentation, similar to the group by function in the database; raw log clustering, in this case, there is no need to rely on field segmentation, and the pre-processed raw logs are directly clustered.

[0101] Provides multiple clustering methods (DBSCAN, progressive clustering with feedback (K-means, Gaussian mixture model clustering, etc.), field value classification), and users can choose the method with the best clustering effect. The purpose of log clustering is to group logs of the same log type into a cluster. The same log usually has the same fields and formats, and can be displayed using a field set.

[0102] Segmentation first is usually helpful for log clustering, but not all situations require segmentation before clustering. For logs where most of the text is in the key:value format, segmentation is generally performed first, and then clustering is performed based on some or all of the fields; for logs where most of the text is in natural language description text, clustering can be performed first, and then segmentation tokenization and parsing can be performed for each type of log.

[0103] After log clustering is completed, the common dynamic contents such as syslog priority, time, IP, MAC, etc. replaced during data preprocessing need to be replaced back to avoid affecting subsequent analysis.

[0104] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization.

[0105] Optionally, the processed sample log data is segmented to obtain a segmentation result corresponding to the processed sample log data, including:

[0106] If the processed sample log data is described in natural language, regular segmentation is used to obtain a regular expression corresponding to the processed sample log data;

[0107] If the processed sample log data is in a preset format, the preset segmentation function is used to segment the data to obtain a segmentation result.

[0108] Specifically, the specific steps of "splitting tokens" in log clustering are: according to the log characteristics, select regular segmentation or function segmentation. When the log is described in natural language, regular segmentation can usually be used. When the log is in key:value format (such as WELF format), function segmentation can be selected.

[0109] Select Regular Split and generate a regular expression to extract the token.

[0110] When selecting function splitting, use the system preset splitting function to split the token.

[0111] Regular expressions can complete the token segmentation of all logs, but when the logs are suitable for segmentation functions, it is simpler to use segmentation functions.

[0112] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve the accuracy of segmentation.

[0113] Optionally, determining a log parsing specification corresponding to the sample log data according to the large language module and the sample log data includes:

[0114] A large language model is used to parse sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data. The log parsing requirement specification includes at least field names, field meanings, data types, value examples, and log samples corresponding to different log types.

[0115] Specifically, the specific steps of "document understanding" described in step (2) generating the log parsing specification are: in the case where the log format document content is standardized and comprehensive, read the document to understand the business significance and field meaning of various logs; in the case where there is a lack of log format documents, the business significance and field meaning of various logs are disassembled through the experience and professional knowledge of the parser.

[0116] The specific steps of "generating log parsing requirement specifications" described in step (2) generating log parsing specifications are: based on the understanding of the log format document and the log, generate the log parsing requirement specifications. The field names in the original log are usually in English, or the field names in the original log are not suitable for direct display. At this time, the field names need to be renamed to field names suitable for display. The correspondence between the original fields and the parsed fields of each log needs to be sorted out. The parsing specification contains the field names, field meanings, data types, value examples, etc. that need to be parsed for all types of logs. It also includes log samples for each log type.

[0117] The specific steps for "determining the log type of each cluster" described in the log parsing specification are as follows:

[0118] After log clustering, similar logs will be grouped into one category. For subsequent analysis, you need to specify a log type for each cluster of logs after clustering.

[0119] For logs in natural language format, the segmentation regular expressions corresponding to each cluster after clustering are used to match all types of log samples included in the log parsing specification. The type corresponding to the successfully matched log sample is designated as the log type of the current cluster. In rare cases, if a log sample matches multiple segmentation regular expressions at the same time, the cluster corresponding to the segmentation regular expression with the most Token objects after segmentation is designated as the log type corresponding to the log sample.

[0120] For logs in key:value format (such as WELF), first, the sample logs are segmented using the same function segmentation method; then, the string consisting of the token names after each cluster of logs is segmented is compared with the string consisting of the token names after the log sample is segmented, and the log category corresponding to the sample with the highest similarity is found and designated as the log category of the current cluster.

[0121] Some embodiments of the present application use the document understanding ability of a large language model to summarize the log parsing requirement specifications, and automatically generate a parsing plug-in and related configurations based on the analysis results of the log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0122] Optionally, according to the log parsing specification and the log type, a log parsing plug-in corresponding to the log type is determined, including:

[0123] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

[0124] Specifically, the specific steps of "generating a parsing plug-in" in step (3) generating a log parsing plug-in are:

[0125] Based on the clustering results and token segmentation results, combined with the requirements for each log analysis specification, and according to the analysis plug-in syntax supported by the log analysis system, an initial analysis plug-in is generated.

[0126] The specific steps of "field adjustment" in step (3) generating the log parsing plug-in are as follows: first, according to the data type of each field in the log parsing requirement specification, the normalized fields supported by the log parsing system are automatically specified. Then, according to the examples of field types and field values, the content of the specific type of field is adjusted.

[0127] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0128] The specific steps of "sending classified logs to the log parsing system" described in step (4) generating the log parsing plug-in are:

[0129] This step verifies whether the generated parsing file is correct. Before sending logs, the log parsing system needs to load the parsing plug-in and related configurations; it is also necessary to add a log source to the log parsing system to receive logs. These are the preparations before verifying the logs.

[0130] Call the log sending tool to send a type of log to the log parsing system.

[0131] The specific steps of "Log Parsing Verification" described in step (4) generating the log parsing plug-in are:

[0132] Obtain the parsing result of the current type of log through the interface, and verify the fields and field values ​​after parsing the current type of log by comparing them with the contents in the log parsing requirement specification to check whether they meet the parsing specification.

[0133] Repeat the two steps of "Sending classified logs to the log analysis system" and "Log analysis verification" in step (4) until all types of logs are verified.

[0134] The embodiments of the present application can be applied to a log audit system to quickly support the parsing of new log types.

[0135] The embodiment of the present application is based on a log audit product, and describes the use of an automatic log parsing method proposed in the embodiment of the present application to simplify the development process of the parsing plug-in and improve the efficiency of new log customization through intelligent analysis and understanding of log samples and log format documents when the log audit product supports new log types.

[0136] This example uses the log parsing process of a vendor's load balancing product as an example to illustrate the process of generating a parsing plug-in. The product contains five types of logs, and this example uses the service log and NAT log as examples to illustrate the process of generating a parsing plug-in.

[0137] A sample service log is:

[0138] <134> Sep 10 21:07:15zdg:RS_DETECT: Virtual service [test_l3] is abnormal, and the default node pool it references is faulty

[0139] <134> Sep 10 21:07:15zdg:RS_DETECT: Virtual service [http] is abnormal, the default node pool it references is faulty

[0140] The sample NAT log is as follows:

[0141] <134> Dec 14 11:35:00src@localhost:[2020-12-14_11-34-59]NAT_LOG_DEL_ENTRY[tcp]10.10 1.1.81:58449(60.164.184.44:58449)->180.163.249.41:80(180.163.249.41:80)

[0142] <134> Dec 14 11:35:00src@localhost:[2020-12-14_11-34-59]NAT_LOG_ADD_ENTRY[tcp]223.1 04.56.5:52163(223.104.56.5:52163)->60.164.184.43:5100(10.99.100.1:5100)

[0143] The above logs are numbered log0, log1, log2, and log3 respectively for the convenience of subsequent description.

[0144] The log format document describes service logs and NAT logs as follows:

[0145] 3. Service log format

[0146] The service log syslog format is as follows:

[0147] {System time host name: module name: log details}

[0148] example:

[0149] <134> Sep 10 21:07:15zdg:RS_DETECT: Virtual service [test_l3] is abnormal, and the default node pool it references is faulty

[0150] 4NAT log format

[0151] The NAT log syslog format is as follows:

[0152] {System time host name: [TIME] ACTION [PROTOCOL] SrcIP: SrcPort (NatSrcIP: NatSrcPort) -> DstIp: DstPort (NatDstIp: NatDstPort)}

[0153] like:

[0154] <134> Sep 10 21:22:02zdg:[2018-09-10_21-22-02]NAT_LOG_DEL_ENTRY[icmp]172.16.1.136:0(200.200.83.7:0)->172.16.1.7:0(200.200.80.136:0)

[0155] 4.1 Introduction to NAT Log Fields

[0156] 1>TIME: Timestamp

[0157] 2>ACTION: divided into create and delete, the value is NAT_LOG_DEL_ENTRY or NAT_LOG_ADD_ENTRY

[0158] 3>PROTOCOL: protocol type, the value range is tcp|icmp|icmpv6|udp|other

[0159] 4>SrcIP: Source IP

[0160] 5>SrcPort: Source port

[0161] 6>NatSrcIP: Source IP after NAT

[0162] 7>NatSrcPort: Source port after NAT

[0163] 8>DstIp: destination IP

[0164] 9>DstPort: destination port

[0165] 10>NatDstIP: Destination IP after NAT

[0166] 11>NatDstPort: destination port after NAT;

[0167] In this example, the log sample is a natural language description. Log clustering can be performed first, and then each cluster of logs can be tokenized. According to the method described in the detailed description of the present invention, the specific process is as follows:

[0168] 1. Log Clustering

[0169] 1.1 Preprocessing

[0170] Replace the dynamic content in the original log, and the replaced log is

[0171] #pri##TIME1#zdg:RS_DETECT: Virtual service [test_l3] is abnormal, and the default node pool it references is faulty

[0172] #pri##TIME1#zdg:RS_DETECT: Virtual service [http] is abnormal, the default node pool it references is faulty

[0173] #pri##TIME1#src@localhost:[#TIME2#]NAT_LOG_DEL_ENTRY[tcp]#IPV4#:#PORT#(#IPV4#:#PORT#)->#IPV4#:#PORT#(#IPV4#:#PORT#)

[0174] #pri##TIME1#src@localhost:[#TIME2#]NAT_LOG_ADD_ENTRY[udp]#IPV4#:#PORT#(#IPV4#:#PORT#)->#IPV4#:#PORT#(#IPV4#:#PORT#)

[0175] 1.2 Log Clustering

[0176] In this example, the DBSCAN clustering algorithm (full name: Density-Based Spatial Clustering of Applications with Noise, which is a relatively representative density-based clustering algorithm) is used to cluster the preprocessed logs. First, the original logs are segmented using the Jieba word segmentation algorithm; then the segmented data is vectorized using TfidfVectorizer (text feature extraction); finally, the log samples are clustered using DBSCAN (parameters such as: eps=0.5, min_samples=1). The clustering result is: [0 0 1 1], and there are 2 clusters in the clustering result, log0 and log1 are one cluster, and log2 and log3 are another cluster. The result can be expressed as:

[0177] Cluster0:[log0,log1]

[0178] Cluster1:[log2,log3]

[0179] When the clustering results are not satisfactory, you can improve them by adjusting the clustering parameters. If satisfactory results are still not achieved after adjusting the parameters, you can choose other clustering algorithms, such as the progressive clustering method with feedback based on K-means and Gaussian mixture model clustering algorithms, to achieve relatively satisfactory clustering results.

[0180] 1.3 Token Splitting

[0181] For each cluster of logs in the above clustering, generate a regular expression for splitting Tokens and split the logs. Here we take the first cluster of logs as an example.

[0182] In the previous step, the segmentation results of log0 and log1 are:

[0183] The word segmentation result of Log0 is: ['#','pri','#',",'#','TIME1','#',",'zdg',",':',",'RS','_','DETECT',':',",'virtual','service',",'[','test','_','l3',']',",'abnormal',',',','its','reference','of','default','node','pool','fault']

[0184] The word segmentation result of Log1 is: ['#','pri','#',",'#','TIME1','#',",'zdg',",':',",'RS','_','DETECT',':',",'virtual','service',",'[','http',']',",'abnormal',',','its','reference','of','default','node','pool','fault']

[0185] For these two sequences, find the longest common subsequence of the two sequences:

[0186] [['#','pri','#',",'#','TIME1','#',",'zdg',",':',",'RS','_','DETECT',':',",'virtual','service',",'['],[']',",'abnormal',',',','its','reference','','default','node','pool','failed']].

[0187] First, concatenate each common subsequence into a string; if there are special characters (regular expression-sensitive special characters) in each subsequence string, escape them; then, use wildcards to concatenate each subsequence to generate a regular expression:

[0188] #pri##TIME1#zdg:RS_DETECT: Virtual service \[([\w\W]+)\] is abnormal, and the default node pool it references is faulty

[0189] Then, use the generated regular expression to capture data in the preprocessing log. For example, the above regular expression captures the following content in two logs: [["test_l3","http"]]. In this example, there is only one capture group (there is only one in this example).

[0190] Next, analyze the components of ["test_l3","http"] and use ([\w_]+) to replace the original wildcard ([\w\W]+).

[0191] Finally, replace the dynamic data identifier replaced in the preprocessing stage with the corresponding regular expression; the regular expression at this time is: "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+)zdg:RS_DETECT: Virtual service\[([\w_]+)\] is abnormal, and the default node pool it references is faulty". Therefore, there are fewer samples in the example, and the host name and module name are the same. The automatically generated regular expression does not extract the host name and module name fields. When the number of log samples increases and these fields have different values, the generated regular expression will automatically extract this part of the content.

[0192] Allow users to adjust and improve the automatically generated segmentation regular expression. In this example, if you are also interested in the host name, module name, and specific log content, you can adjust the regular expression to: "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+)(\w+):([\w-]+):(Virtual service\[([\w\d]+)\] is abnormal, and the default node pool it references is faulty)". Use this regular expression to segment the original log. The default Token name is represented by the grouping sequence number, so an array containing 6 Token objects can be obtained: as shown in Table 1:

[0193] Table 1

[0194]

[0195]

[0196] The second cluster of logs is processed in the same way, and the automatic segmentation regularization result is:

[0197] "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+) ([\w@]+) :\[(\d{4}-\d+-\d+_\d+-\d+-\d+)\] NAT_LOG_\w+_ENTRY\[(\w+)\](\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\) ->(\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\)”.

[0198] Manually add the extraction of the ACTION field and manually adjust the regular expression to:

[0199] "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+) ([\w@]+) :\[(\d{4}-\d+-\d+_\d+-\d+-\d+)\] (NAT_LOG_\w+_ENTRY)\[(\w+)\](\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\) ->(\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\)”. As shown in Table 2:

[0200] Table 2

[0201]

[0202]

[0203] 2. Generate log parsing specifications

[0204] In this example, the log format document is standardized and comprehensive. This step involves reading the document to understand the business significance and field meaning of various logs. The purpose of reading the document is to generate log parsing requirements specifications. The tasks of document understanding and generating log parsing requirements specifications can be done manually or with the help of a large language model.

[0205] The manual generation process is to organize the log parsing requirement specifications and various types of log samples from the documents according to the prescribed format.

[0206] Use the big model to understand the document, design prompt words, then call the interface to input the document into the big model, and output the log parsing requirements specification as required. The following table is part of the log parsing requirements specification generated by the big model that is related to this example, as shown in Table 3.

[0207] Table 3

[0208]

[0209]

[0210] The log types extracted by the large model and part of the log sample table are shown in Table 4.

[0211] Table 4

[0212]

[0213] 2.3 Determine the type of each cluster log

[0214] Since the log in this example is a string described in natural language, the segmentation regularization of each cluster is used to segment each sample log. The results are shown in Table 5:

[0215] Table 5

[0216]

[0217] In this example, the fields after the original log is segmented are named according to the matching log type using the field names in the log parsing specification. The fields in the value example in the log parsing specification can be used to process its token.

[0218] After processing, the segmentation results of Cluster0 are as shown in Table 6:

[0219] Table 6

[0220]

[0221] After processing, the segmentation results of Cluster1 are as shown in Table 7:

[0222] Table 7

[0223]

[0224] Priority and serviceName are fields not included in the specification and are completed manually. The renaming of other token names can be completed automatically by the specification.

[0225] 3. Generate log parsing plug-in

[0226] 3.1 Generate parsing plug-in

[0227] Based on the segmentation regularity and Token name duplication of each cluster log generated above, and combined with the types of each field in the log parsing specification, a parsing plug-in can be generated according to the parsing plug-in syntax rules supported by the current log parsing system. As an example, the key fragments of the parsing plug-in related to this example are shown as follows:

[0228] <? xml version="1.0" encoding="UTF-8"? >

[0229] …

[0230] <exp……reg="<;(\d+)> ;(\w{3}\s\d+\s\d+:\d+:\d+)(\w+):([\w-]+):(Virtual service\[([\w\d]+)\] is abnormal, and the default node pool it references is faulty)"…>

[0231] <! --Service log, CATEGORY = 'Service log' -->

[0232] <ts>

[0233] <t name="1"a="priority" / >

[0234] <t name="2"a="systemTime" / >

[0235] <t name="3"a="hostName" / >

[0236] <t name="4"a="module" / >

[0237] <t name="5"a="message" / >

[0238] <t name="6"a="serviceName" / >

[0239] < / ts>

[0240] <e name="severity"value="_GetSeverity(priority)" / >

[0241] <e name="year"value="_Time($'yyyy')" / >

[0242] <e name="systemTime"value="_Format(year,$”,systemTime)" / >

[0243] <e name="systemTime"value="_FormatTime($'yyyy MMM ddHH:mm:ss',systemTime)" / >

[0244] <m name="SEVERITY" value="severity" / ><!--Log level-->

[0245] <m name="SYSTEM_TIME" value="systemTime" / ><!--System time-->

[0246] <m name="HOST_NAME" value="hostName" / ><!--Host name-->

[0247] <m name="MODULE" value="module" / ><!--Module name-->

[0248] <m name="MESSAGE" value="message" / ><!--Specific log content-->

[0249] <m name="SERVICE_NAME" value="serviceNmae" / ><!--Virtual service-->

[0250] <m name="CATEGORY" value="$'Service log'" / ><!--Log type-->

[0251]

[0252]

[0253] …

[0254] …

[0255] 3.2 Field Adjustment

[0256] According to the field requirements of the log audit system, some fields in the above steps need special processing to be displayed normally in the log audit system. For example, the formatting of the time field, the processing of the log level field, the field naming of the supplementary token, some fields do not need to be parsed (for example, there are two time-related fields in the NAT log, the content is consistent, and only one can be parsed) and comments can be made.

[0257] There are two time fields here, both of which are captured by regular expressions during preprocessing. For supported time modes, corresponding time formatting functions are predefined for processing, and corresponding codes can be directly generated in the parsing plug-in.

[0258] The red part of the XML fragment in the previous step is the adjustment. The annotations of the "Virtual Service" field name and the timestamp field are processed manually, and the formatting of the time field can be completed by the program.

[0259] 4. Parsing verification

[0260] Before starting verification, load the previously generated parsing plug-in and related configurations in the log audit system, and make related configurations for receiving logs.

[0261] 4.1 Sending classified logs to the log parsing system

[0262] Call the simulation log sending tool to send a certain type of log.

[0263] 4.2 Log parsing and verification

[0264] Use the log audit system interface to obtain the parsing results of the current type of log.

[0265] Compare the parsing results obtained by the interface with the fields in the log parsing requirements specification to verify whether the parsing results are correct. Key inspections: 1. Whether all the fields required in the log parsing requirements specification have been parsed; 2. Whether the fields required to be parsed have parsed the correct values.

[0266] Repeat steps 4.1-4.2 to verify all types of log parsing results.

[0267] The embodiments of the present application solve the problem of low efficiency when the existing log parsing system completely relies on manual customization of the parsing of new log types. Log classification is completed through clustering; with the help of the document analysis and summarization capabilities of the large model, the analysis of the log fields is completed through the interpretation of the log format document, and the parsing requirement specifications for each type of log are generated. The parsing requirement specifications in a specific format make it possible to automatically generate parsing plug-ins. An automatic log parsing method is proposed, which automatically generates a parsing plug-in that meets the grammatical requirements of the log parsing system based on the log parsing requirement specifications. Reduce the workload of manually writing and debugging plug-ins, reduce the complexity of manual parsing, use the classified original logs and the requirement specifications for the parsing effect of each type of log, and perform automated verification tests on the loaded plug-ins to ensure that the parsing work of the parsing plug-in meets the requirement specifications.

[0268] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.

[0269] Another embodiment of the present application provides a log parsing device for executing the log parsing method provided in the above embodiment.

[0270] like Figure 3 , which is a schematic diagram of the structure of a log parsing device provided in an embodiment of the present application. The log parsing device includes an acquisition module 301, a determination module 302 and a parsing module 303, wherein:

[0271] The acquisition module 301 is used to acquire the log data to be parsed;

[0272] The determination module 302 is used to determine the target log type of the log data to be parsed according to the log data to be parsed;

[0273] The parsing module 303 is used to parse the log data to be parsed according to a pre-stored log parsing plug-in corresponding to the target log type, and obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and determines the log parsing specification corresponding to the classification result according to the large language model, and determines it according to the classification result and the log parsing specification.

[0274] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0275] Some embodiments of the present application realize automatic classification of logs through log clustering, document understanding ability of large language models, summarize log parsing requirements and specifications, automatically generate parsing plug-ins and related configurations based on the analysis results of log samples, generate log parsing plug-ins, and use the log plug-ins to parse the log data to be parsed to obtain parsing results, thus reducing the difficulty of writing log parsing plug-ins. The grammatical details of the log parsing system are shielded, allowing parsers to focus on the analysis and understanding of parsing requirements, thereby improving log parsing efficiency.

[0276] Another embodiment of the present application further supplements the log parsing device provided in the above embodiment.

[0277] Optionally, the device further comprises an establishing module, the establishing module being configured to:

[0278] Get sample log data;

[0279] The sample log data is classified by log clustering to obtain the log type corresponding to the sample log data;

[0280] Determine the log parsing specification corresponding to the sample log data according to the large language module and the sample log data;

[0281] According to the log parsing specification and log type, determine the log parsing plug-in corresponding to the log type.

[0282] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated in combination with the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, the difficulty of plug-in customization is reduced, and the overall efficiency of parsing customization work is improved.

[0283] Optionally, build modules for:

[0284] Preprocessing the sample log data to obtain processed sample log data;

[0285] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0286] A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0287] Some embodiments of the present application realize automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularization.

[0288] Optionally, create a module for

[0289] If the processed sample log data is described in natural language, regular segmentation is used to obtain a regular expression corresponding to the processed sample log data;

[0290] If the processed sample log data is in a preset format, the preset segmentation function is used to segment the data to obtain a segmentation result.

[0291] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve the accuracy of segmentation.

[0292] Optionally, build modules for:

[0293] A large language model is used to parse sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data. The log parsing requirement specification includes at least field names, field meanings, data types, value examples, and log samples corresponding to different log types.

[0294] Some embodiments of the present application use the document understanding ability of a large language model to summarize the log parsing requirement specifications, and automatically generate a parsing plug-in and related configurations based on the analysis results of the log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0295] Optionally, build modules for:

[0296] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

[0297] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0298] Optionally, the establishing module is used to:

[0299] The log parsing plug-in is verified to obtain a verification result.

[0300] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0301] It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any combination without conflict, and this application is not limited thereto.

[0302] The embodiments of the present application also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the operation of the method corresponding to any embodiment of the log parsing method provided in the above embodiments can be implemented.

[0303] An embodiment of the present application further provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, the operation of the method corresponding to any embodiment of the log parsing method provided in the above embodiments can be implemented.

[0304] like Figure 4 As shown, some embodiments of the present application provide an electronic device 400, which includes: a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420, wherein the processor 420 can implement a method of any embodiment of the log parsing method as described above when reading the program from the memory 410 through a bus 430 and executing the program.

[0305] Processor 420 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 420 can be a microprocessor.

[0306] The memory 410 may be used to store instructions executed by the processor 420 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 420 of the disclosed embodiment may be used to execute instructions in the memory 410 to implement the method shown above. The memory 410 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0307] The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0308] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0309] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A log parsing method, characterized in that: The method comprises: Get the log data to be parsed; Determine, according to the log data to be parsed, a target log type corresponding to the log data to be parsed; According to a pre-stored log parsing plug-in corresponding to the target log type, the log data to be parsed is parsed to obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and according to a large language model, a log parsing specification corresponding to the classification result is determined, and the parsing result is determined based on the classification result and the log parsing specification.

2. The log parsing method according to claim 1, characterized in that: The log parsing plug-in is obtained in the following way: Get sample log data; Classifying the sample log data by log clustering to obtain a log type corresponding to the sample log data; Determining a log parsing specification corresponding to the sample log data according to the large language module and the sample log data; According to the log parsing specification and the log type, a log parsing plug-in corresponding to the log type is determined.

3. The log parsing method according to claim 2, characterized in that: The method of adopting log clustering to classify the sample log data to obtain the log type corresponding to the sample log data includes: Preprocessing the sample log data to obtain processed sample log data; Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data; A preset clustering algorithm is used to perform string clustering on the segmentation result or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

4. The log parsing method according to claim 3, characterized in that: The segmenting of the processed sample log data to obtain a segmentation result corresponding to the processed sample log data includes: If the processed sample log data is described in natural language, regular segmentation is used to obtain a regular expression corresponding to the processed sample log data; If the processed sample log data is in a preset format, a preset segmentation function is used to perform segmentation to obtain the segmentation result.

5. The log parsing method according to claim 2, characterized in that: The step of determining a log parsing specification corresponding to the sample log data according to the large language module and the sample log data includes: The large language model is used to parse the sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data, wherein the log parsing requirement specification includes at least field names, field meanings, data types, value examples and log samples corresponding to different log types.

6. The log parsing method according to claim 2, characterized in that: The step of determining a log parsing plug-in corresponding to the log type according to the log parsing specification and the log type includes: According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

7. The log parsing method according to claim 2, characterized in that: The method further comprises: The log parsing plug-in is verified to obtain a verification result.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can implement the log parsing method described in any one of claims 1 to 7 when executing the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the program, when executed by a processor, can implement the log parsing method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, wherein: When the computer program is executed by a processor, the log parsing method described in any one of claims 1 to 7 can be implemented.

Citation Information

Patent Citations

  • Log analysis method and device

    CN107273269A

  • Information extraction method, device and equipment and storage medium

    CN111708860A

  • Log analysis method and device

    CN116541231A

  • Field content based pattern generation for heterogeneous logs

    US20180307576A1

Cited By

  • Resource consumption intelligent supervision system and method for document data quality detection

    CN120562379A

  • Intelligent monitoring system and method for resource consumption for document data quality detection

    CN120562379B