Log parsing method, electronic device, storage medium, and program product

Abstract

The embodiment of the present application provides a log parsing method, electronic device, storage medium and program product, the method comprising: obtaining log data to be parsed; determining the target log type of the log data to be parsed according to the log data to be parsed; parsing the log data to be parsed according to a pre-stored log parsing plug-in corresponding to the target log type, obtaining a parsing result corresponding to the log data to be parsed, and realizing automatic classification of logs through log clustering, document understanding capability of a large language model, and summarizing log parsing requirement specifications. Combined with the analysis results of the log sample, a parsing plug-in and related configurations are automatically generated, a log parsing plug-in is generated, and the log parsing plug-in is used to parse the log data to be parsed and obtain a parsing result. In this way, the difficulty of writing the log parsing plug-in is reduced, the grammatical details of the log parsing system are shielded, and the parsing personnel are focused on the analysis and understanding of the parsing requirements, thereby improving the efficiency of log parsing.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log parsing method, electronic device, storage medium, and program product. Background Art

[0002] Various computer devices and software systems generate extensive logs during operation. These logs record a rich variety of information, including the device's operating status, operational processes, exception information, and problem records. Analyzing these logs provides insights into the device's operational status. In the event of a device failure, logs can be used to retrace the sequence of operations performed on the device, identify the scenario in which the problem occurred, and locate the specific fault location, aiding in cause analysis. In the cybersecurity industry, security personnel rely on various device operation logs to identify potential security threats. When an attack occurs in a network environment, they also need to review device logs to conduct event tracing and forensic investigations. Clearly, logs have significant business value.

[0003] The log formats generated by these devices and software systems, developed by different manufacturers, vary widely. To facilitate subsequent retrieval and analysis, logs are typically parsed and normalized into a standardized format. Log content recorded by devices or systems in different fields can vary significantly, even including specialized vocabulary. To facilitate user viewing, after log parsing, fields are mapped and enriched as needed, presenting them in a more user-friendly manner and lowering the barrier to entry. Currently, much of this log parsing and field processing is performed by log parsing systems. A mature log parsing system typically handles logs from hundreds or thousands of devices and requires scalability. Therefore, log parsing is typically performed through log parsing plugins or rules, with field mapping performed through configuration files. These log parsing plugins and rules typically utilize a set of agreed-upon grammatical rules combined with regular expressions to normalize logs.

[0004] For a new device, a series of custom parsing files, including plug-ins and configuration files, are required. Currently, starting with log analysis, the development and configuration of parsing templates for each subtype of log are all manually performed. When parsing complex logs, the entire log parsing customization process is extremely time-consuming. Improving log parsing efficiency and reducing manual workload are urgent issues. Summary of the Invention

[0005] The purpose of some embodiments of the present application is to provide a log parsing method, electronic device, storage medium and program product. Through the technical solutions of the embodiments of the present application, by obtaining log data to be parsed; determining the target log type of the log data to be parsed based on the log data to be parsed; parsing the log data to be parsed based on a pre-stored log parsing plug-in corresponding to the target log type, and obtaining a parsing result corresponding to the log data to be parsed, wherein the log parsing plug-in uses log clustering to classify sample log data to obtain a classification result, and determines a log parsing specification corresponding to the classification result based on a large language model, and determines the log parsing specification based on the classification result and the log parsing requirement specifications determined by the log clustering in the embodiments of the present application. Combined with the analysis results of the log sample, a parsing plug-in and related configuration are automatically generated to generate a log parsing plug-in, and the log parsing plug-in is used to parse the log data to obtain a parsing result. In this way, the difficulty of writing the log parsing plug-in is reduced. The grammatical details of the log parsing system are shielded, allowing parsers to focus on analyzing and understanding the parsing requirements, thereby improving log parsing efficiency.

[0006] In a first aspect, some embodiments of the present application provide a log parsing method, including:

[0007] Get the log data to be parsed;

[0008] Determining a target log type corresponding to the log data to be parsed based on the log data to be parsed;

[0009] According to a pre-stored log parsing plug-in corresponding to the target log type, the log data to be parsed is parsed to obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and according to a large language model, a log parsing specification corresponding to the classification result is determined, and determined based on the classification result and the log parsing specification.

[0010] Some embodiments of this application achieve automatic log classification through log clustering, leveraging the document comprehension capabilities of a large language model to summarize log parsing requirements. Combined with the analysis results of log samples, this automatically generates a parsing plug-in and related configurations. This log parsing plug-in is then used to parse the log data to obtain parsing results. This reduces the difficulty of writing a log parsing plug-in. The grammatical details of the log parsing system are shielded, allowing parsers to focus on analyzing and understanding parsing requirements, thereby improving log parsing efficiency.

[0011] Optionally, the log parsing plug-in is obtained in the following manner:

[0012] Get sample log data;

[0013] Classifying the sample log data by log clustering to obtain a log type corresponding to the sample log data;

[0014] Determining a log parsing specification corresponding to the sample log data based on the large language module and the sample log data;

[0015] Determine a log parsing plug-in corresponding to the log type according to the log parsing specification and the log type.

[0016] Some embodiments of the present application achieve automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated based on the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, which reduces the difficulty of plug-in customization and improves the overall efficiency of parsing customization work.

[0017] Optionally, the adopting log clustering to classify the sample log data to obtain a log type corresponding to the sample log data includes:

[0018] Preprocessing the sample log data to obtain processed sample log data;

[0019] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0020] A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0021] Some embodiments of the present application implement automatic classification of logs through log clustering, and then automatically generate segmentation regularizations by analyzing logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularizations.

[0022] Optionally, the segmenting the processed sample log data to obtain a segmentation result corresponding to the processed sample log data includes:

[0023] If the processed sample log data is described in natural language, regular expression segmentation is performed to obtain a regular expression corresponding to the processed sample log data;

[0024] If the processed sample log data is in a preset format, a preset segmentation function is used to perform segmentation to obtain the segmentation result.

[0025] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve segmentation accuracy.

[0026] Optionally, determining a log parsing specification corresponding to the sample log data based on the large language module and the sample log data includes:

[0027] The large language model is used to parse the sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data. The log parsing requirement specification includes at least field names, field meanings, data types, value examples and log samples corresponding to different log types.

[0028] Some embodiments of the present application use the document understanding capabilities of a large language model to process log format description documents. At the same time, they summarize the log parsing requirements through specifications, and automatically generate parsing plug-ins and related configurations based on the analysis results of log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0029] Optionally, determining a log parsing plug-in corresponding to the log type according to the log parsing specification and the log type includes:

[0030] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

[0031] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0032] Optionally, the method further includes:

[0033] The log parsing plug-in is verified to obtain a verification result.

[0034] In a second aspect, some embodiments of the present application provide a log parsing device, including:

[0035] The acquisition module is used to obtain the log data to be parsed;

[0036] A determination module, configured to determine a target log type corresponding to the log data to be parsed based on the log data to be parsed;

[0037] A parsing module is used to parse the log data to be parsed according to a pre-stored log parsing plug-in corresponding to the target log type, and obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and determine the log parsing specification corresponding to the classification result according to the large language model, and determine it according to the classification result and the log parsing specification.

[0038] Some embodiments of this application achieve automatic log classification through log clustering, leveraging the document comprehension capabilities of a large language model to summarize log parsing requirements. Combined with the analysis results of log samples, this automatically generates a parsing plug-in and related configurations. This log parsing plug-in is then used to parse the log data to obtain parsing results. This reduces the difficulty of writing a log parsing plug-in. The grammatical details of the log parsing system are shielded, allowing parsers to focus on analyzing and understanding parsing requirements, thereby improving log parsing efficiency.

[0039] Optionally, the device further includes an establishing module, wherein the establishing module is configured to:

[0040] Get sample log data;

[0041] Classifying the sample log data by log clustering to obtain a log type corresponding to the sample log data;

[0042] Determining a log parsing specification corresponding to the sample log data based on the large language module and the sample log data;

[0043] Determine a log parsing plug-in corresponding to the log type according to the log parsing specification and the log type.

[0044] Some embodiments of the present application achieve automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated based on the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, which reduces the difficulty of plug-in customization and improves the overall efficiency of parsing customization work.

[0045] Optionally, the establishing module is used to:

[0046] Preprocessing the sample log data to obtain processed sample log data;

[0047] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0048] A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0049] Some embodiments of the present application implement automatic classification of logs through log clustering, and then automatically generate segmentation regularizations by analyzing logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularizations.

[0050] Optionally, the establishment module is used to

[0051] If the processed sample log data is described in natural language, regular expression segmentation is performed to obtain a regular expression corresponding to the processed sample log data;

[0052] If the processed sample log data is in a preset format, a preset segmentation function is used to perform segmentation to obtain the segmentation result.

[0053] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve segmentation accuracy.

[0054] Optionally, the establishing module is used to:

[0055] The large language model is used to parse the sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data. The log parsing requirement specification includes at least field names, field meanings, data types, value examples and log samples corresponding to different log types.

[0056] Some embodiments of the present application use the document understanding capabilities of a large language model to summarize log parsing requirement specifications, and automatically generate parsing plug-ins and related configurations based on the analysis results of log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0057] Optionally, the establishing module is used to:

[0058] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

[0059] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0060] Optionally, the establishing module is used to:

[0061] The log parsing plug-in is verified to obtain a verification result.

[0062] In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the log parsing method as described in any embodiment of the first aspect may be implemented.

[0063] In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the log parsing method as described in any embodiment of the first aspect.

[0064] In a fifth aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it can implement the log parsing method as described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following is a brief introduction to the drawings required for use in some embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A flow chart of a log parsing method provided in an embodiment of the present application;

[0067] Figure 2 A flow chart of another log parsing method provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the structure of a log parsing device provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0071] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0072] Various computer devices and software systems generate extensive logs during operation. These logs record a rich variety of information, including the device's operating status, operational processes, exception information, and problem records. Analyzing these logs provides insights into the device's operational status. In the event of a device failure, logs can be used to retrace the sequence of operations performed on the device, identify the scenario in which the problem occurred, and locate the specific fault location, aiding in cause analysis. In the cybersecurity industry, security personnel rely on various device operation logs to identify potential security threats. When an attack occurs in a network environment, they also need to review device logs to conduct event tracing and forensic investigations. Clearly, logs have significant business value.

[0073] The log formats generated by these devices and software systems, developed by different manufacturers, vary widely. To facilitate subsequent retrieval and analysis, logs are typically parsed and normalized into a standardized format. Logs recorded by devices or systems in different fields can also vary significantly, sometimes even containing specialized vocabulary. To facilitate user viewing, after log parsing, fields are mapped and enriched to present them in a more user-friendly manner, lowering the barrier to entry. Currently, much of this log parsing and field processing is performed by log parsing systems. A mature log parsing system typically handles logs from hundreds or thousands of devices and requires scalability. Therefore, log parsing is typically performed using log parsing plugins or rules, with field mapping performed via configuration files. These log parsing plugins and rules typically utilize a set of agreed-upon grammar rules combined with regular expressions to normalize logs.

[0074] For a new device, a series of plug-ins and configuration files such as parsing files need to be customized. Currently, starting from the analysis of logs, the development and configuration of parsing templates for each sub-type of logs are all done manually. When faced with the parsing of complex logs, the entire process of log parsing customization is very time-consuming. How to improve the efficiency of log parsing and reduce the workload of manual work is an urgent problem to be solved. In view of this, some embodiments of the present application provide a log parsing method, which includes obtaining log data to be parsed; determining the target log type of the log data to be parsed based on the log data to be parsed; parsing the log data to be parsed based on a pre-stored log parsing plug-in corresponding to the target log type to obtain a parsing result of the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify and process the sample log data to obtain a classification result, and determines the log parsing specification corresponding to the classification result based on the large language model, and determines the log parsing specification based on the classification result and the log parsing specification. In the embodiment of the present application, the automatic classification of logs is achieved through log clustering, and the document understanding ability of the large language model is used to summarize the log parsing requirement specifications. Based on the analysis results of the log sample, a parsing plug-in and related configuration are automatically generated. This log parsing plug-in is then used to parse the log data to obtain the parsing results. This reduces the difficulty of writing log parsing plug-ins. The grammatical details of the log parsing system are shielded, allowing parsers to focus on analyzing and understanding the parsing requirements, thereby improving log parsing efficiency.

[0075] like Figure 1 As shown, an embodiment of the present application provides a log parsing method, the method comprising:

[0076] S101, obtaining log data to be parsed;

[0077] S102: Determine the target log type of the log data to be parsed based on the log data to be parsed;

[0078] Specifically, the terminal device obtains the log data to be parsed, which can be sent by the user's device. The device (such as a firewall) is configured to send logs, specify the receiving server IP and port, and then send the logs to the log audit server. Alternatively, the logs can be downloaded from the network. The log data to be parsed is parsed to determine the target log type of the log data to be parsed, for example, the log type can be natural language.

[0079] S103. Parse the log data to be parsed according to the pre-stored log parsing plug-in corresponding to the target log type to obtain a parsing result corresponding to the log data to be parsed, wherein the log parsing plug-in adopts a log clustering method to classify the sample log data to obtain a classification result, and determine the log parsing specification corresponding to the classification result according to the large language model, and determine it according to the classification result and the log parsing specification.

[0080] Specifically, log parsing is to extract meaningful information fragments from text described in natural language or following a certain format, name each information fragment, and form an information object list in the form of key:value, which is displayed to the user. Here, the information object list in the form of key:value is called a Token object list, where key represents the name of the Token and value represents the value of the Token.

[0081] The embodiment of the present application classifies the original logs through clustering, analyzes the log fields by interpreting the log format document, generates a log parsing requirement specification, and guides the program to generate a parsing plug-in that meets the grammatical requirements of the log parsing system based on the requirement specification. Finally, the classified original logs and the log parsing requirement specification are used to verify the actual parsing effect of the plug-in. The above method solves the problem of complex process and low efficiency when manually customizing new types of parsing work.

[0082] Some embodiments of this application achieve automatic log classification through log clustering, leveraging the document comprehension capabilities of a large language model to summarize log parsing requirements. Combined with the analysis results of log samples, this automatically generates a parsing plug-in and related configurations. This log parsing plug-in is then used to parse the log data to obtain parsing results. This reduces the difficulty of writing a log parsing plug-in. The grammatical details of the log parsing system are shielded, allowing parsers to focus on analyzing and understanding parsing requirements, thereby improving log parsing efficiency.

[0083] Another embodiment of the present application further supplements the log parsing method provided in the above embodiment.

[0084] like Figure 2 As shown, the embodiment of the present application provides an automatic log parsing method, which includes the following four steps:

[0085] Step (1) clustering of original logs;

[0086] Step (2) analyzing the log format description document and generating log parsing requirement specifications;

[0087] Step (3) Generate a parsing plug-in based on the log parsing requirements specification;

[0088] Step (4) parses the verification of the plug-in.

[0089] Optionally, the log parsing plugin is obtained as follows:

[0090] Get sample log data;

[0091] Use log clustering to classify the sample log data and obtain the log type corresponding to the sample log data;

[0092] Determine the log parsing specification corresponding to the sample log data based on the large language module and the sample log data;

[0093] Determine the log parsing plug-in that corresponds to the log type based on the log parsing specification and log type.

[0094] Some embodiments of the present application achieve automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated based on the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, which reduces the difficulty of plug-in customization and improves the overall efficiency of parsing customization work.

[0095] Optionally, log clustering is used to classify the sample log data to obtain log types corresponding to the sample log data, including:

[0096] Preprocessing the sample log data to obtain processed sample log data;

[0097] Specifically, the "preprocessing" steps described in step (1) of log clustering are: replacing common dynamic fields in the log (such as syslog priority, time, IP, port, MAC, etc.) with specific identifiers through regular expression matching. This facilitates subsequent token segmentation and reduces the interference of these dynamic data during clustering, resulting in more accurate clustering results. Some common regular expressions are pre-set, and users are also supported to specify new patterns and content.

[0098] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0099] A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0100] Specifically, the specific steps of the "log clustering" in step (1) log clustering are: there are three clustering methods: clustering according to the string consisting of some or all field names after segmentation; clustering using one or some field values after segmentation, similar to the group by function in the database; raw log clustering, at this time, there is no need to rely on field segmentation, and the pre-processed raw logs are directly clustered.

[0101] Multiple clustering methods are provided, including DBSCAN, progressive clustering with feedback (K-means, Gaussian mixture model clustering, etc.), and field value classification. Users can select the method that works best. Log clustering is intended to group logs of the same type into a single cluster. Logs of the same type typically share the same fields and formats, and can be displayed using a single field set.

[0102] Segmenting logs first generally helps with log clustering, but this isn't always necessary. For logs where the majority of text is in key:value format, segmentation is generally recommended first, followed by clustering based on some or all fields. For logs where the majority of text is in natural language, clustering can be performed first, followed by tokenization and parsing for each log type.

[0103] After log clustering is completed, common dynamic content such as syslog priority, time, IP, MAC, etc. replaced during data preprocessing needs to be replaced back to avoid affecting subsequent analysis.

[0104] Some embodiments of the present application implement automatic classification of logs through log clustering, and then automatically generate segmentation regularizations by analyzing logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularizations.

[0105] Optionally, the processed sample log data is segmented to obtain segmentation results corresponding to the processed sample log data, including:

[0106] If the processed sample log data is described in natural language, regular expression segmentation is used to obtain a regular expression corresponding to the processed sample log data;

[0107] If the processed sample log data is in a preset format, the preset segmentation function is used to segment the data to obtain a segmentation result.

[0108] Specifically, the "token splitting" step in log clustering involves selecting either regular or function-based splitting based on log characteristics. Regular splitting is typically used when logs are written in natural language. Function-based splitting can be used when logs are in key:value format (such as WELF).

[0109] If you select regular segmentation, you need to generate a regular expression to extract tokens.

[0110] When selecting function splitting, the system preset splitting function is used to split the token.

[0111] Regular expressions can complete token segmentation for all logs, but when the logs are suitable for segmentation functions, it is more concise to use segmentation functions.

[0112] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve segmentation accuracy.

[0113] Optionally, determining a log parsing specification corresponding to the sample log data based on the large language module and the sample log data includes:

[0114] A large language model is used to parse sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data. The log parsing requirement specification includes at least the field name, field meaning, data type, value example and log sample corresponding to different log types.

[0115] Specifically, the specific steps of "document understanding" described in step (2) generating the log parsing specification are: if the log format document content is standardized and comprehensive, read the document to understand the business significance and field meaning of each type of log; if there is a lack of log format document, use the experience and professional knowledge of the parser to disassemble the business significance and field meaning of various logs.

[0116] The specific steps of "generating log parsing requirement specifications" described in step (2) generating log parsing specifications are as follows: based on the understanding of the log format document and the log, generate the log parsing requirement specifications. The field names in the original log are usually in English, or the field names in the original log are not suitable for direct display. In this case, the field names need to be renamed to field names suitable for display. The correspondence between the original fields and the parsed fields of each log needs to be sorted out. The parsing specification includes the field names, field meanings, data types, value examples, etc. that need to be parsed for all types of logs. It also includes log samples for each log type.

[0117] The specific steps for "determining the log type of each cluster" described in the log parsing specification are as follows:

[0118] After log clustering, similar logs are grouped together. For subsequent analysis, each clustered log needs to be assigned a log type.

[0119] For natural language logs, the segmentation regular expressions corresponding to each cluster after clustering are used to match all types of log samples included in the log parsing specification. The type of the log sample that successfully matches is designated as the log type of the current cluster. In rare cases, if a log sample matches multiple segmentation regular expressions at the same time, the cluster corresponding to the segmentation regular expression with the most token objects after segmentation is designated as the log type corresponding to the log sample.

[0120] For logs in the key:value format (such as WELF), first, the sample logs are segmented using the same function segmentation method; then, the string consisting of the token name after the logs in each cluster are segmented is compared with the string consisting of the token name after the log samples are segmented, and the log category corresponding to the sample with the highest similarity is found and designated as the log category of the current cluster.

[0121] Some embodiments of the present application use the document understanding capabilities of a large language model to summarize log parsing requirement specifications, and automatically generate parsing plug-ins and related configurations based on the analysis results of log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0122] Optionally, based on the log parsing specification and the log type, a log parsing plug-in corresponding to the log type is determined, including:

[0123] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a preset log parsing plug-in syntax is used to generate a log parsing plug-in corresponding to the log type.

[0124] Specifically, the specific steps of "generating a parsing plug-in" in step (3) generating a log parsing plug-in are:

[0125] Based on the clustering results and token segmentation results, combined with the requirements for each log parsing specification, and according to the parsing plug-in syntax supported by the log parsing system, an initial parsing plug-in is generated.

[0126] The specific steps for generating the "field adjustment" in the log parsing plug-in in step (3) are as follows: First, based on the data type of each field in the log parsing requirement specification, the normalized fields supported by the log parsing system are automatically specified. Then, based on the examples of field types and field values, the content of the specific type of field is adjusted.

[0127] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0128] The specific steps of "sending classified logs to the log parsing system" in step (4) generating the log parsing plug-in are as follows:

[0129] This step verifies that the generated parsing file is correct. Before sending logs, the log parsing system must load the parsing plugin and related configurations. A log source must also be added to the log parsing system to receive logs. These are the preparatory steps for log verification.

[0130] Call the log sending tool to send a type of log to the log parsing system.

[0131] The specific steps of "Log Parsing Verification" described in step (4) generating the log parsing plug-in are:

[0132] Obtain the parsing results of the current type of log through the interface, and verify the fields and field values after parsing the current type of log by comparing them with the contents of the log parsing requirement specification to check whether they meet the parsing specifications.

[0133] Repeat the two steps of "sending classified logs to the log parsing system" and "log parsing verification" in step (4) until all types of logs are verified.

[0134] The embodiments of the present application can be applied to a log audit system to quickly support the parsing of new log types.

[0135] The embodiment of this application is based on a log audit product, and describes the use of an automatic log parsing method proposed in the embodiment of this application. When the log audit product supports new log types, it simplifies the development process of the parsing plug-in through intelligent analysis and understanding of log samples and log format documents, thereby improving the efficiency of new log customization.

[0136] This example uses the log parsing process for a vendor's load balancing product as an example. This product contains five types of logs, and this example uses service logs and NAT logs as examples to illustrate the process of generating a parsing plug-in.

[0137] A sample service log is:

[0138] <134> Sep 10 21:07:15zdg:RS_DETECT: Virtual service [test_l3] is abnormal, the default node pool it references is faulty

[0139] <134> Sep 10 21:07:15zdg:RS_DETECT: Virtual service [http] is abnormal, the default node pool it references is faulty

[0140] A sample NAT log is as follows:

[0141] <134> Dec 14 11:35:00src@localhost:[2020-12-14_11-34-59]NAT_LOG_DEL_ENTRY[tcp]10.10 1.1.81:58449(60.164.184.44:58449)->180.163.249.41:80(180.163.249.41:80)

[0142] <134> Dec 14 11:35:00src@localhost:[2020-12-14_11-34-59]NAT_LOG_ADD_ENTRY[tcp]223.1 04.56.5:52163(223.104.56.5:52163)->60.164.184.43:5100(10.99.100.1:5100)

[0143] The above logs are numbered log0, log1, log2, and log3 respectively for the convenience of subsequent description.

[0144] The log format document describes service logs and NAT logs as follows:

[0145] 3. Service Log Format

[0146] The service log syslog format is as follows:

[0147] {System time host name: module name: log details}

[0148] example:

[0149] <134> Sep 10 21:07:15zdg:RS_DETECT: Virtual service [test_l3] is abnormal, the default node pool it references is faulty

[0150] 4NAT log format

[0151] The NAT log syslog format is as follows:

[0152] {System time host name: [TIME] ACTION [PROTOCOL] SrcIP:SrcPort (NatSrcIP:NatSrcPort) -> DstIp:DstPort (NatDstIp:NatDstPort)}

[0153] like:

[0154] <134> Sep 10 21:22:02zdg:[2018-09-10_21-22-02]NAT_LOG_DEL_ENTRY[icmp]172.16.1.136:0(200.200.83.7:0)->172.16.1.7:0(200.200.80.136:0)

[0155] 4.1NAT Log Field Introduction

[0156] 1>TIME: Timestamp

[0157] 2>ACTION: divided into create and delete, the value is NAT_LOG_DEL_ENTRY or NAT_LOG_ADD_ENTRY

[0158] 3>PROTOCOL: protocol type, the value range is tcp|icmp|icmpv6|udp|other

[0159] 4>SrcIP: Source IP

[0160] 5>SrcPort: Source port

[0161] 6>NatSrcIP: Source IP after NAT

[0162] 7>NatSrcPort: Source port after NAT

[0163] 8>DstIp: destination IP

[0164] 9>DstPort: destination port

[0165] 10>NatDstIP: Destination IP after NAT

[0166] 11>NatDstPort: destination port after NAT;

[0167] In this example, the log samples are natural language descriptions. We can first cluster the logs and then tokenize each cluster. According to the method described in detail in this invention, the specific process is as follows:

[0168] 1. Log Clustering

[0169] 1.1 Preprocessing

[0170] Replace the dynamic content in the original log, and the replaced log is

[0171] #pri##TIME1#zdg:RS_DETECT: The virtual service [test_l3] is abnormal, and the default node pool it references is faulty

[0172] #pri##TIME1#zdg:RS_DETECT: Virtual service [http] is abnormal, the default node pool it references is faulty

[0173] #pri##TIME1#src@localhost:[#TIME2#]NAT_LOG_DEL_ENTRY[tcp]#IPV4#:#PORT#(#IPV4#:#PORT#)->#IPV4#:#PORT#(#IPV4#:#PORT#)

[0174] #pri##TIME1#src@localhost:[#TIME2#]NAT_LOG_ADD_ENTRY[udp]#IPV4#:#PORT#(#IPV4#:#PORT#)->#IPV4#:#PORT#(#IPV4#:#PORT#)

[0175] 1.2 Log Clustering

[0176] In this example, the DBSCAN clustering algorithm (full name: Density-Based Spatial Clustering of Applications with Noise, a relatively representative density-based clustering algorithm) is used to cluster the preprocessed logs. First, the original log is segmented using the Jieba word segmentation algorithm; then, the segmented data is vectorized using the TfidfVectorizer (text feature extraction); finally, the log samples are clustered using DBSCAN (parameters such as eps = 0.5, min_samples = 1). The clustering result is: [0 0 1 1], with two clusters: log0 and log1 in one cluster, and log2 and log3 in another cluster. The result can be expressed as:

[0177] Cluster0:[log0,log1]

[0178] Cluster1:[log2,log3]

[0179] If the clustering results are unsatisfactory, you can improve them by adjusting the clustering parameters. If satisfactory results are still not achieved after adjusting the parameters, you can use other clustering algorithms, such as the progressive clustering method with feedback based on K-means and Gaussian mixture model clustering algorithms, to achieve relatively satisfactory clustering results.

[0180] 1.3 Token Splitting

[0181] For each cluster of logs in the above cluster, generate a regular expression for token segmentation and segment the logs. This example uses the first cluster of logs as an example.

[0182] The word segmentation results of log0 and log1 in the previous step are:

[0183] The word segmentation result of Log0 is: ['#','pri','#',”,'#','TIME1','#',”,'zdg',”,':',”,'RS','_','DETECT',':',”,'virtual','service',”,'[','test','_','l3',']',”,'abnormal',',',','its','reference','of','default','node','pool','fault']

[0184] The word segmentation result of Log1 is: ['#','pri','#',”,'#','TIME1','#',”,'zdg',”,':',”,'RS','_','DETECT',':',”,'virtual','service',”,'[','http',']',”,'abnormal',',',','its','reference','of','default','node','pool','fault']

[0185] For these two sequences, find the longest common subsequence of the two sequences:

[0186] [['#','pri','#',”,'#','TIME1','#',”,'zdg',”,':',”,'RS','_','DETECT',':',”,'virtual','service',”,'['],[']',”,'exception',',','its','reference','of','default','node','pool','failure']].

[0187] First, concatenate each common subsequence into a string. If there are special characters (regular expression-sensitive special characters) in each subsequence string, escape them. Then, use wildcards to concatenate each subsequence to generate a regular expression:

[0188] #pri##TIME1#zdg:RS_DETECT: Virtual service \[([\w\W]+)\] is abnormal, and the default node pool it references is faulty

[0189] Then, use the generated regular expression to capture data from the preprocessed logs. For example, the regular expression above captures the following content in two logs: [["test_l3","http"]]. In this example, there is only one capture group (also in this example).

[0190] Next, analyze the components of ["test_l3","http"] and replace the original wildcard ([\w\W]+) with ([\w_]+).

[0191] Finally, replace the dynamic data identifiers replaced in the preprocessing phase with the corresponding regular expression; the regular expression in this case is: "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+)zdg:RS_DETECT: Virtual service \[([\w_]+)\] is abnormal, and the default node pool it references is faulty." Because the sample size is small and the host name and module name are identical, the automatically generated regular expression does not extract the host name and module name fields. As the number of log samples increases and these fields begin to vary in value, the generated regular expression automatically extracts these fields.

[0192] This allows users to adjust and refine the automatically generated segmentation regular expression. In this example, if you are also interested in the host name, module name, and specific log content, you can adjust the regular expression to: "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+)(\w+):([\w-]+):(Virtual service \[([\w\d]+)\] is abnormal, and the default node pool it references is faulty)". Using this regular expression to segment the original log, the default token name is represented by the group sequence number, resulting in an array containing 6 token objects, as shown in Table 1:

[0193] Table 1

[0194]

[0195]

[0196] The second cluster of logs is processed in the same way, and the automatic segmentation regularization result is:

[0197] "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+) ([\w@]+) :\[(\d{4}-\d+-\d+_\d+-\d+-\d+)\] NAT_LOG_\w+_ENTRY\[(\w+)\](\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\) ->(\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\)”.

[0198] Manually add the extraction of the ACTION field and manually adjust the regular expression to become:

[0199] "<(\d+)>(\w{3}\s\d+\s\d+:\d+:\d+) ([\w@]+) :\[(\d{4}-\d+-\d+_\d+-\d+-\d+)\] (NAT_LOG_\w+_ENTRY)\[(\w+)\](\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\) ->(\d+\.\d+\.\d+\.\d+):(\d+)\((\d+\.\d+\.\d+\.\d+):(\d+)\)”. As shown in Table 2:

[0200] Table 2

[0201]

[0202]

[0203] 2. Generate log parsing specifications

[0204] In this example, the log format document is standardized and comprehensive. This step involves reading the document to understand the business significance and field meanings of various log types. The goal of reading the document is to generate log parsing requirements specifications. Document understanding and generating log parsing requirements specifications can be accomplished manually or with the help of a large language model.

[0205] The manual generation process is to organize log parsing requirement specifications and various types of log samples from documents in the prescribed format.

[0206] The large model is used to understand the document and design prompts. The document is then fed into the large model through an API call, and the model outputs the log parsing requirements specification as required. The following table shows the portion of the log parsing requirements specification generated using the large model that is relevant to this example, as shown in Table 3.

[0207] Table 3

[0208]

[0209]

[0210] Table 4 shows the log types extracted by the large model and part of the log sample table.

[0211] Table 4

[0212]

[0213] 2.3 Determine the type of each cluster log

[0214] Since the logs in this example are strings described in natural language, we use the segmentation regularization of each cluster to segment each sample log. The results are shown in Table 5:

[0215] Table 5

[0216]

[0217] In this example, the fields after the original log is split are named according to the matching log type using the field names in the log parsing specification. The fields in the log parsing specification with value examples can be used to process their tokens.

[0218] After processing, the segmentation results of Cluster0 are as shown in Table 6:

[0219] Table 6

[0220]

[0221] After processing, the segmentation results of Cluster 1 are as shown in Table 7:

[0222] Table 7

[0223]

[0224] Priority and serviceName are fields not included in the specification and are manually completed. Renaming of other token names can be automatically completed in the specification.

[0225] 3. Generate log parsing plug-in

[0226] 3.1 Generate parsing plug-in

[0227] Based on the segmentation rules and token name duplication of each cluster log generated above, and combined with the types of each field in the log parsing specification, a parsing plug-in can be generated according to the parsing plug-in syntax rules supported by the current log parsing system. As an example, the key snippets of the parsing plug-in related to this example are shown below:

[0228] <? xml version="1.0" encoding="UTF-8"? >

[0229] …

[0230] <exp……reg="<;(\d+)> ;(\w{3}\s\d+\s\d+:\d+:\d+)(\w+):([\w-]+):(Virtual service\[([\w\d]+)\] is abnormal, and the default node pool it references is faulty)"…>

[0231] <!--Service Log, CATEGORY='Service Log'-->

[0232] <ts>

[0233] <t name="1"a="priority" / >

[0234] <t name="2"a="systemTime" / >

[0235] <t name="3"a="hostName" / >

[0236] <t name="4"a="module" / >

[0237] <t name="5"a="message" / >

[0238] <t name="6"a="serviceName" / >

[0239] < / ts>

[0240] <e name="severity"value="_GetSeverity(priority)" / >

[0241] <e name="year"value="_Time($'yyyy')" / >

[0242] <e name="systemTime"value="_Format(year,$”,systemTime)" / >

[0243] <e name="systemTime"value="_FormatTime($'yyyy MMM ddHH:mm:ss',systemTime)" / >

[0244] <m name="SEVERITY" value="severity" / ><!--Log level-->

[0245] <m name="SYSTEM_TIME" value="systemTime" / ><!--System time-->

[0246] <m name="HOST_NAME" value="hostName" / ><!--Host name-->

[0247] <m name="MODULE" value="module" / ><!--Module name-->

[0248] <m name="MESSAGE" value="message" / ><!--Specific log content-->

[0249] <m name="SERVICE_NAME" value="serviceNmae" / ><!--Virtual service-->

[0250] <m name="CATEGORY" value="$'Service log'" / ><!--Log type-->

[0251]

[0252]

[0253] …

[0254] …

[0255] 3.2 Field Adjustment

[0256] Due to the field requirements of the log audit system, some fields in the previous step require special processing to display properly in the log audit system. For example, the time field is formatted, the log level field is processed, and the field naming of the supplementary token is supplemented. Some fields do not need to be parsed (for example, if there are two time-related fields in the NAT log with the same content, only one can be parsed) and can be annotated.

[0257] There are two time fields here, both of which are captured by regular expressions during preprocessing. For supported time modes, corresponding time formatting functions are predefined for processing. The corresponding codes can be directly generated in the parsing plug-in.

[0258] The red section of the XML snippet in the previous step represents the adjustments made. The annotations for the "Virtual Service" field name and timestamp field are manually processed, while the formatting of the time field can be done programmatically.

[0259] 4. Parsing verification

[0260] Before starting verification, load the previously generated parsing plug-in and related configurations in the log audit system, and make related configurations for receiving logs.

[0261] 4.1 Sending classified logs to the log parsing system

[0262] Call the simulated log sending tool to send a certain type of log.

[0263] 4.2 Log parsing and verification

[0264] Use the log audit system interface to obtain the parsing results of the current type of log.

[0265] Comparing the parsing results obtained from the interface with the fields specified in the log parsing requirements specification can verify whether the parsing results are correct. Key checks include: 1. Whether all fields required by the log parsing requirements specification have been parsed; 2. Whether the required fields have been parsed to the correct values.

[0266] Repeat steps 4.1-4.2 to verify all types of log parsing results.

[0267] The embodiments of the present application solve the problem of low efficiency when the existing log parsing system completely relies on manual customization of the parsing of new log types. The classification of logs is completed through clustering; with the help of the document analysis and summarization capabilities of the large model, the analysis of log fields is completed by interpreting the log format document, and the parsing requirement specifications for each type of log are generated. The parsing requirement specifications in a specific format make it possible to automatically generate parsing plug-ins. An automatic log parsing method is proposed, which automatically generates parsing plug-ins that meet the grammatical requirements of the log parsing system based on the log parsing requirement specifications. The workload of manually writing and debugging plug-ins is reduced, and the complexity of manual parsing is reduced. The classified original logs and the requirement specifications for the parsing effect of each type of log are used to perform automated verification tests on the loaded plug-ins to ensure that the parsing work of the parsing plug-in meets the requirement specifications.

[0268] It should be noted that each implementable method in this embodiment can be implemented separately, or can be implemented in combination in any combination without conflict, and this application does not limit it.

[0269] Another embodiment of the present application provides a log parsing device for executing the log parsing method provided in the above embodiment.

[0270] like Figure 3 FIG. 3 is a schematic diagram of the structure of a log parsing device provided in an embodiment of the present application. The log parsing device includes an acquisition module 301, a determination module 302, and a parsing module 303, wherein:

[0271] The acquisition module 301 is used to obtain the log data to be parsed;

[0272] The determination module 302 is used to determine the target log type of the log data to be parsed based on the log data to be parsed;

[0273] The parsing module 303 is used to parse the log data to be parsed according to the pre-stored log parsing plug-in corresponding to the target log type, and obtain the parsing result of the log data to be parsed, wherein the log parsing plug-in adopts the log clustering method to classify the sample log data to obtain the classification result, and determines the log parsing specification corresponding to the classification result according to the large language model, and determines it according to the classification result and the log parsing specification.

[0274] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0275] Some embodiments of this application achieve automatic log classification through log clustering, leveraging the document comprehension capabilities of a large language model to summarize log parsing requirements. Combined with the analysis results of log samples, this automatically generates a parsing plug-in and related configurations. This log parsing plug-in is then used to parse the log data to obtain parsing results. This reduces the difficulty of writing a log parsing plug-in. The grammatical details of the log parsing system are shielded, allowing parsers to focus on analyzing and understanding parsing requirements, thereby improving log parsing efficiency.

[0276] Another embodiment of the present application further supplements the log parsing device provided in the above embodiment.

[0277] Optionally, the device further includes a building module, the building module being configured to:

[0278] Get sample log data;

[0279] Use log clustering to classify the sample log data and obtain the log type corresponding to the sample log data;

[0280] Determine the log parsing specification corresponding to the sample log data based on the large language module and the sample log data;

[0281] Determine the log parsing plug-in that corresponds to the log type based on the log parsing specification and log type.

[0282] Some embodiments of the present application achieve automatic classification of logs through log clustering, and then automatically generate segmentation regularization through analysis of logs in the same cluster, reducing the workload of manual analysis and extraction of regularization. With the help of the document understanding ability of the large language model, the log parsing requirement specifications are summarized, and the parsing plug-in and related configurations are automatically generated based on the analysis results of the log samples; the syntax and configuration details of the log parsing plug-in are hidden, which reduces the difficulty of plug-in customization and improves the overall efficiency of parsing customization work.

[0283] Optionally, build modules for:

[0284] Preprocessing the sample log data to obtain processed sample log data;

[0285] Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data;

[0286] A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

[0287] Some embodiments of the present application implement automatic classification of logs through log clustering, and then automatically generate segmentation regularizations by analyzing logs in the same cluster, thereby reducing the workload of manual analysis and extraction of regularizations.

[0288] Optionally, create a module for

[0289] If the processed sample log data is described in natural language, regular expression segmentation is used to obtain a regular expression corresponding to the processed sample log data;

[0290] If the processed sample log data is in a preset format, the preset segmentation function is used to segment the data to obtain a segmentation result.

[0291] Some embodiments of the present application perform corresponding segmentation according to different types of sample log data to improve segmentation accuracy.

[0292] Optionally, build modules for:

[0293] A large language model is used to parse sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data. The log parsing requirement specification includes at least the field name, field meaning, data type, value example and log sample corresponding to different log types.

[0294] Some embodiments of the present application use the document understanding capabilities of a large language model to summarize log parsing requirement specifications, and automatically generate parsing plug-ins and related configurations based on the analysis results of log samples; hide the syntax and configuration details of the log parsing plug-in, reduce the difficulty of plug-in customization, and improve the overall efficiency of parsing customization work.

[0295] Optionally, build modules for:

[0296] According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a preset log parsing plug-in syntax is used to generate a log parsing plug-in corresponding to the log type.

[0297] Some embodiments of the present application reduce the difficulty of writing log parsing plug-ins, shield the grammatical details of the log parsing system, allow parsers to focus on analyzing and understanding parsing requirements, and improve the efficiency of writing log parsing plug-ins.

[0298] Optionally, the establishing module is used to:

[0299] The log parsing plug-in is verified to obtain a verification result.

[0300] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0301] It should be noted that each implementable method in this embodiment can be implemented separately, or can be implemented in combination in any combination without conflict, and this application does not limit it.

[0302] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the operation of the method corresponding to any embodiment of the log parsing method provided in the above embodiments can be implemented.

[0303] An embodiment of the present application further provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any embodiment of the log parsing method provided in the above embodiments.

[0304] like Figure 4 As shown, some embodiments of the present application provide an electronic device 400, which includes: a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420, wherein the processor 420 reads the program from the memory 410 through the bus 430 and executes the program to implement a method of any embodiment as included in the above-mentioned log parsing method.

[0305] Processor 420 can process digital signals and can include various computing architectures, such as a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, processor 420 can be a microprocessor.

[0306] The memory 410 can be used to store instructions executed by the processor 420 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all functions of one or more modules described in the embodiments of this application. The processor 420 of the embodiment of the present disclosure can be used to execute the instructions in the memory 410 to implement the method shown above. The memory 410 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory known to those skilled in the art.

[0307] The above are merely examples of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application. It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0308] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0309] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A log parsing method, characterized in that: The method comprises: Get the log data to be parsed; Determining a target log type corresponding to the log data to be parsed based on the log data to be parsed; Parsing the log data to be parsed according to a pre-stored log parsing plug-in corresponding to the target log type to obtain a parsing result corresponding to the log data to be parsed, wherein the log parsing plug-in classifies the sample log data using a log clustering method to obtain a classification result, determining a log parsing specification corresponding to the classification result according to a large language model, and determining the parsing specification based on the classification result and the log parsing specification; Wherein: the log parsing plug-in is obtained in the following way: Get sample log data; Classifying the sample log data by log clustering to obtain a log type corresponding to the sample log data; Determine the log parsing requirement specification corresponding to the sample log data based on the large language module and the sample log data; the log parsing requirement specification includes at least the field names, field meanings, data types, value examples, and log samples corresponding to different log types. The log parsing requirement specification is used to rename the field names into suitable display names when the field names in the original log are all in English or the field names in the original log are not suitable for direct display, and to sort out the correspondence between the original fields and the parsed fields of each log; Determining a log parsing plug-in corresponding to the log type according to the log parsing specification and the log type includes: According to the log type, the segmentation result corresponding to the log type and the log parsing specification corresponding to the log type, a log parsing plug-in corresponding to the log type is generated by using a preset log parsing plug-in syntax.

2. The log parsing method according to claim 1, characterized in that: The method of using log clustering to classify the sample log data to obtain a log type corresponding to the sample log data includes: Preprocessing the sample log data to obtain processed sample log data; Segmenting the processed sample log data to obtain segmentation results corresponding to the processed sample log data; A preset clustering algorithm is used to perform string clustering on the segmentation results or the processed sample log data to obtain a log type corresponding to the sample log data, wherein the preset clustering algorithm includes at least one or more of DBSCAN, progressive clustering with feedback, or field value classification.

3. The log parsing method according to claim 2, characterized in that: The segmenting of the processed sample log data to obtain a segmentation result corresponding to the processed sample log data includes: If the processed sample log data is described in natural language, regular expression segmentation is performed to obtain a regular expression corresponding to the processed sample log data; If the processed sample log data is in a preset format, a preset segmentation function is used to perform segmentation to obtain the segmentation result.

4. The log parsing method according to claim 1, wherein: The step of determining a log parsing specification corresponding to the sample log data based on the large language module and the sample log data includes: The large language model is used to parse sample log data corresponding to the log type to obtain a log parsing requirement specification corresponding to the sample log data.

5. The log parsing method according to claim 1, wherein: The method further comprises: The log parsing plug-in is verified to obtain a verification result.

6. An electronic device, characterized in that: The system comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the log parsing method according to any one of claims 1 to 5 when executing the program.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the log parsing method according to any one of claims 1 to 5 can be implemented.

8. A computer program product, comprising a computer program, wherein: When the computer program is executed by a processor, it can implement the log parsing method described in any one of claims 1 to 5.

Images