A log analysis method, device and computing device cluster

Abstract

A log analysis method comprises: receiving a log to be parsed selected by a user and a target field, wherein the log to be parsed comprises at least one original log, and the target field is a field to be extracted from the original log; generating at least one parsing code based on the log to be parsed and the target field, wherein the language of the parsing code is selected by the user, and one parsing code is related to at least one original log; and running the first parsing code to process a first original log related to the first parsing code to obtain a structured log of the first original log, wherein the first parsing code is any one of the at least one parsing code. In this way, by converting the log analysis task into a code generation problem instead of directly generating a log template, efficient and real-time structured and standardized processing of the log is realized, and manual generation of parsing regular statements based on the log template is not required, thereby improving the efficiency and accuracy of log analysis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a log parsing method, apparatus, and computing device cluster. Background Technology

[0002] In today's digital age, security logs have become a critical asset for cybersecurity and operations management. However, raw logs are often unstructured or semi-structured, making them difficult to use directly for analysis and modeling. Only by parsing these logs into standardized structured logs can they serve as effective awareness signals for discovering potential threats, ensuring cybersecurity, and supporting various business needs. However, log standardization is a time-consuming and labor-intensive process, facing numerous challenges. For example, some companies need to parse massive (petaflop-level), thousands of formats, and frequently changing raw logs into multiple types of security standard logs annually. This process requires security operations personnel to deeply understand the log content and write complex parsing code, consuming significant manpower and time, leading to a vacuum in threat awareness and severely impacting the efficiency and response speed of security operations. The importance of log standardization is not limited to the security field; it is equally crucial for other industries. For example, in IT operations and maintenance management, operations logs need to be based on parsed, standardized structured logs to monitor system performance in real time, quickly locate faults, and ensure stable system operation and high availability. In business analytics, various products within an enterprise need to leverage parsed, standardized structured logs to gain in-depth understanding of user preferences and usage patterns, thereby optimizing products and services. In marketing, parsed, standardized structured logs are used to analyze consumer behavior and market trends to develop more targeted and competitive marketing strategies. Therefore, log standardization is crucial for all industries. Improving the efficiency and accuracy of log parsing is a pressing technical challenge that needs to be addressed. Summary of the Invention

[0003] This application provides a log parsing method, apparatus, computing device cluster, computer-readable storage medium, and computer program product containing instructions, which can improve the efficiency and accuracy of log parsing.

[0004] In a first aspect, this application provides a log parsing method, comprising: receiving a log to be parsed and a target field selected by a user, wherein the log to be parsed includes at least one original log, and the target field is a field to be extracted from the original log; generating at least one parsing code based on the log to be parsed and the target field, wherein the language of the parsing code is selected by the user, and a parsing code is associated with at least one original log; running a first parsing code to process a first original log associated with the first parsing code to obtain a structured log of the first original log, wherein the first parsing code is any one of the at least one parsing code.

[0005] In this way, by transforming the log parsing task into a code generation problem, rather than directly generating log templates, efficient and real-time structured and standardized processing of logs is achieved. There is no need for manual generation of parsing regular expressions based on log templates, which can efficiently and accurately process diverse, heterogeneous, high-quality, and frequently changing log data, and can ensure high robustness and high accuracy in parsing in complex and ever-changing log scenarios.

[0006] In one possible implementation, at least one parsing code is generated based on the logs to be parsed and the target field. This includes: clustering the original logs in the logs to be parsed to obtain at least one class of logs; and generating parsing code related to each class of logs based on the target field and the at least one class of logs. In this way, after clustering, only one parsing code needs to be generated for each class of logs, reducing computational overhead.

[0007] In one possible implementation, based on a target field and at least one type of log, parsing code related to each of the at least one log type is generated. This includes: filtering samples in a knowledge base based on the target field and the first type of log to obtain at least one parsing sample related to the first type of log, where the first type of log is any one of the at least one log type; the parsing sample includes: the source log, the structured log of the source log, and parsing code related to the source log; if the first type of log matches any one of the at least one parsing sample, the parsing code in that parsing sample is the parsing code related to the first type of log; if the first type of log does not match any of the at least one parsing sample, at least the target field and at least one parsing sample are input into a large model to obtain parsing code related to the first type of log. This approach leverages existing knowledge and examples to improve the accuracy of parsing code generation.

[0008] In one possible implementation, before generating parsing code related to each of the at least one type of logs based on the target field and at least one type of logs, the method further includes: extracting a first field value related to the target field from at least one original log in the second type of logs, wherein the second type of logs is any one of the at least one type of logs; presenting the first field value to the user; and updating the first field value if the user modifies the first field value. This aligns with the user's parsing target, ensuring the model accurately understands the field requested by the user and preventing confusion when source logs contain different fields of the same type (such as source IP, destination IP, etc.).

[0009] In one possible implementation, after generating parsing code related to each of the at least one type of log based on the target field and at least one type of log, the method further includes: running second parsing code related to the second type of log to process at least one original log of the second type of log to obtain a first structured log of at least one original log of the second type of log; storing the second parsing code in a knowledge base if the first field value matches the second field value, wherein the second field value is the field value related to the target field in the first structured log; and updating the target code statement in the second parsing code if the first field value does not match the second field value, wherein the target code statement is used to extract the field values ​​that differ between the first field value and the second field value. This approach can address the "illusion" of large models and improve the accuracy of parsing code generation.

[0010] In one possible implementation, the method further includes: after multiple updates to the target code statements in the second parsing code, if the value of the first field and the value of the second field extracted by the second parsing code still do not match, prompting for manual intervention. This ensures successful completion of the parsing task even when automated methods fail.

[0011] Secondly, this application provides a log parsing apparatus, comprising: a receiving module and a processing module. The receiving module is configured to receive a log to be parsed and a target field selected by a user, wherein the log to be parsed includes at least one original log, and the target field is a field to be extracted from the original log. The processing module is configured to generate at least one parsing code based on the log to be parsed and the target field, wherein the language of the parsing code is selected by the user, and one parsing code is associated with at least one original log; and to run the first parsing code to process a first original log associated with the first parsing code to obtain a structured log of the first original log, wherein the first parsing code is any one of the at least one parsing code.

[0012] In one possible implementation, when the processing module generates at least one parsing code based on the log to be parsed and the target field, it is specifically used to: perform clustering processing on the original logs in the log to be parsed to obtain at least one type of log; and generate parsing code related to each type of log in the at least one type of log based on the target field and the at least one type of log.

[0013] In one possible implementation, when the processing module generates parsing code related to each of the at least one log category based on the target field and at least one log category, it specifically performs the following: based on the target field and the first log category, it performs sample filtering in the knowledge base to obtain at least one parsing sample related to the first log category, wherein the first log category is any one of the at least one log category, and the parsing sample includes: the source log, the structured log of the source log, and parsing code related to the source log; if the first log category matches any one of the at least one parsing sample, the parsing code in the at least one parsing sample is the parsing code related to the first log category; if the first log category does not match any of the at least one parsing sample, it inputs at least the target field and at least one parsing sample into the large model to obtain the parsing code related to the first log category.

[0014] In one possible implementation, before the processing module generates parsing code related to each of the at least one type of logs based on the target field and at least one type of logs, it is further configured to: extract a first field value related to the target field from at least one original log in the second type of logs, wherein the second type of logs is any one of the at least one type of logs; present the first field value to the user; and update the first field value if the user modifies the first field value.

[0015] In one possible implementation, after the processing module generates parsing code related to each of the at least one type of log based on the target field and at least one type of log, it is further configured to: run second parsing code related to the second type of log to process at least one original log of the second type of log to obtain a first structured log of at least one original log of the second type of log; if the first field value matches the second field value, store the second parsing code in a knowledge base, wherein the second field value is the field value related to the target field in the first structured log; if the first field value does not match the second field value, update the target code statement in the second parsing code, wherein the target code statement is used to extract the field values ​​that are different between the first field value and the second field value.

[0016] In one possible implementation, the processing module is further configured to: prompt for manual intervention if the value of the first field and the value of the second field extracted by the second parsing code still do not match after multiple updates of the target code statement in the second parsing code.

[0017] Thirdly, embodiments of this application provide a log parsing apparatus, including at least one processor and an interface; the at least one processor obtains program instructions through the interface; the at least one processor is used to execute program line instructions to implement the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the log parsing apparatus may be, but is not limited to, a chip.

[0018] Fourthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0019] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0020] Sixthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0021] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating a technical concept for log parsing provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the structure of a log parsing code generation device provided in an embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating a log parsing method provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of a log parsing process provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application;

[0027] Figure 6This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application;

[0030] Figure 9 This is a schematic diagram of another log parsing device provided in an embodiment of this application. Detailed Implementation

[0031] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0032] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0033] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0034] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0035] First, some of the technical terms involved in the embodiments of this application will be introduced.

[0036] (1) Prompt

[0037] A prompt is a text instruction used to guide a neural network model in generating specific content. Its core function is to provide the model with context and task objectives. To provide context, a prompt helps the model understand the semantic environment of the input data by including relevant information or background. To specify the task objective, a prompt tells the model the task to be performed through explicit instructions, such as "check for errors in the data and correct them" or "analyze trends in the data." This instruction guides the model to think and generate content in a specific direction.

[0038] (2) Large Language Model (LLM)

[0039] Large language models are artificial intelligence models based on deep learning techniques. They are trained on large-scale text datasets to learn the syntax, semantics, and pragmatics of a language. These models have numerous parameters and can generate high-quality language output from input text, such as answering questions, translating text, and creating stories, among other natural language processing tasks.

[0040] (3) Log

[0041] Logs are record files generated by systems, applications, devices, etc., during operation. They contain information such as various events, operations, errors, and state changes. For example, server logs record user access requests, system operating status, and error messages; application logs record user actions and program execution. In some embodiments, parsing logs refers to analyzing and processing the contents of log files to extract valuable information, discover potential problems, or achieve specific monitoring and management goals.

[0042] (4) Log Template

[0043] Log templates are patterns obtained by removing dynamic parameters (such as timestamps, IP addresses, user IDs, etc.) from logs and retaining only static content (such as the fixed format and keywords of log messages). For example, the log messages "Download APP1 and install" and "Download App2 and install" can be extracted into the template "Download<*> and install", where "<*>" represents dynamic parameters.

[0044] (4) Retrieval-Augmented Generation (RAG)

[0045] Retrieval-enhanced generation (RAG) is an artificial intelligence technique that combines information retrieval technology with generative models. It aims to enhance the capabilities of generative models (such as large language models) by integrating external knowledge sources. The core idea of ​​RAG is to first retrieve information relevant to the user's query from external data sources, and then input this information as context into a large language model to generate more accurate and context-aware text. Specifically, the RAG workflow includes the following steps: a) Segmenting external knowledge sources (such as documents, databases, etc.) into smaller fragments, vectorizing them, and storing them in a vector database; b) When a user submits a query, the retrieval model retrieves the most relevant knowledge fragments from the vector database; c) The generative model combines the retrieved knowledge fragments to generate the final text answer. The advantage of RAG technology lies in its ability to overcome the limitations of the generative model's own knowledge, reduce the illusion problem, and support the incorporation of real-time or dynamic information.

[0046] Next, the solutions provided in the embodiments of this application will be introduced.

[0047] In automated log parsing technology, log templates are typically extracted first, and then regular expressions are manually written based on the templates. Automated log parsing is mainly divided into two types: unsupervised (not dependent on labeled data) and supervised (dependent on labeled data).

[0048] Unsupervised automated log parsing methods mainly include the following three types:

[0049] Static source code analysis: This method involves statically analyzing the source code that generates logs to extract log templates and then generating corresponding log parsing regular expressions. Static text can be directly inserted with regular expressions (special characters need to be escaped), while dynamic variables require appropriate wildcards for capture. However, this method also has some limitations. First, the unavailability of the source code often limits its application. Second, diverse programming languages, program design and coding styles, and complex log formats increase the complexity of static analysis. This makes fully automated analysis difficult to achieve, usually still requiring manual intervention to analyze the code, which is time-consuming, labor-intensive, and lacks timeliness.

[0050] Frequent pattern mining-based methods utilize frequent itemset mining techniques to extract high-frequency subsequences from logs and summarize them into templates. Tools such as the Simple Log Clustering Tool (SLCT), Log Feature Analysis (LFA), Loogram, and LogCluster effectively extract representative templates from massive log data using this method, providing a solid foundation for log parsing. However, this approach also has some limitations. First, frequent pattern mining-based methods typically assume that high-frequency items are static text and low-frequency items are dynamic variables. However, this assumption does not always hold true for diverse log formats, potentially leading to decreased parsing accuracy. Second, the model's performance is highly sensitive to the threshold of frequent itemsets, currently relying mainly on manual setting. Although some algorithms can estimate the optimal threshold, they cannot completely guarantee against errors. Furthermore, these methods cannot automatically identify and correct errors when general template extraction fails. When log formats change frequently and dynamically, if the model fails to continuously optimize in a timely manner, performance degradation and inability to adapt to new log formats can easily occur.

[0051] Heuristic-based approaches encode domain expert knowledge into general heuristic rules for parsing logs. For example, Automatic Event Logging (AEL) groups log messages by comparing the frequency of occurrence between fixed and variable terms. Iterative Partitioning-based Log Message Modeling (IPLoM) employs an iterative partitioning strategy, grouping log messages based on message length, term position, and mapping relationships. Drain uses a fixed-depth tree structure to extract templates, efficiently processing complex log data through a hierarchical parsing approach. However, these methods also have limitations. First, they rely on domain knowledge, requiring manual writing of heuristic rules and design of features, increasing development and maintenance costs. Second, their performance may be unstable when dealing with complex and heterogeneous log data formats, making it difficult to adapt to diverse log structures. Furthermore, these methods cannot automatically identify and correct errors when general template extraction fails. When log formats change frequently, performance degradation and inability to adapt to new log formats can easily occur if the model is not continuously optimized in a timely manner.

[0052] Supervised automated log parsing primarily relies on deep learning or natural language processing (NLP) algorithms. This approach depends on labeled log data and trains deep learning or NLP algorithms to classify tokens in log messages. Some methods focus on identifying dynamic parameters or static text. For example, the UniParser uses deep learning algorithms to automatically identify dynamic parameters and static text in logs for efficient parsing; log parsing with pattern tree (logPPT) utilizes pattern tree structures to classify tokens in logs, effectively identifying dynamic parameters and static text. Other methods combine semantic understanding to identify specific target fields. For instance, the Semantic Parser (SemParser) uses NLP techniques combined with semantic understanding to identify specific target fields in logs, providing more accurate parsing results. These methods, trained on labeled data, can significantly improve the accuracy and efficiency of log parsing, meeting the needs of different scenarios. However, this approach also has some limitations. First, it typically requires a large amount of labeled data for training, and when log formats are complex and frequently changing, it is difficult to obtain sufficient and high-quality training data. Second, compared to unsupervised methods, supervised methods often require higher computational resources. Furthermore, because models are prone to overfitting to patterns in the training data, they struggle to accurately extract key information when faced with new log formats. If the general template extraction fails, the model cannot automatically identify and correct the error. Moreover, when log formats change dynamically and frequently, if the model fails to continuously optimize in a timely manner, performance degradation can easily occur, making it unable to adapt to new log formats.

[0053] In view of this, the embodiments of this application provide a log parsing scheme that can efficiently and accurately process diverse, heterogeneous, high-quality, and frequently changing log data, and can ensure that it can still achieve high robustness and high accuracy parsing results in complex and ever-changing log scenarios.

[0054] For example, Figure 1 This illustration shows a schematic diagram of a technical concept for log parsing provided in an embodiment of this application. For example... Figure 1As shown, the log parsing framework mainly includes a parsing code generation part and a structured log generation part. The parsing code generation part is primarily responsible for processing the user-selected log to be parsed and the fields (or "target fields") to be extracted from the log using the log parsing code generation device 100, generating parsing code related to each original log entry in the log to be parsed. The parsing code is the code used to parse the original log. The parsing code may contain code statements used to parse the log. Furthermore, the code language of the parsing code may be, but is not limited to, selected by the user. In some embodiments, such as... Figure 2As shown, the log parsing code generation device 100 may include a generator 110, a reflector 120, and a reinforcer 130. The generator 110 can be responsible for clustering preprocessing and target alignment. Specifically, the generator 110 can use a log clustering algorithm (such as Drain) to cluster the user-input logs to be parsed, thereby reducing redundant parsing and generating parsing code only once for each log category, improving efficiency. Of course, the generator 110 may also choose not to perform clustering processing on the logs to be parsed, depending on the actual situation, and is not limited here. Optionally, the generator 110 may also, but is not limited to, initially extracting target field content from the logs for user confirmation or correction to ensure accurate understanding of the requirements. The generator 110 can also be responsible for generating parsing code based on LLM+RAG. Specifically, the generator 110 can retrieve parsing syntax and examples related to the logs to be parsed from a knowledge base. If the knowledge base already has full-match examples, the generator directly extracts the parsing code related to the examples and outputs the extracted parsing code. Alternatively, the generator can construct a Prompt containing standard fields (i.e., target fields), samples, and syntax rules. The LLM then generates corresponding parsing code based on this Prompt. This leverages existing experience to improve the accuracy of the generated parsing code. The reflector 120 is primarily responsible for self-verification and multiple rounds of self-correction. Specifically, the reflector 120 verifies the parsing code generated by the generator 110. If the parsing code contains errors, the reflector 120 optimizes the Prompt based on different error types, and the LLM generates the corresponding parsing code again based on the optimized Prompt. This improves the accuracy of LLM generation by optimizing the Prompt. If correction fails after more than a custom k rounds, manual intervention is prompted. At this point, the reflector 120 can provide auxiliary information to help improve the efficiency of manual processing. The reinforcer 130 is primarily responsible for self-evolving memory upgrades. After the reflector 120 verifies the generated parsing code is correct, the reinforcer 130 adds the parsing code and corresponding logs to the knowledge base, continuously updating the knowledge base to achieve continuous self-optimization of the system. In this way, by leveraging the advantages of LLM in code generation, semantic understanding, general knowledge and generalization ability, and integrating RAG technology for knowledge injection and management, the log parsing code generation device 100 achieves efficient, robust and accurate parsing of diverse, heterogeneous, high-quality and frequently changing enterprise logs.

[0055] The structured log generation section is primarily responsible for running parsing code related to the raw logs in the logs to be parsed within the production environment 200. This parsing process generates structured logs for each raw log. For example, the production environment 200 refers to the environment in which software or applications provide services to users during actual operation. It is a highly stable, secure, and optimized operating environment used to deploy developed and tested code to ensure that the software can efficiently and reliably handle actual user requests and meet business needs. In some embodiments, the production environment 200 may, but is not limited to, consist of hardware (servers, storage, network devices, etc.), software (operating system, database, middleware, application server, etc.), monitoring systems, security mechanisms, backup and recovery tools, and configuration management systems, collectively ensuring the stable, secure, and efficient operation of the system.

[0056] Within this log parsing framework, users input the log to be parsed and select target fields. The system then outputs parsing code that transforms the log into structured log data. This code can then be run directly in the production environment to quickly obtain the required structured logs. In this way, by transforming the log parsing task into a code generation problem, rather than directly generating log templates, this framework achieves efficient and real-time structured and standardized log processing. Furthermore, it eliminates the need for manual generation of parsing regular expressions based on log templates. This allows for efficient and accurate handling of diverse, heterogeneous, high-quality, and frequently changing log data, ensuring high robustness and accuracy even in complex and ever-changing log scenarios.

[0057] The following describes the specific implementation process of the above technical concept.

[0058] For example, Figure 3 This diagram illustrates a flowchart of a log parsing method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, the method can be executed by a log parsing device, which can be implemented in software and / or hardware, and can be, but is not limited to, configured in an electronic device or server. Figure 3 As shown, this log parsing method may include the following steps:

[0059] S301. Receive the log to be parsed and the target field selected by the user, wherein the log to be parsed includes at least one original log, and the target field is the field to be extracted from the original log.

[0060] In this embodiment, the user can select the log to be parsed on the front-end interface of the log parsing device, supporting import of logs from local storage or the cloud. The log to be parsed may include at least one original log. Additionally, the user can select the fields (i.e., "target fields") to be extracted from the original log on the front-end interface of the log parsing device. For example, target fields can be understood as, but are not limited to, fields in the structured log ultimately required by the user. Target fields can be fields from some predetermined standard logs, or they can be user-defined, depending on the actual situation, and are not limited here. After the user completes the selection, the log parsing device can receive the selected log to be parsed and target fields. Furthermore, the user can also select the code language for the parsing code on the front-end interface of the log parsing device, for example, selecting "regular expression," "Grok," "self-developed framework," etc. In some embodiments, the selected target field can also be understood as the selected target log template. In some embodiments, the target field may consist of at least one subfield.

[0061] S302. Based on the log to be parsed and the target field, generate at least one parsing code, wherein the language of the parsing code is selected by the user, and one parsing code is associated with at least one original log.

[0062] In this embodiment, the log parsing device can process the log to be parsed and the target field, for example, through the aforementioned Figure 1 The log parsing code generation device 100 performs processing, etc., to generate at least one parsing code. Each parsing code is associated with at least one original log file.

[0063] As one possible implementation, for each original log in the log to be parsed, the log parsing device can construct a Prompt based on the target field and an original log, and input the Prompt into the large model to generate parsing samples related to the original log through the large model.

[0064] As another possible implementation, for each original log in the logs to be parsed, the log parsing device can perform sample filtering in the knowledge base based on the target field and an original log to obtain at least one parsing sample. The parsing sample can include: the source log, the structured log of the source log, and parsing code related to the source log. Then, the parsing sample is compared with the original log being processed. When the log template of a parsing sample is the same as the log template of the original log, and the target field in the parsing sample is the same as the target field in the original log, the parsing code in the parsing sample can be directly used as the parsing code related to the original log. However, when any one of the following is different: the log template of the selected parsing sample is different from the log template of the original log, and the target field in the selected parsing sample is different from the target field in the original log, a Prompt can be constructed using at least the target field, the original log, and the selected parsing sample. This Prompt is then input into the large model to guide the large model in generating parsing samples related to the original log using existing knowledge and cases, thereby improving the accuracy of the parsing code generation.

[0065] As another possible implementation, the log parsing device can first cluster the raw logs in the logs to be parsed using log clustering algorithms such as Drain to obtain at least one class of logs. Then, based on the target field and the various log classes obtained from the clustering, the log parsing device can generate parsing code related to each log class. Subsequently, the raw logs contained in each log class can be processed using the same parsing code, thus requiring only one parsing code to be generated for each log class, reducing computational overhead.

[0066] At this point, when generating corresponding parsing code for each type of log, the log parsing device can select one or more original logs from each type of log as example logs. Then, for any type of log (hereinafter referred to as "Type 1 log"), the log parsing device can perform sample filtering in the knowledge base based on the target field and the example logs in Type 1 logs to obtain at least one parsing sample related to Type 1 logs. Next, the log parsing device compares the parsing sample with the example logs in Type 1 logs. When the log template of a parsing sample is the same as the log template of the example log, and the target field in the parsing sample is the same as the target field in the example log, the parsing code in the parsing sample can be directly used as the parsing code related to Type 1 logs. If any one of the following is different: the log template of the selected parsing sample is different from the log template of the example log, or the target field in the selected parsing sample is different from the target field in the example log, a Prompt can be constructed using the target field, the example log, and the selected parsing sample. This Prompt can then be input into the large model to guide the large model to generate parsing samples related to the first type of log using existing knowledge and cases, thereby improving the accuracy of the parsing code generation.

[0067] S303. Run the first parsing code to process the first raw log related to the first parsing code to obtain the structured log of the first raw log, wherein the first parsing code is any one of at least one parsing code.

[0068] In this embodiment, after generating parsing code related to the original log in the log to be parsed, for any parsing code (hereinafter referred to as "first parsing code"), the first parsing code can be deployed in the production environment and run in the production environment. In addition, the first original log related to the first parsing code is input into the production environment so that the first original log can be processed by the first parsing code to obtain the structured log of the first original log.

[0069] In this way, by transforming the log parsing task into a code generation problem, rather than directly generating log templates, efficient and real-time structured and standardized processing of logs is achieved. There is no need for manual generation of parsing regular expressions based on log templates, which can efficiently and accurately process diverse, heterogeneous, high-quality, and frequently changing log data, and can ensure high robustness and high accuracy in parsing in complex and ever-changing log scenarios.

[0070] In some embodiments, Figure 3In step S302, when clustering is performed on the logs to be parsed, before this step, for any type of log (hereinafter referred to as "second type logs"), the log parsing device can first extract the first field value related to the target field from at least one original log (e.g., example log) in the second type logs. Then, the log parsing device can present the first field value to the user on its front-end interface so that the user can confirm whether the extraction result is correct. If correct, the user does not need to correct it. If incorrect, the user can correct the content contained in the first field value. If the user modifies the first field value, the log parsing device can update the first field value. This ensures that the model accurately understands the field requested by the user and prevents confusion when the source logs contain different fields of the same type (e.g., source IP, destination IP, etc.). In some embodiments, when the target field consists of multiple subfields, the first field value can contain the values ​​of each subfield.

[0071] Furthermore, after obtaining the parsing code, the log parsing device can verify each parsing code. The following example illustrates the verification of the second parsing code related to the second type of logs. The verification process can involve: running the second parsing code in a production environment to process at least one original log (e.g., a sample log) of the second type of logs to obtain a first structured log of at least one original log of the second type of logs. Then, the log parsing device can compare the first field value with the field value related to the target field in the first structured log (hereinafter referred to as the "second field value"). When the two are the same (i.e., they match), the log parsing device can determine that the second parsing code verification has passed and can store the second parsing code, the corresponding original log, and the first structured log in the knowledge base to update the data in the knowledge base. When the two are different (i.e., they do not match), the log parsing device can update the target code statement in the second parsing code. The target code statement is used to extract the field values ​​that differ between the first field value and the second field value. For example, the log parsing device can re-retrieve parsing samples from the knowledge base and construct a Prompt, which only requires the regeneration of the target code statement. Of course, all the code statements in the second parsed code can also be regenerated; this is not a limitation here.

[0072] Furthermore, if the log parsing device updates the second parsing code multiple times, and the second field extracted by the second parsing code still does not match the value of the first field, the log parsing device can prompt for manual intervention on its front-end interface to ensure successful completion of the parsing task. At this point, the log parsing device can provide multiple nearly correct parsing statements and related syntax to assist the user in quickly generating the correct parsing code.

[0073] The above is a description of the log parsing method provided in the embodiments of this application. To facilitate understanding, the implementation process of the above method will be described in detail below.

[0074] For example, such as Figure 4 As shown in S401, the user inputs the log to be parsed, selects the target field, and the code language of the parsing statement. For example, the user can directly select the standard log, in which case the fields in the standard log are the target fields; the user can also add target fields independently, etc.

[0075] In S402, clustering preprocessing is performed. The purpose of this step is to reduce computational overhead by generating only one parsing code for each log cluster after clustering. The detailed steps are as follows: input a batch of logs to be parsed, perform clustering using a typical log clustering algorithm (such as DRAIN), and display m example logs for each log cluster. For example, log templates can be generated after clustering.

[0076] In S403, target field extraction is performed. The purpose of this step is to align the extracted data with the user's parsing target. The detailed steps are: based on the target field, extract and display the possible field content from various types of sample logs. The user confirms the accuracy of the extraction results; if incorrect, they can be corrected.

[0077] In S404, relevant examples and syntax retrieval are performed. The purpose of this step is to improve the accuracy of parsing code generation by leveraging existing knowledge and examples. The detailed steps are: retrieving the most relevant parsing syntax descriptions (if the user selects a self-developed framework for parsing) and k anonymized parsing examples from the knowledge base (or "knowledge base"), and including source logs, parsing statements, and converted standard logs. For example, such as... Figure 4 As shown, the knowledge base may include, but is not limited to, high-quality grammar manuals, high-quality question and answer samples, high-quality log standard paradigms and introductory documents, etc.

[0078] In S405, this step can be divided into two parts: "outputting existing parsed code" and "generating retrieval enhancements", and these two parts are executed together.

[0079] The purpose of the "Output Existing Parsing Code" section is to improve the accuracy of parsing code generation by utilizing existing knowledge and cases. The detailed steps are as follows: If, after step S404, it is found that the memory knowledge base contains a full-match example (both the log template and the target field must match), then the parsing statement (i.e., parsing code) contained in the full-match example is directly output, and the entire generation process ends.

[0080] The purpose of "retrieval enhancement generation" is to improve the accuracy of parsing code generation by leveraging existing knowledge and cases. The detailed steps are: the planner constructs an initial prompt (including standard fields, relevant examples, and key syntax rules, etc.) based on the information obtained from S404, and generates parsing code based on this.

[0081] In S406, validation and multiple rounds of self-correction are performed. The purpose of this step is to address the "illusion" of large models and improve the accuracy of generated parsing code. The detailed steps are: validating the generated parsing code (running the statements and comparing the parsing results with the standard answer in S403). If errors occur, the Prompt is optimized based on the error type (different error types apply specific Prompts; when the syntax of certain fields is incorrect, the retrieval system retrieves relevant syntax and examples from its memory, and the Prompt only requires regenerating the erroneous field parsing statements). If errors persist after multiple self-corrections, more advanced RAG methods, such as Multi-query Retriever, are used to increase the number of examples, further enhancing the Prompt.

[0082] In S407, the self-evolving memory is upgraded. The purpose of this step is to achieve self-evolution and efficient optimization iteration of the system. The detailed steps are as follows: If the parsing code can be correctly generated after S406, then it is evaluated whether to add this attempt as a new case to the knowledge base (based on diversity indicators). If the user is satisfied with the result, they can also choose to share the relevant Prompt to enrich the system's Prompt library.

[0083] In S408, manual intervention is required. The purpose of this step is to ensure successful completion of the parsing task when automated methods fail. The detailed steps are as follows: if the system fails to generate the correct parsing result after multiple rounds of correction, it prompts for manual intervention. At this point, several near-correct parsing statements and related syntax can be provided to assist the user in quickly generating the correct parsing code.

[0084] After obtaining the parsing code, users can directly deploy it on a big data processing platform to efficiently and in real-time parse logs and obtain standardized logs for downstream applications.

[0085] exist Figure 4 In S402 to S405, it is possible, but not limited to, through the aforementioned Figure 2 The generator 110 shown is executed. S406 can be, but is not limited to, via the aforementioned Figure 2 The reflector 120 shown is executed. S407 can be, but is not limited to, via the aforementioned Figure 2 The enhancer 130 shown is executed.

[0086] As described above, the log parsing method in this embodiment generates log parsing code based on LLM and RAG. It leverages the advantages of LLM in code generation, semantic understanding, and general knowledge, and integrates RAG technology for knowledge injection and management, achieving a high degree of adaptability and robustness: even when faced with messy, non-standard nested, or completely new and unconventional logs, it can accurately generate parsing code by capturing methods for parsing into standard formats.

[0087] Furthermore, this log parsing method supports multiple parsing languages, including regular expressions, Grok, and a self-developed framework. It also features self-verification and multi-round self-correction mechanisms: by combining the Prompt project with a dedicated small verification model, it's easy to verify, optimize, and constrain the parsing code generated by the large model.

[0088] Furthermore, this log parsing method also possesses continuous self-evolution capabilities: it establishes an optimization chain between knowledge base quality, Prompt quality, and LLM response quality. By continuously improving the quality of the knowledge base, it enhances the effectiveness of the Prompt, ultimately improving the quality of the LLM-generated parsing code, thus achieving the system's self-evolution and self-optimization.

[0089] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments and features described in the above embodiments can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0090] Next, based on the methods in the above embodiments, a log parsing device provided in this application will be introduced.

[0091] For example, Figure 5 A schematic diagram of the structure of a log parsing device provided in an embodiment of this application is shown. Figure 5 As shown, the log parsing device 500 includes a receiving module 501 and a processing module 502. The receiving module 501 receives a log to be parsed and a target field selected by the user, wherein the log to be parsed includes at least one original log, and the target field is a field to be extracted from the original log. The processing module 502 generates at least one parsing code based on the log to be parsed and the target field, wherein the language of the parsing code is selected by the user, and one parsing code is associated with at least one original log; and runs the first parsing code to process a first original log associated with the first parsing code to obtain a structured log of the first original log, wherein the first parsing code is any one of the at least one parsing code.

[0092] In some embodiments, when the processing module 502 generates at least one parsing code based on the log to be parsed and the target field, it is specifically used to: perform clustering processing on the original logs in the log to be parsed to obtain at least one type of log; and generate parsing code related to each type of log in the at least one type of log based on the target field and the at least one type of log.

[0093] In some embodiments, when the processing module 502 generates parsing code related to each of the at least one type of log based on the target field and at least one type of log, it specifically performs the following: based on the target field and the first type of log, it performs sample filtering in the knowledge base to obtain at least one parsing sample related to the first type of log, wherein the first type of log is any one of the at least one types of log, and the parsing sample includes: the source log, the structured log of the source log, and parsing code related to the source log; if the first type of log matches any one of the at least one parsing sample, the parsing code in the at least one parsing sample is the parsing code related to the first type of log; if the first type of log does not match any of the at least one parsing sample, it inputs at least the target field and at least one parsing sample into the large model to obtain the parsing code related to the first type of log.

[0094] In some embodiments, before generating parsing code related to each of the at least one type of logs based on the target field and at least one type of logs, the processing module 502 is further configured to: extract a first field value related to the target field from at least one original log in the second type of logs, wherein the second type of logs is any one of the at least one type of logs; present the first field value to the user; and update the first field value if the user modifies the first field value.

[0095] In some embodiments, after generating parsing code related to each of the at least one type of log based on the target field and at least one type of log, the processing module 502 is further configured to: run the second parsing code related to the second type of log to process at least one original log of the second type of log to obtain a first structured log of at least one original log of the second type of log; if the first field value matches the second field value, store the second parsing code in a knowledge base, wherein the second field value is the field value related to the target field in the first structured log; if the first field value does not match the second field value, update the target code statement in the second parsing code, wherein the target code statement is used to extract the field values ​​that are different between the first field value and the second field value.

[0096] In some embodiments, the processing module 502 is further configured to: after updating the target code statement in the second parsing code multiple times, if the value of the first field and the value of the second field extracted by the second parsing code still do not match, prompt for manual intervention.

[0097] In some embodiments, Figure 5 Both the receiving module 501 and the processing module 502 shown can be implemented in software or in hardware. For example, the implementation of the receiving module 501 will be described below. Similarly, the implementation of the processing module 502 can be referenced from the implementation of the receiving module 501.

[0098] As an example of a software functional unit, the receiving module 501 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 501 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0099] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0100] As an example of a hardware functional unit, the receiving module 501 may include at least one computing device, such as a server. Alternatively, the receiving module 501 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0101] The multiple computing devices included in the receiving module 501 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 501 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 501 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0102] It should be noted that, in other embodiments, the receiving module 501 can be used to execute any step in the log parsing method described in the above embodiments, and the processing module 502 can also be used to execute any step in the log parsing method described in the above embodiments. Furthermore, the steps implemented by the receiving module 501 and the processing module 502 can be specified as needed, and different steps in the log parsing method described in the above embodiments can be implemented by the receiving module 501 and the processing module 502 respectively. Figure 5 The log parsing device 500 shown has all the functions of this device.

[0103] This application also provides a computing device 600. For example... Figure 6 As shown, the computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate with each other via the bus 602. The computing device 600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 600.

[0104] Bus 602 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus 604 is represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 604 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608).

[0105] Processor 604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0106] Memory 606 may include volatile memory, such as random access memory (RAM). Processor 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0107] The memory 606 stores executable program code, and the processor 604 executes the executable program code to implement the aforementioned functions. Figure 5 The receiving module 501 and processing module 502 shown herein perform their functions to implement the log parsing method described in the above embodiments. That is, the memory 606 stores instructions for executing the log parsing method described in the above embodiments.

[0108] Alternatively, the memory 606 stores executable code, and the processor 604 executes the executable code to implement the aforementioned functions respectively. Figure 5 The log parsing device 500 shown in the diagram performs the functions of the log parsing method described in the above embodiments. That is, the memory 606 stores instructions for executing the log parsing method described in the above embodiments.

[0109] The communication interface 608 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other devices or communication networks.

[0110] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0111] like Figure 7 As shown, the computing device cluster includes at least one computing device 600. The memory 606 in one or more computing devices 600 in the computing device cluster may store the same instructions for executing the log parsing method described in the above embodiments.

[0112] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the log parsing method described in the above embodiments. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for executing the log parsing method described in the above embodiments.

[0113] It should be noted that the memory 606 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 5 This illustrates some of the functions of the log parsing device 500. Specifically, the instructions stored in the memory 606 of different computing devices 600 can implement the functions of one or more modules in the receiving module 501 and processing module 502.

[0114] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 8 One possible implementation is shown. For example... Figure 8 As shown, two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 606 in computing device 600A stores instructions for performing the functions of the receiving module 501. Simultaneously, the memory 606 in computing device 600B stores instructions for performing the functions of the processing module 502.

[0115] It should be understood that Figure 8The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0116] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 7 and Figure 8 The connection method of the computing device cluster is different in that the memory 606 of one or more computing devices 600 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0117] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the aforementioned log parsing method. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for executing the aforementioned log parsing method.

[0118] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0119] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0120] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices containing at least one computing device, cause the cluster of computing devices to perform the methods in the above embodiments.

[0121] Based on the methods in the above embodiments, this application also provides a log parsing apparatus. Please refer to... Figure 9 , Figure 9 This is a schematic diagram of another log parsing device provided in an embodiment of this application. Figure 9 As shown, the log parsing device 900 includes one or more processors 901 and interface circuitry 902. Optionally, the log parsing device 900 may also include a bus 903. Wherein:

[0122] The processor 901 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of the processor 901 or through software instructions. The processor 901 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0123] The interface circuit 902 can be used to send or receive data, instructions or information. The processor 901 can use the data, instructions or other information received by the interface circuit 902 to process the data, instructions or other information, and can send the processed information out through the interface circuit 902.

[0124] Optionally, the log parsing device 900 also includes memory, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0125] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0126] Optionally, the interface circuit 902 can be used to output the execution results of the processor 901.

[0127] It should be noted that the functions of the processor 901 and the interface circuit 902 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0128] It should be understood that each step of the above method embodiments can be completed by hardware logic circuits or software instructions in a processor.

[0129] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0130] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0131] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0132] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A log parsing method, characterized in that, The method includes: Receive logs to be parsed and target fields selected by the user, wherein the logs to be parsed include at least one original log, and the target field is a field to be extracted from the original log; Based on the log to be parsed and the target field, at least one parsing code is generated, wherein the language of the parsing code is selected by the user, and one parsing code is associated with at least one of the original logs; Run the first parsing code to process the first raw log associated with the first parsing code to obtain a structured log of the first raw log, wherein the first parsing code is any one of the at least one parsing code.

2. The method according to claim 1, characterized in that, The step of generating at least one parsing code based on the log to be parsed and the target field includes: Cluster the original logs in the logs to be parsed to obtain at least one class of logs; Based on the target field and the at least one type of log, generate parsing code related to each type of log in the at least one type of log.

3. The method according to claim 2, characterized in that, The step of generating parsing code related to each type of log in the at least one log category, based on the target field and the at least one log category, includes: Based on the target field and the first type of log, sample filtering is performed in the knowledge base to obtain at least one parsing sample related to the first type of log, wherein the first type of log is any one of the at least one type of log, and the parsing sample includes: source log, structured log of the source log, and parsing code related to the source log; If the first type of log matches any one of the at least one parsing sample, the parsing code in the parsing sample is the parsing code related to the first type of log; If the first type of log does not match any of the parsing samples in the at least one parsing sample, at least the target field and at least one of the parsing samples are input into the large model to obtain parsing code related to the first type of log.

4. The method according to claim 2 or 3, characterized in that, Before generating parsing code related to each of the at least one type of log based on the target field and the at least one type of log, the method further includes: Extract the value of the first field related to the target field from at least one original log in the second type of log, wherein the second type of log is any one of the at least one type of log; The first field value is presented to the user, and the first field value is updated if the user modifies the first field value.

5. The method according to claim 4, characterized in that, After generating parsing code related to each type of log in the at least one log type based on the target field and the at least one log type, the method further includes: Run the second parsing code related to the second type of log to process at least one raw log of the second type of log to obtain a first structured log of at least one raw log of the second type of log; If the first field value matches the second field value, the second parsing code is stored in the knowledge base, wherein the second field value is the field value related to the target field in the first structured log; If the first field value and the second field value do not match, update the target code statement in the second parsing code, wherein the target code statement is used to extract the field values ​​that are different between the first field value and the second field value.

6. The method according to claim 5, characterized in that, The method further includes: If, after updating the target code statement in the second parsing code multiple times, the value of the first field and the value of the second field extracted by the second parsing code still do not match, manual intervention is prompted.

7. A log parsing device, characterized in that, include: A receiving module is used to receive a log to be parsed and a target field selected by the user, wherein the log to be parsed includes at least one original log, and the target field is a field to be extracted from the original log; The processing module is configured to generate at least one parsing code based on the log to be parsed and the target field, wherein the language of the parsing code is selected by the user, and one parsing code is associated with at least one of the original logs; The processing module is further configured to run first parsing code to process the first raw log associated with the first parsing code to obtain a structured log of the first raw log, wherein the first parsing code is any one of the at least one parsing code.

8. A log parsing device, characterized in that, Includes at least one processor and interface; The at least one processor obtains program instructions through the interface; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-6.

9. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-6, wherein the cluster of computing devices includes at least one computing device.

Images