A log analysis method and system based on large language model
Through the log analysis method based on the large language model, the embedded vector and context reasoning capabilities are used to extract variable information in the log data, solving the problem of traditional methods when processing complex logs, and achieving efficient log monitoring and information extraction, reducing the Difficulty in operation and maintenance work.
Patent Information
- Application Number
- CN202411116729.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-15
AI Technical Summary
When the prior art deals with complex and changeable system logs, it is difficult to effectively extract valuable information, especially when facing unknown log formats or information. Traditional methods require a lot of manual maintenance and are prone to errors and lack flexibility.
Using a log analysis method based on a large language model, by obtaining the embedded vector of log data and matching the preset log template library, the primary reasoning ability and context reasoning ability of the large language model are used to extract variable information in the log data, so as to achieve no manual pre-prediction. Set log analysis of patterns and templates.
It greatly reduces the difficulty of operation and maintenance work, improves the efficiency of monitoring the server, can automatically adapt to log changes, process complex log information, and improves the accuracy and adaptability of information extraction.
Smart Images

Figure CN118626359B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a log analysis method and system based on a large language model. Background Art
[0002] Modern business information systems are an important support for enterprises. To ensure their stability, security and effectiveness, system operation and maintenance has gradually become a core component of modern IT architecture. System operation and maintenance is mainly responsible for monitoring and maintaining the daily operation of information systems, including the operating status of servers, network communication conditions, data backup and recovery, etc. In the process of system operation and maintenance, log parsing is an extremely important part. Log parsing is to extract valuable information from various logs generated by the system to facilitate subsequent log analysis and management, such as error detection, performance evaluation, user behavior analysis, etc. However, due to the characteristics of massive system logs, complex sources, variable versions, and diverse formats, how to effectively extract valuable information from them has become a major challenge.
[0003] Traditional log parsing methods mainly include regular expressions and template matching methods. The regular expression method extracts specified information from the log by defining specific pattern matching rules. This method is simple and effective, but when faced with complex and changeable log information, it requires a lot of work to write and maintain rules, and is prone to errors. In addition, since it is a preset template, although it is relatively simple, it lacks flexibility and often cannot effectively process unknown log formats or information. Summary of the invention
[0004] The embodiments of the present application provide a log analysis method and system based on a large language model, which is used to utilize the primary reasoning ability of the large language model to achieve efficient monitoring of the server by streaming and parsing the status information and alarm logs of key components of the server.
[0005] The present application embodiment provides a log analysis method based on a large language model, including:
[0006] Obtaining operation log data, and extracting embedding vectors of the log data;
[0007] Matching the embedding vector of the operation log with the category vector of each log category in a preset log template library to determine the log category that matches the operation log data;
[0008] Obtaining a log template corresponding to the log category in a preset log template library; wherein, in the preset log template library, the log template of each log type corresponds to and is associated with the category vector one by one, and the content of the log template does not contain variable information;
[0009] The log template prompts the large language model to analyze and process the operation log data, extract variable information of the operation log data, and analyze the variable information of the operation log data to obtain the analysis result of the operation log data.
[0010] Optionally, before obtaining the operation log data and extracting the embedding vector of the log data, the method further includes:
[0011] Obtaining log sample data to be analyzed, and extracting an embedding vector of the log sample to be analyzed;
[0012] Clustering the extracted embedding vectors to determine a number of log categories and a category vector corresponding to each of the log categories;
[0013] A plurality of log samples to be processed are selected from each of the log categories, and variable information in the plurality of log samples to be processed is eliminated through the large language model, so as to extract a category template corresponding to the log category.
[0014] Optionally, extracting the embedding vector of the log sample to be analyzed is implemented by an embedded model;
[0015] The extracted embedding vectors are clustered based on the cosine similarity of the embedding vectors.
[0016] Optionally, selecting a plurality of log samples to be processed from each of the log categories, eliminating variable information in the plurality of log samples to be processed by the large language model, and then extracting a category template corresponding to the log category includes:
[0017] Pre-configured demonstration use cases;
[0018] Based on the demonstration use case and the selected log samples to be processed, the large language model is input, and the variable information in the multiple log samples to be processed is eliminated using the large language model to extract a category template corresponding to the log category.
[0019] Optionally, based on the exemplary use case and the selected log sample to be processed, inputting the large language model includes:
[0020] Use the hash table to remove duplicates from the selected log samples to be processed;
[0021] The log samples to be processed that are deduplicated according to the pre-configured demonstration use cases and the hash table are input into the large language model to instruct the large language model to propose the changed variables in the log samples to be processed.
[0022] Optionally, also include:
[0023] The basic information of the category template extracted by the large language model and the embedded vector corresponding to the type of log are encapsulated as metadata to construct the preset log template library.
[0024] Optionally, the large language model is prompted by the log template to analyze and process the operation log data, and the variable information of the operation log data is extracted, including:
[0025] The template entity of the log template and the metadata contained in the log template are used as prompt words and input into the large language model to extract variable information of the operation log data.
[0026] Optionally, the method further includes the following case where a log template corresponding to the log category cannot be obtained from a preset log template library:
[0027] Retrieving a temporary class in a preset log template library by using the embedded vector extracted from the log data, wherein the temporary class includes log data that cannot be clustered based on the preset log template library;
[0028] When the number of log data included in the temporary class meets the clustering requirement, a category template corresponding to the log category is extracted based on the large language model.
[0029] An embodiment of the present application also proposes a log analysis system based on a large language model, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the log analysis method based on the large language model as described above are implemented.
[0030] The embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log analysis method based on a large language model as described above are implemented.
[0031] The log analysis method of the embodiment of the present application contains a log template library that presets and associates log templates and category vectors of each log type, and the content of the log template does not contain variable information. The context reasoning ability of the large language model is further utilized to infer the variable information in the log data, thereby completing the analysis of the log data, achieving the goal of not requiring manual preset modes and templates, greatly reducing the difficulty of operation and maintenance work, and improving the efficiency of server monitoring.
[0032] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0034] Figure 1 The following is a basic flow chart of the log analysis method according to an embodiment of the present application;
[0035] Figure 2 The following is a schematic diagram of the operating architecture of the log analysis method according to an embodiment of the present application. DETAILED DESCRIPTION
[0036] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0037] The present application embodiment provides a log analysis method based on a large language model, such as Figure 1 , Figure 2 As shown, the following steps are included:
[0038] In step S101, operation log data is obtained, and an embedding vector of the log data is extracted. In a specific example, vectorized calculation can be performed through an embedded model to extract the embedding vector of the log data.
[0039] In step S102, the embedding vector of the operation log is matched with the category vector of each log category in the preset log template library to determine the log category that matches the operation log data. In a specific example, a corresponding category vector is configured for each type of log category in the preset log template library in advance. For example, vectors can be extracted from historical logs in advance and clustered to obtain multiple log categories based on historical log data.
[0040] In step S103, a log template corresponding to the log category is obtained in a preset log template library; wherein, in the preset log template library, the log template of each log type corresponds to and is associated with the category vector, and the content of the log template does not contain variable information. In some specific examples, in an information system, each key component will output an alarm message when an error occurs, and the alarm message can be read using tools such as IPMI, and is vectorized and stored in categories through embedded systems to form a preset log template library. The preset log template library contains log templates of each log type and corresponding category (type) vectors, so that the log template and the category vector are in a one-to-one correspondence. In some embodiments, the content of the log template does not contain variable information. For example, in an alarm log, there will be an alarm when the resource occupancy rate is high. One alarm log shows that the CPU occupies 95%, and another log shows that the CPU occupies 99%. The corresponding 95% and 99% are variables. The log template in the embodiment of the present application is a general template that removes similar variable information, which is more referenceable.
[0041] In step S104, the log template prompts the large language model to analyze and process the operation log data, extract variable information of the operation log data, and analyze the variable information of the operation log data to obtain the analysis result of the operation log data. The log analysis method of the present application utilizes the understanding and context reasoning capabilities of the large language model to complete the template parsing and content parsing of the log.
[0042] The log analysis method of the embodiment of the present application contains a log template library that presets and associates log templates and category vectors of each log type, and the content of the log template does not contain variable information. The context reasoning ability of the large language model is further utilized to infer the variable information in the log data, thereby completing the analysis of the log data, achieving the goal of not requiring manual preset modes and templates, greatly reducing the difficulty of operation and maintenance work, and improving the efficiency of server monitoring.
[0043] In some embodiments, before obtaining the operation log data and extracting the embedding vector of the log data, the method further includes:
[0044] Obtain the sample data of the log to be analyzed, and extract the embedding vector of the log sample to be analyzed. For example, the historical log data can be used as the log sample data, and the embedding vector of the log sample to be analyzed can be extracted through the embedding model. It is also possible to establish a channel to communicate with the built-in log API of the server during the system trial operation to collect logs. First, perform a preliminary screening of the logs, retain important logs such as warnings and errors, including system logs, application logs, security logs, access logs, performance logs and other types, and then call the local embedded model to obtain the embedding vector of the log.
[0045] The extracted embedding vectors are clustered to determine several log categories and the category vectors corresponding to each log category. In a specific example, the logs can be classified by setting a suitable threshold based on the cosine similarity of the embedding vectors. After classification, each category includes multiple logs. We can calculate the vectors of these logs and then calculate the average value. The average vector is the vector corresponding to this category and can be used for subsequent retrieval.
[0046] Select multiple log samples to be processed from each of the log categories, and use the large language model to eliminate the variable information in the multiple log samples to be processed, and then extract the category template corresponding to the log category. In some examples, in the template library construction stage, after using the hash table to remove duplicates in each category, random selection is performed to select a small number of samples (3 to 5). The variables referred to in the examples of this application are mainly some variables. For example, in the alarm log, there will be an alarm if the resource occupancy rate is high. In one alarm log, the CPU occupies 95%, and in another log, the hard disk occupies 99%. Then the CPU, hard disk, 95% and 99% are all variables, and each log may be different. Additional variables may be misleading to the large model (with less performance), and the large model with stronger performance has higher fault tolerance. The embodiment of the present application eliminates the variable information in multiple log samples to be processed through the large language model, so that the general log category template of the corresponding category can be extracted, making the category template more referenceable.
[0047] In some embodiments, selecting a plurality of log samples to be processed from each of the log categories, eliminating variable information in the plurality of log samples to be processed by the large language model, and then extracting a category template corresponding to the log category includes:
[0048] Pre-configured demonstration use cases.
[0049] Based on the demonstration case and the selected log samples to be processed, the large language model is input, and the variable information in the multiple log samples to be processed is eliminated by the large language model to extract the category template corresponding to the log category. Specifically, the demonstration case and the log samples to be processed can be input into the large language model together, so as to instruct the large model to replace the variable part that will change in the log with, for example, a placeholder. <blank>Replace, thereby extracting the category template corresponding to the log category.
[0050] In some embodiments, based on the exemplary use case and the selected log sample to be processed, inputting the large language model includes:
[0051] The selected log samples to be processed are deduplicated using a hash table.
[0052] The log samples to be processed that are deduplicated according to the pre-configured demonstration use cases and the hash table are input into the large language model to instruct the large language model to propose the changed variables in the log samples to be processed.
[0053] In some embodiments, it also includes:
[0054] The basic information of the category template extracted by the large language model and the embedded vector corresponding to the type of log are encapsulated as metadata to construct the preset log template library. In a specific example, the large language model outputs a category template that does not contain variables corresponding to the log category, including a brief introduction, combined with basic information such as date and system module. The basic information of the category template and the embedded vector corresponding to the type of log are used as metadata, and the template is standardized and encapsulated with the above information, and persistently stored in the database. The category template can be retrieved and extracted by the embedded vector.
[0055] In some embodiments, the large language model is prompted by the log template to analyze and process the operation log data, and the variable information of the operation log data is extracted, including:
[0056] The template entity of the log template and the metadata contained in the log template are used as prompt words and input into the large language model to extract the variable information of the operation log data. Specifically, when a new log is read, vectorization calculation is first performed through the embedded model to extract the embedded vector of the new log, and then based on the idea of the RAG algorithm, it is retrieved from the database. If the similarity with the nearest class vector is greater than a threshold, it is classified into the log category. And further extract the corresponding category template from the preset log template library, use the metadata of the category template and the template entity as part of the prompt word, input into the large language model, use the contextual reasoning of the large language model to extract the key variable information of the operation log data, and perform problem analysis and report generation.
[0057] In some embodiments, when a log template corresponding to the log category cannot be obtained in a preset log template library:
[0058] The embedded vector of the extracted log data is used to retrieve a temporary class in a preset log template library, wherein the temporary class contains log data that cannot be clustered based on the preset log template library. Based on the above example, further, if the similarity between the new log vector and the closest class vector is less than a threshold, a notification is sent to the administrator, and if it is determined to be a new log category, it is stored in the database, and the new log template is subsequently identified and stored; otherwise, the threshold is adjusted to ensure correct identification.
[0059] In order to reduce the inefficiency caused by the emergence of new log categories and the system's inability to recognize log templates, this embodiment further uses a large language model to regularly update the temporary classes of the database. When the number of log data contained in the temporary class meets the clustering requirements, the category template corresponding to the log category is extracted based on the large language model. For example, when the number of logs in the temporary class is greater than a certain threshold (for example: 5), these log samples are input into the large language model for template extraction. After the extracted template is encapsulated, it is saved as a log template and deleted from the temporary class.
[0060] The log analysis method of the embodiment of the present application utilizes the primary reasoning ability of the large language model, and obtains log meta information by streaming and parsing the status information of the key components of the server and the alarm log. The log meta information mainly includes vectors obtained based on the embedded model. Similarity analysis is performed on the log vectors, and template classification is performed. For each template class, the large language model is used to extract the template, and the template and log meta information are standardized and encapsulated to obtain a log template library. Through vector retrieval, the large language model promptly analyzes the fault logs of various components such as the CPU, memory, and hard disk, and completes the report. By optimizing the parsing results of the large model, the method of the present application can achieve efficient monitoring of the server.
[0061] The method of the present application uses an embedded model to parse the characteristics and semantics of log data, so as to more accurately extract valuable information from it and build a template library. In daily operation, log parsing is completed through a large language model and vector retrieval technology. The invention can automatically adapt to changes in logs, define standardized encapsulation of templates, and complete the construction and update of template libraries without manually presetting modes and templates, greatly reducing the difficulty of operation and maintenance work. Compared with the end-to-end solution of a large language model. The present application has better interpretability, builds a natural thinking chain structure, and is more convenient for subsequent manual review and modification. The method of the present application does not involve regular expression matching, so the large model does not need to strictly comply with the standard of regular expressions when processing logs, and the formatted output requirements of the large model are relatively low. The method of the present application can not only process existing log data, but also cope well with unknown log data, and has strong adaptability.
[0062] An embodiment of the present application also proposes a log analysis system based on a large language model, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the log analysis method based on the large language model as described above are implemented.
[0063] The embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log analysis method based on a large language model as described above are implemented.
[0064] It should be noted that in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0065] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0067] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.< / blank>
Claims
1. A log analysis method based on a large language model, characterized in that: include: Obtaining operation log data, and extracting embedding vectors of the log data; Matching the embedding vector of the operation log with the category vector of each log category in a preset log template library to determine the log category that matches the operation log data; Obtaining a log template corresponding to the log category in a preset log template library; wherein, in the preset log template library, the log template of each log category corresponds to and is associated with the category vector one by one, and the content of the log template does not contain variable information; Prompting the large language model to analyze and process the operation log data through the log template, extracting variable information of the operation log data, and analyzing the variable information of the operation log data to obtain an analysis result of the operation log data; Before obtaining the operation log data and extracting the embedding vector of the log data, the method further includes: Obtaining log sample data to be analyzed, and extracting an embedding vector of the log sample to be analyzed; Clustering the extracted embedding vectors to determine a number of log categories and a category vector corresponding to each of the log categories; Selecting a plurality of log samples to be processed from each of the log categories, eliminating variable information in the plurality of log samples to be processed by using the large language model, and then extracting a log template corresponding to the log category; Selecting a plurality of log samples to be processed from each of the log categories, eliminating variable information in the plurality of log samples to be processed by the large language model, and then extracting a log template corresponding to the log category includes: Pre-configured demonstration use cases; Based on the demonstration use case and the selected log samples to be processed, the large language model is input, and the variable information in the multiple log samples to be processed is eliminated using the large language model to extract a log template corresponding to the log category.
2. The log analysis method based on a large language model as claimed in claim 1, characterized in that: Extracting the embedding vector of the log sample to be analyzed is achieved through an embedded model; The extracted embedding vectors are clustered based on the cosine similarity of the embedding vectors.
3. The log analysis method based on a large language model as claimed in claim 1, characterized in that: Based on the exemplary use case and the selected log sample to be processed, inputting the large language model includes: Use the hash table to remove duplicates from the selected log samples to be processed; The log samples to be processed that are deduplicated according to the pre-configured demonstration use cases and the hash table are input into the large language model to instruct the large language model to propose the changed variables in the log samples to be processed.
4. The log analysis method based on a large language model as claimed in claim 1, characterized in that: Also includes: The basic information of the log template extracted by the large language model and the embedded vector corresponding to the type of log are encapsulated as metadata to construct the preset log template library.
5. The log analysis method based on a large language model as claimed in claim 4, characterized in that: Prompting the large language model to analyze and process the operation log data through the log template, and extracting variable information of the operation log data includes: The entity of the log template and the metadata contained in the log template are used as prompt words and input into the large language model to extract variable information of the operation log data.
6. The log analysis method based on a large language model as claimed in claim 1, characterized in that: It also includes the case where the log template corresponding to the log category cannot be obtained in the preset log template library: Retrieving a temporary class in a preset log template library by using the embedded vector extracted from the log data, wherein the temporary class includes log data that cannot be clustered based on the preset log template library; When the amount of log data included in the temporary class meets the clustering requirement, a log template corresponding to the log category is extracted based on the large language model.
7. A log analysis system based on a large language model, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the log analysis method based on a large language model as described in any one of claims 1 to 6 are implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log analysis method based on a large language model as described in any one of claims 1 to 6 are implemented.
Citation Information
Cited By
Intelligent early warning and decision making system and method based on large model driving
CN120934814A
Dynamic adaptive signaling log analysis method based on large language model
CN121303349A
