An information system log anomaly detection method, system, medium and processor

Abstract

The application discloses an information system log anomaly detection method and system, a medium and a processor, relates to the technical field of information system log analysis, and solves the problem of insufficient reliability of output results caused by lack of vertical field knowledge and self-checking mechanism when directly applying a large language model for log analysis and anomaly detection in the prior art. The application injects vertical field knowledge into the large language model to suppress hallucinations by constructing an enterprise private log knowledge base and introducing a search enhancement generation mechanism. At the same time, a multi-agent collaborative analysis framework is adopted to realize self-correction through a division pipeline of analysis, extraction, verification, understanding, detection and verification, and the search quality is improved by combining reordering and hit rate screening, so that high accuracy, high reliability and continuous evolution of the knowledge base are finally realized in log template automatic extraction and anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of information system log analysis technology, and in particular to an information system log anomaly detection method, system, medium and processor. Background Technology

[0002] As enterprises deepen their digital transformation, information system architectures are becoming increasingly complex, encompassing diverse technology stacks such as cloud computing, microservices, and containerization. Log data is exhibiting characteristics of massive volume explosions, high-frequency writes, and multi-source heterogeneity. In critical industry operation and maintenance scenarios such as power dispatching, financial transactions, telecommunications operations, and the industrial internet, logs, as digital carriers recording key system operation events, carry business flow trajectories, user operation behaviors, system error alarms, performance fluctuations, and security access traces. They are core data assets for operation and maintenance personnel to conduct real-time monitoring, fault root cause localization, and anomaly detection. Faced with daily log increments of TB or even PB levels, traditional methods relying on manually written regular expressions and line-by-line auditing are no longer sustainable in terms of coverage, response speed, and accuracy. Intelligent operation and maintenance has become an inevitable choice for industry transformation. In recent years, large language models, with their powerful semantic understanding, logical reasoning, and long contextual association capabilities, have opened up new paths for the automated parsing and anomaly detection of semi-structured logs. The industry has widely attempted to directly apply general-purpose pre-trained large language models to log analysis, hoping to leverage their learned language rules to achieve unknown template extraction, semantic understanding, and anomaly identification, thereby reducing reliance on expert experience and labeled data.

[0003] However, existing technologies suffer from insufficient reliability of output results when directly applying large language models for log parsing and anomaly detection. The training corpora of general-purpose large language models primarily originate from publicly available internet text, inherently lacking vertical domain knowledge specific to enterprise private systems, including unique log format specifications, internal business logic semantics, the meaning of private protocol fields, and historical fault evolution patterns. When faced with enterprise-specific log templates or anomaly sequences, the model cannot make judgments based on accurate domain knowledge, often relying on subjective assumptions based on general language rules, resulting in factually incorrect outputs—a typical hallucination phenomenon. More critically, existing solutions mostly employ a single-step end-to-end inference model to directly output results, lacking mechanisms for analyzing, extracting, and verifying intermediate steps, and failing to effectively self-verify and cross-correct conclusions. In high-risk scenarios such as power, finance, and telecommunications, this black-box direct output mechanism significantly increases the risk of false positives and false negatives, severely hindering the engineering implementation of large language models in the field of log anomaly detection.

[0004] Therefore, there is a need for a method, system, medium, and processor for detecting anomalies in information system logs. Summary of the Invention

[0005] To address the problem of insufficient reliability of output results when directly applying large language models for log parsing and anomaly detection in existing technologies due to the lack of vertical domain knowledge and self-verification mechanisms, this invention provides a method, system, medium, and processor for information system log anomaly detection. It can automatically construct and iteratively update a log template knowledge base and a log sequence knowledge base, and improve the accuracy and practicality of log anomaly detection through collaborative analysis using large language models. The specific technical solution is as follows: A method for detecting anomalies in information system logs includes the following steps: Collect existing log templates and tagged log sequences from enterprises to build a raw log template library and a raw sequence library, respectively. Then, vectorize the log templates and log sequences to form a template vector library and a sequence vector library. Input the set of logs to be analyzed, parse them using the original log template library, and output the log parsing results directly for successfully parsed logs; for logs that fail to be parsed, vectorize them and retrieve candidate templates from the template vector library and filter them based on similarity. Then, use a multi-agent collaborative analysis framework based on a large language model to analyze, extract and verify the filtered logs that failed to be parsed in sequence, obtain log templates and update them to the original template library and template vector library, and obtain the log parsing results. The log parsing results are sorted by time and reconstructed into a log sequence through a sliding time window. The vectorized log sequence is then used to retrieve candidate sequences from a sequence vector library. After filtering based on similarity and hit rate, the filtered log sequence is anomaly detected and verified using a multi-agent collaborative analysis framework built on a large language model. The anomaly detection results are then output.

[0006] Preferably, constructing the template vector library includes: Collect existing log templates from within the enterprise to build an original log template library: in, This is a log template; Using a pre-trained text embedding model, all log templates in the original log template library are converted into vectors, resulting in vectorized log templates. All vectorized log templates constitute a log template vector library. .

[0007] Preferably, constructing the sequence vector library includes: Collect existing tagged historical log sequence data from enterprises to build a raw log sequence library: in, For log sequences, The parsed log. This is the list of parameters for the log. For parameter values, For log sequences Tags; Using a pre-trained text embedding model, the log sequences in the original log sequence library are converted into vectors, resulting in vectorized log sequences: in for The vectorized representation of, This is the vectorized list of log parameters. These are the vectorized parameter values; the vectorized log sequences and their labels form a tuple, and all tuples consisting of log sequences and their labels constitute the log sequence vector library. .

[0008] Preferably, the logs of failed parsing after filtering are obtained, including: Input the collected log set to be analyzed ,in This is a log entry to be analyzed. Using log templates from the original log template library, All logs to be analyzed are parsed: For If the parsing is successful, output the parsing result. Otherwise, after vectorizing it, retrieve the top K log templates with high semantic similarity to the log from the template vector library to form a candidate log template vector list. ; Using a pre-trained context-delayed interaction model and The semantic similarity of all candidate log templates in the dataset is calculated, and L similar templates are retrieved. Candidate log templates with high semantic similarity are used to form a reordered list of candidate log template vectors. ; Retrieved from the original log template library The original log template corresponding to each log template vector constitutes a candidate log template list. .

[0009] Preferably, for the filtered logs that failed to be parsed, the log parsing results are obtained, including: A multi-agent collaborative analysis framework based on a large language model is constructed, including a log template analysis agent, a log template extraction agent, and a log template verification agent; logs that failed to be parsed and a list of candidate log templates are input into the multi-agent collaborative analysis framework to extract log templates and parse them; First, prompt words are constructed, and a log template analysis agent is used to analyze each log template in the candidate log template list sequentially, providing analysis results. Then, the analysis results of all candidate log templates provided by the log template analysis agent are input into the log template extraction agent. This agent receives the analysis results and the log to be analyzed as input, constructs prompt words to analyze the log, and extracts candidate log templates based on the analysis results of the candidate log templates and the log to be analyzed. Next, the extracted candidate log templates are input into the log template verification agent, establishing a thought chain. All candidate log templates are used to parse the log to be analyzed, verifying the correctness of the extracted log templates, extracting the best log template, and outputting the parsing result. Simultaneously, the best log template is added to the original template library and then vectorized and added to the template vector library. All successfully parsed log results are combined into a log parsing result set and output. The parsing result of each log includes the parsing result of direct successful parsing and the parsing result of successful parsing through the multi-agent collaborative analysis framework.

[0010] Preferably, the filtered log sequence is obtained, including: Get a collection of log parsing results ,right The log parsing results are sorted in chronological order to obtain a sorted log parsing result sequence. ; Set sliding time window ,from Starting with the first log parsing result, a sliding time window is used to extract the log parsing result sequence, forming a log parsing result sequence set. ,in This is a sequence of log parsing results; After vectorizing the log parsing result sequence set, the top K log vector sequences with high similarity to the log parsing result sequence are retrieved from the sequence vector library to obtain the candidate log vector sequence set. ; Further calculate the similarity between the log parsing result sequence and the corresponding logs in the candidate log vector sequence, set a similarity threshold, and consider the corresponding logs to be similar when the similarity is greater than or equal to the threshold. Then, increment the hit rate of the candidate log vector sequence on the log parsing result sequence by one, where the initial value of the hit rate is 0. Finally, select the L candidate log sequences with the highest hit rates as the filtered log sequences.

[0011] Preferably, obtaining anomaly detection results includes: A multi-agent collaborative analysis framework based on a large language model is constructed, including a log sequence understanding agent, a log sequence anomaly detection agent, and a detection result verification agent; the filtered log sequences and candidate log sequences are input into the multi-agent collaborative analysis framework; First, prompt words are constructed, and a log sequence understanding agent is used to analyze candidate log sequences to obtain analysis results. All analysis results constitute the output of the log sequence understanding agent. Next, the output of the log sequence understanding agent and the filtered log sequences are input into a log sequence anomaly detection agent to construct prompt words. Based on the analysis results of the log sequence understanding agent on the candidate log sequences, anomaly detection is performed on the filtered log sequences, and the anomaly detection results are output. Then, the anomaly detection results are used as input to a detection result verification agent to construct a thought chain, verify the anomaly detection results, and output the verified anomaly detection results.

[0012] An information system log anomaly detection system, applied to the method described above, includes: Database construction module: Collect existing log templates and tagged log sequences from the enterprise to build a raw log template library and a raw sequence library respectively, and vectorize the log templates and log sequences to form a template vector library and a sequence vector library respectively; Log parsing module: Input the set of logs to be analyzed, parse them using the original log template library, and directly output the log parsing results for successfully parsed logs; for logs that fail to be parsed, vectorize them and retrieve candidate templates from the template vector library and filter them based on similarity. Then, use a multi-agent collaborative analysis framework built on a large language model to analyze, extract and verify the filtered logs that failed to be parsed in sequence, obtain log templates and update them to the original template library and template vector library, and obtain the log parsing results; Anomaly detection module: Sorts the log parsing results by time and reconstructs them into a log sequence through a sliding time window. It retrieves candidate sequences from the sequence vector library after vectorizing the log sequence, filters them based on similarity and hit rate, and uses a multi-agent collaborative analysis framework based on a large language model to detect and verify anomalies in the filtered log sequence, and outputs the anomaly detection results.

[0013] A computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the information system log anomaly detection method as described above.

[0014] A processor for running a program, wherein the program executes the information system log anomaly detection method as described above.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention first collects existing log templates and tagged historical log sequences from enterprises to construct an original template library, an original sequence library, and their corresponding vector libraries, forming a vertical domain knowledge foundation specific to the enterprise. During the parsing and detection phases, vector retrieval recalls highly similar candidate templates or historical sequences from the private knowledge base, providing accurate domain context references to the large language model through retrieval-enhanced generation, effectively suppressing the illusion caused by the model's lack of private knowledge. Secondly, the solution abandons the traditional single-step LLM direct output mode and innovatively constructs a multi-agent collaborative analysis framework based on the large language model: on the log parsing side, a three-agent pipeline sequentially extracts templates and performs cross-validation on unknown logs; on the anomaly detection side, a three-agent pipeline sequentially performs behavioral judgment and result verification on log sequences, achieving self-correction through task decomposition and role division. Simultaneously, the solution introduces a re-ranking and hit rate filtering mechanism in the retrieval phase, ensuring that the candidate knowledge input to the large language model is highly relevant through fine-grained similarity calculation, avoiding noise interference in the reasoning process. Furthermore, newly extracted log templates are automatically updated to the original template library and vector library, enabling the knowledge base to evolve iteratively. Through these technical means, this solution significantly reduces the risk of false positives and false negatives while ensuring detection efficiency, achieving highly reliable implementation of large language models in enterprise log anomaly detection scenarios. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0017] Figure 1 This is a schematic diagram of the process for detecting log anomalies in an information system based on multi-agent collaboration provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] Example 1 In one embodiment of the present invention, a method for detecting anomalies in information system logs is provided, such as... Figure 1 As shown, the process includes four stages: log template knowledge base construction, log sequence knowledge base construction, log parsing, and log sequence anomaly detection. In the log template knowledge base construction stage, existing log templates within the enterprise are collected to build an initial log template library. A pre-trained text embedding model is then used to vectorize the collected log templates, creating a log template vector library. In the log sequence knowledge base construction stage, existing labeled historical log sequence data from the enterprise is collected to build an initial log sequence library. A pre-trained text embedding model is then used to vectorize the log sequences, creating a log sequence vector library. In the log parsing stage, log templates from the log template library are first used to parse the logs in the input log set. If parsing is successful, the parsing result is given; otherwise, a multi-agent collaborative analysis framework based on LLM is built to analyze logs that cannot be parsed, extract log templates, and provide parsing results. Simultaneously, newly extracted log templates are updated in both the log template library and the log template vector library. In the log sequence anomaly detection phase, a time window is designed to reconstruct the log parsing result set output from the log parsing phase into a log sequence. A multi-agent collaborative analysis framework is built based on LLM to analyze the abnormal behavior of the log sequence and provide an analysis report of the abnormal log sequence.

[0023] Specifically, this method includes the following steps: S1. Log Template Knowledge Base Construction: Collect existing log templates from the enterprise to build an initial log template library; further, use a pre-trained text embedding model to vectorize the log templates in the initial log template library to build a log template vector library; details are as follows: S11. Collect existing log templates within the enterprise to build an original log template library. ,in This is a log template; S12. Using a pre-trained text embedding model, convert all log templates in the original log template library into vectors to obtain vectorized log templates. The pre-trained text embedding model is Qwen3-Embedding-0.6B. By inputting the log template into the Qwen3-Embedding-0.6B model, the model outputs the corresponding embedding vector, as follows: in, This represents the pre-trained Qwen3-Embedding-0.6B model.

[0024] All vectorized log templates constitute a log template vector library. .

[0025] S2. Log Sequence Knowledge Base Construction: Collect existing tagged log sequence data from the enterprise to build an original log sequence library; further, use a pre-trained text embedding model to convert the log sequences in the original log sequence library into vectors to build a log sequence vector library. This includes the following operations: S21. Collect existing tagged historical log sequence data from the enterprise to build a raw log sequence library. ,in For log sequences, The parsed log. This is the list of parameters for the log. For parameter values, For log sequences Tags; S22. Using a pre-trained text embedding model, the log sequences in the original log sequence library are converted into vectors to obtain vectorized log sequences. ,in for The vectorized representation of, This is the vectorized list of log parameters. These are the vectorized parameter values; the pre-trained text embedding model is Qwen3-Embedding-0.6B, which is formed by inputting each parsed log from the log sequence into the Qwen3-Embedding-0.6B model, and the model outputs the corresponding embedding vector; the log vectorization method is as follows: First, the parsed logs log template Inputting into the Qwen3-Embedding-0.6B model yields a vectorized log template. ,as follows: Then, the parsed logs The parameter values ​​from the parameter list are concatenated and then input into the Qwen3-Embedding-0.6B model to obtain a vectorized representation of the parameter list. ,as follows: in, The operator represents the concatenation of parameter values. Used as a separator between log parameters.

[0026] The vectorized log sequences and their labels form a tuple, and all tuples of log sequences and their labels constitute the log sequence vector library. .

[0027] S3. Log Parsing: Input the set of logs to be analyzed. First, use the log templates in the original log template library built in S1 to parse the logs in the log set and obtain the log parsing results. For logs that fail to be parsed, build a multi-agent collaborative analysis framework based on LLM to analyze the logs that fail to be parsed, extract the log templates, update the original log template library and log template vector library built in S1, and obtain the log parsing results. Specifically, the following operations are included: S31. Input the collected log set to be analyzed. ,in This is a log entry to be analyzed. It uses a raw log template library built with S11. The log template in the file, for All logs to be analyzed are parsed. The log parsing method is as follows: In turn Each log entry in the log is parsed: for If the parsing is successful, output the parsing result. Otherwise, execute S32; S32. Use a pre-trained text embedding model to resolve logs that failed to be parsed. Convert to log vector The pre-trained text embedding model is Qwen3-Embedding-0.6B, which is achieved by... When input into Qwen3-Embedding-0.6B, the model outputs the corresponding vectorized representation, as follows: Then, the log template vector library built from S12 The top-K log templates with high semantic similarity to the given log are retrieved and used to form a candidate log template vector list. K is user-defined. Candidate log templates in the middle are based on their relationship with The semantic similarity is sorted from high to low; the semantic similarity is quantified by the cosine similarity between the log template vector and the log vector to be analyzed, denoted as . The formula is as follows: in, for The length of is denoted as ,and equal.

[0028] Furthermore, a pre-trained context-delayed interaction model is used to... and The semantic similarity of all candidate log templates is calculated; the pre-trained context-delayed interaction model is ColBERTv2, and the specific semantic similarity calculation method is as follows: First, to improve the efficiency of semantic similarity calculation, all candidate log templates need to be pre-input into the pre-trained ColBERTv2 model to build an index; Then, Using L as parameters of the ColBERTv2 model, retrieve L values ​​related to... Candidate log templates with high semantic similarity are used to form a reordered list of candidate log template vectors. ,in Customizable by the user; Finally, Searched in The original log template corresponding to each log template vector constitutes a candidate log template list. .

[0029] S33. A multi-agent collaborative analysis framework based on LLM is constructed, including: a log template analysis agent, a log template extraction agent, and a log template verification agent; the LLM is a Qwen3-Max model, which will handle failed parsing... and generated by S32 Input into a multi-agent collaborative analysis framework based on LLM, and extract... The log template is generated and parsed.

[0030] The above log template extraction and parsing process specifically includes the following operations: First, construct prompt words. The log template analysis agent sequentially processes the reordered candidate log template list. Analyze each log template in the log; for Provide analysis results The prompt words The template is shown below: Then, the analysis results of all candidate log templates provided by the log template analysis agent are used. Extract the agent from the log template.

[0031] In the log template extraction agent, the receiver... Logs to be analyzed As input, construct prompt words. right The analysis was performed, and the results were obtained. The prompt words The template is shown below: Next, construct prompt words. ,in accordance with and Extract N Candidate log templates, composed of Candidate log template set , This is a candidate log template, where N is user-defined; the prompt words... The template is as follows: Will Input into the log template verification agent.

[0032] Establish thought chains in the log template validation agent. ,use All candidate log templates respectively The extracted log template is parsed and verified for correctness; the thought chain... The template is as follows: Through thought chain for The extracted template is denoted as , The output is The analysis result is denoted as At the same time, Join In the process, following the S12 operation procedure, the pre-trained text embedding model Qwen3-Embedding-0.6B is used to embed the text. Convert to a vector and add to middle.

[0033] S34. All successfully parsed log results constitute the log parsing result set. ,in For the input to S31 The parsing results of each log entry. Finally, output the log parsing result set. .

[0034] S4. Log Sequence Anomaly Detection: Input a successfully parsed log set and reconstruct it into a log sequence in the time dimension; construct a multi-agent collaborative analysis framework based on LLM, and perform anomaly detection on the reconstructed log sequence based on the original log sequence library and log sequence vector library built in S2.

[0035] Specifically, the following operations are included: S41. Receive the log parsing result set output by S34. ,right The log parsing results are sorted in chronological order to obtain a sorted log parsing result sequence. Set sliding time window ,from Starting with the first log parsing result, a sliding time window is used to extract the log parsing result sequence, forming a log parsing result sequence set. ,in A sequence of log parsing results, consisting of a set of logs falling within a time window. Log parsing results The items are arranged in ascending order according to their chronological order.

[0036] S42. Using a pre-trained text embedding model to... The log parsing results of each log parsing result sequence are converted into vectors to obtain a set of log parsing result vector sequences. ,in , For log parsing results The vectorized representation of, This serves as the separator between the log template vector and the parameter vector. The separator between the log parsing result vectors; the pre-trained text embedding model is Qwen3-Embedding-0.6B, which is achieved by... of Inputting into the Qwen3-Embedding-0.6B model yields... ,Will Each element is concatenated in sequence and input into the Qwen3-Embedding-0.6B model to obtain... The The parameter concatenation method is as follows: in, Used as a separator between parameters.

[0037] Will Inputting it into the Qwen3-Embedding-0.6B model yields... ,as follows: Then, built from S22 Retrieved Top-K results Log vector sequences with high similarity are used to obtain a set of candidate log vector sequences. ,in for The Middle A sequence of log vectors ,for The Middle The parsing results of each log.

[0038] Furthermore, a pre-trained text re-ranking model is used to calculate... middle and middle of The similarity between them is denoted as The pre-trained text re-ranking model is Qwen3-Reranker-0.6B, which is achieved by... and After concatenation, the data is input into the pre-trained Qwen3-Reranker-0.6B model, and the model outputs... User-defined similarity threshold ,when At that time, it was believed and middle of They are similar, therefore right Hit rate Add one, of which The initial value is 0. Finally, the L candidate log sequences with the highest hit rate are selected as... The candidate log sequence, input S43; S43. A multi-agent collaborative analysis framework based on LLM is constructed, including: a log sequence understanding agent, a log sequence anomaly detection agent, and a detection result verification agent; the LLM is the Qwen3-Max model. The L candidate log sequences with the highest hit rates are used as input to construct prompt words. For candidate log sequences The analysis was performed, and the results were obtained. The prompt words The template is as follows: All The log sequence is used to understand the agent's output.

[0039] Next, the output of the log sequence understanding agent and Input log sequence anomaly detection agent, construct prompt words Based on the analysis results of the log sequence understanding agent on high-hit log sequences, for Perform anomaly detection and output the anomaly detection results. The prompt words The template is as follows: The output is used as the input to verify the detection results of the agent.

[0040] In the intelligent agent that verifies the detection results, a thought chain is constructed. ,right The test results were verified; the thought chain The template is as follows: Example 2 In one embodiment of the present invention, an information system log anomaly detection system is provided, comprising: Database construction module: Collect existing log templates and tagged log sequences from the enterprise to build a raw log template library and a raw sequence library respectively, and vectorize the log templates and log sequences to form a template vector library and a sequence vector library respectively; Log parsing module: Input the set of logs to be analyzed, parse them using the original log template library, and directly output the log parsing results for successfully parsed logs; for logs that fail to be parsed, vectorize them and retrieve candidate templates from the template vector library and filter them based on similarity. Then, use a multi-agent collaborative analysis framework built on a large language model to analyze, extract and verify the filtered logs that failed to be parsed in sequence, obtain log templates and update them to the original template library and template vector library, and obtain the log parsing results; Anomaly detection module: Sorts the log parsing results by time and reconstructs them into a log sequence through a sliding time window. It retrieves candidate sequences from the sequence vector library after vectorizing the log sequence, filters them based on similarity and hit rate, and uses a multi-agent collaborative analysis framework based on a large language model to detect and verify anomalies in the filtered log sequence, and outputs the anomaly detection results.

[0041] This invention also provides a computer-readable storage medium, which includes a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute the above-described information system log anomaly detection method.

[0042] This invention also provides a processor for running a program, wherein the program executes the above-described information system log anomaly detection method during runtime.

[0043] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0044] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical functional division. In actual implementation, there may be other division methods, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored.

[0045] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0046] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for detecting anomalies in information system logs, characterized in that, Includes the following steps: Collect existing log templates and tagged log sequences from enterprises to build a raw log template library and a raw sequence library, respectively. Then, vectorize the log templates and log sequences to form a template vector library and a sequence vector library. Input the set of logs to be analyzed, parse them using the original log template library, and output the log parsing results directly for successfully parsed logs; for logs that fail to be parsed, vectorize them and retrieve candidate templates from the template vector library, and filter them based on similarity to obtain candidate log templates. Then, use a multi-agent collaborative analysis framework based on a large language model to analyze, extract and verify the candidate log templates in sequence, obtain log templates and update them to the original template library and template vector library, and obtain the log parsing results. The log parsing results are sorted by time and reconstructed into a log sequence through a sliding time window. The vectorized log sequence is then used to retrieve candidate sequences from a sequence vector library. After filtering based on similarity and hit rate, the filtered log sequence is anomaly detected and verified using a multi-agent collaborative analysis framework built on a large language model. The anomaly detection results are then output.

2. The method for detecting anomalies in information system logs according to claim 1, characterized in that, Constructing the template vector library includes: Collect existing log templates from within the enterprise to build an original log template library: in, This is a log template; Using a pre-trained text embedding model, all log templates in the original log template library are converted into vectors, resulting in vectorized log templates. All vectorized log templates constitute a log template vector library. .

3. The method for detecting anomalies in information system logs according to claim 1, characterized in that, Constructing the sequence vector library includes: Collect existing tagged historical log sequence data from enterprises to build a raw log sequence library: in, For log sequences, The parsed log. This is the list of parameters for the log. For parameter values, For log sequences Tags; Using a pre-trained text embedding model, the log sequences in the original log sequence library are converted into vectors, resulting in vectorized log sequences: in for The vectorized representation of, This is the vectorized list of log parameters. These are the vectorized parameter values; the vectorized log sequences and their labels form a tuple, and all the tuples consisting of log sequences and their labels constitute the log sequence vector library. .

4. The method for detecting anomalies in information system logs according to claim 1, characterized in that, Retrieve candidate log templates, including: Input the collected log set to be analyzed ,in This is a log entry to be analyzed. Using log templates from the original log template library, All logs to be analyzed are parsed: For If the parsing is successful, output the parsing result. Otherwise, after vectorizing it, retrieve the top K log templates with high semantic similarity to the log from the template vector library to form a candidate log template vector list. ; Using a pre-trained context-delayed interaction model and The semantic similarity of all candidate log templates in the dataset is calculated, and L similar templates are retrieved. Candidate log templates with high semantic similarity are used to form a reordered list of candidate log template vectors. ; Retrieved from the original log template library The original log template corresponding to each log template vector constitutes a candidate log template list. .

5. The method for detecting anomalies in information system logs according to claim 4, characterized in that, For candidate log templates, obtain the log parsing results, including: A multi-agent collaborative analysis framework based on a large language model is constructed, including a log template analysis agent, a log template extraction agent, and a log template verification agent; logs that failed to be parsed and a list of candidate log templates are input into the multi-agent collaborative analysis framework to extract log templates and parse them; First, prompt words are constructed, and a log template analysis agent is used to analyze each log template in the candidate log template list sequentially, providing analysis results. Then, the analysis results of all candidate log templates provided by the log template analysis agent are input into the log template extraction agent. This agent receives the analysis results and the log to be analyzed as input, constructs prompt words to analyze the log, and extracts candidate log templates based on the analysis results of the candidate log templates and the log to be analyzed. Next, the extracted candidate log templates are input into the log template verification agent, establishing a thought chain. All candidate log templates are used to parse the log to be analyzed, verifying the correctness of the extracted log templates, extracting the best log template, and outputting the parsing result. Simultaneously, the best log template is added to the original template library and then vectorized and added to the template vector library. All successfully parsed log results are combined into a log parsing result set and output. The parsing result of each log includes the parsing result of direct successful parsing and the parsing result of successful parsing through the multi-agent collaborative analysis framework.

6. The method for detecting anomalies in information system logs according to claim 1, characterized in that, Retrieve the filtered log sequence, including: Get a collection of log parsing results ,right The log parsing results are sorted in chronological order to obtain a sorted log parsing result sequence. ; Set sliding time window ,from Starting with the first log parsing result, a sliding time window is used to extract the log parsing result sequence, forming a log parsing result sequence set. ,in This is a sequence of log parsing results; After vectorizing the log parsing result sequence set, the top K log vector sequences with high similarity to the log parsing result sequence are retrieved from the sequence vector library to obtain the candidate log vector sequence set. ; Further calculate the similarity between the log parsing result sequence and the corresponding logs in the candidate log vector sequence, set a similarity threshold, and consider the corresponding logs to be similar when the similarity is greater than or equal to the threshold. Then, increment the hit rate of the candidate log vector sequence on the log parsing result sequence by one, where the initial value of the hit rate is 0. Finally, select the L candidate log sequences with the highest hit rates as the filtered log sequences.

7. The method for detecting anomalies in information system logs according to claim 1, characterized in that, Obtain anomaly detection results, including: A multi-agent collaborative analysis framework based on a large language model is constructed, including a log sequence understanding agent, a log sequence anomaly detection agent, and a detection result verification agent; the filtered log sequences and candidate log sequences are input into the multi-agent collaborative analysis framework; First, prompt words are constructed, and a log sequence understanding agent is used to analyze candidate log sequences to obtain analysis results. All analysis results constitute the output of the log sequence understanding agent. Next, the output of the log sequence understanding agent and the filtered log sequences are input into a log sequence anomaly detection agent to construct prompt words. Based on the analysis results of the log sequence understanding agent on the candidate log sequences, anomaly detection is performed on the filtered log sequences, and the anomaly detection results are output. Then, the anomaly detection results are used as input to a detection result verification agent to construct a thought chain, verify the anomaly detection results, and output the verified anomaly detection results.

8. An information system log anomaly detection system, characterized in that, The method applied to any one of claims 1 to 7 includes: Database construction module: Collect existing log templates and tagged log sequences from the enterprise to build a raw log template library and a raw sequence library respectively, and vectorize the log templates and log sequences to form a template vector library and a sequence vector library respectively; Log parsing module: Input the set of logs to be analyzed, parse them using the original log template library, and directly output the log parsing results for successfully parsed logs; for logs that fail to be parsed, vectorize them and retrieve candidate templates from the template vector library and filter them based on similarity. Then, use a multi-agent collaborative analysis framework built on a large language model to analyze, extract and verify the filtered logs that failed to be parsed in sequence, obtain log templates and update them to the original template library and template vector library, and obtain the log parsing results; Anomaly detection module: Sorts the log parsing results by time and reconstructs them into a log sequence through a sliding time window. It retrieves candidate sequences from the sequence vector library after vectorizing the log sequence, filters them based on similarity and hit rate, and uses a multi-agent collaborative analysis framework based on a large language model to detect and verify anomalies in the filtered log sequence, and outputs the anomaly detection results.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the information system log anomaly detection method according to any one of claims 1 to 7.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes the information system log anomaly detection method according to any one of claims 1 to 7.

Images