Log template compression-based and large language model-based massive log analysis method

Abstract

The application provides a mass log analysis method based on log template compression and a large language model, comprising the following steps: S1, task receiving and analysis, analyzing a natural language analysis task into structured task information; S2, agent decision and analysis strategy generation, dynamically determining an analysis strategy and tool calling parameters; S3, double-path data acquisition and compression, compressing GB-level original logs into KB-level structured abstracts through template matching statistics, and acquiring detailed information through keyword search; S4, result integration and semantic enhancement, adding business semantic information to the template matching statistics result; S5, agent iterative analysis and reasoning, performing pattern recognition and iterative in-depth analysis based on the semantic enhancement abstract; and S6, generating an analysis report; the application solves the problem that a large language model cannot directly process mass logs, realizes a high compression ratio, completely retains key information, and endows the log data with business semantics.

Description

Technical Field

[0001] This invention relates to the field of intelligent operations and maintenance (AIOps) technology, specifically to a method that combines log template compression technology with a large language model intelligent agent to achieve efficient compression and intelligent analysis of massive log data. Background Technology

[0002] With the rapid development of information technology, the scale and complexity of IT systems are increasing daily, and the amount of log data generated during system operation is exploding, typically reaching GB or even TB levels. Log data directly reflects the system's operating status, fault information, and user behavior, and its effective analysis is crucial for ensuring system stability, quickly locating faults, and optimizing system performance.

[0003] In recent years, large language model-based intelligent agent technology has shown great potential in natural language understanding and complex task processing, providing new ideas for intelligent log analysis. However, directly applying large language model intelligent agents to massive log analysis scenarios faces a series of fundamental technical challenges, and existing technical solutions have the following shortcomings: 1. Infeasibility of Direct Input: Large language models are limited by their fixed context window size, making it difficult even for state-of-the-art models to handle text exceeding one million tokens. GB-level raw logs contain billions of characters, far exceeding the model's processing capacity. Furthermore, the cost of calling the large language model API is directly proportional to the number of input tokens; directly processing massive logs would incur prohibitive economic costs.

[0004] 2. Information loss due to simple sampling strategies: To meet the processing capacity of the model, existing technologies often employ simple truncation or random sampling methods, such as only truncating the header or tail of the log, or performing random sampling. This method is prone to missing key time points that characterize system anomalies, rare error patterns, or the complete execution trajectory of specific entities (such as a task ID), resulting in inaccurate, incomplete, or even completely wrong analysis results, failing to achieve the analysis objectives.

[0005] 3. The Semantic Gap in Traditional Log Analysis: Traditional log analysis techniques, such as pattern matching based on regular expressions, keyword search, and statistical analysis, while capable of handling large-scale data, remain at the syntactic level. They cannot understand the business meaning behind log entries. For example, they cannot distinguish the fundamentally different business impacts of errors like "database connection timeout" and "insufficient disk space," making it difficult to perform deep semantic understanding and intelligent reasoning, resulting in low alert accuracy and high false alarm rates.

[0006] 4. Insufficient flexibility in fixed-process analysis: Existing automated log analysis tools or solutions typically employ preset, fixed processing procedures. This "rigid" analysis model cannot dynamically adjust analysis strategies based on different analysis objectives (such as root cause analysis, performance bottleneck location, and security event tracing) and intermediate analysis results. It lacks the ability to adapt and iteratively analyze in depth, making it difficult to cope with complex and ever-changing operational analysis needs.

[0007] In summary, how to significantly compress log data volume while fully preserving the key semantic information required for analysis, and providing a flexible and intelligent analysis mechanism, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a method for analyzing massive logs based on log template compression and a large language model intelligent agent, so as to achieve efficient, accurate, and low-cost intelligent analysis of massive logs.

[0009] The method includes the following steps: Step S1, Task Reception and Parsing, transforms the diverse and unstructured analysis requests submitted by users in natural language into structured task descriptions that can be understood by the intelligent agent, providing a basis for subsequent decision-making.

[0010] User input analysis tasks, typically in the form of natural language text, such as "Analyze the operation of order-service between 10:00 and 11:00" or "Find the reason for the failure of task-1001". Tasks may include information such as the analysis target, entities of interest (service name, task ID, IP address, etc.), time range, and log level.

[0011] The large language model agent performs semantic parsing on the input task text. The parsing process includes: 1. Intent Recognition: Identify the user's core intent, such as troubleshooting, performance analysis, security auditing, or routine inspection.

[0012] 2. Entity Extraction: Extract entities involved in the task, such as service name (order-service), task ID (task-1001), hostname, time expression ("10:00-11:00"), etc., and convert relative time (such as "last hour") into absolute time range.

[0013] 3. Constraint Identification: Identify implicit constraints in the task, such as log level ("View only error logs"), keywords, etc.

[0014] 4. Task Structuring: Organize the above information into structured task objects, including fields such as: goal (analysis objective), entities (entities list), time_range (time range), and filters (filtering conditions).

[0015] The system outputs structured key task information as input for the next decision-making step of the intelligent agent.

[0016] Step S2: Intelligent agent decision-making and analysis strategy generation. Based on the parsed task information, the optimal analysis path and tool calling strategy are dynamically selected or generated to avoid using rigid fixed processes and enable the analysis process to adapt to the needs of different scenarios.

[0017] This step inputs the key structured task information output from step S1.

[0018] Based on its built-in reasoning capabilities and understanding of the log analysis domain, the large language model agent performs the following decision-making sub-steps: 1. Initial Strategy Selection: Based on the analysis goal, select an initial analysis strategy from a predefined strategy template library. The strategy template library includes: (1) Overall overview strategy: suitable for understanding the overall operation of the system over a certain period of time. Template matching statistical tools should be used first to obtain the distribution and frequency changes of event types.

[0019] (2) Entity tracing strategy: Applicable to tracing the execution process of a specific task ID, user ID, or request chain. Prioritize using keyword search tools to locate logs containing the entity, and obtain the complete trajectory by combining the context.

[0020] (3) Anomaly detection strategy: suitable for finding errors or abnormal events. Prioritize the use of template matching statistics tools and filter templates with event types of "error" or "warning", and pay special attention to templates with a sudden increase in frequency.

[0021] (4) Comparative analysis strategy: Applicable to comparing log differences between two time periods. Prioritize calling time comparison tools (such as compare_time_periods) to obtain template frequency changes.

[0022] 2. Toolchain Orchestration: Based on the selected strategy, orchestrate the tools to be called and their calling order. For example, an overall overview strategy can be orchestrated as follows: first call template_match_stats, and then decide whether to call keyword_search for in-depth analysis of specific templates based on the statistical results.

[0023] 3. Parameter Determination: Determine the specific parameters for each tool to be invoked: (1) Time range: obtained from the task information. If the task is not specified, the agent can set a default range (such as the last 30 minutes) or determine it by asking the user.

[0024] (2) Time window size: The agent dynamically determines the time window size based on the total length of the time range and the required granularity of analysis. For example, a 5-minute window can be used for a 1-hour analysis, and a 1-hour window can be used for a 24-hour analysis. The window size can also be specified by the user or automatically calculated using heuristic rules (such as expecting no more than 20 time points to be output).

[0025] (3) Filtering conditions: generated based on entities and constraints in the task, such as service="order-service".

[0026] (4) Keywords: generated based on the analysis objective, such as using the entity itself as a keyword in an entity tracking strategy.

[0027] This step outputs a decision instruction containing analysis strategies and tool call parameters, which may include a tool call sequence or an executable plan.

[0028] Step S3 involves dual-path data acquisition and compression. While ensuring that key information is not lost, the massive amount of raw log data at the GB level is efficiently compressed into a structured summary at the KB level. At the same time, specific detailed information is acquired, providing the agent with a comprehensive yet refined data foundation.

[0029] Input the decision instruction output in step S2.

[0030] Based on the decision instructions, the appropriate log processing toolset is invoked to perform data acquisition and compression operations in parallel or serially. Core tools include: 1. Template matching statistics tool (template_match_stats): Sub-step 3.1: Log query: Based on the filtering conditions such as time range, service name, and hostname in the command, query the original logs that meet the conditions from the log storage system (such as Elasticsearch, Loki).

[0031] Sub-step 3.2: Template Matching: For each log entry in the query results, match it against a pre-generated log template library. Each template in the library contains a template ID, a regular expression pattern, and an example log entry. The matching process can use the longest common subsequence, tree parsing, or rule-based methods to determine the template ID to which the log belongs.

[0032] Sub-step 3.3: Time window division: Divide the entire time range into fixed-size time windows (the window size is specified by the instruction), and establish a counter mapping (template ID > frequency) for each window.

[0033] Sub-step 3.4: Frequency Statistics: Iterate through each log entry, determine its corresponding window based on its timestamp, and increment the counter of the corresponding template ID by 1. The final result is a three-dimensional data structure: [Time Window][Template ID] = Frequency.

[0034] Sub-step 3.5: Result formatting: Convert the statistical results into a format that is easy for the agent to process, such as a JSON array, where each element contains window_start, window_end, template_id, and count.

[0035] 2. Keyword search tool (keyword_search): Sub-step 3.6: Log query: Query matching log entries from the log storage system based on the time range and keywords (regular expressions are supported).

[0036] Sub-step 3.7: Context acquisition: For each matching log, the N logs before and after it can be acquired as context (the size of the context can be specified by the agent or is the default) in order to understand the environment in which the event occurred.

[0037] Sub-step 3.8: Result formatting: Returns a list containing timestamps, original log content, and context.

[0038] 3. Other auxiliary tools: such as get_log_context, compare_time_periods, etc., can be called according to the command.

[0039] This step outputs results via two paths: Path A outputs: structured template matching statistics, for example: [ {"window": "10:00-10:05", "stats": [{"template_id": "T1", "count":5200}, {"template_id": "T2", "count": 4800}]}, {"window": "10:05-10:10", "stats": [{"template_id": "T1", "count":5100}, {"template_id": "T2", "count": 4700}]} ] Path B output: Keyword search results, including detailed log entries and context.

[0040] Step S4, result integration and semantic enhancement, imbues the compressed statistical summary with business semantics, enabling the agent to understand the actual business meaning represented by each log template, thereby making meaningful inferences and avoiding misjudgments based on surface text.

[0041] Input the template matching statistics and keyword search results output from step S3.

[0042] 1. Semantic Query: Invoke the semantic query tool (get_template_semantics), inputting all template IDs appearing in the statistical results. This tool retrieves the semantic information corresponding to each template ID from the knowledge base. The knowledge base is pre-built and records the business meaning, event type, importance, and associated components of each template.

[0043] 2. Semantic information structure: template_id: Unique identifier for the template.

[0044] business_meaning: A description of the business meaning, such as "order created successfully" or "database connection timed out".

[0045] event_type: An enumeration of event types, such as "normal", "warning", and "error".

[0046] severity: degree of importance, such as "information", "warning", "serious".

[0047] related_components: A list of related components, such as "database" and "order service".

[0048] 3. Result Fusion: The semantic information retrieved is correlated and fused with the template matching statistics. For each record (window, template_id, count) in the statistics, information such as the template's business_meaning, event_type, and severity is appended. Meanwhile, keyword search results may already contain the original logs and do not require additional semantics, but semantic annotation can still be performed based on their matching template ID.

[0049] 4. Construct a semantically enhanced summary: This ultimately generates a comprehensive data structure containing: Template matching statistics organized by time window (with semantics).

[0050] Detailed log entries obtained from keyword searches (with possible matching template semantics).

[0051] Meta-information such as overall time range and window size.

[0052] This step outputs a semantically enhanced structured log summary, which contains both macro-level statistical information and micro-level detailed information, and each statistical item is business-readable.

[0053] Step S5: Iterative analysis and reasoning by the agent. Based on semantically enhanced summarization, the agent performs deep analysis, identifies problems, discovers patterns, and can autonomously decide whether to acquire further data based on intermediate discoveries, thus achieving an iterative reasoning process similar to that of human experts.

[0054] Step S4 outputs a semantically enhanced structured log summary, along with the current analysis context (such as historical analysis steps, established hypotheses, etc.).

[0055] The large language model agent executes an iterative analysis loop, which mainly includes the following sub-steps: Sub-step 5.1: Macro-pattern recognition: Analyze template matching statistics to identify anomalous patterns, such as: Sudden increase / decrease in frequency: The frequency of a template in a certain time window changes significantly relative to the preceding and following windows or the baseline.

[0056] A new template appears: A template that has never appeared in a previous window appears for the first time in the current window, which may indicate that a new event type has occurred.

[0057] Error templates keep appearing: Multiple windows of the "Error" event type template exist consecutively.

[0058] Sub-step 5.2: Semantic Understanding and Hypothesis Generation: Combining semantic information, understand the business meaning behind the abnormal patterns and generate preliminary hypotheses. For example, if the frequency of the "database connection timeout" template surges between 10:30 and 10:40, while the frequency of the "order creation successful" template decreases, the agent can hypothesize that "a database problem caused the order processing to fail."

[0059] Sub-step 5.3: Iterative In-Depth Analysis and Decision Making: Based on the current assumptions and existing information, determine whether more data is needed to verify the assumptions. If so, the agent returns to step S2 and generates new decision instructions. For example, to verify a database problem, the agent might decide to call a keyword search tool to retrieve details of the specific timeout log using "connection timeout" as the keyword, or call get_log_context to retrieve the logs before and after the timeout log to understand other system activities at that time.

[0060] Sub-step 5.4: Execute in a loop: Repeat steps S2-S5, narrowing the focus or broadening the field of vision based on the previous results in each iteration, until the agent believes that it has obtained enough evidence to form a reliable conclusion, or reaches the preset limit of the number of iterations.

[0061] Sub-step 5.5: Conclusion Formation: When the agent deems the analysis sufficient, it integrates all evidence and reasoning processes to form preliminary analytical conclusions. The conclusions should include the root cause of the problem, the scope of its impact, the timeline of its occurrence, and key evidence (such as relevant log samples).

[0062] This step outputs preliminary analysis conclusions formed after intelligent analysis and reasoning, including analysis findings, key evidence chains, and reasoning processes.

[0063] Step S6: Generate an analysis report, presenting the analysis process and conclusions of the intelligent agent to the end user in a clear, structured, and easy-to-read format to support subsequent actions by operations and maintenance personnel.

[0064] Input the preliminary analysis conclusions output from step S5, as well as the key intermediate results generated throughout the analysis process (such as statistical snapshots of abnormal time windows and key log samples).

[0065] The report generation module is invoked. This module can be part of the agent itself or a standalone formatting tool. Report generation includes: 1. Conclusion Summary: The analysis, summarized in concise language, revealed, for example, that "the order-service experienced a decrease in processing capacity due to database connection timeouts between 10:30 and 10:40".

[0066] 2. Evidence Organization: Organize key evidence (such as timelines, template frequency charts, and key logs) to support the conclusions.

[0067] 3. Suggestions: Depending on the type of problem, preliminary suggestions can be provided, such as "Check the network connection and load of the database server db-master-01".

[0068] 4. Report formatting: Output the content in Markdown, HTML, or PDF formats for easy reading and sharing.

[0069] This step outputs the final analysis report, which includes an analysis summary, detailed findings, evidence, timeline, recommendations, etc.

[0070] Compared with the prior art, the present invention has the following beneficial effects: 1. Resolved the contradiction between massive data and model capabilities: Through template matching statistics, a high compression ratio was achieved, making it possible for large language model agents to process GB-level logs, and significantly reducing token consumption and API call costs.

[0071] 2. Ensures the integrity of key information during compression: Unlike simple sampling, this invention retains all event types and their frequency variations, and supplements details by combining keyword search, ensuring that no key clues required for analysis are missed.

[0072] 3. The analysis process is made dynamic and intelligent: The "intelligent agent + tool" architecture makes the analysis process less rigid and can be dynamically adjusted according to the analysis objectives and intermediate results, supporting iterative in-depth analysis and greatly improving the flexibility and accuracy of the analysis.

[0073] 4. It endows machine data with business semantics: Through semantic enhancement mechanisms, the original and difficult-to-understand log text is transformed into analytical units with business meaning, so that the agent's reasoning is based on understanding business logic, and the analysis conclusions are more valuable.

[0074] Other features and advantages of the invention will be set forth in the following description or may be learned by practicing the invention. Attached Figure Description

[0075] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0076] Figure 1 This is the overall system architecture diagram.

[0077] Figure 2 This is a complete analysis flowchart.

[0078] Figure 3 This is a diagram illustrating log template compression. Detailed Implementation

[0079] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention. Any equivalent substitutions, modifications or improvements made by those skilled in the art based on the embodiments of the present invention shall fall within the scope of protection of the present invention.

[0080] Example 1: System hardware and software environment configuration: This embodiment provides a massive log analysis system based on log template compression and large language model intelligence. The system is deployed in an enterprise-level data center or cloud environment and is used to perform intelligent analysis on the operation logs of large-scale distributed systems.

[0081] Hardware environment configuration: Log storage cluster: An Elasticsearch cluster consisting of 3 or more servers, each server is configured with 64 CPU cores, 256GB of memory, and 10TB of SSD storage, used to store raw log data.

[0082] Computing server: configured with 2 Intel Xeon Platinum 8380 processors (40 cores / 80 threads), 512GB of memory, and 4 NVIDIA A100 GPUs (80GB of video memory) for deploying large language model agents and running log processing tools.

[0083] Knowledge base server: configured with 16-core CPU, 64GB memory, and 2TB NVMe SSD, used to store log template library and semantic annotation library, and uses MySQL database for data management.

[0084] Software environment configuration: Operating system: Ubuntu 22.04 LTS Server.

[0085] Log storage system: Elasticsearch 8.11 + Kibana 8.11 + Logstash 8.11.

[0086] Large Language Model: A locally deployed model based on Llama 3 70B Instruction, accelerated for high-performance inference via vLLM.

[0087] Intelligent agent framework: an intelligent agent decision engine based on LangChain + AutoGPT.

[0088] Log processing toolset: Developed using Python 3.11, integrating spaCy for log parsing, and Redis for caching template matching results.

[0089] Front-end display: A visual analysis report interface built with React 18 + ECharts.

[0090] Example 2: System Core Components and Connection Relationships See attached document Figure 1 The system architecture diagram of this invention illustrates the core components and their connections. The system includes the following components: (1) Agent Decision Engine (101): The intelligent agent decision engine is the core decision-making unit of this invention, and it is built based on a large language model. It interacts with external users through an API interface to receive natural language analysis tasks; it connects to the log processing toolset (102) through an internal call interface to send decision instructions and receive execution results; and it connects to the knowledge base (103) to query template semantic information.

[0091] (2) Log processing toolset (102): The log processing toolset is a set of encapsulated functional modules, including a template matching statistics tool (1021), a keyword search tool (1022), a semantic query tool (1023), a log context acquisition tool (1024), and a time comparison tool (1025). These tools are deployed on a computing server and are uniformly exposed to the intelligent agent decision engine for invocation through an API gateway. The toolset connects to the log storage system (104) to perform log query operations and connects to the knowledge base (103) to obtain the data required for template matching and semantic query.

[0092] (3) Knowledge Base (103): The knowledge base consists of a log template library (1031) and a semantic annotation library (1032), deployed on a dedicated database server. The log template library stores log templates extracted from historical logs, with each template containing fields such as template ID, regular expression pattern, creation time, and sample log. The semantic annotation library stores semantic information for each template, including its business meaning, event type, importance, and associated components, which is pre-annotated by operations and maintenance experts or generated through semi-automated tools.

[0093] (4) Log storage system (104): The log storage system uses an Elasticsearch cluster to store all raw log data. Each log entry includes fields such as timestamp, service name, hostname, log level, and log content. Indexes are created daily, supporting efficient time-range queries and full-text search.

[0094] Connections between components: Users (105) submit analysis tasks to the agent decision engine (101) via a web interface or API.

[0095] After the intelligent agent decision engine (101) parses the task, it calls the corresponding tools in the log processing toolkit (102) according to the decision result.

[0096] The log processing toolset (102) sends a query request to the log storage system (104) to obtain the raw log data.

[0097] During the template matching process, the toolset (102) requests template matching from the log template library (1031) of the knowledge base (103).

[0098] During the semantic enhancement process, the toolset (102) requests template semantic information from the semantic annotation library (1032) of the knowledge base (103).

[0099] The processing result is returned to the agent decision engine (101), the agent performs reasoning analysis, and can iteratively call the toolset.

[0100] The final analysis report is returned to the user (105).

[0101] Example 3: Method for constructing a log template library: In this embodiment, the log template library is built using an offline preprocessing method, and the specific steps are as follows: Step T1, Historical Log Collection: Historical logs from the most recent 30 days were extracted from the log storage system (104), with a sampling rate of 10% to ensure coverage of various business scenarios. The sampled log data is approximately 500GB, containing approximately 100 million log records.

[0102] Step T2, Log parsing and template extraction: A log parsing algorithm based on frequent pattern mining (such as the Drain algorithm) is used to parse the sampled logs. The Drain algorithm uses a parse tree structure to separate the constant and variable parts of the log message and extract the log template. The algorithm parameters are set as follows: similarity threshold 0.8, maximum number of child nodes 100, and depth 4 layers.

[0103] Step T3, Template deduplication and merging: The extracted original templates are subjected to similarity calculation, and templates with a similarity higher than 95% are merged. For example, the following two log entries: [2025-01-13 10:21:07] task

[1001] execution completed, duration: 235ms [2025-01-13 10:21:08] task

[1002] execution completed, duration: 187ms They will be merged into the same template: [ <timestamp>] task[<task_id>] execution completed,duration: <duration>ms.

[0104] Step T4, Template import: The finalized template is stored in the log template library (1031) of the knowledge base. Each record contains: template_id: A unique identifier for the template, in the format of T00001, T00002, etc.; pattern: The regular expression pattern of the template used to match the raw log; example: A sample log entry, designed for easy human understanding; created_at: Template creation time; source: source service or module.

[0105] The final template library contains approximately 5,000 templates, covering more than 90% of the system's log types.

[0106] Example 4: Method for constructing a semantic annotation library: In this embodiment, the semantic annotation library is constructed using a combination of manual annotation and semi-automatic propagation. The specific steps are as follows: Step U1, Initial Labeling: The 500 most frequently occurring templates were manually annotated by operations and maintenance experts. The annotations included: business_meaning: Describe the business meaning of the template in natural language, such as "User login request processed successfully"; event_type: An enumeration of values, including "normal", "warning", "error", and "critical error"; severity: An enumeration of values, including "information", "warning", "serious", and "disaster"; related_components: A list of related components or services, such as ["User Service", "Authentication Service"]; suggested_action: Optional field. Suggested action when this template appears.

[0107] Step U2, Feature Extraction and Propagation: For each labeled template, extract its word vector features (using the Sentence-BERT model) and structural features (such as log length, number of variables, etc.). For unlabeled templates, calculate their similarity to labeled templates. When the similarity exceeds 0.85, automatically inherit the semantic annotations of similar templates, and have them manually reviewed and confirmed.

[0108] Step U3, continuous optimization: During system operation, feedback on the use of semantic annotations during agent analysis is recorded. When the agent repeatedly encounters confusion or incorrect reasoning regarding the semantic understanding of the same template, a manual review mechanism is triggered to optimize the semantic annotations.

[0109] The final semantic annotation library (1032) contains semantic information from approximately 5,000 templates, providing a solid business semantic foundation for agent analysis.

[0110] Example 5: Detailed implementation of the complete analysis process: This embodiment takes "analyzing the operation of the order-service between 10:00 and 11:00" as its specific task, and describes in detail the complete execution process of the method of the present invention, with corresponding appendices. Figure 2 The complete analysis flowchart is shown below.

[0111] Step S1, Task Reception and Parsing: The user inputs natural language through a web interface to analyze the operation of the order-service between 10:00 and 11:00 and understand what events occurred during this period.

[0112] After receiving the task text, the agent decision engine (101) calls the large language model for semantic parsing. The model outputs the following structured JSON object: { "goal": "operational_analysis", "entities": { "service": "order-service" }, "time_range": { "start": "2025-01-13 10:00:00", "end": "2025-01-13 11:00:00", "type": "absolute }, "filters": {}, "original_task": "Analyze the operation of the order-service between 10:00 and 11:00 to understand what events occurred during this period." } This step outputs the key information of the structured task mentioned above, which serves as the input for step S2.

[0113] Step S2, Agent decision-making and analysis strategy generation: Input the structured task information output from step S1.

[0114] The agent decision engine (101) analyzes the task objective as operational_analysis, which falls under the scenario of understanding the overall operational status. Therefore, it executes the following decision: 1. Initial strategy selection: Select "Overall overview strategy".

[0115] 2. Toolchain orchestration: Determine the tool call order as follows: First, template_match_stats is called to obtain the overall event distribution; If any anomalies are found based on the statistical results, keyword_search will be called for further analysis.

[0116] 3. Parameter Determination: Time range: start="2025-01-13 10:00:00", end="2025-01-13 11:00:00"; Time window size: One hour is divided into 12 five-minute windows, therefore window_size="5m"; Filter condition: service="order-service"; Other parameters: default values; The agent generates the following decision instructions: { "strategy": "overview_strategy", "tool_calls": [ { "tool": "template_match_stats", "order": 1, "params": { "start_time": "2025-01-13 10:00:00", "end_time": "2025-01-13 11:00:00", "service": "order-service", "window_size": "5m } } ], "iteration_config": { "max_iterations": 5, "stop_condition": "conclusion_reached" } } This step outputs the above decision instruction and sends it to the log processing toolset (102).

[0117] Step S3, Dual-path data acquisition and compression: like Figure 3 As shown, input the decision instruction output in step S2.

[0118] After receiving the instruction, the log processing toolset (102) calls the template_match_stats tool to perform data acquisition and compression: Sub-step 3.1, log query: The tool sends a query to the Elasticsearch cluster. The DSL query statement is as follows: { "query": { "bool": { "must": [ {"range": {"@timestamp": {"gte": "2025-01-13T10:00:00", "lte": "2025-01-13T11:00:00"}}}, {"term": {"service.keyword": "order-service"}} ] } }, "size": 10000, "sort": [{"@timestamp": "asc"}] } The query returned approximately 600,000 log entries, totaling about 300MB of data.

[0119] Sub-step 3.2, template matching: For each log entry, a template matching algorithm is invoked. The algorithm first extracts the log content field, removes the timestamp prefix, and then matches it against 5000 templates in the template library. The matching process employs an optimized Aho-Corasick automaton algorithm, achieving a matching speed of 100,000 log entries per second. Each log entry is assigned a unique `template_id` based on its matching result.

[0120] Sub-step 3.3, Time window division: Divide the timeframe from 10:00 to 11:00 into 12 five-minute windows: [10:00-10:05], [10:05-10:10], ..., [10:55-11:00]. Initialize an empty dictionary for each window: window_stats[window] = {}.

[0121] Sub-step 3.4, frequency statistics: Iterate through 600,000 log entries, determine the window to which the log belongs based on its timestamp, and increment the counter value of the corresponding template by 1. For example, if a log entry at 10:02 matches template T1, then window_stats["10:00-10:05"]["T1"] += 1.

[0122] Sub-step 3.5, result formatting: After the statistics are completed, a structured result is generated. In this embodiment, there are 3 main templates (T1, T2, T3), so the final statistical result is 3 templates × 12 windows = 36 statistical records, with a data volume of approximately 3KB.

[0123] Output: Path A output (template matching statistics): [ {"window": "10:00-10:05", "stats": [{"template_id": "T1", "count":5200}, {"template_id": "T2", "count": 4800}]}, {"window": "10:05-10:10", "stats": [{"template_id": "T1", "count":5100}, {"template_id": "T2", "count": 4700}]}, / / ... Middle window omitted... {"window": "10:30-10:35", "stats": [{"template_id": "T1", "count":3200}, {"template_id": "T2", "count": 3000}, {"template_id": "T3", "count":45}]}, {"window": "10:35-10:40", "stats": [{"template_id": "T1", "count":2800}, {"template_id": "T2", "count": 2600}, {"template_id": "T3", "count":120}]}, {"window": "10:40-10:45", "stats": [{"template_id": "T1", "count":4900}, {"template_id": "T2", "count": 4600}]} / / ... Remaining windows... ] Path B Output (Keyword Search Results): This step has not yet invoked keyword search, so it is empty.

[0124] Step S4, Result Integration and Semantic Enhancement: Input the template matching statistics output from step S3.

[0125] The agent calls the `get_template_semantics` tool, taking as input a list of template IDs ["T1", "T2", "T3"] appearing in the statistical results. The semantic query tool retrieves the results from the semantic annotation library (1032) of the knowledge base (103): { "T1": { "business_meaning": "HTTP request processed successfully", "event_type": "normal", "severity": "information", "related_components": ["order-service", "api-gateway"] }, "T2": { "business_meaning": "Database query complete", "event_type": "normal", "severity": "information", "related_components": ["order-service", "database"] }, "T3": { "business_meaning": "Database connection timed out", "event_type": "Error", "severity": "severe" "related_components": ["order-service", "database", "network"] } } The agent fuses semantic information with statistical results to generate a semantically enhanced structured log summary. The fusion process uses an inner join approach, adding semantic fields to each statistical record. For example, {"window": "10:30-10:35", "template_id": "T3", "count": 45} becomes the fused record: { "window": "10:30-10:35", "template_id": "T3", "count": 45, "business_meaning": "Database connection timed out", "event_type": "Error", "severity": "severe" "related_components": ["order-service", "database", "network"] } This step outputs a semantically enhanced structured log summary, which serves as the input for step S5.

[0126] Step S5, Iterative analysis and reasoning of the agent: Input the semantically enhanced summary output from step S4.

[0127] The agent executes an iterative analysis loop: Sub-step 5.1, Macroscopic Pattern Recognition: The agent analyzes the template matching statistics of each window and identifies the following patterns: Between 10:00 and 10:30, only T1 and T2 occurred, with a frequency of around 5000 / 4800. The window from 10:30 to 10:35 shows the first occurrence of T3 (database connection timeout, error, critical), with a count of 45. During the 10:35-10:40 window, the T3 count increases to 120, while the T1 and T2 counts decrease to 2800 / 2600. From 10:40 to 10:45, T3 disappears, and T1 and T2 return to 4900 / 4600. Sub-step 5.2, Semantic Understanding and Hypothesis Generation: Based on semantic information, the agent understands that T3 (database connection timeout) occurs between 10:30 and 10:40, and its frequency is increasing; during the same period, the frequency of T1 (request processing success) decreases from 5000 to 2800. The agent generates the hypothesis: "Database connection problems cause a decrease in the order-service processing capacity."

[0128] Sub-step 5.3, iterative in-depth analysis and decision-making: To verify the hypothesis, the agent decides to retrieve the specific log content corresponding to template T3. Returning to step S2, a new decision instruction is generated: { "strategy": "entity_tracking_strategy", "tool_calls": [ { "tool": "keyword_search", "order": 1, "params": { "start_time": "2025-01-13 10:30:00", "end_time": "2025-01-13 10:40:00", "service": "order-service", "keyword": "connection timeout" "max_results": 200 } } ] } Sub-step 5.4, executed repeatedly: Perform the second round of steps S3-S5: S3': Calls the keyword_search tool, returning approximately 165 matching log entries, including timestamps, log content, and context. Key log entries include: [10:30:12] Database connection timeout after 3000ms, host: db-master-01; [10:30:15] Database connection timeout after 3000ms, host: db-master-01; [10:31:08] Database connection timeout after 3000ms, host: db-master-01; ... [10:38:45] Database connection restored, host: db-master-01.

[0129] S4': The agent integrates the keyword search results with existing information, confirms that all timeouts occurred on the same database host db-master-01, and recovers at 10:38:45.

[0130] S5': The agent analyzes new information, confirms the hypothesis, and refines the conclusion: Between 10:30:12 and 10:38:45, the database master node db-master-01 experienced a connection timeout, resulting in a decrease of approximately 40% in the order-service processing capacity.

[0131] The agent believes it has obtained sufficient evidence and terminates the iteration.

[0132] Sub-step 5.5, conclusion reached: The agent integrates all evidence to form preliminary analytical conclusions, including: Root cause: Connection timeout on database master node db-master-01; Impact: Order-service processing capacity is reduced, affecting approximately 8 minutes; Timeline: First appearance at 10:30:12, recovered at 10:38:45, timeout 45 times from 10:30 to 10:35, timeout 120 times from 10:35 to 10:40; Key evidence: T3 template matching statistics, keyword search results; Output the above preliminary analysis conclusions.

[0133] Step S6, generate the analysis report: Input, preliminary analysis conclusions and key intermediate results of step S5.

[0134] The agent invokes the report generation module to generate an analysis report in Markdown format: order-service operation status analysis report (10:00-11:00) I. Summary of Analysis Conclusions: The order-service experienced a 40% drop in processing capacity between 10:30 and 10:40 due to a database connection timeout, but automatically recovered after 10:38:45.

[0135] II. Detailed findings: 2.1 Overall Operational Overview: Between 10:00 and 11:00, the order-service processed approximately 550,000 requests and completed approximately 500,000 database queries.

[0136] Overall, the system was operating normally, but an anomaly occurred between 10:30 and 10:40.

[0137] 2.2 Analysis of Abnormal Time Period (10:30-10:40): | Time Window | Number of Successful Requests | Number of Database Queries | Connection Timeouts | Status | |---------|-----------|------------|-----------|------| | 10:30-10:35 | 3200 | 3000 | 45 | Error started | | 10:35-10:40 | 2800 | 2600 | 120 | Abnormally worsening | | 10:40-10:45 | 4900 | 4600 | 0 | Back to normal | 2.3 Key Timeline: 10:30:12: First database connection timeout occurred; 10:30:12 - 10:38:45: Connection timeouts occurred continuously, totaling 165 times; 10:38:45: Database connection restored; After 10:40:00: Service resumed normal operation.

[0138] 2.4 Root Causes of the Problem: Problem type: Database connection timeout; Affected components: order-service, database master node db-master-01; Possible causes: network fluctuations, excessive database load, or connection pool exhaustion.

[0139] III. Recommended Measures: 1. Check the network connection status and load of database server db-master-01 between 10:30 and 10:40.

[0140] 2. Consider increasing the database connection pool size to improve concurrent processing capabilities.

[0141] 3. It is recommended to configure database connection timeout alerts to detect similar issues in a timely manner.

[0142] IV. Key Log Samples: [10:30:12] Database connection timeout after 3000ms, host: db-master-01; [10:35:22] Database connection timeout after 3000ms, host: db-master-01; [10:38:45] Database connection restored, host: db-master-01.

[0143] The above analysis report is output and displayed to users through a web interface, and can be exported as a PDF.

[0144] Example 6: Actual verification of compression effect: This embodiment quantifies and verifies the compression effect in the above analysis process: Verification results show that the method of the present invention achieves effective compression of more than 3,000 times without losing key information, enabling large language model agents to efficiently process massive log data that were originally impossible to process.

[0145] Example 7: Specific implementation and interface definition of the toolset: This embodiment details the specific implementation methods and API interface definitions of each tool in the log processing toolset: 1. Template matching statistics tool (template_match_stats): Function Description: Based on time range, filter conditions, and window size, perform log queries, template matching, and frequency statistics, and return template hit statistics grouped by time window.

[0146] API interface: RESTful API, POST / tools / template_match_stats.

[0147] Request body example: { "start_time": "2025-01-13T10:00:00Z", "end_time": "2025-01-13T11:00:00Z", "service": "order-service", "host": null, "log_level": null, "window_size": "5m", "options": { "include_sample_logs": false, "min_count": 1 } } Internal processing flow: (1) Parse the request parameters and construct the Elasticsearch query DSL; (2) Execute the search_after deep pagination query to retrieve logs in batches (10,000 records per batch); (3) For each batch of logs, the template matching engine is called in parallel, and the LRU cache is used to accelerate repeated matching; (4) Calculate the window index based on the timestamp and update the counter in memory; (5) After all logs have been processed, format the results and return them.

[0148] Example of response body: Same as the output example of step S3.

[0149] 2. Keyword search tool (keyword_search): Function Description: Search for matching log entries based on time range and keywords, and supports obtaining context.

[0150] API interface: POST / tools / keyword_search.

[0151] Request body example: { "start_time": "2025-01-13T10:30:00Z", "end_time": "2025-01-13T10:40:00Z", "service": "order-service", "keyword": "connection timeout" "is_regex": true, "context_lines": 5, "max_results": 200 } Internal processing flow: (1) Construct an Elasticsearch query and perform regular expression matching on the message field; (2) Execute the query to obtain the matching log; (3) For each matching log, based on its timestamp, additionally query the N logs before and after it as context; (4) Sort by time and return.

[0152] Example response body: Contains an array of results, where each element contains timestamp, message, context_before, and context_after.

[0153] 3. Semantic query tool (get_template_semantics): Function Description: Query the corresponding semantic information based on the template ID list.

[0154] API interface: POST / tools / get_template_semantics.

[0155] Request body example: { "template_ids": ["T1", "T2", "T3"] } Internal processing flow: (1) Connect to the knowledge base MySQL database; (2) Execute SELECT FROM template_semantics WHERE template_id IN (...); (3) Organize the results into a dictionary by template ID; (4) Return the results.

[0156] Response body example: Same as the S4 semantic query output example 4. Log context retrieval tool (get_log_context): Function Description: Retrieves the N preceding and following logs of a specific log entry for in-depth analysis.

[0157] API interface: POST / tools / get_log_context.

[0158] Request body example: { "timestamp": "2025-01-13T10:30:12Z", "service": "order-service", "host": null, "context_lines": 10, "around": "both" } Internal processing flow: (1) Query the logs before and after the specified time point based on the timestamp and filtering conditions; (2) Obtain the first N log entries and the last N log entries respectively; (3) After merging, return the results in a sorted order by time.

[0159] 5. Time comparison tool (compare_time_periods): Function Description: Compares the template hit differences between two time periods for change analysis.

[0160] API interface: POST / tools / compare_time_periods.

[0161] Request body example: { "period1": {"start": "2025-01-13T10:00:00Z", "end": "2025-01-13T11:00:00Z"}, "period2": {"start": "2025-01-13T09:00:00Z", "end": "2025-01-13T10:00:00Z"}, "service": "order-service", "window_size": "1h" } Internal processing flow: (1) Call template_match_stats to obtain statistical results for the two time periods respectively; (2) Calculate the frequency difference and percentage change of each template in the two time periods; (3) Sort by the magnitude of change and return the list of templates with the largest differences.

[0162] Example 8: A method for dynamically determining the time window: This embodiment details how the agent dynamically determines the time window size in step S2: The agent dynamically determines the size of the time window according to the following rules: Rule 1, based on the total length of the time range: Time range ≤ 1 hour: Window size = 5 minutes; 1 hour < Time range ≤ 6 hours: Window size = 15 minutes; 6 hours < Time range ≤ 24 hours: Window size = 1 hour; 24 hours < Time range ≤ 7 days: Window size = 6 hours; Time range > 7 days: Window size = 1 day; Rule 2, based on the expected number of time points: The agent can accept the number of desired output time points N specified by the user, with a default value of N=12. This rule is to dynamically determine the appropriate time window size based on the user's desired analysis granularity. The specific calculation steps are as follows: (1) Calculate the theoretical window size: Divide the length of the total time range (e.g., 1 hour, 24 hours, etc.) by the expected number of time points N to obtain the theoretical window size. .

[0163] (2) Round up to the commonly used time unit: round the theoretical window size up. Round up to the nearest predefined, commonly used time unit. The list of commonly used time units includes: 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 6 hours, and 1 day. The rounding rule is: select units greater than or equal to [the nearest predefined unit]. The smallest unit of time.

[0164] For example, if the total duration =2 hours (120 minutes), N=12, then =10 minutes. Round up to the nearest common time unit. The smallest unit greater than or equal to 10 minutes is 15 minutes, therefore the window size is set to 15 minutes.

[0165] If the total duration =1 hour (60 minutes), N=12, then =5 minutes, which is exactly equal to the commonly used time unit of 5 minutes, so the window size is 5 minutes.

[0166] If the total duration =7 days, N=12, then =0.583 days ≈ 14 hours. Rounding up to the commonly used time unit, the smallest unit greater than or equal to 14 hours is 1 day (24 hours). Therefore, the window size is determined to be 1 day.

[0167] (3) Final window size: The commonly used time unit after rounding up is used as the actual time window size. This satisfies the user's expected number of analysis points (the actual number of points may be slightly less than N, but closer) and ensures that the window size is a commonly used time unit that is easy to understand and process, which is convenient for subsequent statistics and visualization.

[0168] Rule 3, based on the analysis granularity requirement: If the task implicitly specifies the required granularity of analysis (such as "view changes every minute"), then the granularity specified by the user will be given priority.

[0169] In this embodiment of the invention, the agent adopts rule 1 as the default strategy. For a 1-hour analysis, a 5-minute window is selected, resulting in 12 time points, which ensures sufficient analytical granularity while avoiding overly sparse output.

[0170] Example 9: Specific implementation of the template matching algorithm: This embodiment details the specific implementation of the template matching algorithm in sub-step 3.2 of step S3: Algorithm input: log_message: The original log message string; template_library: A pre-generated log template library containing 5000 templates, each containing template_id and pattern (regular expression).

[0171] Algorithm output: the matching template_id, or null if no match is found.

[0172] Algorithm steps: (1) Preprocessing: Remove the timestamp prefix from the log (regular expression matching ^[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\]) to get the pure log content.

[0173] (2) Cache query: Query the Redis cache with the key match:${content}. If a match is found, the cached template_id is returned directly. The cache expiration time is set to 1 hour.

[0174] (3) Index matching: Use the Aho-Corasick automaton to build an inverted index of template keywords. Extract keywords (such as "execution", "completed", "database") from the log content to quickly filter the candidate template set, narrowing the matching range from 5000 to an average of about 50.

[0175] (4) Exact Match: Iterate through the candidate templates and match the regular expression pattern of each template. Wildcards are used for the variable part in the regular expression. Replacement. For example, template [ <timestamp>] task[<task_id>]execution completed, duration: <duration>The regular expression corresponding to ms is ^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\] task\[\d+\] execution completed, duration: \d+ms$.

[0176] (5) Matching degree calculation: For templates that are successfully matched by regular expressions, calculate the similarity between the log and the template (optional). The similarity is based on the ratio of variable part to constant. If there are too many variable parts (e.g., more than 50%), the template is considered to be too generalized, and the matching priority is reduced.

[0177] (6) Result selection: Select the first template that is a perfect match and meets the similarity standard as the final result. If there are multiple matching templates, select the template with the fewest variables (i.e., the most specific template).

[0178] (7) Cache update: Write the matching result to the Redis cache for quick matching of subsequent identical logs.

[0179] Performance metrics: Under the hardware configuration of this embodiment, the algorithm can achieve a matching speed of 100,000 logs per second, meeting the requirements for real-time processing.

[0180] Example 10: Comparison and verification with existing technologies: This embodiment verifies the advantages of the method of the present invention compared with the prior art through comparative experiments, corresponding to the beneficial effects in the "Summary of the Invention" section.

[0181] Experimental setup: Dataset: 600,000 logs from order-service between 10:00 and 11:00 on January 13, 2025.

[0182] Analysis task: Identify anomalous events and their impact within this time period.

[0183] Comparison method: 1. Method A (existing technology): Simple sampling, randomly selecting 1% of logs (6000 records) and inputting them into the large model; 2. Method B (existing technology): Extract 0.5% of the logs from the beginning and end (3000 logs each) and input them into the large model; 3. Method C (existing technology): Traditional log analysis, based on regular expression matching to count the number of errors; 4. Method D (the present invention): The method described in the present invention.

[0184] Experimental results: Experimental conclusions: The method of this invention achieves a token consumption level comparable to simple sampling while ensuring the accuracy and completeness of the analysis, but the accuracy and depth of the analysis conclusions far exceed those of existing technologies. Compared with traditional analysis methods, this invention has semantic understanding and intelligent reasoning capabilities, and can provide more valuable analysis conclusions.

[0185] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.< / duration> < / timestamp> < / duration> < / timestamp>

Claims

1. A method for analyzing massive logs based on log template compression and a large language model, characterized in that, Includes the following steps: Step S1, Task reception and parsing, parses the natural language analysis task input by the user into structured key task information; Step S2: Intelligent agent decision-making and analysis strategy generation. Based on the structured key task information, the large language model intelligent agent dynamically determines the analysis strategy and tool calling parameters to generate decision instructions. Step S3, dual-path data acquisition and compression: according to the decision instruction, the log processing toolset is invoked to compress the GB-level raw log into a KB-level structured summary through template matching statistical path, while specific log details are obtained through keyword search path; Step S4, result integration and semantic enhancement: add template semantic information to the structured summary to generate a semantically enhanced structured log summary; Step S5: The agent iteratively analyzes and reasons, performs pattern recognition and hypothesis generation based on the semantically enhanced structured log summary, and iteratively returns to step S2 to obtain more data as needed for analysis, until a preliminary analysis conclusion is formed. Step S6: Generate an analysis report. Based on the preliminary analysis conclusions, generate and output a final analysis report.

2. The method for massive log analysis based on log template compression and large language model according to claim 1, characterized in that, Step S1 further includes: The large language model agent performs semantic parsing on the input task text, including intent recognition, entity extraction, constraint recognition, and task structuring, and outputs a structured task object containing the analysis target, entity list, time range, and filtering conditions.

3. The method for massive log analysis based on log template compression and large language model according to claim 1, characterized in that, Step S2 further includes: Sub-step S2.1, Initial strategy selection: Select an initial analysis strategy from a predefined strategy template library based on the analysis objective. The strategy template library includes an overall overview strategy, an entity tracking strategy, anomaly detection strategy, and comparative analysis strategy. Sub-step S2.2, Toolchain Orchestration: Based on the selected strategy, determine the tools to be called and their calling order; Sub-step S2.3, parameter determination: Determine specific parameters for each tool to be called, including time range, time window size, filtering conditions, and keywords.

4. The method for massive log analysis based on log template compression and large language model intelligent agents according to claim 3, characterized in that, The size of the time window is dynamically determined based on the total length of the time range, according to the following rules: When the time range is ≤1 hour, the window size is 5 minutes; When the time range is 1 hour to 6 hours, the window size is 15 minutes. When the time range is 6 hours to 24 hours, the window size is 1 hour. When the time range is 24 hours to 7 days, the window size is 6 hours. When the time range is greater than 7 days, the window size is 1 day; Alternatively, based on the user-specified desired number of output time points N, determine the time window size using the following steps: Step a, calculate the theoretical window size = ÷N, where The total time range is the length of N, where N is the number of expected output time points specified by the user, with a default value of 12. Step b, adjust the theoretical window size. Round up to the nearest commonly used time unit, which includes 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 6 hours, and 1 day. The rounding rule is to select a time unit greater than or equal to the nearest common time unit from the list of commonly used time units. The smallest unit of time; Step c: Determine the actual time window size by rounding down the commonly used time unit.

5. The method for massive log analysis based on log template compression and large language model according to claim 1, characterized in that, The template matching statistical path in step S3 further includes: Sub-step 3.1, Log query: Query the original logs from the log storage system according to the time range and filtering conditions in the decision instruction; Sub-step 3.2, Template matching: Match each log entry with the pre-generated log template library to determine the template ID to which the log belongs; Sub-step 3.3, Time window division: Divide the time range into time windows of fixed size; Sub-step 3.4, Frequency Statistics: Count the number of times each template ID is hit within each time window; Sub-step 3.5, Result Formatting: Generate template matching statistics grouped by time window.

6. The method for massive log analysis based on log template compression and large language model according to claim 5, characterized in that, The template matching uses Aho-Corasick automaton for index matching, combined with regular expressions for precise matching, and Redis caching to accelerate the matching of duplicate logs.

7. The method for massive log analysis based on log template compression and large language model according to claim 1, characterized in that, The keyword search path in step S3 further includes: Sub-step 3.6, Log Query: Query matching log entries from the log storage system based on the time range and keywords; Sub-step 3.7, Context Acquisition: For each matching log, obtain the N logs before and after it as the context; Sub-step 3.8, Result Formatting: Returns a list containing timestamps, original log content, and context.

8. The method for massive log analysis based on log template compression and large language model according to claim 1, characterized in that, Step S4 further includes: The semantic query tool is invoked to query the corresponding semantic information from the pre-built knowledge base based on the template ID. The semantic information includes business meaning, event type, importance level, and related components. The semantic information retrieved is combined with the template matching statistics to add semantic fields to each statistical record; Construct semantically enhanced structured log summaries that include both macro-level statistical information and micro-level detailed information.

9. The method for massive log analysis based on log template compression and large language model according to claim 1, characterized in that, Step S5 further includes: Sub-step 5.1, Macro-pattern recognition: Identify anomalous patterns from the semantically enhanced summary, including sudden increases / decreases in frequency, the appearance of new templates, and the persistence of erroneous templates; Sub-step 5.2, Semantic Understanding and Hypothesis Generation: Combine semantic information to understand the business meaning behind the abnormal patterns and generate preliminary hypotheses; Sub-step 5.3, Iterative in-depth analysis and decision-making: Based on the current assumptions, determine whether more data is needed; if so, return to step S2 to generate new decision instructions. Sub-step 5.4, Execute repeatedly: Repeat steps S2 to S5 until sufficient evidence is obtained or the iteration limit is reached; Sub-step 5.5, Conclusion Formation: Integrate all evidence to form a preliminary analytical conclusion that includes the root cause, scope of impact, timeline, and key evidence.

10. The method for massive log analysis based on log template compression and large language model according to claim 4, characterized in that, The size of the time window is also dynamically determined based on the implicit analysis granularity requirements in the analysis task, prioritizing the granularity specified by the user.

Images