Adaptive log processing method and system based on large model and storage medium

Abstract

This invention relates to an adaptive log processing method, system, and storage medium based on a large model. The method includes: acquiring log samples; performing semantic structure analysis and format feature recognition on the log samples to generate structured extraction templates; associating the original log messages with the structured extraction templates and injecting template identifiers into the messages, writing them into a message middleware; reading messages from the message middleware and distributing them to corresponding processing pipelines for parallel stream processing according to the template identifiers; collecting log processing process data; constructing a multi-dimensional analysis model based on the process data to allocate resources to the processing pipelines; performing alarm matching on the processed logs; and combining alarm content with a local operation and maintenance knowledge graph to generate diagnostic suggestions, which are sent together with the alarm information. This invention can reduce log processing latency, optimize storage costs, reduce manual intervention, improve alarm accuracy, and achieve intelligent allocation of processing resources.

Description

Technical Field

[0001] This invention relates to the fields of data processing and intelligent operation and maintenance technology, and in particular to an adaptive log processing method, system and storage medium based on a large model. Background Technology

[0002] In modern distributed systems and cloud-native architectures, log data serves as a data source for system observability, and its collection, processing, and analysis are crucial for operation and maintenance monitoring, fault diagnosis, and business analysis. With the advent of the Internet of Things (IoT) era, especially in fields such as new energy vehicles, the continuous generation of data from various digital components of massive intelligent devices has led to an explosive growth in the total volume of logs. Currently, log management mainly relies on traditional centralized log systems such as ELK (Elasticsearch, Logstash, Kibana) and Splunk, which can achieve basic log collection, storage, and query functions. Existing log processing technologies are mainly divided into two paradigms: batch collection and streaming collection. Batch collection processes logs through periodic rotation, while streaming collection can achieve lower latency log processing. In terms of log analysis, traditional solutions mainly use regular expression matching and template matching methods, extracting specific information from logs by defining specific pattern matching rules. In addition, existing technologies have also emerged that utilize Natural Language Processing (NLP) for log analysis, enabling semantic understanding and key information extraction from log content.

[0003] However, existing log processing solutions exhibit numerous limitations when dealing with petabyte-scale log volumes. First, batch processing architectures result in high log processing latency; the time from log generation to queryability is typically lengthy, making it difficult for operations teams to troubleshoot business issues in real time. Second, storing all raw logs increases storage costs, often requiring enterprises to pay exorbitant fees for historical logs accessed only infrequently. Third, traditional systems lack intelligent analysis capabilities, offering only basic full-text search or keyword matching, unable to further analyze log-related issues, and heavily reliant on the skills of operations engineers. During log access, traditional solutions require operations engineers to manually assess log content, extract data structures based on experience, and allocate processing resources according to business scenarios. This approach is prone to errors due to the accuracy of engineer predictions. Furthermore, existing solutions generally lack intelligent data processing mechanisms, generating alerts based on rigid keyword matching, resulting in numerous invalid alerts and failing to effectively assist operations teams in troubleshooting. Regarding resource allocation, most existing data acquisition systems adopt an indiscriminate full-data acquisition mode, lacking the ability to intelligently judge the volume and quality of logs. This leads to uneven resource allocation within the data processing pipeline, resulting not only in wasted storage and transmission resources but also potential log delays or loss due to resource contention. Therefore, a more intelligent log processing solution is needed to address these issues. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an adaptive log processing method, system and storage medium based on a large model. By introducing large model technology, adaptive processing and intelligent analysis of logs can be achieved, which can reduce log processing latency, optimize storage costs, reduce manual intervention, improve alarm accuracy and realize intelligent allocation of processing resources.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] Firstly, the present invention provides an adaptive log processing method based on a large model, which adopts the following technical solution: Randomly sample the access logs to obtain log samples; The log samples are analyzed for semantic structure and format features based on natural language processing technology. The processed logs are then clustered using an unsupervised clustering algorithm to automatically generate a structured extraction template that is suitable for this type of log. The original log message is associated with the structured extraction template, and the template identifier is injected into the message and written into the message middleware; Read messages from the message middleware and distribute them to the corresponding processing pipelines for parallel stream processing according to the template identifier; During the parallel stream processing, log processing data is collected, including the number of logs per second, the average log size, and log peak and trough information. A multi-dimensional analysis model is constructed based on the process data, and the resource allocation of the processing pipeline is dynamically adjusted by the multi-dimensional analysis model. The processed logs are matched for alarms, and the alarm regular expressions are continuously optimized using a large model; and By combining alarm content with the local operations and maintenance knowledge graph, diagnostic suggestions are generated through a large model and sent together with the alarm information.

[0007] Furthermore, in the above method, the random sampling of the access logs includes: For log streams with unknown formats, uniform random sampling is performed based on a preset sampling rate; Meta-features are extracted from the sampled logs, including text length distribution, timestamp pattern, level keywords, and unique identifiers; Calculate the disorder index based on the aforementioned meta-features; and The sampling rate is dynamically adjusted based on the disorder index, wherein the sampling rate is reduced for log streams with low disorder and increased for log streams with high disorder.

[0008] Furthermore, in the above method, the unsupervised clustering algorithm includes the DBSCAN algorithm or the K-Means algorithm, and the automatic generation of a structured extraction template adapted to this type of log includes: Perform text cleaning and word segmentation on the log lines, and remove random variables; The unsupervised clustering algorithm is used to cluster the processed log lines; For each cluster, calculate the common prefix sequence and generate a log template; and The entity types in the template are identified and labeled based on the pre-trained model.

[0009] Furthermore, in the above method, the structured extraction template also includes processing strategy information, which includes concurrency level and processing priority. High-throughput logs are adapted to high-concurrency pipelines, and low-throughput important logs are adapted to low-concurrency but high-priority pipelines.

[0010] Furthermore, in the above method, the parallel stream processing also includes data processing, which includes converting unstructured raw log text into structured JSON objects according to the extraction strategy and data structure defined in the structured extraction template.

[0011] Furthermore, in the above method, the process data also includes the processing capacity per second of each pipeline, the number and frequency of supermodel logs, processing latency, and system resource utilization.

[0012] Furthermore, in the above method, the dynamic adjustment of resource allocation of the processing pipeline by the multi-dimensional analysis model includes: sending a request to the resource manager to allocate or reclaim memory, CPU and node resources for the pipeline.

[0013] Furthermore, in the above method, the alarm matching of the processed logs includes: Precompile the regular expression; Pre-filter logs, performing regular expression matching only on logs that match a preset level; and An alarm is triggered based on the matching results.

[0014] Furthermore, in the above method, the continuous optimization of the alarm regular expression through a large model includes: Collect information on the regular expression rules executed within a preset time period, their performance overhead, and user feedback. The regular expression rules are merged and simplified, logically optimized, and rule failure identified using a large language model; and Deploy the optimized regular expression rule set to the production environment.

[0015] Furthermore, in the above method, the step of generating diagnostic suggestions through a large model includes: sending the context information of the alarm log to a large language model, wherein the large language model retrieves the local operation and maintenance knowledge graph through RAG technology and generates diagnostic suggestions in natural language.

[0016] Secondly, the present invention provides an adaptive log processing system based on a large model, which adopts the following technical solution: The data acquisition and access module is configured to randomly sample the accessed logs to obtain log samples, and write the original log messages to the message middleware after associating them with the structured extraction template. The data acquisition and access module includes a log acquisition plugin, a dynamic sampler, a log stream template, and a writing middleware. The intelligent analysis module is configured to perform semantic structure analysis and format feature recognition on the log samples based on natural language processing technology, automatically generate structured extraction templates adapted to this type of log through unsupervised clustering algorithm, and build a multi-dimensional analysis model based on the log processing process data to dynamically adjust the resource allocation of the processing pipeline. The intelligent analysis module includes an NLP structure analysis engine, a multi-dimensional analysis model and a large model optimizer. A parallel stream processing module is configured to read messages from the message middleware, distribute the messages to the corresponding processing pipelines for parallel stream processing based on template identifiers, and collect process data during processing. The parallel stream processing module includes a consumer middleware, data processing, process data collection, alarm matching, alarm service, and hierarchical storage routing. The storage module is configured to perform hierarchical storage of the processed logs; The large model optimizer is configured to continuously optimize the alarm regular expression and combine the alarm content with the local operation and maintenance knowledge graph to generate diagnostic suggestions, which are then sent together with the alarm information.

[0017] Thirdly, the present invention provides a readable storage medium, which adopts the following technical solution: A readable storage medium storing computer instructions that, when executed by a processor, implement the method as described in any one of the first aspects above.

[0018] In summary, compared with the prior art, the present invention has at least one of the following beneficial technical effects: The adaptive log processing method based on a large model provided by this invention reduces manual configuration workload and improves log access efficiency by intelligently sampling access logs and automatically generating structured extraction templates using natural language processing technology. Efficient log classification is achieved by injecting template identifiers into messages and distributing them to corresponding processing pipelines for parallel stream processing. Dynamic resource allocation is adjusted by collecting process data and building a multi-dimensional analysis model to match processing resources with actual load, thus avoiding resource waste or processing bottlenecks. Continuous optimization of alarm regular expressions using a large model improves alarm accuracy and reduces invalid alarms. Diagnostic suggestions generated by combining local operation and maintenance knowledge graphs provide reference information for operation and maintenance personnel, further assisting in problem investigation and repair. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The diagram shows a structural view of an embodiment of an adaptive log processing system based on a large model according to the present invention.

[0021] Figure 2 The flowchart of an embodiment of an adaptive log processing method based on a large model according to the present invention is shown.

[0022] Figure 3 This diagram illustrates the log structured feature extraction and cluster-based template discovery process of the present invention.

[0023] Figure 4 The system block diagram of the data processing flow in the adaptive log processing system based on a large model of the present invention is shown.

[0024] Figure 5 A flowchart illustrating the alarm analysis and self-optimization process of this disclosure is shown. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, it should be understood that the specific embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application.

[0026] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments of this application. Furthermore, the descriptions of each embodiment in the following embodiments have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0027] The method steps described in this embodiment of the invention can be executed in the order described in the specific implementation, or the execution order of each step can be adjusted according to actual needs, provided that the technical problem can be solved. These are not listed one by one here.

[0028] The present invention will be further described in detail below with reference to the accompanying drawings.

[0029] Reference Figure 1This invention provides an adaptive log processing system based on a large model, including a data acquisition and access module, an intelligent analysis module, a parallel stream processing module, and a storage module. The data acquisition and access module is configured to randomly sample the accessed logs to obtain log samples, and then write the original log messages to a message middleware after associating them with structured extraction templates. The intelligent analysis module is configured to perform semantic structure analysis and format feature recognition on the log samples based on natural language processing technology, automatically generate structured extraction templates adapted to this type of log through an unsupervised clustering algorithm, and construct a multi-dimensional analysis model based on the log processing process data to dynamically adjust the resource allocation of the processing pipeline. The parallel stream processing module is configured to read messages from the message middleware, distribute the messages to the corresponding processing pipelines for parallel stream processing according to the template identifier, and collect process data during processing. The storage module is configured to perform hierarchical storage of the processed logs.

[0030] Continue to refer to Figure 1 The data acquisition and access module includes a log acquisition plugin, a dynamic sampler, a log stream template, and a writing middleware. The log acquisition plugin transmits the collected logs to the dynamic sampler, which is connected to the log stream template, which in turn connects to the writing middleware. The dynamic sampler simultaneously sends the sampled data to the NLP structural analysis engine in the intelligent analysis module. The log acquisition plugin can run as a Sidecar container deployed as a DaemonSet, automatically collecting the standard output (stdout / stderr) and log files within the mounted volume of each Pod. This deployment is suitable for Kubernetes container environments. In some cases, the log acquisition plugin can run as an agent deployed as a system service, such as a Systemd Service, collecting logs by listening to a specified log file path, such as files in the / var / log / directory. This deployment is suitable for virtual machine or physical machine environments. The log acquisition plugin supports various common logging protocols, such as Syslog, HTTP / S endpoints, and Kafka protocols, to directly receive data from applications, network devices, or other log sources.

[0031] like Figure 1 As shown, the collected raw log streams can be uniformly encapsulated into a message format, such as Apache Avro or Protobuf, and supplemented with source information, which may include fields such as app_name, host_ip, and log_path. The encapsulated message is then pushed to the message middleware. The message middleware can be implemented using Apache Kafka or Pulsar. This system employs a lock-free circular queue, a double-buffering mechanism, and an asynchronous pipeline architecture to solve the thread blocking and performance bottleneck problems of traditional data collection chains.

[0032] Continue to refer to Figure 1 The intelligent analysis module includes an NLP structural analysis engine, a multi-dimensional analysis model, and a large model optimizer. The NLP structural analysis engine receives data from the dynamic sampler and transmits the analysis results to the multi-dimensional analysis model. The multi-dimensional analysis model is connected to the large model optimizer. The large model optimizer is associated with an alarm rule base and an operations and maintenance knowledge graph. The large model optimizer is configured to continuously optimize alarm regular expressions and combine alarm content with the local operations and maintenance knowledge graph to generate diagnostic suggestions, which are sent along with the alarm information. The intelligent analysis module also includes a dynamic allocation engine, which is connected to the parallel stream processing module for dynamically adjusting processing resources.

[0033] like Figure 1 As further illustrated, the parallel stream processing module includes a consumption middleware, data processing, process data acquisition, alarm matching, alarm service, and storage hierarchical routing. The consumption middleware obtains log data from the write middleware, and data processing transforms the raw data. Process data acquisition collects processing metrics during data processing and submits them to the multi-dimensional analysis model. Alarm matching performs keyword matching based on regular expressions, and the alarm service notifies the user upon successful matching. Storage hierarchical routing categorizes logs into hot and cold data.

[0034] Continue to refer to Figure 1 The storage module includes a time-series database and low-cost storage. Hot data is stored in the time-series database, such as ClickHouse, while cold data is stored in low-cost storage. For the same log file, tiered storage can be implemented in both hot and archive storage. The storage module employs asynchronous batch writes, aggregating and compressing data within a pipeline before writing to reduce I / O pressure. In some cases, if users need to query historical data, they can use the data unfreezing function to migrate historical data from low-cost storage to hot storage for querying.

[0035] Based on the aforementioned adaptive log processing system, this embodiment of the invention also provides an adaptive log processing method based on a large model. (Refer to...) Figure 2 The method begins execution from application access. During the application access phase, the method first performs pre-access sampling by randomly sampling the access logs to obtain log samples. The system determines whether the sampled logs conform to an existing template. If the sampled logs do not conform to an existing template, a new template is dynamically created; if the sampled logs conform to an existing template, the process proceeds to the preprocessing phase.

[0036] Continue to refer to Figure 2The preprocessing stage includes data structure transformation, date formatting, and adding processing procedure fields. Data structure transformation converts the raw log data into a unified data format. Date formatting standardizes the timestamps in the logs. Adding processing procedure fields appends metadata information required for the processing flow to the log messages. After preprocessing, the method associates the raw log messages with a structured extraction template, injects the template identifier into the message, and writes it to the message middleware.

[0037] like Figure 2 As further illustrated, this method reads messages from the message middleware and distributes them to the corresponding processing pipelines for parallel stream processing based on template identifiers. During parallel stream processing, the method performs process data sampling analysis to collect process data from log processing. The results of the process data sampling analysis can be used for alarm keyword matching. In the alarm keyword matching stage, the system determines whether a keyword is matched. If a keyword is matched, RAG retrieval is performed using LLM keyword extraction; if no keyword is matched, no processing is performed.

[0038] Continue to refer to Figure 2 After the RAG search, the system determines whether any content was found. If content was found, a remedial suggestion and the scope of impact are generated, and an alarm is issued. If no content was found, the situation is recorded, and the system suggests optimizing the local operations and maintenance knowledge base in the weekly report, also issuing an alarm.

[0039] like Figure 2 As shown, in terms of resource allocation, this method performs dynamic allocation, adjusting resources based on different business peaks and troughs. The multi-dimensional analysis model can execute differentiated processing strategies, such as allocating less processing resources to logs from low-complexity businesses during the day and allocating resources in advance for batch tasks running at night. Dynamic resource allocation can handle various types of logs, including logs with small volume but each log being large, logs with uncertain frequency, logs with large volume but each log being small, and meaningless logs such as heartbeats.

[0040] Continue to refer to Figure 2 After resource allocation, this method performs database classification routing. Hot data is stored in a time-series database via a hot data adapter and can be queried directly. Cold data is stored in low-frequency, inefficient storage via a cold data adapter. Cold data needs to be unfrozen before it can be accessed by users.

[0041] Furthermore, refer to Figure 3This method randomly samples incoming logs to obtain log samples. For new log streams with unknown formats, the dynamic sampler performs uniform random sampling based on a preset sampling rate. In the initial stage, the preset sampling rate can be set to 5%. The dynamic sampler extracts meta-features from the sampled logs, including text length distribution, timestamp patterns, level keywords, and unique identifiers. Based on these meta-features, the dynamic sampler calculates a disorder index. The disorder index reflects the degree of format consistency of the log stream.

[0042] Continue to refer to Figure 3 The dynamic sampler dynamically adjusts the sampling rate based on the disorder index. For log streams with low disorder, i.e., log streams with highly consistent meta-features, the dynamic sampler reduces the sampling rate, which can be as low as 0.1%. For log streams with high disorder, i.e., log streams with varied and messy formats, the dynamic sampler increases the sampling rate, which can be as high as 15%. The dynamic adjustment range of the sampling rate is from 0.1% to 15%. This dynamic sampling mechanism can reduce sampling overhead while ensuring the diversity of log samples.

[0043] like Figure 3 As shown, the NLP structural analysis engine performs semantic structural analysis and format feature recognition on log samples based on natural language processing technology. Semantic structural analysis includes log structured feature extraction, used to identify constant and variable parts in the log. The constant part may include error level and error message, while the variable part may include timestamps and identifiers.

[0044] Continue to refer to Figure 3 The NLP structural analysis engine performs text cleaning and tokenization on the log lines, removing random variables. During the text cleaning and tokenization stage, the NLP structural analysis engine replaces random variables with placeholders. For example, it can replace IP addresses with placeholders and numbers with placeholders. Text cleaning and tokenization also include noise filtering, removal of irrelevant characters, case neutralization, and handling abbreviations.

[0045] like Figure 3 As further illustrated, the NLP structural analysis engine clusters the processed logs using an unsupervised clustering algorithm. This unsupervised clustering algorithm can include DBSCAN or K-Means. DBSCAN is a density-based spatial clustering algorithm that can discover clusters of arbitrary shapes and identify noise points. K-Means is a classic unsupervised learning clustering algorithm that discovers the inherent structure of data by dividing data points into K clusters. The NLP structural analysis engine uses this unsupervised clustering algorithm to cluster the processed log lines, grouping logs with similar semantics and structure into the same category.

[0046] Continue to refer to Figure 3The NLP structural analysis engine calculates a common prefix sequence for each cluster to generate a log template. For example, from the clusters ["User 123 logged in", "User 456 logged in", "User 789 loginfailed"], the NLP structural analysis engine can extract the template "User <*> logged in". Based on a pre-trained model, the NLP structural analysis engine identifies and labels the entity types in the template, further labeling <*> as...<User_ID> , <ip> 、 <timestamp>Fields such as [example fields]. When a log cannot match an existing template, the NLP structure analysis engine creates a new template.

[0047] like Figure 3 As shown, the NLP structure analysis engine automatically generates a structured extraction template adapted to this type of log. The structured extraction template also includes processing strategy information, including concurrency level and processing priority. High-throughput logs (such as heartbeat logs) are adapted to high-concurrency pipelines, while low-throughput important logs (such as error logs) are adapted to low-concurrency but high-priority pipelines. The generated log stream template is defined as a complete data structure, such as {"timestamp": "datetime", "level": "string", "user_id": "int", "assignpipeline": "string"......}. This data structure contains information such as the keyword extraction strategy, concurrency level, and matching data structure.

[0048] Reference Figure 4 In parallel stream processing, this method also includes data processing. Data processing involves transforming unstructured raw log text into structured JSON objects based on the extraction strategy and data structure defined in the structured extraction template. Data processing extracts structured data from the text using Grok patterns. For example, data processing can extract structured JSON data from the text "2023-10-01 ERROR [auth] User 123 login failed": {"@timestamp": "2023-10-01T00:00:00Z", "level": "ERROR", "module": "auth", "user\_id": 123, "event": "login failed"}.

[0049] Continue to refer to Figure 4 During parallel stream processing, this method collects process data for log processing. This process data includes the number of logs per second, average log size, and log peak and trough information. It also includes the processing capacity per second for each pipeline, the number and frequency of supermodel logs, processing latency, and system resource utilization. System resource utilization can include CPU utilization and memory utilization. Processing latency can be defined as the time from log collection to processing completion. Supermodel logs refer to logs whose length exceeds a preset threshold.

[0050] like Figure 4 As shown, the system includes a feature engineering processing stage. This stage comprises multiple processing units. The real-time log generation rate calculation unit is used to statistically analyze the real-time log generation rate. The peak / off-peak time analysis unit is used to identify the time distribution characteristics of log generation and determine the start and end times of peak and off-peak periods. The supermodel / error log probability analysis unit is used to analyze the probability of abnormal log occurrences in the log content, including the number and frequency of supermodel logs. The historical data aggregation and statistical feature unit is used to calculate statistical indicators such as mean and quantiles. After feature engineering processing, the system outputs a feature vector.

[0051] Continue to refer to Figure 4 The feature vectors then enter the preprocessing stage. This stage includes a standardization / normalization unit and a feature selection unit. The standardization / normalization unit performs numerical standardization on the feature vectors, converting features with different dimensions into a uniform numerical range. The feature selection unit filters the features to obtain valid features and removes redundant or irrelevant features. After preprocessing, the processed feature vectors are output.

[0052] like Figure 4 As further illustrated, this method constructs a multi-dimensional analysis model based on process data. The processed feature vectors are fed into the inference model at the core of the machine learning model. The inference model takes the log generation rate, the length of a single log entry, and the volume of logs generated at different times as input parameters. The multi-dimensional analysis model dynamically adjusts the resource allocation of the processing pipeline. The inference model outputs three types of results through regression prediction: concurrency level, allocated CPU / membrane resource configuration, and queue depth.

[0053] Continue to refer to Figure 4 The system dynamically adjusts resource allocation in the log processing pipeline using a multi-dimensional analysis model. This includes sending requests to the resource manager to allocate or reclaim memory, CPU, and node resources for the pipeline. Three types of prediction results are aggregated to form concurrent parameter transmission, which guides the dynamic allocation of resources in the log processing pipeline. This system models and analyzes data during log processing using a multi-dimensional analysis model, enabling data-driven dynamic resource allocation. This allows the system to adjust the log processing resources required by different business systems at a more granular level.

[0054] Reference Figure 5 This method can further perform alarm matching on the processed logs and continuously optimize the alarm regular expressions through a large model. The alarm analysis and self-optimization process demonstrates the complete processing of log content from input to alarm output, as well as the alarm optimization mechanism based on large model technology.

[0055] Continue to refer to Figure 5 The method performs alert matching on the processed logs, including pre-compiling regular expressions. Regular expressions are pre-compiled periodically to reduce their resource overhead. The regular expression matcher contains multiple matching regular expression objects and receives input from multiple alert keywords, which are provided to the matcher through periodic pre-compilation.

[0056] like Figure 5 As shown, alarm matching of the processed logs also includes pre-filtering the logs, performing regular expression matching only on logs that meet a preset level. In the pre-matching stage, this method performs rapid pre-filtering on each log entry. The pre-filtering stage only performs subsequent regular expression matching on logs with level="ERROR" or level="FATAL", directly ignoring logs with level="INFO". This pre-filtering mechanism can reduce the CPU overhead of a large number of regular expression matches.

[0057] Continue to refer to Figure 5 The alarm matching process for processed logs also includes triggering alarms based on the matching results. The regular expression matcher performs the matching operation and outputs the regular expression to the regular expression processing module, which then performs the matching operation and outputs the matching timeliness information. When a log matches an alarm rule, the alarm service is triggered.

[0058] like Figure 5 As further illustrated, this method continuously optimizes the alert regular expression using a large model. The large model optimizer runs an optimization task periodically. Continuous optimization of the alert regular expression using the large model includes collecting regular expression rules executed within a preset time period, their performance overhead, and user feedback information. The large model optimizer collects all regular expression rules executed in the past 24 hours and their performance overhead, which may include the number of matches and CPU time. The large model optimizer also collects work order feedback information from users after receiving alerts, which is used to optimize the regular expression rules. The human feedback results generated from work order feedback are passed to the LLM large model optimization module.

[0059] Continue to refer to Figure 5 Continuous optimization of alarm regular expressions through a large model also includes merging and simplifying regular expression rules, optimizing logic, and identifying rule failures using a large language model. This method calls a large language model (LLM), inputting regular expression rules and log samples, and requires the LLM to complete the following tasks: merging and simplifying, combining multiple functionally similar regular expressions into a more efficient one; optimizing logic, suggesting adjustments to the matching order, placing high-hit-rate, low-cost rules first; and identifying rule failures, recognizing invalid rules that have never been hit or have always been hit. The LLM large model optimization module receives matching timeliness information from the regular expression processing module, overriding regular expression information from the regular expression matcher, and manual feedback results from work order feedback.

[0060] like Figure 5 As shown, continuous optimization of alert regular expressions through the large model also includes deploying the optimized regular expression rule set to the production environment. After optimization, the LLM large model optimization module outputs the optimized regular expression object and keywords. The optimization results are fed back to the regular expression matcher and multiple alert keyword modules, forming a closed loop for continuous optimization of alert rules. The optimized regular expression rule set is automatically deployed to the production environment, replacing the old rules.

[0061] Continue to refer to Figure 5 This method combines alarm content with a local operations and maintenance knowledge graph, generates diagnostic suggestions through a large model, and sends them along with the alarm information. Generating diagnostic suggestions through the large model includes sending contextual information from the alarm logs to the large language model. Contextual information may include error messages, service names, and timestamps. The LLM large model connects to the RAG (Retrieval of Operations and Maintenance Graph) module, which is used to retrieve relevant information from the local operations and maintenance knowledge base.

[0062] like Figure 5 As further illustrated, the large language model retrieves the local operations and maintenance knowledge graph using RAG technology to generate diagnostic suggestions in natural language. The output of the RAG retrieval module for the operations and maintenance knowledge graph is passed to the LLM summary module, which generates diagnostic suggestions. The diagnostic suggestions generated by the large model are in natural language form, such as "This database connection error usually occurs when there is a version mismatch. Please check whether the client driver version is compatible with the database server version."

[0063] Continue to refer to Figure 5 The alarm service sets different notification priorities based on log levels and sends alarm information and diagnostic suggestions through preset notification channels. The alarm service sets different notification priorities based on log levels (such as ERROR, FATAL). Alarm information (original error) and diagnostic suggestions are sent together through preset notification channels. The system sends alarm information through a multi-channel alarm module and also sets up a work order feedback mechanism to collect user feedback.

[0064] In summary, the adaptive log processing method based on a large model provided in this embodiment intelligently samples access logs through a dynamic sampling mechanism. It automatically generates structured extraction templates by combining natural language processing technology and unsupervised clustering algorithms, reducing manual configuration workload and improving log access efficiency. By injecting template identifiers into messages and distributing them to corresponding processing pipelines for parallel stream processing, log classification can be achieved. By collecting process data and constructing a multi-dimensional analysis model, the resource allocation of processing pipelines is dynamically adjusted based on characteristics such as log generation rate, single log length, and peak / off-peak periods, ensuring that processing resources match the actual load and avoiding resource waste or processing bottlenecks. By pre-compiling and pre-filtering regular expressions and using a large language model to merge, simplify, logically optimize, and identify rule failures in alarm regular expressions, alarm matching efficiency can be improved and invalid alarms reduced. By combining a local operation and maintenance knowledge graph and using RAG technology to generate diagnostic suggestions in natural language form, reference information can be provided to operation and maintenance personnel, assisting in the problem-solving process. A hierarchical storage mechanism divides logs into hot and cold data for separate storage, optimizing storage costs.

[0065] This invention also discloses a readable storage medium.

[0066] A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any of the above embodiments. The computer-readable storage medium may include any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. The computer program includes computer program code. The computer program code may be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable storage medium may include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0067] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0068] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0069] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / timestamp> < / ip>

Claims

1. An adaptive log processing method based on a large model, characterized in that, include: Randomly sample the access logs to obtain log samples; The log samples are analyzed for semantic structure and format features based on natural language processing technology. The processed logs are then clustered using an unsupervised clustering algorithm to automatically generate a structured extraction template that is suitable for this type of log. The original log message is associated with the structured extraction template, and the template identifier is injected into the message and written into the message middleware; Read messages from the message middleware and distribute them to the corresponding processing pipelines for parallel stream processing according to the template identifier; During the parallel stream processing, log processing data is collected, including the number of logs per second, the average log size, and log peak and trough information. A multi-dimensional analysis model is constructed based on the process data, and the resource allocation of the processing pipeline is dynamically adjusted by the multi-dimensional analysis model. The processed logs are matched for alarms, and the alarm regular expressions are continuously optimized using a large model. as well as By combining alarm content with the local operations and maintenance knowledge graph, diagnostic suggestions are generated through a large model and sent together with the alarm information.

2. The method according to claim 1, characterized in that, The random sampling of the access logs includes: For log streams with unknown formats, uniform random sampling is performed based on a preset sampling rate; Meta-features are extracted from the sampled logs, including text length distribution, timestamp pattern, level keywords, and unique identifiers; Calculate the disorder index based on the aforementioned meta-features; and The sampling rate is dynamically adjusted based on the disorder index, wherein the sampling rate is reduced for log streams with low disorder and increased for log streams with high disorder.

3. The method according to claim 2, characterized in that, The unsupervised clustering algorithm includes the DBSCAN algorithm or the K-Means algorithm, and the automatically generated structured extraction template adapted to this type of log includes: Perform text cleaning and word segmentation on the log lines, and remove random variables; The unsupervised clustering algorithm is used to cluster the processed log lines; For each cluster, calculate the common prefix sequence and generate a log template; and The entity types in the template are identified and labeled based on the pre-trained model.

4. The method according to claim 3, characterized in that, The structured extraction template also includes processing strategy information, which includes concurrency level and processing priority. High-throughput logs are adapted to high-concurrency pipelines, while low-throughput important logs are adapted to low-concurrency but high-priority pipelines.

5. The method according to claim 1, characterized in that, The parallel stream processing also includes data processing, which involves converting unstructured raw log text into structured JSON objects according to the extraction strategy and data structure defined in the structured extraction template.

6. The method according to claim 1, characterized in that, The process data also includes the processing capacity per second for each pipeline, the number and frequency of supermodel logs, processing latency, and system resource utilization.

7. The method according to claim 6, characterized in that, The dynamic adjustment of resource allocation for the processing pipeline by the multi-dimensional analysis model includes: sending a request to the resource manager to allocate or reclaim memory, CPU, and node resources for the pipeline.

8. The method according to claim 1, characterized in that, The alarm matching of the processed logs includes: Precompile the regular expression; Pre-filter logs, performing regular expression matching only on logs that match a preset level; and An alarm is triggered based on the matching results.

9. The method according to claim 8, characterized in that, The continuous optimization of alarm regular expressions through a large model includes: Collect information on the regular expression rules executed within a preset time period, their performance overhead, and user feedback. The regular expression rules are merged and simplified, logically optimized, and rule failure identified using a large language model; and Deploy the optimized regular expression rule set to the production environment.

10. The method according to claim 1, characterized in that, The process of generating diagnostic suggestions through a large model includes: sending the contextual information of alarm logs to a large language model, which then retrieves the local operation and maintenance knowledge graph using RAG technology to generate diagnostic suggestions in natural language.

11. An adaptive log processing system based on a large model, characterized in that, include: The data acquisition and access module is configured to randomly sample the accessed logs to obtain log samples, and write the original log messages to the message middleware after associating them with the structured extraction template. The data acquisition and access module includes a log acquisition plugin, a dynamic sampler, a log stream template, and a writing middleware. The intelligent analysis module is configured to perform semantic structure analysis and format feature recognition on the log samples based on natural language processing technology, automatically generate structured extraction templates adapted to this type of log through unsupervised clustering algorithm, and build a multi-dimensional analysis model based on the log processing process data to dynamically adjust the resource allocation of the processing pipeline. The intelligent analysis module includes an NLP structure analysis engine, a multi-dimensional analysis model and a large model optimizer. The parallel stream processing module is configured to read messages from the message middleware, distribute the messages to the corresponding processing pipelines for parallel stream processing according to the template identifier, and collect process data during the processing. The parallel stream processing module includes a consumer middleware, data processing, process data collection, alarm matching, alarm service, and storage hierarchical routing. as well as The storage module is configured to perform hierarchical storage of the processed logs; The large model optimizer is configured to continuously optimize the alarm regular expression and combine the alarm content with the local operation and maintenance knowledge graph to generate diagnostic suggestions, which are then sent together with the alarm information.

12. A readable storage medium, characterized in that, The readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-10.

Images