Log analysis method and system based on big and small model collaboration

Abstract

The invention provides a log analysis method and system based on cooperation of large and small models, and belongs to the technical field of intelligent operation and maintenance of software systems. Through constructing a diversified training sample fine-tuning small model in an offline training stage and combining the small model and a large model in an online stage to cooperatively analyze a real-time log of a software system and dynamically optimize a cache template library, efficient log analysis and template management are realized. According to the method, the large model and the small model work cooperatively, the small model directly completes analysis of most logs, the large model only corrects a small number of logs, the calling frequency of the large model is reduced, and the privacy disclosure risk is reduced; through multi-level feature clustering and diversified sample selection, it is ensured that the small model can adapt to multiple log modes; by dynamically optimizing the cache template library and combining the structure and semantic analysis of the log template by the large model, the real-time dynamic updating and optimization of the template library are realized; the log analysis efficiency and accuracy are improved, and the method is suitable for log processing of a large-scale software system.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent operation and maintenance of software systems, and specifically relates to a log parsing method and system based on the collaboration of large and small models. The method aims to achieve efficient log parsing and template management by combining offline training and online parsing, thereby improving the accuracy and real-time performance of log analysis. The method is suitable for log processing of large-scale software systems. Background Art

[0002] With the widespread adoption of software systems, the volume of system log data has grown exponentially, making log parsing a critical component of system operations and maintenance. Log parsing is the process of extracting structured information from unstructured log data. It supports log-based analysis tasks such as anomaly detection and fault diagnosis, and directly impacts the efficiency and quality of system operations and maintenance. Inaccurate log parsing can lead to false positives or false negatives, delaying troubleshooting and even causing more serious problems. In large-scale software systems, log formats are diverse and complex, with significant differences between logs from different modules and services. Furthermore, log formats may change with version updates, making log parsing a particularly challenging task. In recent years, log parsing methods based on deep learning (DL) have gained increasing attention, particularly those based on large language models (LLMs) and small language models (SLMs).

[0003] Although both large and small language models show great potential in log parsing tasks, they face their own challenges in practical applications. While large language models' large number of parameters can capture complex semantic information, a single inference requires significant computational resources. This significantly increases the computational burden in high-throughput scenarios, leading to reduced system performance. Furthermore, the frequent invocation of large language models poses a serious risk of privacy leakage. Log data often contains a large amount of sensitive information, such as user identities, access IP addresses, and system keys. Frequently transmitting log data to the cloud or remote servers to invoke large models for parsing can lead to information leakage through data interception and storage vulnerabilities. Small language models, while offering low computational resource consumption and high computational efficiency, can mitigate privacy risks. However, due to their limited training data coverage and small model capacity, they struggle to fully capture the semantic information of complex logs. Consequently, they perform poorly when processing diverse log data, often lacking accuracy and generalization compared to large language models. In particular, when faced with new log formats or rare log events, the parsing capabilities of small language models significantly decline, making them difficult to meet the demands of practical applications.

[0004] Therefore, in order to improve the availability and reliability of software systems, there is an urgent need for an intelligent log parsing method that has high accuracy and efficiency, strong generalization ability, and effectively reduces the risk of privacy leakage. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned existing methods, the present invention provides a log parsing method and system based on the collaboration of large and small models. A small model (small language model) is fine-tuned by constructing diversified training samples in the offline training phase, and the small model is combined with the large model (large language model) in the online parsing phase to collaboratively parse the real-time logs of the software system, dynamically optimize the cache template library, and realize efficient log parsing and template management.

[0006] The technical solutions provided by the present invention are as follows:

[0007] A log parsing method based on the collaboration of large and small models includes building a small model with diversified training samples in the offline training phase, and combining the small model with the large model to collaboratively parse logs and dynamically optimize the cache template library in the online phase. Specifically:

[0008] 1) In the offline phase, the software system's historical logs are converted into vector representations and hierarchically clustered. Diverse training samples are selected from each log category to fine-tune the small model to adapt it to the log parsing task.

[0009] 2) In the online phase, real-time log processing of the software system is achieved through cache matching, collaborative parsing of large and small models, and template merging, and the cache template library is dynamically optimized.

[0010] In the above-mentioned log parsing method based on collaboration between large and small models, the offline phase of step 1) includes the following operations:

[0011] 11) Input historical log data set D off ={l1,l2,…,l N}, where l i As a historical log, each historical log l is extracted by feature extraction algorithm. i Convert to vector To capture the statistical features of historical logs; the feature extraction algorithm is the TF-IDF algorithm, which converts the log text into a vector representation by calculating the term frequency-inverse document frequency value of each word in the log;

[0012] 12) Perform hierarchical clustering on historical logs based on multi-level features to generate a log category set C = {C1, C2, ..., C K Each category corresponds to a log pattern; the multi-level features include: frequent words in the log, the length of the log, and special characters in the log;

[0013] 13) From each log category Ck Randomly select several samples as the initial candidate set Select some representative samples from them, and then further filter out samples with the greatest difference from the selected samples to ensure the diversity of the training set; repeat this step until the number of selected samples reaches the preset value;

[0014] 14) Using sample sets As a training set, fine-tune the small model to adapt it to the log parsing task; specifically, the Masked Language Model (MLM) task is used to fine-tune the small model; for each training sample s i , randomly mask part of its vocabulary and train a small model to predict the masked vocabulary; optimize the model parameters by minimizing the cross entropy loss function.

[0015] In the log parsing method based on collaboration between large and small models, step 13) specifically includes the following steps:

[0016] For each log category C k , randomly select several samples as the initial candidate set Calculate the initial candidate set Each sample e i The sum of the vector similarities with other samples is denoted as SimSum(e i ), the formula is as follows:

[0017]

[0018] Among them, cosine_similarity(e i ,e j ) represents the vector e i and e j Cosine similarity between, select SimSum(e i ) The first m smallest samples are used as representative sample sets

[0019] For each sample Calculate its difference with the selected sample set The sum of the vector similarities of all samples in is recorded as SimSumSelected(e i ), the formula is as follows:

[0020]

[0021] Select SimSumSelected(e i ) is the smallest sample as the sample with the largest difference and is added to middle.

[0022] In the log parsing method based on large and small model collaboration, the online phase of step 2) includes the following operations:

[0023] 21) Receive real-time log stream L on ={l1,l2,…}, for each log l t , priority is given to the cache template library T cache Find out whether there is a regular match l t If the match succeeds, the parsing result is output directly; if the match fails, the subsequent steps are continued;

[0024] 22) Parsing logs using a small model t , generate log template T t and variable annotation V t , and call the large model to log template T according to the preset trigger conditions t Make corrections; the preset trigger conditions include: grammatical error detection, keyword matching, and confidence judgment;

[0025] 23) Generate the final template T based on the analysis results of the small model or large model t ', from the cache template library T cache Select T t 'For candidate templates whose edit distance is less than the set similarity threshold β, let the large model analyze T t 'Determine whether to merge the candidate template's structural information and semantic information into a unified template;

[0026] 24) Update cache template library T cache .

[0027] In the log parsing method based on collaboration between large and small models, the preset triggering conditions in step 22) are specifically:

[0028] Syntax Error Detection: Log Template T t There are syntax errors in the brackets or quotation marks, which are manifested as inconsistent numbers or incorrect nesting order;

[0029] Keyword matching: log text t Contains at least one keyword from the preset keyword set K = {error, exception, warning, ...};

[0030] Confidence judgment: small model analysis result T t The confidence level C t The following conditions are met:

[0031]

[0032] where c i∈[0,1] is the log template T t The i-th variable v in i The confidence level, |V t |For log template T t is the number of variables in , and γ is the preset confidence threshold.

[0033] In the log parsing method based on collaboration between large and small models, the step 23) updates the cache template library, which is specifically divided into the following situations:

[0034] Template merging: If the large model determines that the currently generated log template T t 'With cache template library T cache A candidate template T in candidate Can be merged, then T t 'With T candidate Merge into a unified template T merged , and update the cache template library T cache The corresponding template in; the merged template T merged The common features of the two will be retained, while the annotations of the variable parts will be optimized to ensure the universality and accuracy of the template;

[0035] New template: If the large model determines the currently generated log template T t 'With cache template library T cache If any template in cannot be merged, then T t 'Add to cache template library T as a new template cache Ensure that the cache template library can cover more log patterns and improve the matching efficiency of subsequent log parsing.

[0036] On the other hand, the present invention also provides a log parsing system based on large and small model collaboration, which is characterized by comprising a model training module, a log parsing module and a parsing result display module;

[0037] The model training module is used to extract representative log templates from system logs and train a small model. By converting historical logs into vector representations and performing hierarchical clustering, a diverse set of training samples are selected from each category to fine-tune the small model to adapt it to the log parsing task.

[0038] The log parsing module is used to parse the system log stream and dynamically optimize the template library. It realizes real-time log processing through cache template matching, large and small model collaborative parsing and template merging. It first searches the cache template library for a template that can regularly match the current log. If the match fails, it calls the small model to parse the log, and calls the large model to correct the parsing result according to the preset trigger conditions. Finally, it generates a log template and uses the large model to determine whether merging is required.

[0039] The analysis result display module is used to display the log analysis results to system operation and maintenance personnel, including the analyzed log template, variable annotations and analysis confidence, to help operation and maintenance personnel manage logs.

[0040] The above-mentioned log parsing system based on the collaboration of large and small models, the model training module includes a log data preprocessor, a training data selector and a small model trainer; the log data preprocessor is used to convert historical logs into vector representations; the training data selector is used to hierarchically cluster logs based on multi-level features and select diversified training samples; the small model trainer is used to fine-tune the small model so that it adapts to the log parsing task.

[0041] The above-mentioned log parsing system based on the collaboration of large and small models, the log parsing module includes a cache template matcher, a small model parser, a large model parser and a template dynamic optimizer; the cache template matcher is used to preferentially search for matching log templates in the cache template library; the small model parser is used to parse logs and generate preliminary templates and variable annotations; the large model parser is used to correct the parsing results; the template dynamic optimizer is used to determine whether the current log template needs to be merged with the existing log template in the cache.

[0042] In the log parsing system based on collaboration between large and small models, the parsing result display module is used to generate parsed log templates, variable annotations, and parsing confidence, and display the parsing results to operation and maintenance personnel through a visual interface, making it easier for them to manage logs.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The system and method provided by the present invention work together through the collaboration of large and small models. The small model directly completes the parsing of most logs, while the large model only corrects a small number of logs. This significantly reduces the number of times the large model is called, improves the efficiency of log parsing, and reduces the risk of privacy leakage.

[0045] (2) The system and method provided by the present invention ensure that the small model can adapt to various log modes through multi-level feature clustering and diversified sample selection, thereby improving the generalization ability of the model.

[0046] (3) The system and method provided by this invention dynamically optimizes the cache template library and combines it with a large model to analyze the structure and semantics of log templates. This enables real-time dynamic updating and optimization of the template library, reduces the generation of redundant templates, improves the accuracy of the template library, and thus enhances the efficiency and accuracy of log parsing. This system is suitable for log processing in large-scale software systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of the log parsing method based on collaboration of large and small models provided by the present invention;

[0048] Figure 2 This is a structural block diagram of the log parsing system based on collaboration of large and small models provided by the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described below by way of examples in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0050] Figure 1 It is a flowchart of the log parsing method based on the collaboration of large and small models provided by the present invention, which is divided into an offline stage and an online stage. In the offline stage, first, the model training module extracts features from the historical logs and converts them into vector representations, and then performs hierarchical clustering to select diverse training samples from each log category to train the small model. In the online stage, the log parsing module receives the real-time logs of the system and preferentially searches the cache template library for a matching log template. If the match is successful, the parsing result is directly output; if the match fails, the small model is called to parse the log, and the large model is called to correct the parsing result according to the preset trigger conditions. After the log template is generated, the large model is used to determine whether the template needs to be merged with the template in the cache template library, and the cache template library is dynamically updated. Finally, the parsing result display module displays the log parsing results to the system operation and maintenance personnel, including the parsed log template, variable annotations, and parsing confidence.

[0051] The log parsing method based on the collaboration of large and small models of the present invention includes:

[0052] For the historical log data set of the software system, the offline phase process is executed, which includes the following operations:

[0053] 11) Input historical log data set D off ={l1,l2,…,l N}, where l i As a historical log, each historical log l is extracted by feature extraction algorithm. i Convert to vector To capture the statistical features of historical logs; the feature extraction algorithm is the TF-IDF algorithm, which converts the log text into a vector representation by calculating the term frequency-inverse document frequency (TF-IDF) value of each word in the log; the calculation formula of the TF-IDF value is as follows:

[0054] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0055] Where TF(t,d) represents the word frequency of word t in log d, and IDF(t) represents the inverse document frequency of word t. The calculation formula is:

[0056]

[0057] Where N is the total number of logs, DF(t) is the number of logs containing word t;

[0058] 12) Perform hierarchical clustering on historical logs based on multi-level features to generate a log category set C = {C1, C2, ..., C K Each category corresponds to a log pattern; the multi-level features include: frequent words in the log, the length of the log, and special characters in the log;

[0059] 13) From each log category C k Select a number of samples randomly selected as the initial candidate set Then, we select some representative samples from them, and then further screen the samples with the greatest differences from the selected samples to ensure the diversity of the training set. This includes the following steps:

[0060] For each log category C k , randomly select several samples as the initial candidate set Calculate the initial candidate set Each sample e i The sum of the vector similarities with other samples is denoted as SimSum(e i ), the formula is as follows:

[0061]

[0062] Among them, cosine_similarity(e i ,e j ) represents the vector e i and e j Cosine similarity between, select SimSum(e i ) The first m smallest samples are used as representative sample sets

[0063] For each sample Calculate its difference with the selected sample set The sum of the vector similarities of all samples in is recorded as SimSumSelected(e i ), the formula is as follows:

[0064]

[0065] Select SimSumSelected(e i ) is the smallest sample as the sample with the largest difference and is added to Repeat this process until The number of samples in reaches the preset value;

[0066] 14) Using sample sets As a training set, we fine-tune the small model to adapt it to the log parsing task. Specifically, we use the Masked Language Model (MLM) task to fine-tune the small model. i , randomly mask some of its words and train a small model to predict the masked words. The model parameters are optimized by minimizing the cross entropy loss function, as follows:

[0067]

[0068] Where |V| is the vocabulary size, y i is the true label, p i The probability distribution predicted by the model. Through the masked language model task, the small model can better capture the semantic information of the log text, thereby improving the accuracy of log parsing.

[0069] For the online real-time log stream of the software system, execute the online phase process, which includes the following operations:

[0070] 21) Receive real-time log stream L on ={l1,l2,…}, for each log l t , priority is given to the cache template library T cache Find out whether there is a regular match l t If the match succeeds, the parsing result is output directly; if the match fails, the subsequent steps are continued;

[0071] 22) Parsing logs using a small model t , generate log template T t and variable annotation V t , and call the large model to log template T according to the preset trigger conditions t Make corrections; the preset trigger conditions include:

[0072] Syntax Error Detection: Log Template T t There are syntax errors in the brackets or quotation marks, which are manifested as inconsistent numbers or incorrect nesting order;

[0073] Keyword matching: log text t Contains at least one keyword from the preset keyword set K = {error, exception, warning, ...};

[0074] Confidence judgment: small model analysis result T t The confidence level C t The following conditions are met:

[0075]

[0076] where c i ∈[0,1] is the log template T t The i-th variable v in i The confidence level, |V t |For log template T t The number of variables in , γ is the preset confidence threshold;

[0077] 23) Generate the final log template T based on the analysis results of the small model or large model t ', from the cache template library T cache Select T t 'For candidate templates whose edit distance is less than the set similarity threshold β, let the large model analyze T t 'Determine whether to merge the candidate template's structural information and semantic information into a unified template;

[0078] 24) Update cache template library T cache , specifically divided into the following situations:

[0079] Template merging: If the large model determines that the currently generated log template T t 'With cache template library T cache A candidate template T in candidate Can be merged, then T t 'With T candidate Merge into a unified template T merged , and update the cache template library T cache The corresponding template in . The merged template T merged The common features of the two will be retained, while the annotations of the variable parts will be optimized to ensure the universality and accuracy of the template;

[0080] New template: If the large model determines the currently generated log template T t 'With cache template library T cache If any template in cannot be merged, then T t 'Add to cache template library T as a new template cache Ensure that the cache template library can cover more log patterns and improve the matching efficiency of subsequent log parsing;

[0081] This completes the log parsing based on the collaboration of large and small models.

[0082] Figure 2 This is a block diagram of the log parsing system based on large and small model collaboration provided by the present invention. The system includes a model training module, a log parsing module, and a parsing result display module. The following describes each module in detail.

[0083] S1) Model training module

[0084] The model training module extracts representative log templates from historical logs and trains small models. This module includes the following submodules:

[0085] S11) Log data preprocessor: used to convert historical logs into vector representations. Specifically, the TF-IDF algorithm is used to calculate the term frequency-inverse document frequency (TF-IDF) value of each word in the log and convert the log text into a vector representation.

[0086] S12) Training Data Selector: This is used to perform hierarchical clustering of logs based on multi-level features and select diverse training samples. Specifically, it selects representative samples by calculating the vector similarity of the samples and further filters out samples that are most different from the selected samples to ensure the diversity of the training set.

[0087] S13) Small model trainer: used to fine-tune the pre-trained language model using training samples. Specifically, the small model is fine-tuned using the masked language model task, and the model parameters are optimized by minimizing the cross-entropy loss function to make it suitable for the log parsing task.

[0088] S2) Log parsing module

[0089] The function of the log parsing module is to parse the log stream in real time and dynamically optimize the template library. This module includes the following submodules:

[0090] S21) Cache Template Matcher: This is used to prioritize searching for matching log templates in the cache template library. If a match is successful, the parsing result is directly output; if a match fails, the small model is called to parse the log.

[0091] S22) Small model parser: used to parse logs and generate preliminary templates and variable annotations.

[0092] S23) Large model parser: used to call the large model to correct the parsing results according to preset trigger conditions (such as syntax error detection, keyword matching and confidence judgment).

[0093] S24) Template Dynamic Optimizer: This determines whether the current log template needs to be merged with an existing log template in the cache. If so, the templates are merged into a unified template and the cache template library is updated. If not, the current template is added to the cache template library as a new template.

[0094] S3) Analysis result display module

[0095] The parsing result display module displays log parsing results to system operators. This module generates parsed log templates, variable annotations, and parsing confidence scores, and displays the parsing results to operators through a visual interface, making it easier for them to manage logs.

[0096] The log parsing method based on the collaboration of large and small models provided by this invention offers the advantages of high efficiency, accuracy, and strong adaptability. By combining offline training with online parsing, it significantly improves the accuracy and efficiency of log parsing. It also dynamically optimizes the template library, reduces the generation of redundant templates, and lowers maintenance costs. This method is suitable for log processing scenarios in large-scale software systems, effectively addressing dynamic changes in log formats and meeting the needs of complex operation and maintenance environments.

[0097] It should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A log parsing method based on collaboration between large and small models, characterized in that: This includes building a small model with diversified training samples for fine-tuning during the offline training phase, and combining the small model with the large model for collaborative log parsing and dynamic optimization of the cache template library during the online phase. Specifically: 1) In the offline phase, the software system's historical logs are converted into vector representations and hierarchically clustered. Diverse training samples are selected from each log category to fine-tune the small model to adapt it to the log parsing task. 2) In the online phase, real-time log processing of the software system is achieved through cache matching, collaborative parsing of large and small models, and template merging, and the cache template library is dynamically optimized.

2. The log parsing method based on large and small model collaboration according to claim 1, characterized in that: The offline phase of step 1) includes the following operations: 11) Input historical log data set D off ={l1,l2,…,l N }, where l i As a historical log, each historical log l is extracted by feature extraction algorithm. i Convert to vector To capture the statistical characteristics of historical logs; The feature extraction algorithm is the TF-IDF algorithm, which converts the log text into a vector representation by calculating the term frequency-inverse document frequency value of each word in the log; 12) Perform hierarchical clustering on historical logs based on multi-level features to generate a log category set C = {C1, C2, ..., C K Each category corresponds to a log pattern; the multi-level features include: frequent words in the log, the length of the log, and special characters in the log; 13) From each log category C k Randomly select several samples as the initial candidate set Select some representative samples from them, and then further filter out samples with the greatest difference from the selected samples to ensure the diversity of the training set; repeat this step until the number of selected samples reaches the preset value; 14) Using sample sets As a training set, fine-tune the small model to adapt it to the log parsing task; specifically, the Masked Language Model (MLM) task is used to fine-tune the small model; for each training sample s i , randomly mask part of its vocabulary and train a small model to predict the masked vocabulary; optimize the model parameters by minimizing the cross entropy loss function.

3. The log parsing method based on large and small model collaboration as claimed in claim 2, characterized in that: The step 13) specifically includes the following steps: For each log category C k , randomly select several samples as the initial candidate set Calculate the initial candidate set Each sample e i The sum of the vector similarities with other samples is denoted as SimSum(e i ), the formula is as follows: Among them, cosine_similarity(e i ,e j ) represents the vector e i and e j Cosine similarity between, select SimSum(e i ) The first m smallest samples are used as representative sample sets For each sample Calculate its difference with the selected sample set The sum of the vector similarities of all samples in is recorded as SimSumSelected(e i ), the formula is as follows: Select SimSumSelected(e i ) is the smallest sample as the sample with the largest difference and is added to middle.

4. The log parsing method based on large and small model collaboration according to claim 1, characterized in that: The online stage of step 2) includes the following operations: 21) Receive real-time log stream L on ={l1,l2,…}, for each log l t , priority is given to the cache template library T cache Find out whether there is a regular match l t Log template; if the match is successful, the parsing result is directly output; If the match fails, continue with the next steps; 22) Parsing logs using a small model t , generate log template T t and variable annotation V t , and call the large model to log template T according to the preset trigger conditions t Make corrections; the preset trigger conditions include: grammatical error detection, keyword matching, and confidence judgment; 23) Generate the final template T based on the analysis results of the small model or large model t ', from the cache template library T cache Select T t 'For candidate templates whose edit distance is less than the set similarity threshold β, let the large model analyze T t 'Determine whether to merge the candidate template's structural information and semantic information into a unified template; 24) Update cache template library T cache .

5. The log parsing method based on large and small model collaboration according to claim 4, characterized in that: The preset triggering conditions in step 22) are specifically: Syntax Error Detection: Log Template T t There are syntax errors in the brackets or quotation marks, which are manifested as inconsistent numbers or incorrect nesting order; Keyword matching: log text t Contains at least one keyword from the preset keyword set K = {error, exception, warning, ...}; Confidence judgment: small model analysis result T t The confidence level C t The following conditions are met: where c i ∈[0,1] is the log template T t The i-th variable v in i The confidence level, |V t |For log template T t is the number of variables in , and γ is the preset confidence threshold.

6. The log parsing method based on large and small model collaboration according to claim 4, characterized in that: The step 23) updates the cache template library, which is specifically divided into the following situations: Template merging: If the large model determines that the currently generated log template T t 'With cache template library T cache A candidate template T in candidate Can be merged, then T t 'With T candidate Merge into a unified template T merged , and update the cache template library T cache The corresponding template in; the merged template T merged The common features of the two will be retained, while the annotations of the variable parts will be optimized to ensure the universality and accuracy of the template; New template: If the large model determines the currently generated log template T t 'With cache template library T cache If any template in cannot be merged, then T t 'Add to cache template library T as a new template cache Ensure that the cache template library can cover more log patterns and improve the matching efficiency of subsequent log parsing.

7. A log parsing system based on collaboration between large and small models, characterized in that: Includes model training module, log parsing module and parsing result display module; The model training module is used to extract representative log templates from system logs and train a small model. By converting historical logs into vector representations and performing hierarchical clustering, a diverse set of training samples are selected from each category to fine-tune the small model to adapt it to the log parsing task. The log parsing module is used to parse the system log stream and dynamically optimize the template library. It realizes real-time log processing through cache template matching, large and small model collaborative parsing and template merging. It first searches the cache template library for a template that can regularly match the current log. If the match fails, it calls the small model to parse the log, and calls the large model to correct the parsing result according to the preset trigger conditions. Finally, it generates a log template and uses the large model to determine whether merging is required. The analysis result display module is used to display the log analysis results to system operation and maintenance personnel, including the analyzed log template, variable annotations and analysis confidence, to help operation and maintenance personnel manage logs.

8. The log parsing system based on large and small model collaboration as claimed in claim 7, characterized in that: The model training module includes a log data preprocessor, a training data selector and a small model trainer; the log data preprocessor is used to convert historical logs into vector representations; the training data selector is used to hierarchically cluster logs based on multi-level features and select diverse training samples; the small model trainer is used to fine-tune the small model to adapt it to the log parsing task.

9. The log parsing system based on large and small model collaboration according to claim 7, characterized in that: The log parsing module includes a cache template matcher, a small model parser, a large model parser and a template dynamic optimizer; the cache template matcher is used to preferentially search for matching log templates in the cache template library; the small model parser is used to parse the log and generate preliminary templates and variable annotations; the large model parser is used to correct the parsing results; The template dynamic optimizer is used to determine whether the current log template needs to be merged with the existing log template in the cache.

10. The log parsing system based on large and small model collaboration according to claim 7, characterized in that: The analysis result display module is used to generate the analyzed log template, variable annotation and analysis confidence, and display the analysis results to the operation and maintenance personnel through a visual interface, so that the operation and maintenance personnel can manage the logs more easily.

Images