LLM-based software package construction failure log classification system and method
The failure log classification system is built through LLM-based software packages, diverse samples are generated and redundant information is cleaned, and key features are captured in combination with Word2Vec and Transformer modules, data imbalance and redundancy problems are solved, and classification accuracy and development efficiency are improved.
Patent Information
- Application Number
- CN202510494842.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-21
- Publication Date
- 2025-08-01
AI Technical Summary
In the prior art, the classification of software package construction failed logs has problems such as unbalanced data distribution, redundant log content and complex semantics, resulting in reduced classification accuracy and dependence on developer experience.
The failure log classification system is built using LLM-based software packages, and a variety of samples are generated through the sample generation module and redundant information is cleaned using the semantic alignment module. The Word2Vec and Transformer modules are used to capture error characteristics and key information for classification.
It improves the classification accuracy of the model's software package construction failure log, improves the generalization ability and classification performance of the model, reduces interference with redundant information, and improves development efficiency.
Smart Images
Figure CN120407256A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep learning technology, and particularly to a software package build failure log classification system and method based on LLM. Background Art
[0002] With the update of the Linux distribution based on the RISC-V architecture and the increase in the number of users, its software package ecosystem has developed rapidly, and the build logs have increased sharply. Software package build failures occur frequently, and developers usually spend a lot of time manually analyzing the build failure logs and trying to fix them. Accurate classification of software package build failure logs can help developers quickly and efficiently locate build failures, thereby improving the efficiency of the development process and accelerating the ecological construction.
[0003] Regarding the related work on software package build analysis, the literature (Huang Tangkun, Liu Jie, Zhu Jiaxin. Empirical Research on Software Package Build Problems in RISC-V Linux Distribution [J]. Journal of Guangxi University (Natural Science Edition), 2023, 48(04): 943-957. DOI: 10.13624 / j.cnki.issn.1001-7445.2023.0943.) describes a build error log classification framework. In this framework, log preprocessing is first performed, and the build failure logs are binary-classified into architecture-related error logs and architecture-unrelated error logs. Then, text preprocessing is carried out. Before constructing the text vector model, the original log text is further simplified, including removing stop words and punctuation marks, converting all words to lowercase, and then performing text tokenization, stemming, and lemmatization on the words. Next is the construction of feature vectors. For each log block after text preprocessing, each word is converted into a vector representation. Finally, the feature vectors of the logs are used as the input of the logistic regression algorithm model.
[0004] The problems of the above method are as follows:
[0005] (1) The distribution of error log data is unbalanced. Build failure logs usually contain multiple error types, such as missing dependencies, test case failures, memory overflows, etc. The distribution of these error types is extremely uneven. Build configuration problems, test case problems, and incompatibility problems account for 72% of the whole, while plugin problems and network problem samples only account for 3% of the dataset. In the case of such unbalanced data, traditional deep learning models may not be able to fully learn their features, resulting in misclassification and a decrease in their classification accuracy.
[0006] (2) Log texts often contain redundant information. To record the entire build process, a large amount of redundant information such as timestamps, file paths, and process numbers needs to be output in the log; moreover, the language expression in the log is not fluent natural language, which forms a complex context and further increases the difficulty of classification. Deep learning methods have certain defects in understanding semantics. For example, the error message in the log of the python - pytest - jupyter package build failure is "File " / usr / lib64 / python3.8 / threading.py", line 559, in wait", which is misclassified by the deep learning model as a file missing problem, but actually it should belong to a timeout problem. The reason for this misclassification is that deep learning only focuses on the keyword "File" in this log and cannot understand the meaning of "in wait", because most timeout problems are related to "time" such as "timeout". This indicates that traditional models may only rely on local keywords and shallow features and cannot accurately capture the deep semantics and context relationships in the log. This package was built on August 24, 2023, but was not fixed until March 6, 2024, showing the complexity of the software package build error repair process and its dependence on developers' empirical knowledge.
[0007] Problems such as the unbalanced distribution of build failure logs, redundant log content, and complex semantics need to be solved by this application. Summary of the Invention
[0008] This application provides a software package build failure log classification system and method based on LLM. Its technical purpose is to combine sample generation and semantic alignment to alleviate the problem of unbalanced failure log data and improve the classification accuracy of the model for display feature categories.
[0009] The above - mentioned technical purpose of this application is achieved through the following technical solutions:
[0010] A software package build failure log classification system based on LLM, including:
[0011] The LLM module, including a sample generation module and a semantic alignment module. The sample generation module generates the first log sample data according to the Prompt instruction and the original log data set; the semantic alignment module cleans the redundant information in the first log sample data, and then extracts the error features and key information therein to obtain the second log sample data;
[0012] The model classification module captures and outputs the classification labels and local keyword features of the second log sample data;
[0013] The performance evaluation module evaluates the performance of the model classification module according to the classification labels and local keyword features;
[0014] Among them, the Prompt instruction includes a sample generation instruction and a semantic alignment instruction. The sample generation instruction includes a log classification criterion, and the semantic alignment instruction includes a CoT method and a few-shot learning method.
[0015] Further, the sample generation module swaps the names and positions of the logs in the original log dataset according to the log classification criterion, generates diverse samples for the logs in the case of construction failure, avoids intersections with the logs of other labels, and finally controls the distribution of the generated samples according to the balance parameter to obtain the first log sample data.
[0016] Further, the semantic alignment module analyzes the timestamps, stack traces, and other irrelevant parts in the first log sample data according to the semantic alignment instruction, extracts the error features and key information from the first log sample data, and outputs the second log sample data.
[0017] Further, the model classification module includes a Word2Vec module and a Transformer module. The Word2Vec module captures the error features and key information in the second log sample data and generates a static word vector representation sequence. The Transformer module captures the long-distance dependence relationships in the word vector representation sequence to obtain the classification labels and local keyword features of the second log sample data.
[0018] Further, the performance evaluation module generates a feature visualization graph of the local keyword features according to the t-SNE method, and evaluates the clustering effect of the model classification module according to the feature visualization graph.
[0019] Further, the performance evaluation module assigns manual annotation labels to the construction failure logs in the original log dataset, calculates the P, R, and F1 metrics according to the manual annotation labels and the classification labels, and evaluates the performance of the model classification module according to the P, R, and F1 metrics.
[0020] Further, the performance evaluation module draws the confusion matrix of the classification labels according to the confusion matrix generation function, and evaluates the misclassification situation of the model classification module according to the confusion matrix.
[0021] A method for classifying software package construction failure logs based on an LLM, including:
[0022] Generating first log sample data according to the Prompt instruction and the original log dataset;
[0023] Cleaning the redundant information in the first log sample data and extracting the error features and key information therein to obtain the second log sample data;
[0024] Capture and output the classification labels and local keyword features of the second log sample data;
[0025] Evaluate the classification performance according to the classification labels and local keyword features;
[0026] Among them, the Prompt instruction includes a sample generation instruction and a semantic alignment instruction. The sample generation instruction includes a log classification standard, and the semantic alignment instruction includes a CoT method and a few-shot learning method.
[0027] The beneficial effect of this application is that the LLM-based software package construction failure log classification system and method described in this application generate diverse samples through the LLM to adjust the balance. Without overfitting, it can effectively improve the model's feature learning ability for rare sample categories, thereby improving the model's classification performance and enhancing the model's generalization ability.
[0028] At the same time, through the powerful semantic understanding ability of the LLM, irrelevant information in the log is removed, and error features are extracted, making it easier for the model to capture key error patterns, and the cleaned log reduces interference with model training.
[0029] This application significantly improves the classification accuracy of the model for software package construction failure logs, providing not only strong method support for developers in the RISC-V ecosystem but also new ideas for log classification tasks in other fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a module diagram of the LLM-based software package construction failure log classification system in the embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0031] The technical solution of this application will be described in detail below with reference to the accompanying drawings.
[0032] As Figure 1 shown, the LLM-based software package construction failure log classification system described in this application includes an LLM module, a model classification module, and a performance evaluation module. The LLM module includes a sample generation module and a semantic alignment module.
[0033] The sample generation module is used to generate the first log sample data according to the Prompt instruction and the original log data set. Among them, the Prompt instruction includes a sample generation instruction and a semantic alignment instruction. The sample generation instruction includes a log classification standard, and the semantic alignment instruction includes a CoT algorithm and a few-shot learning algorithm.
[0034] Furthermore, the sample generation module swaps the names and locations of the logs in the original log dataset according to the log classification criteria, generates diverse samples for the logs with build failure situations, avoids intersections with the logs of other tags, and finally controls the distribution of the generated samples according to the balance parameter to obtain the first log sample data.
[0035] Specifically, the specific implementation steps of the sample generation module are as follows:
[0036] (1) By providing classification criteria, the LLM can understand the boundaries of different error categories and avoid generating ambiguous samples. For example, when generating logs for plugin problems, it is necessary to ensure that they do not contain descriptions of missing dependencies for build configuration problems.
[0037] (2) During the generation process, while swapping names and locations, the LLM should also boldly carry out diversified innovation without intersecting with other tags. For example, for plugin problems, the LLM can generate logs describing different plugin loading failure scenarios, such as "Plugin 'X' failed to load due to missing dependency 'Y'" or "Plugin 'Z' encountered an initialization error: undefined symbol 'A'". These generated samples have no similar data in the original dataset but strictly conform to the classification criteria for plugin problems.
[0038] (3) Control the distribution of the generated samples by adjusting the balance parameter (0.1, 0.2, 0.3). For example, a balance of 0.1 means that the sample size of the low-frequency category is increased to 10% of the highest-frequency category. Direct oversampling may cause model overfitting, while gradually increasing the balance can verify the marginal contribution of the generated samples to the classification effect. For example, in the case of plugin problems (only 13 original training samples), the generated sample size is gradually increased from 26 (balance 0.1) to 105 (balance 0.3) to observe the changing trend of the subsequent model performance.
[0039] The semantic alignment module analyzes the timestamps, stack traces, and other irrelevant parts in the first log sample data according to the semantic alignment instructions, extracts the error features and key information from the first log sample data, and outputs the second log sample data.
[0040] Furthermore, the semantic alignment module improves the model's sensitivity to key patterns by cleaning redundant information (such as timestamps, process IDs) in the first log sample data and refining core error features. The processing of the content of the first log sample data includes the following dimensions:
[0041] (1) Log cleaning: Remove irrelevant and redundant information from the logs.
[0042] (2) Feature extraction: Extract the main error features from the cleaned log information.
[0043] The specific implementation steps of the semantic alignment module include:
[0044] (1) Coarse-grained cleaning of the first log sample data after sample generation, deleting noise fields with fixed formats. For example, remove the timestamp (such as [2023-03-02 15:16:08]) and process ID (such as PID: 1234).
[0045] (2) Utilize the semantic understanding ability of the LLM to extract the core error features from the cleaned logs. For example, refine the original log: "[INFO]2023-03-02 15:16:08, PID 5678: Building package 'A'... [ERROR] Failed to locate file ' / path / to / B.c': No such file or directory." to: "File missing error: No such file or directory."
[0046] Furthermore, the design steps of the Prompt instruction are as follows:
[0047] (1) The design of the Prompt instruction in the sample generation part provides the log classification standard for the LLM, enabling the LLM to generate diverse log samples that conform to the real scenario. The advantage of this design is that the LLM can utilize its powerful semantic understanding ability to generate samples with high diversity, thus effectively supplementing the data distribution of low-frequency categories. The sample generation Prompt design is as follows:
[0048] Role: You are a sample generation assistant
[0049] Task: Please generate samples that conform to the E6 incompatibility problem classification standard described by me before.
[0050] Requirements:
[0051] · Avoid intersections with other tags.
[0052] · Generate as diverse samples as possible.
[0053] · Maintain the following data format:
[0054] – One column is the integer 6
[0055] – One column is the log content
[0056] Reference log content: {log_text}
[0057] By generating the Prompt design through the above samples, the classification performance of the model for error categories with scarce samples can be improved.
[0058] (2) For the Prompt instruction design in the semantic alignment part, the CoT method (Chain-of-Thought) and the few-shot learning method are adopted to train the LLM. The CoT method helps the LLM more accurately extract the core error description by guiding it to gradually reason about the error patterns in the log. The few-shot learning algorithm enables the LLM to quickly adapt to the log cleaning tasks in a specific domain by providing a small number of high-quality samples. The goal of this design is to enhance the LLM's ability to clean and refine log texts, ensuring that key error patterns are not lost or generalized. The semantic alignment Prompt design is as follows:
[0059] Role: You are a log processing assistant
[0060] Task: Filter the log by removing irrelevant content and only retaining the key error information.
[0061] Thought process:
[0062] (a) Analyze the timestamps, stack traces, and other irrelevant parts in the log.
[0063] (b) Extract the error features and key information.
[0064] (c) Output the concise log after filtering
[0065] Example:
[0066] · Input example 1:
[0067] 2023-02-18 10:05:25,200 WARNING[thread-1]-FileNotFoundException: / path / to / file not found
[0068] · Output example 1:
[0069] FileNotFoundException: / path / to / file not found
[0070] ·...
[0071] Log content to be processed: {log_text}
[0072] Through the above semantic alignment Prompt design, the classification accuracy of the model for explicit feature categories can be improved.
[0073] Furthermore, by using an LLM (Large Language Model) to generate diverse log samples, the problem of data distribution imbalance can be solved, especially for error categories with scarce samples (such as plugin problems and network problems). This application provides the LLM with an original log dataset and detailed classification criteria to ensure that the generated samples strictly follow the category definitions and do not have semantic overlap with other categories and describe specific scenarios of plugin loading or compilation failures.
[0074] The model classification module is used to capture and output the classification labels and local keyword features of the second log sample data.
[0075] Furthermore, the model classification module adopts a dual-channel architecture, combining the advantages of Transformer and Word2Vec to capture the global semantics and local keyword features of log texts.
[0076] Specifically, the model classification module includes a Word2Vec module and a Transformer module. The Word2Vec module captures the error features and key information in the second log sample data and generates a static word vector representation sequence. The Transformer module captures the long-distance dependencies in the word vector representation sequence to obtain the classification labels and local keyword features of the second log sample data.
[0077] The specific implementation steps of the model classification module are as follows:
[0078] (1) Input the semantically aligned training set into the deep learning model.
[0079] (2) As a classic word embedding model, the Word2Vec module can generate static word vector representations according to a specific log corpus. These static word vector representations capture the semantic information of local keywords in the log text, such as the semantics of fixed phrases like "No such file" or "timeout".
[0080] (3) The Transformer module endows the model with the extraordinary ability to capture long-distance dependencies in the sequence through the self-attention mechanism, and can understand the long-distance dependencies and complex semantics in the log, such as the causal relationship between "dependency missing" and "build failure".
[0081] (4) Input the test set into the deep learning model to obtain the dataset after model classification.
[0082] The performance evaluation module is used to evaluate the performance of the model classification module according to the classification label and the local keyword feature.
[0083] Furthermore, the performance evaluation module generates a feature visualization graph of the local keyword feature according to the t-SNE method, and evaluates the clustering effect of the model classification module according to the feature visualization graph.
[0084] Furthermore, the performance evaluation module marks the construction failure logs in the original log dataset with manual annotation labels, calculates the P, R, and F1 metrics according to the manual annotation labels and the classification labels, and evaluates the performance of the model classification module according to the P, R, and F1 metrics. Among them, P represents precision, and R represents recall.
[0085] Furthermore, the performance evaluation module draws the confusion matrix of the classification label according to the confusion matrix generation function, and evaluates the misclassification situation of the model classification module according to the confusion matrix.
[0086] The software package construction failure log classification method based on LLM described in this application is implemented based on the software package construction failure log classification system based on LLM, and specifically includes:
[0087] Generate the first log sample data according to the Prompt instruction and the original log dataset;
[0088] Clean the redundant information in the first log sample data, and extract the error features and key information therein to obtain the second log sample data;
[0089] Capture and output the classification label and the local keyword feature of the second log sample data;
[0090] Evaluate the classification performance according to the classification label and the local keyword feature;
[0091] Among them, the Prompt instruction includes a sample generation instruction and a semantic alignment instruction. The sample generation instruction includes a log classification standard, and the semantic alignment instruction includes the CoT method and the few-shot learning method.
[0092] The above is a demonstration embodiment of this application, and the protection scope of this application is realized by the claims and their equivalents.
Claims
1. A software package build failure log classification system based on LLM, characterized in that, Including: The LLM module, including a sample generation module and a semantic alignment module. The sample generation module generates the first log sample data according to the Prompt instruction and the original log dataset; The semantic alignment module cleans the redundant information in the first log sample data, and then extracts the error features and key information therein to obtain the second log sample data; The model classification module captures and outputs the classification labels and local keyword features of the second log sample data; The performance evaluation module evaluates the performance of the model classification module according to the classification labels and local keyword features; Among them, the Prompt instruction includes a sample generation instruction and a semantic alignment instruction. The sample generation instruction includes a log classification criterion, and the semantic alignment instruction includes the CoT method and the few-shot learning method.
2. The software package build failure log classification system according to claim 1, characterized in that The sample generation module swaps the names and positions of the logs in the original log dataset according to the log classification criterion, generates diverse samples for the logs in the case of construction failure, avoids intersections with the logs of other labels, and finally controls the distribution of the generated samples according to the balance parameter to obtain the first log sample data.
3. The software package build failure log classification system according to claim 2, wherein The semantic alignment module analyzes the timestamps, stack traces, and other irrelevant parts in the first log sample data according to the semantic alignment instruction, and extracts the error features and key information of the first log sample data according to this analysis to obtain and output the second log sample data.
4. The software package build failure log classification system according to claim 3, characterized in that, The model classification module includes a Word2Vec module and a Transformer module. The Word2Vec module captures the error features and key information in the second log sample data and generates a static word vector representation sequence. The Transformer module captures the long-distance dependencies in the word vector representation sequence to obtain the classification labels and local keyword features of the second log sample data.
5. The software package build failure log classification system according to claim 4, wherein The performance evaluation module generates a feature visualization graph of the local keyword features according to the t-SNE method, and evaluates the clustering effect of the model classification module according to the feature visualization graph.
6. The software package build failure log classification system according to claim 5, wherein The performance evaluation module assigns manual annotation labels to the construction failure logs in the original log dataset, calculates the P, R, and F1 metrics according to the manual annotation labels and classification labels, and evaluates the performance of the model classification module according to the P, R, and F1 metrics.
7. The software package build failure log classification system according to claim 6, wherein The performance evaluation module draws the confusion matrix of the classification labels according to the confusion matrix generation function, and evaluates the misclassification situation of the model classification module according to the confusion matrix.
8. A method for classifying software package build failure logs based on LLM, characterized in that, Including: Generate the first log sample data according to the Prompt instruction and the original log dataset; Clean the redundant information in the first log sample data, and extract the error features and key information therein to obtain the second log sample data; Capture and output the classification labels and local keyword features of the second log sample data; Evaluate the classification performance according to the classification labels and local keyword features; Among them, the Prompt instruction includes a sample generation instruction and a semantic alignment instruction. The sample generation instruction includes a log classification standard, and the semantic alignment instruction includes a CoT method and a few-shot learning method.