Construction method and device of software defect prediction data set based on large language model

Abstract

The invention relates to the technical field of software, in particular to a method and device for constructing a software defect prediction data set based on a large language model.The method comprises the steps that the software defect prediction data set is obtained, a plurality of large language models are selected to annotate the same software defect prediction data set from different annotation dimensions respectively, and the annotation dimensions of the software defect prediction data set are obtained; obtaining a plurality of annotation results; a voting model is selected to evaluate the annotation result so as to screen out a target large language model under each annotation dimension, and the target large language model has a better annotation effect in the corresponding annotation dimension compared with other large language models; constructing the target large language model under each annotation dimension into a model pool, wherein the model pool is used for constructing a subsequent software defect prediction data set; and selecting a comparative large language model from the target large language model, and comparing the data under a certain annotation dimension corresponding to the comparative large language model with the sampling data selected from the model pool to verify the data quality in the model pool.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software, in particular to a method and device for constructing a software defect prediction dataset based on a large language model. BACKGROUND

[0002] Software defect prediction is a crucial research direction in the field of software engineering. Its main goal is to predict potential future defects at an early stage of the development process, guiding the optimal allocation of testing resources, thereby reducing repair costs and improving software quality. In the context of the continuous expansion of open source projects and the proliferation of automated development tools, defect prediction technology should be integrated into the entire software development life cycle as early as possible to maximize software quality.

[0003] Existing research has made progress in software defect prediction; however, the increasing complexity of models highlights interpretability issues: developers cannot accurately locate the position of software defects, and it is difficult to understand the decision logic of the model, which limits trust and debugging efficiency in practical applications. In practical applications, there is a disconnect between detected software defects and their explanations or modification suggestions. Most methods only stop at ranking the importance of static code features, failing to provide actionable repair suggestions.

[0004] Therefore, there is an urgent need for a construction method that can simultaneously output defect type, location, and repair suggestions to improve the interpretability and practical value of the model. SUMMARY

[0005] The purpose of the present application is to provide a method and device for constructing a software defect prediction dataset based on a large language model, to enhance the interpretability of software defect prediction by constructing a high-quality multilingual dataset.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows: In a first aspect, an embodiment of the present application provides a method for constructing a software defect prediction dataset based on a large language model, the method comprising: obtaining a software defect prediction dataset; selecting a plurality of large language models to respectively annotate the same software defect prediction dataset from different annotation dimensions to obtain a plurality of annotation results; selecting a voting model to evaluate the annotation results to filter out a target large language model under each annotation dimension, the target large language model being better than other large language models in annotation effect on the corresponding annotation dimension; constructing the target large language model under each annotation dimension into a model pool, the model pool being used to construct a subsequent software defect prediction dataset; selecting a comparative large language model from the target large language models, and comparing data under a certain annotation dimension corresponding to the comparative large language model with sampling data selected from the model pool to verify the quality of data in the model pool.

[0007] In a second aspect, an embodiment of the present application also provides a device for constructing a software defect prediction dataset based on a large language model, the device comprising: an obtaining module configured to obtain a software defect prediction dataset; an annotation module configured to select a plurality of large language models to respectively annotate the same software defect prediction dataset from different annotation dimensions to obtain a plurality of annotation results; an evaluation module configured to select a voting model to evaluate the annotation results to filter out a target large language model under each annotation dimension, the target large language model being better than other large language models in annotation effect on the corresponding annotation dimension; a construction module configured to construct the target large language model under each annotation dimension into a model pool, the model pool being used to construct a subsequent software defect prediction dataset; and a comparison module configured to select a comparative large language model from the target large language models, and compare data under a certain annotation dimension corresponding to the comparative large language model with sampling data selected from the model pool to verify the quality of data in the model pool.

[0008] The embodiment of the application provides a kind of based on the construction method and device of software defect prediction data set of large language model, which comprises obtaining software defect prediction data set, selecting multiple large language models respectively to same software defect prediction data set from different annotation dimensions are annotated, obtain multiple annotation results;Select the voting model to evaluate the annotation result, to filter out the target large language model under each annotation dimension, the target large language model is compared with other large language models in the annotation effect of corresponding annotation dimension is more excellent;The target large language model under each annotation dimension is constructed into model pool, and the model pool is used to construct subsequent software defect prediction data set;Select a comparative large language model from target large language model, and compare the data under certain annotation dimension corresponding to comparative large language model with the sampling data selected from model pool, to verify the quality of data in model pool;The large language model in the model pool finally obtained by the scheme can be used to predict software defects, and more accurate prediction data can be obtained.

[0009] In order to make the above-mentioned purpose, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0011] Figure 1 The flowchart of the construction method of the software defect prediction data set based on the large language model provided by the embodiment of the present application is shown.

[0012] Figure 2 The function module schematic diagram of the construction device of the software defect prediction data set based on the large language model provided by the embodiment of the present application is shown.

[0013] FIG. 100 - construction device of software defect prediction data set based on large language model; 110 - acquisition module; 120 - annotation module; 130 - evaluation module; 140 - construction module; 150 - comparison module. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0015] It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0016] Please refer to Figure 1 is a flowchart of a method for constructing a software defect prediction dataset based on a large language model provided by an embodiment of the present application. The method comprises: S110, obtaining a software defect prediction dataset.

[0017] Specifically, a plurality of publicly available software defect identification and bug fixing datasets are selected, which cover four mainstream programming languages: Java, C / C++, Python and JavaScript. The goal is to construct a diversified large-scale software defect prediction dataset.

[0018] Further, due to the problems of abnormal values, duplicate data and inconsistent code granularity in the original code dataset, preprocessing of the dataset is crucial. The preprocessing steps are as follows: Abnormal value removal: delete excessively long or short code segments and verify the integrity of the remaining code.

[0019] Code granularity unification: use regular expressions to identify and standardize the granularity of classes and functions.

[0020] Duplicate data deletion: delete duplicate data entries to ensure uniqueness.

[0021] Token calculation: estimate the average Token size of each subset in the dataset.

[0022] Finally, all data is converted to a unified JSON format for subsequent processing and model input, resulting in a clean, standardized dataset suitable for software defect prediction tasks.

[0023] S120, Select multiple large language models to annotate the same software defect prediction dataset from different annotation dimensions to obtain multiple annotation results.

[0024] Specifically, these multiple large language models include, but are not limited to, GPT-4o, DeepSeekV3, Claude-3.5-Sonnet, Gemini-1.5-Pro-latest, and Yi-lightning. These models are then selected to annotate the same software defect prediction dataset from multiple dimensions, such as functional description generation, CWE (defect enumeration) defect prediction, and defect remediation suggestions, resulting in multiple annotation results. For example, if 500 data points are randomly selected from the software defect prediction dataset, and each of the five large language models is independently annotated, the resulting dataset will have different annotations from the five models, laying the foundation for subsequent model evaluation and selection.

[0025] The annotation dimension—functional description generation—can be understood as follows: the large language model generates a functional description for a given code segment. The user instruction structure is as follows: "Analyze the following code and concisely describe its core function in no more than 30 words:" followed by the target code. This setup requires the model to reconstruct the expected behavior from the implementation pattern. The prediction range under the conditions of unknown function (ϕ1) and known function (ϕ2) must meet the following requirements: ϕ1 ∩ ϕ2 = ϕ3 and ϕ3 ̸≠ ∅ This constraint requires explicit requirements integration, guiding the predictor to identify defects within a constrained operational context. This targeted detection enables context-aware remediation generation, preserving functional specifications while adhering to practical software engineering constraints.

[0026] The annotation dimension—CWE defect prediction—can be understood as follows: after the functional description is generated, the framework initiates defect detection within the formally defined context ϕ3. The system prompts strategically configure three key elements for large language models: (a) simulated code review roles, (b) applicable static analysis techniques, and (c) security-centric operational constraints.

[0027] The annotation dimension—defect repair suggestions can be understood as follows: Based on the technical overlap between defect prediction and code repair, the system prompts for CWE defect prediction have been optimized through three key modifications: (1) context role specification, (2) elimination of redundant instruction components, and (3) integration of enhanced development context and technical background specification.

[0028] S130, Select a voting model to evaluate the annotation results, so as to select the target large language model under each annotation dimension. The target large language model has better annotation effect than other large language models in the corresponding annotation dimension.

[0029] To select a pool of large-scale annotation models that achieves the optimal balance between performance and cost, this scheme implements a voting mechanism to systematically eliminate poorly performing annotation models. This voting model is configured to simulate the role of a "teacher," responsible for evaluating the quality of the "students" (large language models used in the annotation phase) and identifying the "least suitable" items. Priority is given to the first task, "Functional Description," and the second task, "CWE Defect Prediction," while the third task ("Defect Repair Suggestion Generation"), which is highly dependent on the outputs of the first two tasks, is not involved in the voting process.

[0030] Specifically, at least one voting model is selected to vote on the annotation results multiple times. The voting results of each model are combined to select the target large language model for each annotation dimension. This target large language model performs better than other large language models in the corresponding annotation dimension. An example is given below: The first round of voting was conducted using "DeepSeek V3" and "Yi-lightning". Each voting model underwent three rounds of voting, and the results are as follows: Yi-lightning: The variance for the first round was 24.5333, and the variance for the second round was 81.5333. DeepSeek V3: The average variance for the first round is 15.2, and the average variance for the second round is 19.15, making it more reliable than Yi-lightning.

[0031] Claude-3.5-Sonnet: Performed poorly in the first round and is not suitable for the "Function Description" task.

[0032] Gemini-1.5-Pro-latest: was deemed "inappropriate" and effectively rejected by both voting models.

[0033] GPT-4o: Performed best in the evaluation of both voting models.

[0034] Yi-lightning exhibits relatively high variance and frequently self-eliminates itself during the voting process, leading to its elimination and leaving DeepSeek V3.

[0035] GPT-4o performed well during the evaluation process; therefore, GPT-4o was used for the second round of voting, and the voting results are as follows: GPT-4o: The average variance of the two rounds was 27.8333 and 64.1833, respectively.

[0036] DeepSeek V3: Showed lower variance than GPT-4o, further supporting the reliability of DeepSeek V3 variance. Similar voting trends indicated that its bias was also acceptable.

[0037] Claude-3.5-Sonnet: Should not be completely ruled out as it performed well in "defect prediction."

[0038] S140, the target large language model under each batch annotation dimension is constructed into a model pool, which is used to construct subsequent software defect prediction datasets.

[0039] Specifically, when the batch annotation dimension is function description generation, GPT-4o and DeepSeekV3 are selected to construct a model pool, and the sampling ratio of the subsequent software defect prediction dataset is 1:2; When the batch annotation dimension is CWE defect prediction, GPT-4o, DeepSeekV3, and Claude-3.5-Sonnet are selected to construct a model pool, and the sampling ratio of the subsequent software defect prediction dataset is 4:3:3; When the batch annotation dimension is defect repair suggestion, GPT-4o, DeepSeekV3, Claude-3.5-Sonnet, and Gemini-1.5-Pro-latest are selected to construct a model pool, and the sampling ratio of the subsequent software defect prediction dataset is 1:1:1:1.

[0040] S150, a comparison large language model is selected from the target large language model, and the data corresponding to a certain batch annotation dimension of the comparison large language model is compared with the sampling data selected from the model pool to verify the quality of the data in the model pool.

[0041] Specifically, for the "function description" task, the "m3e" embedding model is used to process the synthetic data from GPT-4o and the sampling data from the model pool. Then, the cosine similarity of 500 pairs of data entries was calculated, and the average similarity score was 0.74.

[0042] In the "CWE Defect Prediction" task, CWE type numbers were extracted from the outputs of GPT-4o and the model pool. The number of occurrences of each CWE defect type was counted. GPT[1]4o and the sampled data both contained 34 different CWE types, 18 of which were common to both. The most frequently occurring CWE type was "CWE-20 (Improper Input Validation)." Instances where no defect was predicted were recorded as "CWE-0." This process resulted in two binary sparse matrices with a shape of (500, 50). We calculated the Jaccard similarity between these matrices, scoring 0.54. Additionally, the number of "pass!" instances in our model pool results was 75, while GPT-4o had 92, further demonstrating the superiority of our model pool strategy.

[0043] For the "Defect Fix Suggestion" task, we compared the fix outputs of large language models at the "line" level with the fix_code in the original dataset and with the differences from the original code, calculating the intra-line repetition rate. GPT-4o and our model pool scored 0.6062 and 0.6190, respectively. This indicates that our approach is superior to GPT-4o compared to the true situation in the original dataset, with a final "Defect Fix Suggestion" task score of 1.62.

[0044] Finally, the overall quality assessment score of the subsequent software defect prediction dataset constructed by the model pool is about 72.5, indicating that the dataset is very satisfactory.

[0045] Therefore, the technical solutions of the present application mainly contribute in three key dimensions: 1) a multilingual, interpretable defect prediction dataset for large language model training, Hand[1]Pick, is proposed; 2) a thought chain framework designed specifically for the defect prediction task, TriCogVuln-LLM, is developed, which balances data synthesis costs, data quality, and diversity; 3) an evaluation method is designed for TriCogVuln-LLM. It is worth noting that using the designed evaluation method, the dataset obtained a high score of 72.5, highlighting its effectiveness in defect prediction for different programming languages and addressing the lack of interpretability in existing defect prediction datasets.

[0046] In addition, the embodiment of the application also provides a software defect prediction dataset construction device 100 based on a large language model, as shown in Figure 2 , which comprises: An acquisition module 110 is configured to acquire a software defect prediction dataset.

[0047] In the embodiment of the present application, S110 is executed by the acquisition module 110.

[0048] The comment module 120 is configured to select a plurality of large language models to comment on the same software defect prediction dataset from different comment dimensions to obtain a plurality of comment results.

[0049] In the embodiment of the present application, S120 is executed by the comment module 120.

[0050] The evaluation module 130 is configured to select a voting model to evaluate the comment results to screen out a target large language model under each comment dimension, wherein the target large language model has a better comment effect than other large language models in the corresponding comment dimension.

[0051] In the embodiment of the present application, S130 is executed by the evaluation module 130.

[0052] The construction module 140 is configured to construct the target large language model under each comment dimension into a model pool, wherein the model pool is used to construct a subsequent software defect prediction dataset.

[0053] In the embodiment of the present application, S140 can be executed by the construction module 140.

[0054] The comparison module 150 is configured to select a comparison large language model from the target large language model, and compare data under a certain comment dimension corresponding to the comparison large language model with sample data selected from the model pool to verify the quality of the data in the model pool.

[0055] In the embodiment of the present application, S150 can be executed by the comparison module 150.

[0056] Since the construction method of the software defect prediction dataset based on the large language model has been described in detail in the above, it will not be repeated here.

[0057] To sum up, the embodiment of the application provides a method and device for constructing a software defect prediction data set based on a large language model, the method comprising: obtaining a software defect prediction data set, selecting a plurality of large language models to respectively annotate the same software defect prediction data set from different annotation dimensions to obtain a plurality of annotation results; selecting a voting model to evaluate the annotation results to filter out a target large language model under each annotation dimension, the target large language model being better than other large language models in annotation effect in the corresponding annotation dimension; constructing the target large language model under each annotation dimension into a model pool, the model pool being used to construct a subsequent software defect prediction data set; selecting a comparative large language model from the target large language models, and comparing data in a certain annotation dimension corresponding to the comparative large language model with sampling data selected from the model pool to verify the quality of data in the model pool; and predicting software defects by using the large language models in the finally obtained model pool, so that more accurate prediction data can be obtained.

[0058] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are only schematic, for example, the flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementation manners, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0059] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0060] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. It should be noted that, in this document, relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0061] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0062] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

Claims

1. A method for constructing a software defect prediction dataset based on a large language model, characterized in that, The method comprises: obtaining a software defect prediction dataset; selecting multiple large language models to respectively annotate the same software defect prediction dataset from different annotation dimensions to obtain multiple annotation results; selecting a voting model to evaluate the annotation results to filter out a target large language model under each annotation dimension, the target large language model being better than other large language models in annotation effect on the corresponding annotation dimension; constructing the target large language model under each annotation dimension into a model pool, the model pool being used to construct a subsequent software defect prediction dataset; selecting a comparative large language model from the target large language models, and comparing data under a certain annotation dimension corresponding to the comparative large language model with sampling data selected from the model pool to verify the quality of data in the model pool.

2. The method of claim 1, wherein the method is based on a large language model. The step of obtaining a software defect prediction dataset comprises: selecting multiple publicly available software defect identification and bug fixing datasets to constitute the software defect prediction dataset, the software defect prediction dataset covering multiple mainstream programming languages.

3. The method of claim 1, wherein the method further comprises: After obtaining the software defect prediction dataset, the method further comprises the step of: preprocessing the software defect prediction dataset, the preprocessing comprising: abnormal value removal, including deleting code segments with lengths not within a preset value range; code granularity unification, including identifying and standardizing the granularity of classes and functions using regular expressions; repeated data deletion, including deleting repeated data entries.

4. The method of claim 1, wherein the method is based on a large language model. The step of selecting multiple large language models to respectively annotate the same software defect prediction dataset from different annotation dimensions to obtain multiple annotation results comprises: The multiple large language models comprise GPT-4o, DeepSeekV3, Claude-3.5-Sonnet, Gemini-1.5-Pro-latest, and Yi-lightning; and the annotation dimensions comprise function description generation, CWE defect prediction, and defect fixing suggestions. Randomly selecting a specific number of data points from the software defect prediction dataset, and selecting multiple large language models to respectively annotate the same specific number of data points from different annotation dimensions to obtain multiple annotation results.

5. The method of claim 1, wherein the method is based on a large language model. The step of selecting a voting model to evaluate the annotation results to filter out a target large language model under each annotation dimension comprises: selecting at least one voting model to vote multiple times on the annotation results, and combining each voting result of each voting model to filter out a target large language model under each annotation dimension.

6. The method of claim 4, wherein the method further comprises: The step of constructing the target large language model under each annotation dimension into a model pool, the model pool being used to construct a subsequent software defect prediction dataset comprises: When the annotation dimension is function description generation, GPT-4o and DeepSeekV3 are selected to construct a model pool, and the sampling ratio of the subsequent software defect prediction dataset is 1:

2. When the annotation dimension is CWE defect prediction, GPT-4o, DeepSeekV3, and Claude-3.5-Sonnet are selected to build a model pool, and the sampling ratio of the subsequent software defect prediction dataset is 4:3:3; When the annotation dimension is defect repair suggestion, GPT-4o, DeepSeekV3, Claude-3.5-Sonnet, and Gemini-1.5-Pro-latest are selected to build a model pool, and the sampling ratio of the subsequent software defect prediction dataset is 1:1:1:

1. 7.A device for constructing a software defect prediction dataset based on a large language model, characterized in that, The device comprises: an acquisition module configured to acquire a software defect prediction dataset; an annotation module configured to select multiple large language models to respectively annotate the same software defect prediction dataset from different annotation dimensions, and obtain multiple annotation results; an evaluation module configured to select a voting model to evaluate the annotation results, so as to screen out a target large language model under each annotation dimension, the target large language model being better than other large language models in annotation effect under the corresponding annotation dimension; a construction module configured to construct the target large language model under each annotation dimension into a model pool, the model pool being used to construct a subsequent software defect prediction dataset; a comparison module configured to select a comparison large language model from the target large language models, and compare data under a certain annotation dimension corresponding to the comparison large language model with sampled data selected from the model pool, so as to verify the quality of the data in the model pool. 8.The device for constructing a software defect prediction dataset based on a large language model according to claim 7, wherein, The acquisition module is specifically configured to select multiple publicly available software defect identification and error repair datasets to constitute the software defect prediction dataset, the software defect prediction dataset covering multiple mainstream programming languages. 9.The software defect prediction dataset construction apparatus based on a large language model of claim 7, wherein, The annotation module is specifically configured to randomly select a specific number of data points from the software defect prediction dataset, and select multiple large language models to respectively annotate the same specific number of data points from different annotation dimensions, and obtain multiple annotation results. 10.The device for constructing a software defect prediction dataset based on a large language model of claim 7, wherein, The evaluation module is specifically configured to select at least one voting model to vote on the annotation results multiple times, and combine each voting result of each voting model to screen out the target large language model under each annotation dimension.

Images