Software project defect prediction method and system based on pre-training model

Abstract

The invention discloses a software project defect prediction method and system based on a pre-training model. According to the method, the pre-training language model and the multi-modal data are represented through the codes, so that the adaptability and generalization ability of the model to different items can be improved, and high-quality semantic and grammatical features of the codes can be obtained. Multiple data enhancement is realized through a conditional generative adversarial network pre-training generator, so that a semi-supervised generative adversarial network generator can construct training data according to a specific defect category proportion. The semi-supervised generative adversarial network is trained by using the labeled data and the unlabeled data, so that the data distribution difference between different items can be reduced, and the generalization ability of the model is improved. And finally, integrated learning is realized by storing combined weight files in different training stages, a plurality of model states can be integrated during prediction to reduce the variance of the model, and the stability and reliability of a prediction result are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software defect prediction, and in particular to a software project defect prediction method and system based on a pre-training model. Background Art

[0002] Current software defect prediction technologies fall into two main categories: those based on manually extracted static features and those based on feature extraction from the source code itself. The former struggles to fully capture the syntactic and semantic information of the code, while the latter, while more directly leveraging the code itself, still lacks sufficient leverage on code annotations.

[0003] In recent years, existing research has tended to extract features directly from source code, particularly using pre-trained models (such as CodeBERT, GraphCodeBERT, and CodeT5) to directly extract syntactic and semantic features of the code. Pre-trained models do not require specific tools to parse the code structure and can more directly utilize source code information for defect prediction.

[0004] However, most existing research uses only the code itself as input, ignoring the information in code comments. While some studies have attempted to incorporate code comments into feature extraction, these approaches are relatively simple, such as directly using all comments within a file or inputting them into a pre-trained model along with the code after sorting the comments. However, pre-trained models (such as CodeBERT) have token length limitations, and excessively long comments can hinder the effective use of code text information.

[0005] Secondly, software defect prediction is inherently a classification problem, typically relying on datasets to train classification models. However, existing open-source datasets commonly suffer from an imbalance in defect categories and significant data distribution differences. Because defective samples are far fewer than non-defective samples, model training is susceptible to class imbalance, resulting in poor prediction capabilities for minority class samples. Currently, with limited data resources, a single technical approach is insufficient to fully address data imbalance and distribution differences. Therefore, existing technologies require further improvement and development. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a software project defect prediction method and system based on a pre-trained model in response to the above-mentioned defects of the prior art, aiming to solve the problem of poor prediction effect of the software defect prediction method based on the pre-trained model in the prior art.

[0007] The technical solutions adopted by the present invention to solve the problem are as follows:

[0008] In a first aspect, an embodiment of the present invention provides a software project defect prediction method based on a pre-trained model, the method comprising:

[0009] Obtain a preprocessed source project dataset; each source project data includes: code text, corresponding natural language text, and corresponding defect category label;

[0010] Pre-training a first generator and a first discriminator of a conditional adversarial generative network based on the source project dataset and the code representation pre-trained language model to obtain a pre-trained first generator;

[0011] The second generator and the second discriminator of the semi-supervised adversarial generative network are trained based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator; in each round of training, the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator respectively;

[0012] The defect prediction category of each target project data is obtained by using the code to represent the pre-trained language model, the second discriminator and all the combined weight files.

[0013] In a second aspect, an embodiment of the present invention further provides a software project defect prediction system based on a pre-trained model, the system comprising:

[0014] The data processing module is used to obtain the pre-processed source project data set; each source project data includes: code text, corresponding natural language text and corresponding defect category label;

[0015] A pre-training module, configured to pre-train a language model based on the source project dataset and the code representation, and pre-train a first generator and a first discriminator of a conditional adversarial generative network to obtain a pre-trained first generator;

[0016] A training module is configured to train a second generator and a second discriminator of a semi-supervised adversarial generative network based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator; in each round of training, the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator respectively;

[0017] An integrated learning module is used to obtain the defect prediction category of each target project data through the code representation pre-trained language model, the second discriminator and all the combined weight files.

[0018] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which multiple instructions are stored, characterized in that the instructions are suitable for being loaded and executed by a processor to implement any of the steps of the above-mentioned software project defect prediction method based on a pre-trained model.

[0019] Beneficial effects of the present invention: The embodiments of the present invention can improve the adaptability and generalization ability of the model to different projects by using code representation pre-trained language models and multimodal data, and can obtain high-quality semantic and grammatical features of the code. Multiple data enhancements are achieved through the conditional generative adversarial network pre-training generator, so that the generator of the semi-supervised generative adversarial network can construct training data according to the specific defect category ratio. By using labeled data and unlabeled data for training through the semi-supervised generative adversarial network, the data distribution differences between different projects can be narrowed and the generalization ability of the model can be improved. Finally, integrated learning is achieved by saving the combined weight files of different training stages. During prediction, multiple model states can be integrated to reduce the variance of the model and improve the stability and reliability of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a software project defect prediction method based on a pre-training model provided by an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the main architecture of the software defect prediction model provided by an embodiment of the present invention.

[0023] Figure 3 2 is a schematic diagram of the structure of the conditional generative adversarial network provided by an embodiment of the present invention.

[0024] Figure 4 2 is a schematic diagram of the structure of a semi-supervised generative adversarial network provided by an embodiment of the present invention.

[0025] Figure 5 This is a module diagram of software project defect prediction based on a pre-trained model provided by an embodiment of the present invention.

[0026] Figure 6 This is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention discloses a method and system for predicting software project defects based on a pre-trained model. To clarify the objectives, technical solutions, and effects of the present invention, the present invention is further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the present invention and are not intended to limit the present invention.

[0028] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0029] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0030] In response to the above-mentioned defects in the prior art, the present invention provides a software project defect prediction method based on a pre-trained model, the method comprising: obtaining a pre-processed source project dataset; each source project data comprises: code text, corresponding natural language text and corresponding defect category label; based on the source project dataset and the code representation pre-trained language model, pre-training the first generator and the first discriminator of the conditional adversarial generative network to obtain a pre-trained first generator; based on the source project dataset, the target project dataset to be predicted and the pre-trained first generator, training the second generator and the second discriminator of the semi-supervised adversarial generative network; in each round of training, optimizing the parameters of the Transformer model, the second generator and the second discriminator of the code representation pre-trained language model to obtain combined weight files of different training stages; each of the combined weight files comprises: weight files corresponding to the Transformer model and the second discriminator respectively; obtaining the defect prediction category of each target project data through the code representation pre-trained language model, the second discriminator and all the combined weight files. The present invention can improve the adaptability and generalization ability of the model to different projects by pre-training language models and multimodal data through code representation, and can obtain high-quality semantic and grammatical features of the code. Multiple data enhancements are achieved through the conditional generative adversarial network pre-training generator, so that the generator of the semi-supervised generative adversarial network can construct training data according to the specific defect category ratio. By using labeled data and unlabeled data for training through the semi-supervised generative adversarial network, the data distribution differences between different projects can be narrowed and the generalization ability of the model can be improved. Finally, by saving the combined weight files of different training stages to achieve integrated learning, multiple model states can be integrated during prediction to reduce the variance of the model and improve the stability and reliability of the prediction results.

[0031] like Figure 1 As shown, the method specifically includes the following steps:

[0032] Step S100: Obtain a pre-processed source project data set; each source project data includes: code text, corresponding natural language text and corresponding defect category label.

[0033] Specifically, in this embodiment, the source project dataset is used as pre-training data and / or training data. The source project dataset is obtained by pre-processing the source project code files. The pre-processing is mainly to obtain a data combination in the form of code text + natural language text + defect category labels based on the source project code files, which is used for the subsequent pre-training / training process.

[0034] It should be noted that the method of this embodiment can achieve two prediction goals:

[0035] Cross-project version prediction targets refer to situations where the source project code files and the target project code files to be predicted are different versions of the same software project, and the source project dataset is derived based on the lowest version of the software project. For example, if software project A has two versions, 1.1 and 1.2, all code files of version 1.1 are used as pre-training / training data to predict defects in the code files of version 1.2.

[0036] The cross-project prediction target means that the source project code file and the target project code file to be predicted are different software projects. The lowest versions of other project projects excluding the target project are used as pre-training data / training data to predict defects in the target project's code files.

[0037] In one implementation, obtaining a preprocessed source project dataset includes:

[0038] Obtain a source project code file, remove comment information from the code in the source project code file, and remove package names and package import statements using a regular expression to obtain the code text;

[0039] Parsing the annotation information through regular matching;

[0040] If the annotation information meets the preset conditions, the class name and function name are extracted according to the parsed data to construct the natural language text;

[0041] If the annotation information does not meet the preset requirements, extracting the class name and part of the class-level annotation information based on the parsed data to construct the natural language text;

[0042] The source project data is obtained according to the code text, the natural language text and the corresponding defect category label.

[0043] In summary, this embodiment needs to process the source project code file to obtain source project data in the form of a combination of code text + natural language text + defect category label. Specifically, first remove the comment information in the code (including single-line comments, multi-line comments, Javadoc), and then use regular expressions to remove the package name and package import statements, retaining only the main content of the code to obtain the code text. Use regular matching to parse some of the comments, such as Javadoc comments. Based on the amount of information in the comment information, preset conditions can be established to determine which natural language text construction method is currently suitable: if the amount of information in the comment information is large, especially the method comments are rich, then the class name and function name are extracted to reduce redundant information interference and obtain natural language text; if the amount of information in the comment information is small, especially the lack of method comments, then retain the class name and some class-level comment information to ensure that the model still has certain semantic input and obtain natural language text. At the same time, removing the package name and import information makes the model more focused on the code logic and enhances the stability of cross-project predictions.

[0044] For example, using Java files as an example, the source project code files are preprocessed: comments (including single-line comments, multi-line comments, and Javadoc) are removed from the code. Then, regular expressions are used to remove package names and import statements, retaining only the main code content to reduce noise and highlight the code's logical structure, thereby obtaining the code text. Furthermore, Javadoc comments are parsed using regular expression matching. If methods have rich comments, class names, class-level comments, and method names are extracted to construct a textual representation of the code, namely, natural language text (also known as comment text information). If methods lack comments, class-level comments are used by default to supplement the natural language text. Finally, the extracted defect category labels (label: defective corresponds to 1, no defect corresponds to 0), code text (PL or code), and natural language text (NL or comment) are combined into a label+PL+NL format and stored in a formatted format to obtain the source project dataset, providing high-quality input data for subsequent feature extraction and model training.

[0045] Step S200: Pre-train the first generator and the first discriminator of the conditional adversarial generative network according to the source project data set and the code representation pre-trained language model to obtain a pre-trained first generator.

[0046] Specifically, the code representation pre-trained language model in this embodiment can adopt the CodeBERT pre-trained model. The following mainly uses the CodeBERT pre-trained model as an example to illustrate the implementation process of the method. The CodeBERT pre-trained model can extract features from the source project data to obtain the corresponding real code feature vector.

[0047] After feature extraction is completed, Figure 2 As shown, the main architecture of the software defect prediction model includes: data enhancement based on the conditional generative adversarial network (CGAN). Due to the class imbalance problem in the data set, this embodiment introduces a conditional generative adversarial network pre-training generator, which enables it to generate code feature samples that meet specific defect category labels to make up for the lack of minority class defect samples. This method not only improves the model's ability to recognize minority class samples, but also enables the model to learn defect features more comprehensively and improve generalization capabilities by generating diverse synthetic samples. In actual application scenarios, the conditional generative adversarial network includes a first generator and a second generator. The first generator and the first discriminator are pre-trained by the real code feature vector extracted by the CodeBERT pre-training model, and the pre-trained first generator is obtained after the training is completed.

[0048] In one implementation, pre-training a first generator and a first discriminator of a conditional adversarial generative network based on the source project dataset and the code representation pre-trained language model includes:

[0049] For each source project data, construct bimodal input data based on the code text and the corresponding natural language text, input the bimodal input data into the code representation pre-trained language model, and obtain a corresponding true code feature vector;

[0050] The first generator and the first discriminator of the conditional adversarial generative network are pre-trained according to the real code feature vector and the corresponding defect category label.

[0051] Specifically, in terms of feature input, this embodiment adopts a natural language-programming language (NL-PL) bimodal strategy: code comment information and code logic body are used as feature extraction sources. The specific implementation method is: a fixed prefix text (use_info = "These codes have corresponding class or function information:") is spliced in front of the natural language text, and then together with the code text, it constitutes the bimodal input data of NL-PL. By refining the input text, the ability to understand the code semantics can be improved, ensuring that the pre-trained model can obtain high-quality semantic and grammatical features. The CodeBERT pre-training model is used to extract features from the bimodal input data, and the output feature representation is the real code feature vector. The conditional adversarial generative network includes a first generator and a first discriminator. The conditional adversarial generative network is pre-trained using the real code feature vector and the corresponding defect category label as pre-training data, so that the first generator continuously optimizes the quality of the generated samples, while improving the identification ability of the first discriminator.

[0052] In one implementation, the code represents a pre-trained language model, including:

[0053] A word segmentation and embedding module is used to perform subword segmentation on the bimodal input data to obtain an input sequence containing several code blocks; perform word vector embedding and position encoding on the input sequence to obtain an initial vector representation;

[0054] A Transformer model is used to obtain the true code feature vector based on the initial vector representation through a multi-head self-attention mechanism and a feedforward neural network.

[0055] Specifically, bimodal input data is fed into the CodeBERT pre-trained model as input text for feature extraction. First, the text is fed into the word segmentation and embedding module, where it undergoes tokenization. This involves tokenizing both the code text and the natural language text. The input text is segmented at the subword level using CodeBERT's pre-trained BPE (Byte-Pair Encoding) tokenizer to ensure it fits the input format of the pre-trained model, resulting in an input sequence. The input sequence then passes through two embedding layers for token embedding and position encoding, resulting in an initial vector representation. The token embedding captures word-level semantic information, while the position encoding helps the model understand positional information within the code structure, ensuring it can learn the relative relationships between code blocks. Furthermore, the CodeBERT pre-trained model utilizes a Transformer model (a multi-layer Transformer architecture) to perform deep semantic modeling on the input text. The initial vector representation uses a multi-head self-attention mechanism to globally capture long-range dependencies within the code and, combined with annotation information, helps understand the code's semantics. Then, a feed-forward neural network (FFN) performs a nonlinear transformation to extract higher-level code feature representations, resulting in a vector representing the code's feature information—the true code feature vector. After processing by the Transformer model, the generated true code feature vector is a high-dimensional vector that incorporates code logic, grammatical structure, and semantic features, providing accurate feature input for subsequent defect prediction tasks. The code representations learned by the CodeBERT pre-trained model on large-scale code-annotation data enable it to have stronger generalization capabilities.

[0056] In one implementation, pre-trained models that support long text processing, such as the LongCode model and the CodeT5 model, can also be used to utilize more complete code and its annotation information for feature extraction, thereby improving the model's ability to understand the syntax and semantics of long code segments.

[0057] In one implementation, pre-training the first generator and the first discriminator of the conditional adversarial generative network based on the real code feature vector and the corresponding defect category label includes:

[0058] Generate a pseudo code feature vector according to the noise vector and the defect category label corresponding to the real code feature vector by the first generator;

[0059] Generate first prediction data based on the pseudo code feature vector and the corresponding defect category label through the first discriminator; generate second prediction data based on the real code feature vector and the corresponding defect category label;

[0060] Parameters of the first generator and the first discriminator are optimized according to the first prediction data and the second prediction data.

[0061] Specifically, the conditional generative adversarial network (CGAN) consists of a first generator and a first discriminator. Both the first generator and the first discriminator use label embedding to integrate the defect category label (or category information) into the input to improve the learning ability of category-specific features. Among them, the first generator receives the noise vector and the defect category label of the real code feature vector as input, and after multi-layer linear transformation, layer normalization, LeakyReLU activation and Dropout structure, it finally outputs a high-dimensional feature representation for generating synthetic samples, thereby obtaining a pseudocode feature vector. The first discriminator receives the pseudocode feature vector and the corresponding defect category label, performs feature extraction through a similar layer structure, and outputs a scalar (validity) for judging the authenticity of the input data, that is, obtaining the first predicted data; in addition, the first discriminator also receives the corresponding real code feature vector and defect category label, performs feature extraction through a similar layer structure, and outputs a scalar, that is, obtaining the second predicted data. Through conditional adversarial training, the first generator continuously optimizes the quality of generated samples, that is, converges the gap between the first predicted data and the second predicted data; at the same time, the first discriminator's discrimination ability is improved, that is, the gap between the first predicted data and the second predicted data is widened. The structure of the conditional adversarial generation network is detailed in Figure 3 .

[0062] In practical application scenarios, the training process of a conditional adversarial generative network involves first inputting the prepared source project dataset into the CodeBERT pre-trained model for feature extraction, obtaining a true code feature vector. Simultaneously, a random noise vector is used to generate a pseudo-code feature vector through the first generator. This, along with the true code feature vector, is input into the first discriminator for adversarial training. During training, 30 epochs are used, with a learning rate of 1e-5 and a dropout rate of 0.3. AdamW is used as the optimizer, and a cosine annealing hard-start scheduling strategy is used for learning rate adjustment to smooth the training process and improve model convergence stability. After training convergence, the weights of the first generator are fixed for subsequent experimental training.

[0063] In another implementation, to further optimize the CodeBERT pre-trained model, WGAN (Wasserstein distance-based generative adversarial network) or WGAN-GP (Wasserstein distance-based generative adversarial network with gradient penalty mechanism) can be introduced during the CGAN training process to enhance the stability of the generated feature vectors and avoid mode collapse. In addition, CVAE (conditional variational autoencoder) can be considered as an alternative to CGAN to generate code feature vectors that conform to the specified labels, while introducing latent variable modeling to improve the diversity and quality of data generation.

[0064] Step S300: Train the second generator and the second discriminator of the semi-supervised adversarial generative network based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator; in each round of training, optimize the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator, respectively.

[0065] Specifically, the target project dataset comes from the target project file to be predicted. Similar to the data processing method for extracting the source project dataset from the source project file, the target project dataset acquisition process also requires processing the target project file into bimodal input data including code text and corresponding natural language text. Only then can the constructed bimodal input data and the CodeBERT pre-training model be used to extract the corresponding code feature vector. Unlike each source project data, each target project data does not have a corresponding defect category label, and prediction is required to obtain the defect category. In short, this embodiment uses the code feature vector corresponding to the source project dataset to train the prediction model, and then predicts the defect category of the code feature vector corresponding to the target project dataset.

[0066] like Figure 2As shown, the main architecture of the software defect prediction model also includes: cross-project feature alignment using a semi-supervised generative adversarial network (SSGAN). In the cross-project defect prediction task, there are often large differences in the feature distributions of the source project and the target project to be predicted, which affects the migration ability of the model. Therefore, this embodiment adopts a semi-supervised generative adversarial network (SSGAN), takes the pre-trained first generator as the initial input, and combines the code feature vectors of the target project data (unlabeled data) and the source project data (labeled data), and gradually reduces the distribution differences through adversarial training. On the one hand, the knowledge transfer ability of the source project is retained, and on the other hand, the feature space of the target project is gradually adapted, thereby improving the prediction performance of the model on the target project. In order to make the extracted features more suitable for the software defect prediction task, the embodiment of the present invention will also combine the Transformer model of the CodeBERT pre-trained model with the semi-supervised generative adversarial network, and use reverse gradient propagation to perform parameter optimization together, so as to achieve fine-tuning of the pre-trained model in a specific project. Finally, this embodiment also introduces an early stopping strategy during the training process to fix the optimal parameters, so that after multiple rounds of training, multiple combined weight files of different training stages are obtained, each combined weight file includes a set of Transformer model weights and a second discriminator weight.

[0067] In one implementation, training a second generator and a second discriminator of a semi-supervised adversarial generative network based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator includes:

[0068] Dividing the source project data set into several training-validation experimental sets using the BRHS method, each training-validation experimental set including a training set and a validation set, and adjusting the defect category ratio of each training set based on an oversampling technique;

[0069] The target project data set is added to each of the training sets to train the second generator and the second discriminator.

[0070] like Figure 2As shown, the main architecture of the software defect prediction model also includes ensemble learning based on block regularized repeated half-sampling (BRHS). Specifically, due to limited available training data, this embodiment adopts an optimization mechanism for ensemble learning based on data partitioning. This mechanism uses the BRHS (block regularized repeated half-sampling) method to partition the training set to enhance the robustness of the model and improve the model training effect. The BRHS method divides the original training set (i.e., the training set obtained based on the source project dataset, each training data includes: real code feature vectors extracted after feature engineering based on the source project data and corresponding defect labels) into multiple training and validation sets. The BRHS method ensures that each training sample appears evenly across all training sets and the amount of overlap between groups is equal, effectively reducing performance fluctuations caused by group imbalance and ensuring that the defect rate of each group after grouping is similar to the defect rate in the original source project dataset, thereby fully utilizing limited data resources. Oversampling technology is then used to adjust the defect category (defect / non-defect) ratio in the training set to a preset ratio, while the defect category ratio in the validation set remains unchanged to control the distribution differences in the training set. The division of multiple training and validation experimental sets helps to learn the defect correlation characteristics between different code files, while ensuring the stability of the integration strategy and improving the prediction accuracy.

[0071] The semi-supervised generative adversarial network includes a second generator and a second discriminator. When using a semi-supervised generative adversarial network for training, the target project data can be added to the training set as a test set for training. The test set here is unlabeled data (or semi-supervised samples). The purpose is to narrow the data distribution difference between the training set and the test set, and it can also play a role in expanding the training data. Ultimately, it is possible to continuously fit the characteristics of specific project code data in adversarial learning, while retaining the generalization and migration capabilities for new content in subsequent test sets, so that the adaptability and accuracy of the prediction model on new projects are improved. This strategy not only reduces the variance in small sample scenarios and improves the uniformity of the training process, but also enhances the stability of the model in unbalanced data and cross-project tasks.

[0072] For example, first determine the software defect prediction target, namely, cross-project version prediction or cross-project engineering prediction. The project code files to be predicted are the target project code files. The source project dataset, serving as training data, is then divided into several training-validation sets using the BRHS technique, for example, 12 training-validation sets. Oversampling is then used to adjust the defect category (defect / non-defect) ratio in the training set to 1:1, while the defect category ratio in the validation set remains unchanged. Suppose that the original training set (source project data) consists of 100 rows and is divided into 50 rows for the training set and 50 rows for the validation set. The label distribution in the training set after this division is unbalanced. For example, 15 rows belong to defect category label 1 (defective) and 35 rows belong to defect category label 0 (non-defective). To balance the label distribution, oversampling is used to increase the number of samples with defect category label 1 to the same number as those with defect category label 0, that is, 35 rows for each. The total number of samples in the training set now becomes 70 rows. During the training process, 120 rows of the target project dataset are added to the 70 rows of the training set, resulting in 190 rows, which are used as training data for training the semi-supervised generative adversarial network (used to extract the true code feature vector input to the second discriminator), and the validation set remains at 50 rows.

[0073] In practical application scenarios, the BRHS method can be to divide the code features based on the source project into multiple mutually exclusive sub-blocks, randomly select some sub-blocks as training sets and other parts as validation sets in each round of training, and oversample the training samples to make full use of limited data resources.

[0074] In one implementation, during each round of training, parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages, including:

[0075] For each set of the training-validation experimental set, extracting the true code feature vector corresponding to each training sample in the training set through the code representation pre-trained language model;

[0076] Generate a pseudocode feature vector based on the noise vector and the defect category label through the pre-trained first generator;

[0077] generating, by the second generator, a target pseudocode feature vector according to the pseudocode feature vector;

[0078] Generate third prediction data according to the target pseudo code vector through the second discriminator; generate fourth prediction data according to the real code feature vector;

[0079] Based on the third prediction data and the fourth prediction data, parameters of the Transformer model, the second generator, and the second discriminator of the code representation pre-trained language model are optimized, and verification is performed on the corresponding validation set to obtain a combined weight file corresponding to the training-validation experimental set.

[0080] In general, the main body of the semi-supervised generative adversarial network (SSGAN) is similar to the conditional generative adversarial network used in pre-training, but the second generator and the second discriminator of the semi-supervised generative adversarial network do not have defect category labels embedded. During the training process of the semi-supervised generative adversarial network, this embodiment adopts a defect prediction mechanism based on multiple data enhancements: the first generator is pre-trained through the conditional generative adversarial network (CGAN) to enable it to generate code feature vectors with generalization capabilities and associate them with defect category labels. The pre-trained first generator provides initial input during the training process of the semi-supervised generative adversarial network (SSGAN), ensuring that the second generator can construct training data according to a specific defect category ratio, thereby effectively alleviating the problem of data category imbalance.

[0081] Specifically, the second generator takes the pseudocode feature vector generated by the pre-trained first generator as input. After undergoing Tanh normalization, the input pseudocode feature vector passes through a network structure consisting of fully connected layers, layer normalization, LeakyReLU activation, and Dropout, and is gradually mapped to the target output dimension to further generate feature representations that are similar to the distribution of real project data.

[0082] The second discriminator is similar to the first discriminator mentioned above. The input feature vectors are the pseudocode feature vector and the code feature vector extracted based on the real project code file. The two feature vectors are respectively input into the second discriminator for prediction. Specifically, the second discriminator adopts the same multi-layer structure, receives the input feature vector, and passes it through a series of network layers. Finally, it calculates the category score through a fully connected layer (logit layer), and outputs the probability distribution through the Softmax layer to generate prediction data. The prediction data is used to reflect the defect category (defective / non-defective) to which the input feature vector judged by the second discriminator belongs, as well as to additionally predict its authenticity (that is, to determine whether it is a generated fake sample or real data). The prediction data output by the second discriminator based on the target pseudocode feature vector is the third prediction data, and the prediction data output based on the real code feature vector is the fourth prediction data. For details of the network structure of the semi-supervised generative adversarial network, see Figure 4 .

[0083] In order to make the feature vectors extracted by the CodeBert pre-training model more suitable for software defect prediction, this embodiment combines the Transformer model parameters of the CodeBert pre-training model with the second discriminator parameters, and optimizes the second generator parameters together using the AdamW optimizer. The optimizer uses the gradient descent method to adjust these parameters to reduce the loss. During the training process, the epoch is set to 30 rounds, the learning rate is 1e-5, the dropout rate is 0.3, and the cosine annealing hard start scheduling strategy is also used for learning rate adjustment. After each round of training, it is verified on the validation set. The overall training process adopts an early stopping strategy on the validation set to fix the optimal parameters, and finally loads it into the defect prediction task of the target project, thereby further improving the prediction performance and generalization ability of the model. After training each training-validation experimental set, a combined weight file (i.e., a set of Transfomer weights and discriminator weights) can be obtained. Finally, after training and verification of the training-validation experimental set of all groups, multiple combined weight files can be obtained for defect prediction in the subsequent integrated testing phase. It should be noted that this embodiment uses unlabeled data for semi-supervised adversarial training. This unlabeled data is used only to reduce the distribution gap between the source and target data, ensuring that the feature vectors generated by the second generator ultimately also contain the distribution characteristics of the target data. The labels predicted by the second discriminator for this portion are not used; only the labels of the labeled data are used to calculate the loss.

[0084] Another implementation approach involves optimizing the training model. In semi-supervised generative adversarial networks, the VAE-GAN architecture can be leveraged to improve the similarity between generated samples and real-world samples, thereby enhancing the discriminator's capabilities. Furthermore, semi-supervised training can be augmented with data that more closely resembles the target project's characteristics to improve generalization. Data partitioning strategies, such as group sampling, block partitioning, or cross-validation, can enhance model training effectiveness and stability.

[0085] Step S400: Obtain the defect prediction category of each target project data through the code representation pre-trained language model, the second discriminator and all the combined weight files.

[0086] Specifically, this embodiment saves the combined weight files of different training stages, and can integrate multiple model states during prediction, which is similar to the integrated learning mechanism. It can reduce the variance of the model, improve the stability and reliability of the prediction results, and reduce the errors that may be caused by the configuration of a single model parameter.

[0087] In one implementation, obtaining a defect prediction category for each target project data by using the code representation pre-trained language model, the second discriminator, and all the combined weight files includes:

[0088] For each target project data in the target project data set, after configuring the weights of the code representation pre-trained language model and the second discriminator according to each combined weight file, defect prediction is performed on the target project data to obtain an initial defect prediction result;

[0089] The target defect prediction result is comprehensively determined by using the initial defect prediction results corresponding to each of the combined weight files.

[0090] Specifically, this embodiment uses each combined weight file to set weights for the CodeBert pre-trained language model and the second discriminator. The combined weight file is saved during the model training process and records the parameter states at different stages of the model training. By loading different combined weight files, the CodeBert pre-trained language model and the second discriminator will exhibit different performance states. After completing the weight configuration, the target project data is input into the configured CodeBert pre-trained language model for feature extraction. The extracted code feature vector is then input into the configured second discriminator for defect prediction to determine the possible defect categories of the target project data, thereby obtaining an initial defect prediction result. Due to the use of different combined weight files, multiple different initial defect prediction results may be obtained for the same target project data. After obtaining the initial defect prediction results corresponding to each combined weight file, it is necessary to comprehensively consider these initial defect prediction results to determine the final target defect prediction result. For example, the initial defect prediction result reflects a prediction category and corresponding probability value output by the second discriminator. The prediction categories and probability values of all initial defect prediction results are weighted and integrated to obtain the final probability values corresponding to each prediction category. The prediction category with the highest final probability value is the target defect prediction result. Because different weight files represent the model's learning at different training stages or under different data distributions, a single prediction result may be biased or unstable. By combining multiple prediction results for ensemble learning, this bias and instability can be reduced, improving prediction accuracy and reliability.

[0091] For example, this embodiment generates multiple classifiers in each round of training, and performs weighted voting integration on the test samples of the target project to improve the accuracy of the prediction. Specifically, taking a combined weight file as an example, the Transformer weights fixed in the training phase are first loaded into the CodeBert pre-training model, and then the target project data is input into it for feature extraction. The extracted code feature vector is then input into the discriminator network (the architecture is consistent with the second discriminator in the semi-supervised adversarial stage) using the discriminator weight file fixed in the training phase to perform defect prediction binary classification. It should be noted that this stage does not need to consider the classification of whether the sample is real or not. After the code feature vector of the target project data is processed through the above steps using all the combined weight files, it is equivalent to the prediction classification of all the training-verification experimental sets, and finally a plurality of prediction categories equal to the number of groups in the training-verification experimental set are obtained, and the probability of each prediction category can also be obtained. The F1 value obtained in the training and verification phase is used as the weight for weighted integration, and the prediction category with the highest probability is the target defect prediction result.

[0092] In another implementation, the ensemble strategy can be optimized: in addition to weighted voting based on the F1 value, it can also be combined with indicators such as accuracy and precision, or statistical methods can be used for ensemble optimization to improve the reliability and robustness of the final prediction.

[0093] Other possible technical variations of the present invention:

[0094] 1. At the information utilization level: If relevant data such as source code static metrics and project architecture dependency information is readily available, consider more fully integrating these three types of information (source code, static metrics, and architecture dependencies) for multi-level analysis and modeling. This comprehensive utilization of information can improve prediction accuracy and model robustness.

[0095] 2. At the method-level code prediction level: Using code text alone as input requires a large amount of training data, so careful evaluation of data availability and its impact on model training is necessary. If sufficient data and computing resources allow, more complex models can be considered to fully exploit the potential of code text.

[0096] 3. For prediction projects using code in other languages (such as C and Python), the data processing process must be adjusted to suit the characteristics of each language. The syntax, programming style, and frameworks of different languages can all impact prediction tasks. Therefore, when working with projects in these languages, appropriate adaptation and adjustments must be made to ensure the scalability and effectiveness of the model across multiple language environments.

[0097] The advantages of the present invention are:

[0098] 1. Based on the Java language project data in the software defect prediction dataset PROMISE, combined with the project source code and its annotation information, this paper uses the CodeBert pre-training model for feature extraction and designs a deep learning method suitable for cross-project defect prediction.

[0099] 2. This invention achieves excellent defect prediction results by utilizing only source code and its annotations, without relying on artificial static features or complex code parsing. It is suitable for file-level defect prediction across versions and projects. The NL-PL bimodal input strategy enables the model to fully integrate code text and annotation information, overcoming the limitations of existing methods that rely solely on code text. By extracting class and function names to reduce redundant information and removing package names and import information, the stability of defect prediction is enhanced, making the model more adaptable in different project environments.

[0100] 3. The present invention introduces the CGAN generator to learn the vector representation of code features, thereby enhancing the generalization ability of the generated data and effectively alleviating the problem of class imbalance. Compared with traditional data resampling, the data generated by CGAN is more in line with the actual distribution, which improves the ability to identify minority class defects. At the same time, SSGAN reduces the feature distribution deviation between different projects through adversarial training, enabling the model to better adapt to the target project and improve the accuracy of cross-version and cross-project predictions. In addition, BRHS combines data partitioning with ensemble learning, which not only optimizes the small sample training effect, but also improves the robustness of the prediction through a multi-classifier voting mechanism, so that the model can still maintain good performance when there is insufficient data.

[0101] Experimental results show that this method outperforms traditional methods in both F1-Score and AUC on the PROMISE dataset, and demonstrates stronger generalization capabilities in cross-version and cross-project prediction tasks. Overall, this paper achieves more accurate and stable software defect prediction through efficient feature extraction, data generation, and adversarial training strategies.

[0102] Based on the above embodiments, the present invention also provides a software project defect prediction system based on a pre-trained model, such as Figure 5 As shown, the system includes:

[0103] Data processing module 01 is used to obtain the pre-processed source project data set; each source project data includes: code text, corresponding natural language text and corresponding defect category label;

[0104] Pre-training module 02, configured to pre-train a language model based on the source project dataset and code representation, and pre-train a first generator and a first discriminator of a conditional adversarial generative network to obtain a pre-trained first generator;

[0105] Training module 03 is used to train the second generator and the second discriminator of the semi-supervised adversarial generative network based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator; in each round of training, the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator respectively;

[0106] The integrated learning module 04 is used to obtain the defect prediction category of each target project data through the code representation pre-trained language model, the second discriminator and all the combined weight files.

[0107] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 6 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a software project defect prediction method based on a pre-trained model is implemented. The display screen of the terminal can be a liquid crystal display or an electronic ink display.

[0108] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0109] In one implementation, the terminal stores one or more programs in its memory and is configured to be executed by one or more processors. The one or more programs include instructions for performing a software project defect prediction method based on a pre-training model.

[0110] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0111] In summary, the present invention discloses a method and system for predicting software project defects based on a pre-trained model. The method includes: obtaining a pre-processed source project data set; each source project data includes: code text, corresponding natural language text and corresponding defect category label; according to the source project data set and the code representation pre-trained language model, pre-training the first generator and the first discriminator of the conditional adversarial generative network to obtain a pre-trained first generator; according to the source project data set, the target project data set to be predicted and the pre-trained first generator, training the second generator and the second discriminator of the semi-supervised adversarial generative network; in each round of training, optimizing the parameters of the Transformer model, the second generator and the second discriminator of the code representation pre-trained language model to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator respectively; through the code representation pre-trained language model, the second discriminator and all the combined weight files, obtaining the defect prediction category of each target project data. The present invention can improve the adaptability and generalization ability of the model to different projects by pre-training language models and multimodal data through code representation, and can obtain high-quality semantic and grammatical features of the code. Multiple data enhancements are achieved through the conditional generative adversarial network pre-training generator, so that the generator of the semi-supervised generative adversarial network can construct training data according to the specific defect category ratio. By using labeled data and unlabeled data for training through the semi-supervised generative adversarial network, the data distribution differences between different projects can be narrowed and the generalization ability of the model can be improved. Finally, by saving the combined weight files of different training stages to achieve integrated learning, multiple model states can be integrated during prediction to reduce the variance of the model and improve the stability and reliability of the prediction results.

[0112] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A software project defect prediction method based on a pre-trained model, characterized in that: The method comprises: Obtain a preprocessed source project dataset; each source project data includes: code text, corresponding natural language text, and corresponding defect category label; Pre-training a first generator and a first discriminator of a conditional adversarial generative network based on the source project dataset and the code representation pre-trained language model to obtain a pre-trained first generator; The second generator and the second discriminator of the semi-supervised adversarial generative network are trained based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator; in each round of training, the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator respectively; The defect prediction category of each target project data is obtained by using the code to represent the pre-trained language model, the second discriminator and all the combined weight files.

2. The software project defect prediction method based on the pre-training model according to claim 1 is characterized in that: Get the preprocessed source project dataset, including: Obtain a source project code file, remove comment information from the code in the source project code file, and remove package names and package import statements using a regular expression to obtain the code text; Parsing the annotation information through regular expression matching; if the annotation information meets the preset conditions, extracting the class name and function name based on the parsed data to construct the natural language text; If the annotation information does not meet the preset requirements, extracting the class name and part of the class-level annotation information based on the parsed data to construct the natural language text; The source project data is obtained according to the code text, the natural language text and the corresponding defect category label.

3. The software project defect prediction method based on the pre-training model according to claim 1 is characterized in that: Pre-training a first generator and a first discriminator of a conditional adversarial generative network based on the source project dataset and the code representation pre-trained language model includes: For each source project data, construct bimodal input data based on the code text and the corresponding natural language text, input the bimodal input data into the code representation pre-trained language model, and obtain a corresponding true code feature vector; The first generator and the first discriminator of the conditional adversarial generative network are pre-trained according to the real code feature vector and the corresponding defect category label.

4. The software project defect prediction method based on the pre-training model according to claim 3 is characterized in that: The code represents a pre-trained language model, including: A word segmentation and embedding module is used to perform subword segmentation on the bimodal input data to obtain an input sequence containing several code blocks; perform word vector embedding and position encoding on the input sequence to obtain an initial vector representation; A Transformer model is used to obtain the true code feature vector based on the initial vector representation through a multi-head self-attention mechanism and a feedforward neural network.

5. The software project defect prediction method based on the pre-training model according to claim 3 is characterized in that: Pre-training a first generator and a first discriminator of the conditional adversarial generative network according to the real code feature vector and the corresponding defect category label includes: Generate a pseudo code feature vector according to the noise vector and the defect category label corresponding to the real code feature vector by the first generator; Generate first prediction data based on the pseudo code feature vector and the corresponding defect category label through the first discriminator; generate second prediction data based on the real code feature vector and the corresponding defect category label; Parameters of the first generator and the first discriminator are optimized according to the first prediction data and the second prediction data.

6. The software project defect prediction method based on the pre-training model according to claim 1 is characterized in that: Training a second generator and a second discriminator of a semi-supervised adversarial generative network according to the source project dataset, the target project dataset to be predicted, and the pre-trained first generator, comprising: Dividing the source project data set into several training-validation experimental sets using the BRHS method, each training-validation experimental set including a training set and a validation set, and adjusting the defect category ratio of each training set based on an oversampling technique; The target project data set is added to each of the training sets to train the second generator and the second discriminator.

7. The software project defect prediction method based on the pre-training model according to claim 6 is characterized in that: During each round of training, the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages, including: For each set of the training-validation experimental set, extracting the true code feature vector corresponding to each training sample in the training set through the code representation pre-trained language model; Generate a pseudocode feature vector based on the noise vector and the defect category label by the pre-trained first generator; generate a target pseudocode feature vector based on the pseudocode feature vector by the second generator; Generate third prediction data according to the target pseudo code vector through the second discriminator; generate fourth prediction data according to the real code feature vector; Based on the third prediction data and the fourth prediction data, parameters of the Transformer model, the second generator, and the second discriminator of the code representation pre-trained language model are optimized, and verification is performed on the corresponding validation set to obtain a combined weight file corresponding to the training-validation experimental set.

8. The software project defect prediction method based on the pre-training model according to claim 1 is characterized in that: Obtaining a defect prediction category for each target project data by using the code to represent the pre-trained language model, the second discriminator, and all the combined weight files, includes: For each target project data in the target project data set, after configuring the weights of the code representation pre-trained language model and the second discriminator according to each combined weight file, defect prediction is performed on the target project data to obtain an initial defect prediction result; The target defect prediction result is comprehensively determined by using the initial defect prediction results corresponding to each of the combined weight files.

9. A software project defect prediction system based on a pre-trained model, characterized in that: The system comprises: The data processing module is used to obtain the pre-processed source project data set; each source project data includes: code text, corresponding natural language text and corresponding defect category label; A pre-training module, configured to pre-train a language model based on the source project dataset and the code representation, and pre-train a first generator and a first discriminator of a conditional adversarial generative network to obtain a pre-trained first generator; A training module is configured to train a second generator and a second discriminator of a semi-supervised adversarial generative network based on the source project dataset, the target project dataset to be predicted, and the pre-trained first generator; in each round of training, the parameters of the Transformer model of the code representation pre-trained language model, the second generator, and the second discriminator are optimized to obtain combined weight files of different training stages; each of the combined weight files includes: weight files corresponding to the Transformer model and the second discriminator respectively; An integrated learning module is used to obtain the defect prediction category of each target project data through the code representation pre-trained language model, the second discriminator and all the combined weight files.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the software project defect prediction method based on a pre-trained model as described in any one of claims 1-8 above.

Images