Retrieval enhancement instant defect prediction method based on pre-training model

Abstract

The invention provides a retrieval enhancement instant defect prediction method based on a pre-training model, and belongs to the field of software security. Firstly, a vector encoder with a small number of parameters is constructed, and vector representation is performed on feature data of an instant defect prediction task; and then, the retrieval recall of the L2 distance is completed based on the FAISS. And finally, designing two architectures of information stacking (RIS) and cross-model vector fusion (RVF) through a retrieval enhancement strategy, and completing an instant defect prediction task. The RIS architecture enriches context information by stacking similar submission codes to an input sequence, and the internal structure of a model does not need to be changed; the RVF architecture directly fuses a retrieval vector and a backbone model embedding feature, information interaction flow is enhanced, and the precision and generalization ability of an instant defect prediction task are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of software security, and in particular is a retrieval-enhanced instant defect prediction method based on a pre-training model. Background Art

[0002] Modern software development cycles tend to release software products in a short period of time. In order to cope with the significant challenges posed by such short software development cycles to modern software quality assurance (SQA) practices, continuous code quality tools (e.g., CI / CD, modern code review, static analysis) are used to detect software defects early. However, due to limited SQA resources, the SQA team cannot effectively check every submission. The proposal of just-in-time defect prediction is mainly to predict whether a submission will introduce defects in the future. This submission-level prediction is very useful for helping practitioners prioritize limited SQA resources on the submissions with the highest risks during the software development process.

[0003] When developers submit a project, they often modify multiple files, and each modified file has a unique id. A code submission includes the change code itself and the commit message. The more important part of the commit message is the developer's annotation, which is composed of natural language text and can largely reflect the developer's intention for this submission. It is also an effective feature commonly used in deep real-time defect prediction tasks.

[0004] Although the current research on defect prediction tasks based on pre-trained models has made great improvements compared to previous technologies, there is still a problem of insufficient available defect data in the real-time defect prediction task, that is, the information that can be learned through pre-training is limited and the performance improvement brought by simple fine-tuning is limited. Retrieval enhancement, as a way to introduce additional information, has achieved good performance in generative tasks, but the application of retrieval enhancement to understanding tasks such as defect prediction is still an unexplored direction. Summary of the invention

[0005] In view of the problem that available defect information in the prior art is insufficient and the performance improvement brought by simple fine-tuning is limited, the present invention proposes a retrieval-enhanced instant defect prediction method based on a pre-trained model.

[0006] A retrieval-enhanced instant defect prediction method based on a pre-trained model, the specific implementation includes the following steps:

[0007] Step 1: Obtain feature data for the real-time defect prediction task, including the commit code of the change history code in the software development process and the commit message attached when the developer submits the code.

[0008] Step 2: Prune the large language model based on the Llama architecture to obtain a lightweight vector encoder and perform vector representation on the commit code.

[0009] After model pruning, the embed_token, lm_head, and final normlayer of the selected large language model, as well as the first and last decoder layers, are retained to construct a lightweight vector encoder containing only two layers of decoders.

[0010] The specific method of constructing the commit code vector is:

[0011] Select the 4 most recently modified files from each commit code, denoted as C = [P1, P2, ..., P |C| ]; if the number of modified files in the commit is less than 4, the remaining positions are filled with "empty patch".

[0012] The code text in the i-th file is taken as input, and a hidden state Hissen_state_i with a tensor shape of [6144, 4096] is obtained through the constructed vector encoder s. Then, an average pooling operation is performed on Hissen_state_i to obtain the vector representation mean_embedding_i of the i-th file, as shown in formula (1):

[0013]

[0014] Since a commit contains 4 files, the final vector representation of a commit is obtained by averaging the 4 vectors to obtain the final commit code vector representation F_llm.

[0015]

[0016] Step three: retrieve similar information based on the vector, obtain the retrieved information, and implement L2 distance similarity calculation and recall based on FAISS.

[0017] The implementation based on L2 distance is completed through the FAISS library in Python. The calculation formula of L2 distance (Euclidean distance) d is:

[0018]

[0019] Among them, xi and i are the i-th dimension components of the original code vector and the retrieved vector, respectively, and n is the dimension of the vector;

[0020] Step 4: Establish a retrieval enhancement model based on information stacking or cross-model vector fusion, process the commit code vectors separately, and obtain the most significant feature vector.

[0021] (1) The process of processing the commit code vector by the information stacking-based retrieval enhancement model is as follows:

[0022] The closest commit code retrieved is merged into the commit code input process. The new commit code is represented by C = [P1, P2, ..., P |2C| ], and then each patch P i As a sequence in the submitted code, as code input to the bone intervention training model to obtain the embedding representation e i , as shown in the following formula:

[0023] e i =BackboneModel(P i ) (5)

[0024] After each patch in the submitted code is processed by the pre-trained model, use The vector representation of the commit code is:

[0025]

[0026] Use CNN and max pooling to extract features of code embedding, specifically:

[0027] Define a filter f∈R kxd , and use k windows to extract new features for each k files of the submitted code. For a code patch e i , according to formula (6) we get the new vector x i .

[0028] x i =ReLU(f*e i:i+k-1 +b i ) (6)

[0029] Among them, the symbol * represents the sum of element products, ReLU is a nonlinear activation function, and b i ∈R is the bias value.

[0030] New Featuresx i They are connected in series to form a vector X, as shown in formula (7).

[0031] X=[x1,x2,...x |2C|-k+1 ] (7)

[0032] For the new feature vector X, use the maximum pooling method to obtain the most significant feature Z c ;

[0033] (2) The process of processing the commit code vector by the retrieval enhancement model based on cross-model vector fusion is as follows:

[0034] This architecture directly uses the vector embedding encoded by the vector encoder model as the internal feature F_llm of the defect prediction model. The commit code processing of this submission follows the original model processing method, that is, selecting the most recent 4 modified files from each submission. The commit code is represented by C = [P1, P2, ..., P |C| ], where |C| represents the number of modified files in the commit, and |C| = 4. If the number of modified files in the commit is less than 4, then "empty patch" is used to fill the remaining positions. Then each patch P i As a sequence in the submitted code, as code input the bone intervention training model to obtain the embedding representation of the backbone model e i , as shown in the following formula.

[0035] e i =BackboneModel(P i ) (8)

[0036] After each patch in the submitted code is processed by the pre-trained model, use The vector representation of the commit code obtained by the operation follows formula (9).

[0037]

[0038] For the cross-modal fusion vector, we use The operation stacks the internal features F_llm, where n is the number of recalled vector features.

[0039]

[0040] For the obtained retrieval vector F_llm all , the model uses a layer of full connection for dynamic weighting operation and maps it to the same dimension as the backbone model embedding feature, following the following formula:

[0041] weighted t =W fc ·F_llm all +b fc (10)

[0042] For the two-part code vector obtained, use The final representation of the vector of commit codes obtained by the operation is (11).

[0043]

[0044] At this point, the features of the commit code include the backbone model embedding and the weighted fusion. Then, CNN and maximum pooling are used to extract the features of the code embedding. A filter f∈R is defined. kxd And use k windows to extract the submission code C e '. For code patch e i , according to formula (12) we get the new vector x i .

[0045] x i =ReLU(f*e i:i+k-1 +b i ) (12)

[0046] Among them, the symbol * represents the sum of element products, ReLU is a nonlinear activation function, and b i ∈R is the bias value. Generate new features x by applying filter f to each k files of the submitted code i . New Features x i They are connected in series to form a vector X, as shown in formula (13).

[0047] X=[x1,x2,...x |C+1|-k+1 ] (13)

[0048] For the new feature vector X, use the maximum pooling method to obtain the most significant feature Z c .

[0049] Step 5: Process the commit message and combine the most significant features of the commit code to complete the immediate defect prediction task.

[0050] Process the commit message to get the feature vector Z m , specifically:

[0051] Consider the entire commit message as a sequence M = [w1,w2,w3,...,w |m| ]. M is a sequence of words, and |m| is the length of the sequence. The submission information M is then fed into the backbone model to obtain the embedding representation, following formula (14).

[0052] e m=BackboneModel(M) (14)

[0053] Only one layer of fully connected network is used in feature extraction of commit message embedding. According to formula (15), we can obtain the commit message feature vector Z m .

[0054] Z m =W m ·e m +b (15)

[0055] The feature vector Z of the commit code c and the feature vector Z of the commit message m Fusion, to obtain a complete feature vector Z submitted, as shown in formula (16):

[0056]

[0057] Input Z into a fully connected network as a classifier, and map the high-dimensional vector to a one-dimensional vector through the sigmoid function to complete the binary classification prediction task, as shown in formula (17).

[0058] y = sigmoid(ReLU(W h Z+b h )) (17)

[0059] Compared with the prior art, the advantages and positive effects of the present invention are:

[0060] 1) Improve the accuracy of defect prediction: Through the architecture design based on retrieval enhancement, combined with information stacking and vector fusion methods, it can make full use of historical submission data and improve the accuracy and robustness of the instant defect prediction model. The model uses information from similar historical submission codes, making defect detection more stable and accurate in a variety of scenarios.

[0061] 2) Improve the generalization ability of the model: Based on the vector encoder obtained after pruning the large language model, efficient vector representation is achieved with fewer parameters and supports longer input text. This optimization method not only improves the efficiency of vector encoding, but also enhances the generalization ability of the model on different data sets.

[0062] 3) Significant information enhancement effect: Through information stacking and cross-model vector fusion methods, the model can enrich context information and features at the input stage, enhancing the accuracy of code representation. The information stacking method introduces retrieval information without changing the internal model structure, simplifies the complexity of the model, and retains the effect of context expansion.

[0063] 4) Strong dynamic adaptability: The architecture based on cross-model vector fusion can perform weighted adjustment on the retrieved vector features, which has strong dynamic adaptability. By directly introducing external feature vectors in this way, the model can more flexibly utilize historical data and enhance information flow, thus significantly improving the defect prediction capability.

[0064] 5) The present invention achieves high efficiency, accuracy and robustness of instant defect prediction by introducing a retrieval enhancement strategy and its optimized design, which significantly improves the practicability and application potential of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of a retrieval-enhanced instant defect prediction method based on a pre-trained model of the present invention;

[0066] Figure 2 It is a specific implementation framework diagram of the information stack-based retrieval enhancement model (Retrieval-basedInformation Stack) proposed in the present invention;

[0067] Figure 3 It is a specific implementation framework diagram of the retrieval-based vector fusion model proposed in the present invention;

[0068] Figure 4 It is a framework diagram of the no-retrieval model used in the embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described in detail below with reference to the accompanying drawings and implementation examples.

[0070] The core of the present invention is to construct a classification model by using retrieval enhancement in defect prediction tasks, and to examine the specific performance of the proposed scheme on different performance evaluation indicators such as AUC, F1 score and MCC to achieve the improvement of the performance of real-time defect prediction tasks.

[0071] A retrieval-enhanced instant defect prediction method based on a pre-trained model, the process is as follows Figure 1 As shown, the specific steps are as follows:

[0072] Step 1: Obtain feature data for the immediate defect prediction task;

[0073] For the real-time defect prediction task, the feature data includes the commit code of the change history code in the software development process and the commit message attached when the developer submits the code. The present invention uses the commit code and commit message as the model input.

[0074] Step 2: Prune the large language model based on the Llama architecture to obtain a lightweight vector encoder and perform vector representation on the commit code.

[0075] The present invention performs model pruning based on the large language model of the current Llama architecture (such as Codellama-7B) to obtain a model with a small number of parameters. The model pruning retains the embed_token, lm_head, and the last norm layer of the selected large language model, as well as the first and last decoder layers, and constructs a lightweight vector encoder containing only two layers of decoders.

[0076] The current Llama architecture model can achieve length extension through positional coding interpolation. The original Llama2 only supports an input length of 4096, while the updated large language model (such as Codellama-7B) can naturally handle longer text inputs based on continued training. Therefore, when constructing the commit code vector representation using the lightweight model obtained after trimming, the present invention selects a code length and text length of up to 6144 (i.e., 1024*6), which can basically cover the input length in the data set without truncation when constructing the vector. At the same time, due to the small amount of parameters after trimming, the present invention has a faster construction efficiency than the original large language model when constructing the vector representation of the data set.

[0077] The specific method of constructing the commit code vector of the present invention is to extract four modified file codes from one commit, take the code text in the i-th file as input, obtain a tensor with a shape of [6144, 4096] hidden state Hissen_state_i through the constructed vector encoder s, and then perform an average pooling operation on Hissen_state_i as shown in the following formula to obtain the vector representation mean_embedding_i of the i-th file, as shown in formula (1):

[0078]

[0079] Since a commit contains four files, the final vector representation of a commit is obtained by averaging the four vectors to obtain the final commit code vector representation F_llm:

[0080]

[0081] Step three: retrieve similar information based on the vector, obtain the retrieved information, and implement L2 distance similarity calculation and recall based on FAISS.

[0082] After obtaining the vector representation, similar information needs to be retrieved based on the vector. The selection of the retrieval method and the retrieved information are also a part that needs to be considered in model optimization. It is very important to use the retrieved information reasonably. The present invention selects the recall based on L2distance (Euclidean distance) for experimental research.

[0083] The implementation based on L2 distance is completed by the FAISS library in Python. The calculation formula of L2 distance (Euclidean distance) d can be expressed as follows, where x i and i are the i-th dimension components of the original code vector and the retrieved vector, respectively, and n is the dimension of the vector, as shown in formula (3).

[0084]

[0085] Step 4: Establish a retrieval enhancement model based on information stacking or cross-model vector fusion, process the commit code vectors separately, and obtain the most significant feature vector.

[0086] (1) Retrieval-based Information Stack (RIS) architecture Figure 2 As shown in the figure, the retrieved code is the n commit codes that are close to the current input commit code. That is, the model uses two different code information when constructing the code input, the code for this commit and the code for the retrieval. Different from the architecture without retrieval model, this retrieval enhancement based on information stacking only adds the retrieval code when constructing the input, and does not involve changes in the internal model structure.

[0087] In the process of processing the commit code, the present invention first selects the four most recently modified files from the current commit. The commit code is represented as C = [P1, P2, ..., P |C| ], where |C| represents the number of modified files in the commit, |C| = 4. If the number of modified files in the commit is less than 4, "empty patch" is used to fill the remaining positions. In addition to the code of the current commit, the closest commit code retrieved will also be merged into the input process of the commit code. At this time, the new commit code is represented by C = [P1, P2, ..., P |2C| ], and then each patch P i As a sequence in the submitted code, as code input to the bone intervention training model to obtain the embedding representation e i , as shown in the following formula:

[0088] e i=BackboneModel(P i ) (4)

[0089] After each patch in the submitted code is processed by the pre-trained model, the present invention uses The vector representation of the commitcode of the operation.

[0090]

[0091] Use CNN and max pooling to extract features of code embedding, specifically:

[0092] Define a filter f∈R kxd , and use k windows to extract new features for each k files of the submitted code. For a code patch e i , according to formula (5) we get the new vector x i .

[0093] x i =ReLU(f*e i:i+k-1 +b i ) (5)

[0094] Among them, the symbol * represents the sum of element products, ReLU is a nonlinear activation function, and b i ∈R is the bias value.

[0095] New Featuresx i They are connected in series to form a vector X, as shown in formula (6).

[0096] X=[x1,x2,...x |2C|-k+1 ] (6)

[0097] For the new feature vector X, use the maximum pooling method to obtain the most significant feature Z c ;

[0098] (2) The architecture of the retrieval-based vector fusion model (hereinafter referred to as RVF for ease of description) is as follows: Figure 3 As shown in the figure, different from the information stacking method, this architecture directly embeds the vector encoded by the vector encoder model as the internal feature F_llm of the defect prediction model. The commit code processing of this submission follows the original model processing method, that is, selecting the four most recently modified files from each submission. The commit code is represented by C = [P1, P2, ..., P |C| ], where |C| represents the number of modified files in the commit, and |C| = 4. If the number of modified files in the commit is less than 4, then "empty patch" is used to fill the remaining positions. Then each patch Pi As a sequence in the submitted code, as code input the bone intervention training model to obtain the embedding representation of the backbone model e i , as shown in the following formula.

[0099] e i =BackboneModel(P i ) (7)

[0100] After each patch in the commit code is processed by the pre-trained model, the present invention uses the ⊕ operation to obtain the vector representation of the commit code, following formula (8).

[0101]

[0102] For the cross-modal fusion vector, we use The operation stacks the internal features F_llm, where n is the number of recalled vector features.

[0103]

[0104] For the obtained retrieval vector F_llm all , the model uses a layer of full connection for dynamic weighting operation and maps it to the same dimension as the backbone model embedding feature, following the following formula:

[0105] weighted t =W fc ·F_llm all +b fc (9)

[0106] For the two-part code vector obtained, use The final representation of the vector of commit codes obtained by the operation is (10).

[0107]

[0108] At this point, the features of the commit code include the backbone model embedding and weighted fusion. After feature extraction through CNN, the final commit code representation Z is obtained. c .

[0109] Step 5: Process the commit message and combine the most significant features of the commit code to complete the immediate defect prediction task.

[0110] For the processing of commit messages, the present invention only performs a simple weighted fusion operation.

[0111] After obtaining the commit code’s feature vector Z c and the feature vector Z of the commit message m Then, Z c and Z m Fusion obtains the complete feature vector Z for one submission.

[0112] Input Z into a fully connected network as a classifier, and map the high-dimensional vector to a one-dimensional vector through the sigmoid function to complete the binary classification prediction task.

[0113] The increase in retrieval based on cross-model vector fusion is mainly achieved by retrieving several vectors F_llm all After redistributing the weights, the code features are added by fusing them with the vector features of the original input code represented by the small model. The fused code features and message features are then fused again to complete the construction of the real-time defect prediction model. Different from the retrieval enhancement model based on information stacking, the retrieval enhancement based on vector fusion directly introduces the vector representation to bring about the representation of the code by the external model, thereby enhancing the information flow during model training.

[0114] The proposed method can be universally implemented on models with different parameter values ​​and different architectures to demonstrate the effectiveness and generalization of the proposed method.

[0115] From the above, the proposed scheme RIS and RVF combined with the pre-trained model as the backbone model to construct the classification task can achieve performance improvement of the instant defect prediction task compared with the existing defect prediction technology.

[0116] Experimental implementation and result analysis

[0117] First, we build a retrieval-free model and use the pre-trained model as the backbone model to build a retrieval-free model. To obtain the vector representation of the input, we use Figure 4 The framework of the no-retrieval enhancement model is shown in Figure 1. In the process of processing commit codes, the four most recently modified files are selected from each commit. The commit code is represented by C = [P1, P2, ..., P |C| ], where |C| represents the number of modified files in the commit, and |C| = 4. If the number of modified files in the commit is less than 4, then "empty patch" is used to fill the remaining positions. Then each patch P i As a sequence in the submitted code, as code input to the bone intervention training model to obtain the embedding representation e i , as shown in formula (10).

[0118] e i =BackboneModel(P i ) (10)

[0119] After each patch in the submitted code is processed by the pre-trained model, use The vector representation of the commit code obtained by the operation follows formula (11).

[0120]

[0121] Use CNN and maximum pooling to extract the features of code embedding. Define a filter f∈R kxd And use k windows to extract new features of the submitted code. For the code patch P i , according to formula (12) we get the new vector x i .

[0122] x i =ReLU(f*P i:i+k-1 +b i ) (12)

[0123] Among them, the symbol * represents the sum of element products, ReLU is a nonlinear activation function, and b i ∈R is the bias value. Generate new features x by applying filter f to each k files of the submitted code i . New Features x i They are connected in series to form a vector X, as shown in formula (13).

[0124] X=[x1,x2,...x |C|-k+1 ] (13)

[0125] For the new feature vector X, use the maximum pooling method to obtain the most significant feature Z c .

[0126] For the processing of commit message, the entire commit message is regarded as a sequence M = [w1,w2,w3,...,w |m| ]. M is a sequence of words, and |m| is the length of the sequence. The submission information M is then fed into the backbone model to obtain the embedding representation, following formula (14).

[0127] e m =BackboneModel(M) (14)

[0128] Because the sequence length of the commit message is shorter than that of the commit code, only one layer of fully connected network is used in the feature extraction of the commit message embedding. According to formula (15), the commit message feature vector Z is obtained. m .

[0129] Zm =W m ·e m +b (15)

[0130] After obtaining the commit code’s feature vector Z c and the feature vector Z of the commit message m After that, the present invention will Z c and Z m The fusion is performed to obtain a complete feature vector Z for a submission, as shown in formula (16).

[0131]

[0132] Input Z into a fully connected network as a classifier, and map the high-dimensional vector to a one-dimensional vector through the sigmoid function to complete the binary classification prediction task, as shown in formula (17).

[0133] y = sigmoid(ReLU(W h Z+b h )) (17)

[0134] This no-retrieval model is mainly used to compare the performance of the retrieval enhancement model. The experimental environment is configured and the experiment is designed to evaluate the instant defect prediction effects of the two retrieval enhancement models using indicators such as AUC, F1, and MCC, and to compare and analyze the performance with the no-retrieval model to verify the advantages and applicability of each model architecture. The method can be generalized to a model with a larger number of parameters to verify the effectiveness and generalization of the two proposed retrieval enhancement schemes (RIS and RVF).

[0135] Table 1 Performance of the original CodeGPT model without retrieval in six projects

[0136]

[0137] Table 2 Performance of RIS+CodeGPT(1) retrieval enhancement model with the number of recall vectors being 1

[0138]

[0139] Table 3 Performance of retrieval enhancement model with RVF+CodeGPT(4) when the number of recall vectors is 4

[0140]

[0141] Tables 1, 2 and 3 respectively show the effect data obtained by applying the no-retrieval model and the solution proposed in the present invention in different projects. The evaluation indicators AUC, F1 and MCC obtained by applying the solution proposed in the present invention are improved to varying degrees. It can be seen that the solution proposed in the present invention has advantages in performance compared with the no-retrieval enhancement model.

[0142] Table 4 AUC score of the proposed optimization scheme compared with the baseline model

[0143]

[0144] From the comparison in Table 4, we can see that the average AUC of RVF+CodeGPT(4) is 0.764, which is 4.80% higher than DeepJIT and 4.95% higher than CC2Vec. The RIS and RVF architectures show comprehensive improvements in the AUC index compared to the baseline model, and are consistent across different projects. This shows that retrieval enhancement significantly promotes the predictive ability of the model in different dataset projects.

[0145] As shown in Table 5, for the f1 index, the average f1 values ​​of DeepJIT and CC2Vec are 0.425 and 0.406 respectively, and the average F1 value of RIS+CodeGPT(1) is 0.489, which is about 15.06% higher than DeepJIT and 20.44% higher than CC2Vec. The average F1 value of RVF+CodeGPT(4) is 0.491, which is 15.53% higher than DeepJIT and 20.94% higher than CC2Vec. In terms of f1 index, RIS and RVF have a more significant improvement over the baseline model, especially in projects with worse code retrieval results, showing stronger generalization ability and robustness.

[0146] Table 5 F1 score of the proposed optimization scheme compared with the baseline model

[0147]

[0148] As shown in Table 6, for the MCC index, the average MCC values ​​of DeepJIT and CC2Vec are 0.243 and 0.230 respectively, and the average MCC value of RIS+CodeGPT(1) is 0.306, which is about 25.93% higher than DeepJIT and 33.04% higher than CC2Vec. The average MCC value of RVF+CodeGPT(4) is 0.314, which is about 29.22% higher than DeepJIT and 36.52% higher than CC2Vec. In terms of MCC, the improvement of RIS and RVF structures is particularly significant, especially in terms of measuring the adaptability of the model to different projects and the robustness of the model, which provides higher numerical support, indicating that the two retrieval enhancement methods have a higher accuracy rate.

[0149] Table 6 MCC of the proposed optimization scheme compared with the baseline model

[0150]

[0151] Combining the performance of the three indicators of auc, f1 and MCC, RIS+CodeGPT(1) and RVF+CodeGPT(4) both show significant improvements compared to the baseline model, especially in F1 and MCC indicators. This shows that in the real-time defect prediction task scenario, the retrieval-enhanced model can effectively improve the model's prediction performance and generalization ability.

[0152] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. It is pointed out here that the above description helps those skilled in the art to understand the invention, but does not limit the protection scope of the invention. Any equivalent replacement, modification and / or simplification of the above description without departing from the essence of the invention falls within the protection scope of the invention.

Claims

1. A retrieval-enhanced instant defect prediction method based on a pre-trained model, characterized in that: The steps include: Step 1: Obtain feature data for the real-time defect prediction task, including the commit code of the change history during the software development process and the commit message attached by the developer when submitting the code; Step 2: Prune the large language model based on the Llama architecture to obtain a lightweight vector encoder and perform vector representation on the commit code. The specific method of constructing the commit code vector is: Select the 4 most recently modified files from each commit code, denoted as C = [P1, P2, ..., P |C| ]; If the number of modified files in the commit is less than 4, use "empty patch" to fill the remaining positions; The code text in the i-th file is taken as input, and a hidden state Hissen_state_i with a tensor shape of [6144, 4096] is obtained through the constructed vector encoder s. Then, an average pooling operation is performed on Hissen_state_i to obtain the vector representation mean_embedding_i of the i-th file, as shown in formula (1): Since a commit contains 4 files, the final vector representation of a commit is obtained by averaging the 4 vectors to obtain the final commit code vector representation F_llm: Step 3: Retrieve similar information based on the vector, obtain the retrieved information, and implement L2 distance similarity calculation and recall based on FAISS; Step 4: Establish a retrieval enhancement model based on information stacking or cross-model vector fusion, process the commit code vectors separately, and obtain the most significant feature vectors; (1) The process of processing the commit code vector by the information stacking-based retrieval enhancement model is as follows: The closest commit code retrieved is merged into the commit code input process. The new commit code is represented by C = [P1, P2, ..., P |2C| ], and then each patch P i As a sequence in the submitted code, as code input to the bone intervention training model to obtain the embedding representation e i , as shown in the following formula: e i =BackboneModel(P i ) (5) After each patch in the submitted code is processed by the pre-trained model, use The vector representation of the commit code is: Use CNN and max pooling to extract features of code embedding, specifically: Define a filter f∈R kxd , and use k windows to extract new features for each k files of the submitted code; for the code patch e i , according to formula (6) we get the new vector x i : x i =ReLU(f*e i:i+k-1 +b i ) (6) Among them, the symbol * represents the sum of element products, ReLU is a nonlinear activation function, and b i ∈R is the bias value; New Featuresx i Concatenate into vector X, as shown in formula (7): X=[x1,x2,...x |2C|-k+1 ] (7) For the new feature vector X, use the maximum pooling method to obtain the most significant feature Z c ; (2) The process of processing the commit code vector by the retrieval enhancement model based on cross-model vector fusion is as follows: This architecture directly uses the vector embedding encoded by the vector encoder model as the internal feature F_llm of the defect prediction model; the commit code processing of this submission follows the original model processing method, that is, the four most recent modified files are selected from each submission, and the commit code is represented by C = [P1, P2, ..., P |C| ], where |C| represents the number of modified files in the submission, |C| = 4; if the number of modified files in the submission is less than 4, use "empty patch" to fill the remaining positions; then each patch P i As a sequence in the submitted code, as code input the bone intervention training model to obtain the embedding representation of the backbone model e i : e i =BackboneModel(P i ) (8) After each patch in the submitted code is processed by the pre-trained model, use The vector representation of the commit code obtained by the operation follows formula (9): For the cross-modal fusion vector, we use The operation stacks the internal features F_llm, where n is the number of recalled vector features: For the obtained retrieval vector F_llm all , the model uses a layer of full connection for dynamic weighting operation and maps it to the same dimension as the backbone model embedding feature, following the following formula: weighted t =W fc ·F_llm all +b fc (10) For the two-part code vector obtained, use The final representation of the commit code vector obtained by the operation (11) is: At this point, the features of the commit code include the backbone model embedding and the weighted fusion. Then, CNN and maximum pooling are used to extract the features of the code embedding, and a filter f∈R is defined. kxd And use k windows to extract the submission code C e ', for code patch e i , according to formula (12) we get the new vector x i : x i =ReLU(f*e i:i+k-1 +b i ) (12) Among them, the symbol * represents the sum of element products, ReLU is a nonlinear activation function, and b i ∈R is the bias value; a new feature x is generated by applying the filter f to each k files of the submitted code i ; New Featuresx i Concatenate into vector X, as shown in formula (13): X=[x1,x2,...x |C+1|-k+1 ] (13) For the new feature vector X, use the maximum pooling method to obtain the most significant feature Z c ; Step 5: Process the commit message and combine the most significant features of the commit code to complete the immediate defect prediction task; The feature vector Z of the commit code c and the feature vector Z of the commit message m Fusion, to obtain a complete feature vector Z submitted, as shown in formula (16): Input Z into a fully connected network as a classifier, and map the high-dimensional vector to a one-dimensional vector through the sigmoid function to complete the binary classification prediction task, as shown in formula (17): y=sigmoid(ReLU(W h Z+b h )) (17)。 2. The retrieval-enhanced instant defect prediction method based on a pre-trained model according to claim 1, characterized in that: The pruned large language model retains embed_token, lm_head and the last norm layer, as well as the first and last decoder layers, to construct a lightweight vector encoder containing only two layers of decoders.

3. The retrieval-enhanced instant defect prediction method based on a pre-trained model according to claim 1, characterized in that: The calculation formula of L2 distance d is: Among them, x i and i They are the i-th dimension components of the original code vector and the retrieved vector, respectively, and n is the dimension of the vector.

4. The retrieval-enhanced instant defect prediction method based on a pre-trained model according to claim 1, characterized in that: Process the commit message to get the feature vector Z m , specifically: Consider the entire commit message as a sequence M = [w1,w2,w3,...,w |m| ], M is a sequence of words, and |m| is the length of the sequence; the submission information M is then input into the backbone model to obtain the embedded representation, following formula (14): e m =BackboneModel(M) (14) Only one layer of fully connected network is used in the feature extraction of commit message embedding; according to formula (15), the commit message feature vector Z is obtained. m : Z m =W m ·e m +b (15)。

Images