A method for detecting recurring defect reports based on BERT and BILSTM
By combining BERT and BILSTM models, global semantic and contextual features of defect report pairs are extracted, solving the problem of low accuracy in duplicate defect report detection in existing technologies and achieving more efficient duplicate defect report detection.
Patent Information
- Application Number
- CN202310718582.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-16
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-06-16
AI Technical Summary
Existing technologies are unable to effectively detect duplicate content in software defect reports, resulting in low efficiency in subsequent processing. Furthermore, existing methods cannot effectively extract deep semantic and temporal information from the text, leading to low detection accuracy.
We employ a method based on BERT and BILSTM, using the BERT model to extract global semantic features and contextual temporal features between defect report pairs. By combining multi-head attention mechanism and Softmax classification, we can achieve accurate detection of duplicate defect reports.
It improves the accuracy of duplicate defect report detection, better captures the semantic dependencies and contextual information of words, avoids gradient vanishing and exploding, and improves the accuracy and efficiency of detection.
Smart Images

Figure CN116629271B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of information management technology and relates to the detection of software development defect reports, specifically to a method for detecting duplicate defect reports based on BERT and BILSTM. Background Technology
[0002] With the rapid development of the information industry, the continuous growth of the software industry, and the increasing complexity of software development processes, software defects are unavoidable. More and more projects are using defect tracking systems such as Bugzilla and Debian to store and manage software defect reports. Because developers, testers, and users all contribute to and submit defect reports throughout the process, a single defect may appear in multiple reports simultaneously. Furthermore, since error reports are always written in natural language, the same defect may be described differently in different reports. As the number of reports stored in the defect tracking system increases, a large number of duplicate defect reports inevitably emerge, affecting subsequent defect report assignment and defect remediation. Manually detecting duplicate defect reports in the system is time-consuming and labor-intensive. Therefore, effectively detecting duplicate defect reports is crucial for reducing the number of defect reports, decreasing the workload of defect report assignment, preventing assignment errors, and improving defect remediation efficiency.
[0003] To improve the efficiency of detecting duplicate defect reports and reduce the workload of defect report assignment, researchers have proposed many techniques for automatically detecting duplicate defect reports in recent years. These techniques can be broadly categorized into three types: information retrieval-based methods, machine learning-based methods, and deep learning-based methods. The first two types of methods use techniques such as BM25, TF-IDF, and topic modeling to determine whether two defect reports are duplicates. However, these methods mostly use static word embeddings, which only consider the word frequency information of the defect report text and cannot extract deep semantic and temporal information, thus failing to address issues such as polysemy. While the third type of methods can dynamically extract text features based on word embeddings, they still suffer from insufficient feature extraction and an inability to effectively handle complex word dependencies, resulting in low detection accuracy. Therefore, accurately detecting duplicate defect reports remains a challenge. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention proposes a duplicate defect report detection method based on BERT and BILSTM, which combines the interaction features and contextual information between defect report pairs to achieve accurate detection of duplicate defect reports.
[0005] The method for detecting duplicate defect reports based on BERT and BILSTM specifically includes the following steps:
[0006] Step 1: Obtain the public dataset of defect reports and construct pairs of duplicate and non-duplicate defect reports.
[0007] Step 2: Perform noise removal and text preprocessing on the defect reports.
[0008] Step 3: Extract the global semantic features and contextual temporal features between defect report pairs. The specific steps are as follows:
[0009] Step 3-1: Input the processed defect report text into the BERT model, obtain word vectors with location information through BERT word embedding, and then extract global semantic features containing the interaction information of the defect report from the word vectors based on the multi-head attention mechanism.
[0010] Step 3-2: Input the word vectors obtained in Step 3-1 into the BILSTM model to extract contextual features.
[0011] Step 4: Concatenate and fuse the global semantic features and contextual features obtained in Step 3, and input them into the fully connected layer. Calculate the probability of duplicate defect reports using Softmax.
[0012] Step 5: Use the cross-entropy loss function to train the model parameters from Step 3, so that the repetition probability of duplicate defect report pairs in Step 4 is close to 1, and the repetition probability of non-duplicate defect report pairs is close to 0.
[0013] Step 6: After preprocessing the defect report to be processed, check whether the defect report is duplicated with the defect reports in the dataset using the methods in steps 3 and 4. If they are not duplicated, add them to the dataset. If they are duplicated, mark them as duplicate defect reports and add them to the dataset.
[0014] The present invention has the following beneficial effects:
[0015] 1. Feature extraction based on BERT and BILSTM can not only obtain the long-distance dependencies and global features of defect report text, but also better capture the bidirectional semantic dependencies between defect report pairs. When extracting semantic information of words, it can better combine the context and avoid phenomena such as gradient vanishing and gradient exploding.
[0016] 2. The BERT model is used for next sentence prediction (NSP) to embed vectors and extract features from defect report pairs. The extracted features combine the interaction information between the two defect reports, which can identify the situation where the same words represent different meanings in different positions, thereby further improving the accuracy of duplicate defect report detection. Attached Figure Description
[0017] Figure 1 Flowchart of the method for detecting recurring defect reports;
[0018] Figure 2 This is a schematic diagram of the detection model used in the embodiment. Detailed Implementation
[0019] The present invention will be further explained below with reference to the accompanying drawings;
[0020] This invention proposes a method for detecting recurring defect reports based on BERT and BILSTM, such as... Figure 1 As shown, a pair of preprocessed defect report texts are input into the trained network model to determine whether the content is duplicated. The network model is as follows: Figure 2 As shown, it includes an input layer, an embedding layer, a feature extraction layer, a feature fusion layer, and an output layer. The specific detection steps are as follows:
[0021] Step 1: Obtain defect reports from public datasets. A complete defect report includes information such as a summary, description, product, component, comments, and duplicate defect report ID. The summary is a brief description of the defect, and the description is a detailed description of the defect by the submitter. Select the content of the summary and description sections from the defect reports as the defect report text data. Then, construct duplicate defect report pairs and non-duplicate defect report pairs based on the duplicate defect report IDs, and divide them into training and test sets proportionally.
[0022] Step 2: In the description section of the defect report, in addition to a detailed description of the defect written in natural language, it also includes a large amount of non-natural language information, such as stack traces, URLs, code snippets, etc. This content does not contain semantic information, so it is treated as noise and removed using regular expressions. Then, the remaining content undergoes operations such as word segmentation, stemming, lowercase conversion, and stop word removal.
[0023] Step 3: Train the duplicate defect report detection model using supervised learning.
[0024] Step 3-1: Input the preprocessed defect report text into the embedding layer. The embedding layer uses BERT word embedding, which distinguishes two defect report texts by inserting a special marker [SEP] between them, and then converts the text content into 768-dimensional word vectors. BERT word embedding adds positional embedding to word embedding, taking into account the influence of text positional information on text semantics when representing text vectors. The formula for calculating the positional embedding vector is as follows:
[0025]
[0026]
[0027] Where p represents the position of the word, d pos This represents the dimension of the position embedding vector, where i represents the range from 0 to d. pos If the integer is 2^(-1), then 2^i represents the even dimension of the position embedding vector, 2^i+1 represents the odd dimension of the position embedding vector, and PE is the position matrix. 2i (p) is the value in the p-th row and 2i-th column of matrix PE. PE 2i+1 (p) is the value in the p-th row and 2i+1-th column of matrix PE. That is, different formulas are used to calculate the values in odd-numbered columns and even-numbered columns respectively.
[0028] Summing the word vectors obtained from BERT word embeddings with the positional embedding vectors yields word vectors with positional information. These word vectors possess not only semantic textual information but also positional information, which is beneficial for attention calculations in subsequent steps.
[0029] Step 3-2: Concatenate the word vectors of the two defect reports output by the word embedding layer into a word embedding matrix in sequence. Multiply the word embedding matrix by three randomly initialized matrices, which are used as the Query, Key, and Value matrices, respectively, denoted as . Calculate self-attention (Attention(Q,K,V)):
[0030]
[0031] Where n represents the length of the input text.
[0032] In one layer of BERT, projection matrices are used respectively. Linear mapping is performed on Q, K, and V to calculate self-attention. This process is repeated 8 times, and the results of the 8 calculations are concatenated to obtain the multi-head attention weights.
[0033]
[0034] MultiHead(Q,K,V)=Concat(head1,…,head8)W O
[0035] Among them, the weight matrix Therefore, the multi-head attention weight MultiHead(Q,K,V) is a function with dimension n×(8d). v The sequence is ). In this embodiment, d is set. k =d v =d model / 8 = 64.
[0036] The multi-head attention weights are multiplied by the word embedding matrix and used as input for the next BERT layer. After 12 layers of the BERT model, global semantic features containing interaction information of defect reports are obtained.
[0037] Step 3-3: Input the word vectors obtained from the word embedding layer in Step 3-1 into the BILSTM model to extract the contextual temporal features of the text. The BILSTM model consists of two LSTM structures, which process the input data in forward and reverse order respectively. Finally, the outputs of the two LSTM structures are concatenated as the output of the BILSTM model.
[0038] The LSTM structure includes a forget gate, an input gate, and an output gate. The forget gate simultaneously passes information from the previous hidden state and the current input information to the sigmoid function. The output value of the sigmoid function is between 0 and 1, and the output value determines which information should be discarded or retained. The closer the output value is to 0, the more likely it should be discarded, and the closer it is to 1, the more likely it should be retained.
[0039] The input gate passes information from the previous hidden state and the current input to the sigmoid function, determining the importance of the information based on the sigmoid function's output. Simultaneously, it passes the information from the previous hidden state and the current input to the tanh function, creating a new candidate value vector. Finally, the outputs of the two functions are multiplied to determine the information to be retained in the candidate value vector.
[0040] The output gate passes the information from the previous hidden state and the current input to the sigmoid function, and the newly obtained state information to the tanh function. Finally, the output values of the two functions are multiplied to determine the information carried by the hidden state, and the hidden state is used as the output of the current cell. The new cell state and the new hidden state are then passed to the next time step.
[0041] Step 4: The features extracted by the BERT model in Step 3-2 and the features extracted by the BILSTM model in Step 3-3 are merged by concatenation to obtain higher-dimensional classification features, making the features more discriminative.
[0042] Step 5: Input the classification features obtained in Step 4 into the fully connected layer for classification. After Softmax calculation, the output is the probability that the input defect report is a duplicate or not. Based on the obtained probability, determine whether the contents of the two defect reports are duplicated.
[0043] We conducted defect report detection experiments using data from three publicly available datasets: Eclipse, Mozilla, and Thunderbird. Our method was compared with four common models from existing technologies, and the results are shown below:
[0044] Table 1 Experimental results on the Mozilla dataset
[0045] Models Accuracy Precision Recall F1-score BERT-BILSTM 0.94 0.92 0.93 0.92 BERT 0.92 0.88 0.92 0.90 S-BERT 0.83 0.83 0.83 0.83 DC-CNN 0.83 0.83 0.69 0.75 RoBERTa 0.65 0.72 0.72 0.72
[0046] Table 2 Experimental Results of the Eclipse Dataset
[0047] Models Accuracy Precision Recall F1-score BERT-BILSTM 0.95 0.86 0.92 0.88 BERT 0.94 0.79 0.94 0.86 S-BERT 0.82 0.85 0.82 0.83 DC-CNN 0.83 0.52 0.82 0.64 RoBERTa 0.72 0.70 0.55 0.60
[0048] Table 3 Experimental results on the Thunderbird dataset
[0049] Models Accuracy Precision Recall F1-score BERT-BILSTM 0.91 0.94 0.88 0.91 BERT 0.86 0.90 0.87 0.89 S-BERT 0.80 0.80 0.80 0.80 DC-CNN 0.80 0.84 0.75 0.79 RoBERTa 0.68 0.66 0.65 0.64
[0050] The data in the table shows that the BERT-BILSTM model has the highest precision, accuracy, recall, and F1 score on the Mozilla and Thunderbird datasets. On the Eclipse dataset, it has the highest precision, accuracy, and F1 score, while its recall ranks second.
[0051] A comparison of the BERT and S-BERT models reveals that the results appear to be significantly influenced by the training method. The BERT model's training process involves two methods: 1) MLM (Mask Language Model), which helps the model understand the relationships between words; and 2) NSP (Next Sentence Prediction), which helps the model understand long-term dependencies between sentences. Given two sentences A and B, NSP determines the positional relationship between sentence B and sentence A. If the model determines that sentence B follows sentence A, a term dependency exists between the two sentences; otherwise, no term dependency exists. The BERT model uses the NSP method for training. In contrast, S-BERT, which relies on the Siamese architecture, uses MLM. For two input defect reports, it generates fixed-size feature vectors for each, then concatenates these two feature vectors to obtain a feature vector representing the defect report pair. The results show that the BERT model trained using NSP performs better than the S-BERT model trained using MLM. This is because the NSP method allows the model to understand long-term dependencies between sentences, thus better determining whether two defect reports are duplicates.
Claims
1. A method for detecting duplicate defect reports based on BERT and BILSTM, characterized in that: Specifically, the following steps are included: Step 1: Obtain the public dataset of defect reports, select the content of the summary and description sections of the defect reports, and construct pairs of duplicate and non-duplicate defect reports; Step 2: Perform noise removal and text preprocessing on the defect reports; Step 3: Extract the global semantic features and contextual temporal features between defect report pairs. The specific steps are as follows: Step 3-1: Input the processed defect report text into the BERT model, obtain word vectors with location information through BERT word embedding, and then extract global semantic features containing the interaction information of the defect report from the word vectors based on the multi-head attention mechanism. Step 3-2: Input the word vectors obtained in Step 3-1 into the BILSTM model to extract contextual features; Step 4: Concatenate and fuse the global semantic features and contextual features obtained in Step 3, and input them into the fully connected layer. Calculate the probability of duplicate defect reports using Softmax. Step 5: Use the cross-entropy loss function to train the model parameters in Step 3, so that the repetition probability of duplicate defect report pairs in Step 4 is close to 1, and the repetition probability of non-duplicate defect report pairs is close to 0. Step 6: After preprocessing the defect report to be processed, check whether the defect report is duplicated with the defect reports in the dataset using the methods in steps 3 and 4. If they are not duplicated, add them to the dataset. If they are duplicated, mark them as duplicate defect reports and add them to the dataset.
2. The method for detecting duplicate defect reports based on BERT and BILSTM as described in claim 1, characterized in that: Non-natural language in the missing phase report pairs is treated as noise information and removed using regular expressions.
3. The method for detecting duplicate defect reports based on BERT and BILSTM as described in claim 1 or 2, characterized in that: For the defect report content after noise removal, we perform word segmentation, stemming, lowercase conversion, and stop word removal.
4. The method for detecting duplicate defect reports based on BERT and BILSTM as described in claim 1, characterized in that: The BERT model uses BERT word embedding to generate word vectors and positional embedding vectors for the input data. The positional embedding vector is generated based on the word's position using the following formula: Where p represents the position of the word, d pos This represents the dimension of the position embedding vector, where i represents the range from 0 to d. pos If the integer is 2 - 1, then 2i represents the unit being in an even position, 2i+1 represents the word being in an odd position, and PE is the position embedding vector; PE 2i (p), PE 2i+1 (p) represents the values at the 2ith and (2i+1)th positions in the pth row of PE, respectively; The word vectors obtained from BERT word embeddings and the position embedding vectors are summed to obtain word vectors with position information.
5. The method for detecting duplicate defect reports based on BERT and BILSTM as described in claim 1 or 4, characterized in that: The BERT model concatenates the word vectors of the two defect reports in a defect report pair into a word embedding matrix, and then passes it through a 12-layer multi-head attention mechanism to obtain global semantic features containing the interaction information of the defect report pair. Eight self-attention weight calculations are performed in each layer of the multi-head attention mechanism.
6. The method for detecting duplicate defect reports based on BERT and BILSTM as described in claim 5, characterized in that: The method for calculating the self-attention weights is as follows: Multiply the input data by three randomly initialized matrices, and denote the results as Q, K, and V. Calculate the self-attention (Attention(Q,K,V)) corresponding to the input data using the following formula: Where n represents the length of the input data, d k Let Q be the dimension of the Q matrix and K matrix.
7. The method for detecting duplicate defect reports based on BERT and BILSTM as described in claim 1 or 4, characterized in that: The BILSTM model uses two LSTM structures in opposite directions to extract contextual temporal features of word vectors.
Citation Information
Patent Citations
Duplicate defect report detection method based on dual-channel convolutional neural network
CN110188047A
Software defect report dispatching method and device based on knowledge graph and semantic role labeling
CN113138920A
