A pre-trained code-enhanced answer recommendation method for programming question-and-answer communities
By combining code features and tag recommendations through pre-trained code enhancement, this approach addresses the problem of neglecting code features in existing technologies, achieving more efficient and accurate answer recommendations for programming question-and-answer communities.
Patent Information
- Application Number
- CN202311555457.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-21
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-11-21
AI Technical Summary
Existing answer recommendation methods in programming Q&A communities neglect the programming expertise of these communities, especially the characteristics of the code, resulting in a poor user experience. Furthermore, existing methods rely on expert answers, making it difficult to achieve real-time recommendations.
We employ a pre-trained code augmentation method, incorporating code embedding as one of the features, and combine it with the label recommendation task. By using a pre-trained model and fine-tuning techniques, we improve the answer recommendation performance and leverage the Lucene search engine to increase the speed of candidate answer determination.
It significantly improves the accuracy and efficiency of answer recommendation, especially performing well in question-and-answer questions containing code. It can effectively handle noisy answer lists and maintain high-quality recommendation results.
Smart Images

Figure CN117591647B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of code search technology, and in particular to a method for recommending answers in a programming question-and-answer community using pre-trained code enhancement. Background Technology
[0002] In recent years, programming Q&A communities have developed rapidly, allowing developers to quickly find answers to their questions on platforms like Stack Overflow and CSDN. However, the large number of unresolved questions in these communities leads to a poor user experience. Answer recommendation methods, which can help users quickly obtain accurate answers, have become a hot topic in programming Q&A communities.
[0003] Existing answer recommendation methods in programming Q&A communities typically focus on the textual semantics of questions and answers, neglecting the programming professionalism of the community itself. For Stack Overflow: 1) Linkable questions: Hyperlinks in Q&A can connect multiple questions, and the implicit links are often overlooked; 2) Code features: Code blocks in Q&A are often removed as noise, but code is actually one of the core features of programming Q&A communities.
[0004] Domestic and international researchers have conducted extensive research on programming question-and-answer communities, mainly focusing on three aspects: answer recommendation based on question retrieval, answer recommendation based on answer ranking, and answer recommendation based on expert recommendation.
[0005] Answer recommendation based on question retrieval has studied the representation of questions from multiple aspects, including models, features, and metrics, and has achieved certain results in question retrieval tasks. Existing research also considers the question link relationships in programming question-answering communities. Xu et al. defined four types of link relationships between questions and used a convolutional neural network model for prediction. Fu et al. focused on efficiency issues and proposed a method using machine learning fine-tuning to efficiently predict link relationships between questions. Xu et al. proposed a more effective method for predicting inter-question relationships based on feature engineering. However, the above methods only focus on the semantics of neighboring words, ignoring the global features of words in a single question and the joint features of words in another question.
[0006] Answer recommendation methods based on answer ranking improve the effectiveness of answer recommendation from multiple aspects, including word matching, semantic matching, and feature enhancement. However, they all treat programming Q&A communities as ordinary Q&A communities, only considering common features of communities such as text users, and ignoring the code information unique to programming Q&A communities. However, code is an important component for expressing semantics in programming Q&A communities, and existing research treats it as noise and deletes it, which will result in the loss of some semantic information.
[0007] Expert-recommendation-based answer recommendation methods combine user and text features to recommend suitable experts for answers. While this approach can guarantee answer quality, it heavily relies on expert responses and makes real-time answer recommendation difficult. Summary of the Invention
[0008] To address the aforementioned problems in existing technologies, this invention proposes a pre-training code-enhanced answer recommendation method for programming question-and-answer communities (PT-CARec). This method embeds code separately as one of the features used to determine the relationship between questions and answers. Furthermore, PT-CARec leverages pre-training techniques combined with a label recommendation task, building upon code enhancement, to guide the generation of feature vectors for questions and answers. Specifically, it fine-tunes the parametric model of label recommendation using a question-and-answer dataset to obtain a model suitable for answer recommendation. By utilizing the dual enhancement of labels and code, the effectiveness of answer recommendation is improved.
[0009] To achieve the above objectives, the present invention adopts the following technical solution: a method for recommending answers in a programming question-and-answer community with pre-trained code enhancement, comprising the following steps:
[0010] S1: Tag recommendation pre-training stage, build recommendation pre-training model TagRec. TagRec includes three feature extractors: Title Extractor, Desc Extractor and Code Extractor.
[0011] TagRec takes the question title, question description, and question code as input and the question label as the truth value. The output of TagRec is the confidence probability distribution for each question label for that question. The distribution is the predicted value for each question. The cross-entropy loss function is used to calculate the cross-entropy between the predicted value and the truth value, and the parameters of TagRec are updated according to the value of the loss function. When the set number of iterations is reached, a pre-trained TagRec is obtained.
[0012] S2: In the question-answering classification fine-tuning stage, the pre-trained TagRec is reused. The parameters of the pre-trained TitleExtractor, Desc Extractor, and Code Extractor are frozen. The question title, question description, question code, answer description, and answer code are used as input samples for the pre-trained TagRec. The label of each input sample is used as the true value. The output of the pre-trained TagRec is the confidence probability of four categories: positive, neural+, neural-, and negative. The confidence probabilities of the four categories are used as the predicted values of the input samples. The cross-entropy loss function is used to calculate the cross-entropy between the predicted and true values of the input samples. The parameters of the pre-trained TagRec are updated according to the value of the loss function. When the set number of iterations is reached, the trained TagRec is obtained.
[0013] S3: In the recommendation stage, for a new question, first use a search engine to query similar question titles and select the top Q as candidate questions. Next, embed words in the candidate question titles using pre-trained Word2vec word vectors and use the average vector of the words as a temporary representation of the question title. Calculate the cosine similarity between the candidate question title vector and the new question title vector, sort and select the top k questions as the filtered candidate questions, where k is a hyperparameter. Then extract the answers corresponding to the k questions as candidate answers.
[0014] The answers in the candidate answers and the new question are used as input to the trained TagRec to obtain the confidence probabilities of the new question and each candidate answer in four categories: positive, neural+, neural-, and negative. The matching score Score of the new question and each candidate answer is calculated according to formula (24). After sorting the matching scores from largest to smallest, the candidate answers corresponding to the top Q' matching scores are used as the final recommended answers.
[0015] Score = 2 * p positive +p nerual+ -p nerual- -2*p negative (twenty four).
[0016] Preferably, the step in S1 to calculate the confidence probability distribution of each question label for that question is as follows:
[0017] Given a question title containing M words, a question description containing N words, and a question code containing K words, let w iLet be the i-th word in title, desc, and code, where i = 1, 2, ..., M+N+K. Record the number of each word. Using random initialization techniques, each word is transformed into a discrete vector representation. title, desc, and code are mapped to a high-dimensional matrix space, denoted as T∈R. M×d ,D∈R N×d ,C∈R K×d As input to TagRec, T represents the title matrix, D represents the question description matrix, C represents the question code matrix, and d is the encoding dimension of the word vectors.
[0018] After using TagRec, we obtain the confidence probability of each label in the current space. Represents the set of real numbers with dimension |tags|.
[0019] For the title matrix T, as shown in formula (3), the convolution operation is performed between each position and p randomly initialized convolution kernels, and the results are combined into a feature matrix, denoted as . Where i is the convolution kernel with the i-th stride, and |stride| refers to the number of stride categories;
[0020]
[0021] Among them, Conv i This is the i-th convolution operation, and no padding is applied to the edge data. Next, we will... Max pooling is performed along the sequence dimension, as shown in Equation (4), to obtain the feature vector after max pooling of the i-th convolutional kernel.
[0022]
[0023] The title extractor finally concatenates the feature vectors obtained from max pooling of convolutional kernels of different lengths along the word vector dimension using a concatenation operation to obtain the title feature vector. As shown in the following formula:
[0024]
[0025] in, It is a join operation.
[0026] The feature vector of the problem description is obtained using the same method as that used to obtain the feature vector of the title. and the feature vector of the problem code
[0027] By splicing FV T FV D and FV C This yields the representation vector of the entire problem.
[0028]
[0029] The FV is modeled using a ReLU-based nonlinear transformation to obtain the hidden layer vector. Finally, Softmax is used to map the hidden layer vectors to the label distribution space, obtaining the confidence probability of each question label for that question. As shown in (7)(8):
[0030] hidden = Rule(FV·W) H +b H (7)
[0031] distribution=Sigmoid(hidden·W D +b D (8)
[0032] in All of them are trainable parameters.
[0033] Preferably, the S2 adjustment process is as follows: The specific steps for calculating the confidence probabilities of the four categories (positive, neural+, neural-, and negative) in S2 are as follows:
[0034] The inputs to the pre-trained TagRec are: question title, question description, question code, answer description, and answer code. These are then encoded into dense matrix representations, which are input into their respective feature extractors and concatenated to obtain the feature vector representation of the question title. Describe the feature vector representation Problem code feature vector representation Answer description: Feature vector representation and answer code feature vector representation The problem feature vector FV is obtained by concatenating the features of each feature vector. q and the answer feature vector FV a , As shown in formulas (9)-(15):
[0035]
[0036]
[0037]
[0038]
[0039]
[0040]
[0041]
[0042] FV q and FV a By connecting and using a multilayer perceptron to mine deep features and their correlations, a new hidden layer vector is obtained. Finally, Softmax is used to map hidden′ to the label distribution space to obtain the confidence probability of each question label for that question, as shown in equations (16)-(18):
[0043]
[0044] hidden′=Rule(FV·W H +b H (17)
[0045] distribution′=Sigmoid(hidden′·W D +b D (18)
[0046] in These are trainable parameters;
[0047] Use formula (18) to calculate the confidence probabilities of the four categories: positive, neural+, neural-, and negative.
[0048] Compared with the prior art, the present invention has at least the following advantages:
[0049] 1. Unlike previous programming Q&A community answer recommendation methods that do not use the code in the posts, this method uses code as an additional feature to enhance the answer recommendation effect.
[0050] 2. In addition to the code, this method incorporates the important element of labels into the answer recommendation model and combines it with the label recommendation pre-trained model, taking advantage of the pre-training fine-tuning method in feature extraction.
[0051] 3. This method introduces the Lucene search engine based on inverted index, which can efficiently improve the speed of determining candidate answers.
[0052] 4. This method has been thoroughly tested on a sufficient dataset and shows significant improvement over existing methods for answer recommendation tasks. Attached Figure Description
[0053] Figure 1 A pre-trained code-enhanced answer recommendation framework.
[0054] Figure 2 Tag recommendation for pre-trained model structure.
[0055] Figure 3 Code-enhanced question-answering classification model structure. Detailed Implementation
[0056] The present invention will now be described in further detail.
[0057] See Figure 1-3 For ease of description, this invention, a pre-trained code-enhanced answer recommendation method for a programming question-and-answer community, is abbreviated as PT-CARec (Pre-Training Code enhanced Answer Recommendation).
[0058] A method for recommending answers in a programming question-and-answer community using pre-trained code enhancement includes the following steps:
[0059] S1: Label recommendation pre-training phase Figure 2 The overall process of building a tag recommendation pre-trained model is shown. The recommendation pre-trained model TagRec is constructed. TagRec includes three feature extractors: Title Extractor, Desc Extractor, and Code Extractor.
[0060] TagRec takes the question title, description, and code as input, and the question label as the ground truth. The output of TagRec is the confidence probability (distribution) of each question label for that question; the distribution is the predicted value for each question. A cross-entropy loss function is used to calculate the cross-entropy between the predicted and ground truth values, and the parameters of TagRec are updated based on the loss function value. After a set number of iterations, a pre-trained TagRec is obtained. Specifically, before inputting data into the model, a label dictionary is constructed by counting all labels appearing in the entire dataset. Therefore, the final output vector of TagRec, i.e., the distribution, has a length equal to the length of this label dictionary. Assuming the length of this vector is 50, the value at the i-th position in the vector represents the probability that the post belongs to that category. Regarding the update iteration of model parameters: the cross-entropy between this predicted vector and the ground truth value (also a vector) is calculated as the loss function, thereby updating the model parameters.
[0061] S2: In the question-answering classification fine-tuning stage, the pre-trained TagRec is reused. The parameters of the pre-trained TitleExtractor, Desc Extractor, and Code Extractor are frozen. The question title, question description, question code, answer description, and answer code are used as input samples for the pre-trained TagRec. The label of each input sample is used as the true value. The output of the pre-trained TagRec is the confidence probability of four categories: positive, neural+, neural-, and negative. The confidence probabilities of the four categories are used as the predicted values of the input samples. The cross-entropy loss function is used to calculate the cross-entropy between the predicted and true values of the input samples. The parameters of the pre-trained TagRec are updated according to the value of the loss function. When the set number of iterations is reached, the trained TagRec is obtained.
[0062] The calculation of four confidence probabilities—positive, neural+, neural-, and negative—measures the degree of matching between question-answer pairs. Higher confidence probabilities indicate a more suitable answer for the question. In the fine-tuning phase, the task of fine-tuning the model is essentially a classification task. In the fine-tuning dataset, each sample is a question-answer pair, and each pair has a label (different from the tag mentioned above). For the same question, there are three different answers that form pairs (i.e., three different samples). The label of each sample represents the matching category (or degree of matching) between the answer and the question. There are four categories, corresponding to the four confidence categories mentioned above. Because it is a multi-class classification task, the model does not directly return a specific category but rather the probabilities of different categories. After obtaining the confidence probabilities of the four categories, this result is also a vector. This vector is used to calculate the cross-entropy with the true label, which serves as the loss function for the fine-tuning phase, and then iteratively updates the model parameters.
[0063] S3: In the recommendation stage, for a new question, first use a search engine to query similar question titles, and select the top Q [Q can be 10000] as candidate questions. Next, embed words from the candidate question titles using pre-trained Word2vec word vectors, and use the average vector of the words as a temporary representation of the question title. Calculate the cosine similarity between the candidate question title vector and the new question title vector, sort and select the top k questions as the filtered candidate questions, where k is a hyperparameter, and extract the answers corresponding to the k questions as candidate answers.
[0064] The answers in the candidate answers and the new question are used as input to the trained TagRec to obtain the confidence probabilities of the new question and each candidate answer in four categories: positive, neural+, neural-, and negative. The matching score Score of the new question and each candidate answer is calculated according to formula (24). After sorting the matching scores from largest to smallest, the candidate answers corresponding to the top Q' matching scores are used as the final recommended answers.
[0065] Score = 2 * p positive +p nerual+ -p nerual- -2*p negative (twenty four).
[0066] Specifically, the steps in S1 for calculating the confidence probability distribution of each question label for that question are as follows:
[0067] Given a question title containing M words Problem description containing N words Problem code containing K words Let w i Let be the i-th word in title, desc, and code, where i = 1, 2, ..., M+N+K. Record the number of each word. Using random initialization techniques, each word is transformed into a discrete vector representation. title, desc, and code are mapped to a high-dimensional matrix space, denoted as T∈R. M×d ,D∈R N×d ,C∈R K×d As input to TagRec, where T represents the title matrix, D represents the question description matrix, C represents the question code matrix, and d is the encoding dimension of the word vectors, the above operations can be represented by formulas (1) and (2):
[0068] After using TagRec, we obtain the confidence probability of each label in the current space. Represents the set of real numbers with dimension |tags|.
[0069] T,D,C=embedding(title,desc,code) (1)
[0070] distribution=TagRec(T,D,C) (2)
[0071] like Figure 2As shown, the three proposed feature extractors have similar structures. Taking the title feature extractor as an example, a multi-kernel convolutional neural network is used to mine deep features carried in the title sequence from multiple angles. Pooling technology is used to sample the complex high-dimensional feature matrix and concatenate multiple feature segments to obtain a concise title feature vector rich in feature information.
[0072] For the title matrix T, as shown in formula (3), the convolution operation is performed between each position and p randomly initialized convolution kernels, and the results are combined into a feature matrix, denoted as . Where i is the convolution kernel with the i-th stride, and |stride| refers to the number of stride categories;
[0073]
[0074] Among them, Conv i This is the i-th convolution operation, and no padding is applied to the edge data. Next, we will... Max pooling is performed along the sequence dimension, as shown in Equation (4), to obtain the feature vector after max pooling of the i-th convolutional kernel.
[0075]
[0076] The title extractor finally concatenates the feature vectors obtained from max pooling of convolutional kernels of different lengths along the word vector dimension using a concatenation operation to obtain the title feature vector. As shown in the following formula:
[0077]
[0078] in, It is a join operation.
[0079] The feature vector of the problem description is obtained using the same method as that used to obtain the feature vector of the title. and the feature vector of the problem code
[0080] By splicing FV T FV D and FV C This yields the representation vector of the entire problem.
[0081]
[0082] The FV is modeled using a ReLU-based nonlinear transformation to obtain the hidden layer vector. Finally, Softmax is used to map the hidden layer vectors to the label distribution space, obtaining the confidence probability of each question label for that question. As shown in (7)(8):
[0083] hidden = Rule(FV·W) H +b H (7)
[0084] distribution=Sigmoid(hidden·W D +b D (8)
[0085] in All of them are trainable parameters.
[0086] Specifically, the steps for calculating the confidence probabilities of the four categories—positive, neural+, neural-, and negative—in step S2 are as follows:
[0087] Reuse the pre-trained TagRec, freeze the parameters of the pre-trained Title Extractor, DescExtractor, and Code Extractor, and use the tag information contained therein to guide the representation of subsequent questions and answers. For example... Figure 3 As shown, Title Extractor, Desc Extractor, and Code Extractor are used to extract title features, description features, and code features respectively, and these three are concatenated to form the representation vector of the question. For the answer, since the answer does not contain title information, Desc Extractor and Code Extractor are used to extract description features and code features respectively, and these two are concatenated to form the representation vector of the answer. It should be noted that if the question or answer does not contain a code segment, then the code portion will be padded with a zero vector of equal length. Then, the output is mapped to the classification space through multiple concatenation and fully connected linear transformation operations.
[0088] The inputs to the pre-trained TagRec are: question title, question description, question code, answer description, and answer code. These are then encoded into dense matrix representations, which are input into their respective feature extractors and concatenated to obtain the feature vector representation of the question title. Describe the feature vector representation Problem code feature vector representation Answer description: Feature vector representation and answer code feature vector representation The problem feature vector FV is obtained by concatenating the features of each feature vector. q and the answer feature vector FV a , As shown in formulas (9)-(15):
[0089]
[0090]
[0091]
[0092]
[0093]
[0094]
[0095]
[0096] To map answers to the question-and-answer category space, FV q and FV a By connecting and using a multilayer perceptron to mine deep features and their correlations, a new hidden layer vector is obtained. Finally, Softmax is used to map hidden′ to the label distribution space to obtain the confidence probability of each question label for that question, as shown in equations (16)-(18):
[0097]
[0098] hidden′=Rule(FV·W H +b H (17)
[0099] distribution′=Sigmoid(hidden′·W D +b D (18)
[0100] in These are trainable parameters;
[0101] Use formula (18) to calculate the confidence probabilities of the four categories: positive, neural+, neural-, and negative.
[0102] The calculation process of formula (18) is actually a very common linear layer calculation process in deep learning. For this kind of classification task, the linear layer is often used as the last layer of the neural network to calculate the probability distribution. By setting its output dimension, the dimension of its output result can be controlled. In this model, because it is necessary to calculate the confidence probability of four categories, its output dimension is set to 4.
[0103] Example 1: This example 1 statistically analyzed 873,918 questions and answers tagged as Java on Stack Overflow up to July 2021, and 1,767,134 answers. The proportion of questions and answers containing code blocks was calculated. As shown in Table 1, the number of questions containing code blocks is approximately three times that of answers without code blocks, and the number of answers containing code blocks is approximately 1.3 times that of answers without code blocks, indicating that code blocks frequently appear in questions and answers. In fact, the code block content in Stack Overflow questions and answers is not entirely code text; it also includes non-code text such as configuration information and stack traces. To further investigate the content present in the code blocks, 200 randomly selected post samples were manually categorized, as shown in Table 2. They were divided into 8 categories. It should be noted that each post may contain code blocks from multiple categories. Here, "several lines of code," "a function," and "a class" are considered code, while other categories are considered non-code. It can be seen that 85% of the samples contain code information, with "several lines of code" being the most numerous, accounting for 40%. For Java, compilable code accounts for 34% (at the class level). Overall, Stack Overflow's code has poor compilability and usually exists in the form of code snippets. The above data shows that code information does exist in most questions and answers, but due to its incompleteness, it cannot effectively utilize the characteristics of code, such as abstract syntax trees, data dependencies, and control relationships. Therefore, in this embodiment of the invention, code is processed as text. Non-code categories such as "configuration," "XML format," "output results," "error messages," and "others" are also processed as text. Since both code and non-code are processed as text, no further distinction is made; they are all considered code snippets and processed using the same embedding method.
[0104] Table 1. Distribution of samples containing code blocks in the Java dataset.
[0105]
[0106] Table 2: Category Distribution of Code Blocks
[0107]
[0108] Experimental setup
[0109] 1. Dataset
[0110] The experimental dataset used was the Stack Exchange dataset up to July 2021. This invention experiment used the SAX streaming XML parsing method to extract a total of 873,918 questions marked 'java' from a 7z compressed file of over 16GB called Posts, with a total of 1,767,134 corresponding answers. For ease of processing, the original XML data was stored in a relational database using a MySQL database, and an index was created based on PostId and AcceptedAnswerId (question only). In addition, to increase retrieval speed, Lucene was used to create an index on the question titles.
[0111] The extracted questions include: ID, Title, Body, Tags, and AcceptedAnswerId. During data preprocessing, the title and body undergo text segmentation, deduplication, stop word removal, and HTML block removal. Since the body contains text and code segments, regular expressions are needed to match the extracted text during data processing. <pre> <code>…< / code> < / pre> The pattern extracts the code segment (Code) from the main body, and the remaining text segment is used as the description (Desc). For the answer to the question, the identifier (Id), body, and dependency identifier (ParentId) are extracted. The operation on the body is similar to that on the question, extracting the description and code.
[0112] For the tag recommendation pre-training phase, the experiment extracted four attributes from the question set: title, body, code, and tags. Dictionaries were built for the title, body, and code, removing words with a frequency less than 50 and tags with a usage frequency less than 50. During the training and testing phase, 10,000 samples from the question set were used as the test set, and the remainder as the training set.
[0113] For the fine-tuning phase of question-answering classification, the experiment sampled 10,000 questions as test questions, and the rest were used for training the question-answering classification task. The classification categories included positive, neural+, neural-, and negative. Based on the description by Gu et al., the experiment used accepted answers to questions as positive, answers belonging to the question but not accepted as neural+, accepted answers to similar questions as neural-, and irrelevant answers as negative. For each question, four question-answering classification samples were generated, one for each category, for a total of 3,463,868 data points.
[0114] 2. Benchmark Method
[0115] To compare the effectiveness of PT-CARec in answer recommendation, the experiment used DeepAns and Post2vec as benchmark methods:
[0116] 3 Evaluation Indicators
[0117] To evaluate the effectiveness of PT-CARec in answer recommendation, we continue to use the evaluation metrics from previous related work: the precision (Precision@k) of the top k recommended answers and the Discounted Cumulative Gain (DCG@k) of the top k recommended answers. Given a list A of recommended answers of length m (m>k). m ={a1,a2,…,a m}, define index truth The position of the ground truth in the answer list (1≤index) truth If ≤m), then Precision@k represents the proportion of test samples containing true values among the first k recommendations, as shown in the following formula:
[0118]
[0119]
[0120] DCG@k measures recommender system performance based on the hierarchical relevance of the truth and its position in the candidate set. Unlike Precision@k, the earlier the truth appears in the recommendation list, the greater its impact on DCG@k. DCG@k is defined as follows:
[0121]
[0122]
[0123] 4 Experimental Setup
[0124] The parameter settings in the experiment have a significant impact on the results. Here, we present the complete parameter settings according to the experimental procedure. First, the title, description, and code text elements are input into the deep learning model (i.e., the three feature extractors). The sequence lengths of these three elements are corrected to 20, 300, and 800 respectively using a tail-truncation method. Placeholders are used for elements with insufficient length.<PAD_TOKEN> Complete the missing information; secondly, the word vector dimension of each word is 128.
[0125] For the pre-trained model of tag recommendation, the experiment used one-dimensional convolution. Three types of convolutional kernels of size 1, 2, and 3 were used for the title, three types of convolutional kernels of size 1, 2, and 3 were used for the description, and three types of convolutional kernels of size 2, 3, and 4 were used for the code. The number of kernels in each type was 100. The hidden layer dimension of tag recommendation was set to 900. The batch size during training was 128, the number of iterations was 20, and the learning rate of the Adam optimizer was 0.00005. The pre-trained model of tag recommendation followed the existing tag recommendation work and selected cross-entropy as the loss function of the model, as shown in Equation (23), so that the confidence probability of each tag approaches the existence of the actual tag.
[0126] For the question-and-answer classification fine-tuning stage, the feature extractor for tag recommendation was reused and its parameters were frozen. The dimensions of the subsequent question feature vector and answer feature vector were set to 1000, and the dimension of the question-and-answer representation hidden layer vector was set to 500. The batch size during training was 256, the number of iterations was 100, and the learning rate of the Adam optimizer was 0.0001. The code enhancement question-and-answer classification model [i.e., the question-and-answer classification fine-tuning stage] also uses cross-entropy as the model's loss function, as shown in Equation (23), which makes the probability distribution of the predicted question-and-answer categories closer to the real categories.
[0127]
[0128] In the recommendation stage, the top k questions out of the 10,000 retrieved questions were filtered by word vectors, and k was set to 5 in the experiment. For the weighted summation step of calculating the final scores of questions and answers, the experiment used fixed parameters to aggregate the confidence probabilities of the four categories, as shown in Equation (24).
[0129] Score = 2 * p positive +p nerual+ -p nerual- -2*p negative (twenty four)
[0130] Where, p positive p nerual+ p nerual- p negative These represent the confidence probabilities of positive, neural+, neural-, and negative for the current question-answer pair, respectively.
[0131] 5. Experimental Results and Analysis of PT-CARec
[0132] (1) Recommended performance analysis of PT-CARec
[0133] This experiment aims to analyze the performance of PT-CARec on the answer recommendation task and compare it with benchmark models. PT-CARec fully leverages the programming expertise of Stack Overflow, incorporating tags and code as two important elements into the answer recommendation model. It combines the advantages of a pre-trained tag recommendation model and code text features. Guided by tag features, it generates effective feature vectors for titles, descriptions, and code for answer classification. Furthermore, it uses a weighted summation method to calculate scores and re-rank the answers. Since answer recommendation is a subjective task, the experiment presents two approaches for extracting the answer list and determining the truth value: ① Extracting the accepted answers (A) from the question as a candidate answer list, and using the accepted answers for the test question as the truth value; ② Extracting all answers to the question (Complete answer, C), and using the corresponding answer for any test question as the truth value. The effectiveness of answer recommendation is analyzed from rigorous and lenient perspectives, denoted as Answer. accepted and Answer complete Unless otherwise specified, Option ① is the default recommended answer.
[0134] The experimental results are shown in Table 3. PT-CARec showed excellent results in all indicators under different schemes. Except for Pre@4 under scheme ② which was slightly lower than Post2vec, the other indicators significantly outperformed the benchmark method. Specifically, for scheme ①, PT-CARec improved performance compared to DeepAns by 25.55%, 12.16%, 7.56%, 3.65%, 25.55%, 15.7%, 12.37%, and 9.74% on Pre@1, Pre@2, Pre@3, Pre@4, DCG@1, DCG@2, DCG@3, and DCG@4, respectively. Compared to Post2vec, it improved performance by 56.83%, 36.25%, 22.68%, 10.5%, 56.83%, 41.58%, 31.9%, and 23.41%, respectively. This indicates that if answer recommendation is only performed on high-quality answers (acceptable answers), PT-CARec can significantly outperform the benchmark method. For scheme ②, PT- Compared to DeepAns, CARec improved performance by 53.78%, 32.3%, 22.25%, 16.78%, 53.78%, 38.26%, 30.95%, and 27.08% on Pre@1, Pre@2, Pre@3, Pre@4, DCG@1, DCG@2, DCG@3, and DCG@4, respectively. Compared to Post2vec, CARec improved performance by 50.76%, 23.32%, 8.48%, -2.45%, 50.76%, 30.6%, 19.61%, and 11.66%, respectively. This indicates that the model's performance did not change significantly under conditions of poor-quality, noisy answer lists and relatively lenient truth selection. PT-CARec, in particular, maintained a high recommendation performance.
[0135] Experiments show that PT-CARec outperforms the baseline models DeepAns and Post2vec in answer recommendation, regardless of the quality of the answer.
[0136] Table 3 in Answer accepted With Answer complete A performance comparison of accuracy and cumulative gain loss for the first 1, 2, 3, and 4 terms of DeepAns, Post2vec, and PT-CARec under two schemes. The highest value for each evaluation result is shown in bold.
[0137]
[0138] (2) Impact analysis of Lucene retrieval on the performance and efficiency of PT-CARec recommendations
[0139] This experiment aims to investigate the impact of introducing a search engine retrieval mechanism on the performance and efficiency of subsequent answer recommendation. Among the three steps of similar question retrieval, candidate answer selection, and answer re-ranking, similar question retrieval generally consumes the most time and directly affects the quality of answer recommendations. For similar question retrieval of new questions, the experiment designed two schemes: ① Introducing the Lucene retrieval mechanism, using an inverted index to retrieve the question titles, selecting the top 10,000 questions as candidates, and then using the average word vector of each word in the question title as the title vector. The cosine similarity between the title vectors of the new question and the candidate questions is calculated, and the top k are selected as similar questions; ② Directly calculating the cosine similarity between the new question and all questions in the question database, and selecting the top k as similar questions. Both schemes optimize the title vector calculation process through a caching mechanism. These are denoted as Retrival. lucene and Retrival all Unless otherwise specified, the default solution for similar question retrieval is solution ①.
[0140] As shown in Table 4, PT-CARec in Retrival lucene The solution significantly improves efficiency, reducing the retrieval time per question from 44 seconds to 1 second, greatly increasing the feasibility of the overall answer recommendation. Simultaneously, in terms of answer recommendation performance, Retrival... lucene Compared to Retrival all The performance of Pre@1, Pre@2, Pre@3, Pre@4, DCG@1, DCG@2, DCG@3, and DCG@4 increased by 2.4%, 2.8%, 3.12%, 3.15%, 2.4%, 2.72%, 2.91%, and 2.92% respectively, indicating that Retrival... all Searching across the entire dataset introduces more noise, which slightly reduces the effectiveness of recommendations.
[0141] Experiments show that Retrival lucene The introduction of a search engine can shorten the answer recommendation time, reduce the introduction of noisy data, and improve the answer recommendation performance.
[0142] Table 4 in Retrival lucene and Retrival all Comparison of accuracy and cumulative gain performance and efficiency of the first 1, 2, 3, and 4 terms of PT-CARec under the two schemes.
[0143]
[0144] (3) Analysis of the impact of code features and pre-trained models on PT-CARec performance
[0145] The experiment aims to analyze the impact of code features and pre-trained models on overall recommendation performance in PT-CARec. To this end, two variants of PT-CARec were designed: ① PT-ARec (Pre-training Answer Recommedation), which removes code enhancement by removing code blocks from the question and answer body, utilizing only the question title, description, and question description for the experiment; ② CARec (Code Enhanced Answer Recommedation), which does not use a pre-trained model, allowing the entire model parameters to be optimized by the answer classification task. For ease of distinction, both methods are renamed AnswerRec. no_code and AnswerRec no_pretrain In addition, the experiment also explored the impact of including and not including code on answer recommendations.
[0146] As shown in Table 5, removing code features generally leads to a decrease in performance. Pre@1, Pre@2, Pre@3, Pre@4, DCG@1, DCG@2, DCG@3, and DCG@4 show decreases of 15.32%, 9.04%, 5.48%, 2.27%, 15.32%, 10.84%, 8.51%, and 6.59%, respectively. Removing the pre-trained model further reduces the performance of Pre@1, Pre@2, DCG@1, DCG@2, DCG@3, and DCG@4 by 10.1%, 1.41%, 10.1%, 3.9%, 2.07%, and 2.32%, respectively. However, Pre@3 and Pre@4 show performance improvements of 1.22% and 0.44%, respectively. Compared to the decrease in performance metrics, these two performance improvements are almost negligible. Experimental results indicate that code features and pre-trained models play a significant role in improving the recommendation performance of PT-CARec.
[0147] Table 5 PT-CARec, AnswerRec no_code and AnswerRec no_pretrain Comparison of accuracy and cumulative gain performance of the first four items.
[0148]
[0149] To further explore the impact of code features on answer recommendations for samples without code blocks, the test set was divided into sets containing code and sets without code. A comparison was made with DeepAns and AnswerRec. no_code And the performance of PT-CARec in two scenarios, including DeepAns and AnswerRec. no_code Neither of them used code features.
[0150] The experimental results are shown in Table 6. For the test set containing code, PT-CARec outperforms DeepAns and AnserRec. no_code It exhibits significant advantages. For DeepAns, PT-CARec outperformed by 62.67%, 28.77%, 14.76%, 6.09%, 62.67%, 37.53%, 27.27%, and 21.07% in Pre@1, Pre@2, Pre@3, Pre@4, DCG@1, DCG@2, DCG@3, and DCG@4, respectively; for AnswerRec... no_code PT-CARec outperformed Pre@1, Pre@2, Pre@3, Pre@4, DCG@1, DCG@2, DCG@3, and DCG@4 by 47.35%, 24.03%, 11.68%, 4.43%, 47.35%, 30.33%, 21.69%, and 16.71%, respectively. For test sets without code, PT-CARec's performance was slightly worse, but the decrease was not significant. For DeepAns, PT-CARec decreased by 6.31%, 3.06%, 6.31%, 3.95%, 1.55%, and 1.09% on Pre@1, Pre@2, DCG@1, DCG@2, DCG@3, and DCG@4, respectively, but slightly outperformed Pre@3 and Pre@4, by 0.6% and 1.19%, respectively. For AnswerRec... no_code PT-CARec showed reductions of 8.88%, 3.44%, 8.88%, 4.92%, 2.63%, and 2.36% in Pre@1, Pre@2, DCG@1, DCG@2, DCG@3, and DCG@4, respectively. It maintained the same performance for Pre@3, while exceeding the performance of Pre@4 by 0.18%. In conclusion, PT-CARec significantly outperforms DeepAns and AnswerRec in recommending answers to coded samples. no_code The performance was slightly worse for samples without code, but the decline was within an acceptable range.
[0151] Table 6 shows the results of DeepAns and AnswerRec tests in subsets of the test set that include and do not contain code. no_code Performance comparison of accuracy and cumulative gain of the first four terms of PT-CARec.
[0152]
[0153] (4) Analysis of the impact of answer list length on PT-CARec performance
[0154] Generally, a longer answer list can incorporate more information. To investigate the impact of answer list length on PT-CARec performance, the experiment modified the k value during similar question retrieval to change the answer list length, using 5, 6, 7, 8, 9, and 10 as candidate data for comparison. As discussed earlier, the answer selection scheme also affects the answer list length; factors already discussed will not be repeated here. As shown in Table 7, as k increases, all indicators show a downward trend. Therefore, the experiment uses 5 as the optimal number of similar questions to retrieve.
[0155] Table 7 shows the impact of the k-value for similar question retrieval on the precision and cumulative gain of the first 1, 2, 3, and 4 items in PT-CARec.
[0156]
[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for recommending answers in a programming question-and-answer community using pre-trained code enhancement, characterized in that: Includes the following steps: S1: Tag recommendation pre-training stage, build recommendation pre-training model TagRec. TagRec includes three feature extractors: Title Extractor, Desc Extractor and Code Extractor. TagRec takes the question title, question description, and question code as input, and the question label as the truth value. The output of TagRec is the confidence probability of each question label for that question. , That is, for each problem, the cross-entropy loss function is used to calculate the cross-entropy between the predicted value and the true value, and the parameters of TagRec are updated according to the value of the loss function. When the set number of iterations is reached, the pre-trained TagRec is obtained. In step S1, the confidence probability of each question label for that question is calculated. The steps are as follows: Given a question title containing M words, a question description containing N words, and a question code containing K words, let... yes , and Let i be the i-th word in the array, i = 1, 2, ..., M+N+K. Record the number of each word. Using random initialization techniques, convert each word into a discrete vector representation. , , Mapped to a high-dimensional matrix space, denoted as As input to TagRec, T represents the title matrix, D represents the question description matrix, C represents the question code matrix, and d is the encoding dimension of the word vectors; After using TagRec, we obtain the confidence probability of each label in the current space. , The dimension is The set of real numbers; For the title matrix As shown in formula (3), calculate the relationship between each position and... Convolution operations are performed on 1 randomly initialized convolution kernel, and the results are combined into a feature matrix, denoted as . ,in It is a convolution kernel with the i-th stride. This refers to the number of step size categories; (3) in, This is the i-th convolution operation, and no padding was applied to the edge data; the next operation is... Max pooling is performed along the sequence dimension, as shown in Equation (4), to obtain the feature vector after max pooling of the i-th convolutional kernel. ; (4) The title extractor finally concatenates the feature vectors obtained from max pooling of convolutional kernels of different lengths along the word vector dimension using a concatenation operation to obtain the title feature vector. As shown in the following formula: (5) in, It is a join operation; The feature vector of the problem description is obtained using the same method as that used to obtain the feature vector of the title. and the feature vector of the problem code ; By splicing , and This yields the representation vector of the entire problem. ; (6) Using a ReLU-based nonlinear transformation pair Modeling is performed to obtain hidden layer vectors. Finally, Softmax is used to map the hidden layer vectors to the label distribution space to obtain the confidence probability of each question label for that question. As shown in (7)(8): (7) (8) in All of them are trainable parameters; S2: In the question-answering classification fine-tuning stage, the pre-trained TagRec is reused. The parameters of the pre-trained TitleExtractor, Desc Extractor, and Code Extractor are frozen. The question title, question description, question code, answer description, and answer code are used as input samples of the pre-trained TagRec. The label of each input sample is used as the true value. The output of the pre-trained TagRec is the confidence probability of four categories: positive, neural+, neural-, and negative. The confidence probabilities of the four categories are used as the predicted values of the input samples. The cross-entropy loss function is used to calculate the cross-entropy between the predicted values and the true values of the input samples. The parameters of the pre-trained TagRec are updated according to the value of the loss function. When the set number of iterations is reached, the trained TagRec is obtained. S3: In the recommendation stage, for a new question, first use a search engine to query similar question titles and select the top Q as candidate questions. Next, embed words in the candidate question titles using pre-trained Word2vec word vectors and use the average vector of the words as a temporary representation of the question title. Calculate the cosine similarity between the candidate question title vector and the new question title vector, sort and select the top k questions as the filtered candidate questions, where k is a hyperparameter. Then extract the answers corresponding to the k questions as candidate answers. Using the answers from the candidate answers and the new question as input to the trained TagRec, the confidence probabilities of the new question and each candidate answer in four categories (positive, neural+, neural-, and negative) are obtained. The matching score between the new question and each candidate answer is calculated according to formula (24). After sorting the matching scores from largest to smallest, the candidate answers corresponding to the top Q' matching scores are used as the final recommended answers. (24)。 2. The method for recommending answers in a programming question-and-answer community with pre-trained code enhancement as described in claim 1, characterized in that: The S2 adjustment process is as follows: The specific steps for calculating the confidence probabilities of the four categories (positive, neural+, neural-, and negative) in S2 are as follows: The inputs to the pre-trained TagRec are: question title, question description, question code, answer description, and answer code. These are then encoded into dense matrix representations, which are input into their respective feature extractors and concatenated to obtain the feature vector representation of the question title. Describing the feature vector representation Problem code feature vector representation Answer description: Feature vector representation and answer code feature vector representation The feature vectors of the problem are obtained by connecting them respectively. and the answer feature vector , As shown in formulas (9)-(15): (9) (10) (11) (12) (13) (14) (15) Will and By connecting and using a multilayer perceptron to mine deep features and their correlations, a new hidden layer vector is obtained. Finally, Softmax is used to... Mapping to the label distribution space, we obtain the confidence probability of each question label for that question, as shown in equations (16)-(18): (16) (17) (18) in These are trainable parameters; Use formula (18) to calculate the confidence probabilities of the four categories: positive, neural+, neural-, and negative.
Citation Information
Patent Citations
Recommendation method for technical labels in software question and answer community
CN107798624A
Providing explainable product recommendation in a session
WO2021066903A1
