Collaborative learning-based annotation code library purification method and device
Through the annotation code base purification method based on collaborative learning, combined with wavelet time-frequency transformation and self-attention variation autoencoder model, the noise processing and global information capture problems in software engineering data quality management are solved, and the code search is improved Efficiency and effect achieve effective verification of different programming languages.
Patent Information
- Application Number
- CN202411058661.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-19
AI Technical Summary
Existing software engineering data quality management technologies have shortcomings in noise data processing and code search, resulting in model performance deviations and low query quality. In particular, GRU neural networks are slow to execute and difficult to capture global information, and lack of different programming languages. extensive verification.
Using annotation code base purification method based on collaborative learning, combining unsupervised driver components and supervised filter components, using wavelet time-frequency transformation fusion algorithm and self-attention variational autoencoder model to build a high-quality guided query corpus and non- Supervised filtering comments - code pairs, initially filtered and refined through manually defined heuristic rules and syntax rules.
It improves the efficiency and effectiveness of code search tasks, enhances the performance of the model in handling long-distance dependence and global semantic associations, ensures the quality of annotations and the proximity of natural language queries, and improves the verification capabilities of different programming languages.
Smart Images

Figure CN118981331A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image data processing, and in particular to a method and device for purifying an annotation code library based on collaborative learning. Background Art
[0002] With the continuous development of software engineering, many open source communities for software developers have emerged on the Internet. These open source communities cover huge code bases, including thousands of comments, queries, and accompanying codes, and can use search engines to find code solutions related to the problems they are solving. It is becoming more and more popular to use these code bases to build the next generation of intelligent pre-trained language models (PLMs) and large language models (LLMs). These models can be trained based on code datasets in specific fields to generate new code solutions. Therefore, deep learning (DL) models, especially large language models (LLMs), are widely used in the field of intelligent software engineering due to their advantages in semantic modeling and understanding, and many related studies have emerged one after another, including retrieving, generating, and annotating code snippets through natural language (NL) descriptions.
[0003] However, existing data quality management techniques for software engineering data are still understudied: 1. Previous work mainly focused on analyzing semantic or grammatical data to eliminate noise. However, using noisy data benchmarks to evaluate models is difficult to reflect their performance in real applications and may introduce significant biases in model design, evaluation, and application; 2. Second, existing code search datasets, mainly derived from GitHub's CodeSearchNet (CSN) dataset, usually use alternative methods for queries, such as using code comments to replace queries. Although the CSN corpus has been filtered, it still contains noise and unnatural elements. 3. Directly using annotation-based datasets for search tasks may be problematic because there are differences between annotations and natural language queries; 4. The GRU neural network architecture used results in slow execution and sequential processing, making it difficult to capture global information, especially when processing longer sequences; 5. The guided query corpus collected from StackOverflow is randomly sampled, which may result in low query quality and a tendency to sample low-quality annotations; 6. There is a lack of extensive validation for different programming languages, and the evaluation is limited to Java datasets. Therefore, there is still room for further improvement in improving the efficiency and effectiveness of NLQF. Summary of the invention
[0004] In order to solve the above problems, the present invention proposes a method for purifying annotated code bases based on collaborative learning. Specifically, the unsupervised driving component combines the WTFF algorithm to extract high-quality posts to build a guided query corpus, and uses manually defined heuristic rules to perform preliminary filtering on the original comment-code pairs. The supervised filtering component introduces a semantic filter based on the self-attention variational autoencoder model, which takes the preliminarily cleaned comment-code pairs from the unsupervised driving component as input, and further selects the comment-code pairs that are semantically close to the guided query corpus.
[0005] A method for purifying an annotation code library based on collaborative learning, comprising the following steps:
[0006] S1. Collect query-code matching pairs of Java and Python programming language types through software question-answering websites to form a guided query corpus. The specific steps are as follows:
[0007] S1.1. Use software question-answering websites to crawl query-code matching pairs containing Python and Java programming language types;
[0008] S1.2, extracting the social attribute values according to the query-code matching pairs, constructing a social attribute feature matrix, and filling missing values and normalizing all features;
[0009] S1.3, using the wavelet time-frequency transform fusion algorithm to score the corpus, selecting the top 10 corpora as high-quality query-code pairs according to the score ranking, and obtaining a set of query results starting with "how to do" through the query classifier, which are used to construct the guided query corpus;
[0010] The wavelet time-frequency transform fusion algorithm specifically includes the following steps:
[0011] S1.3.1, the feature matrix of the query-code matching pair corpus is regarded as a signal, and converted into a wavelet tree using wavelet transform;
[0012] S1.3.2, based on the wavelet tree, by fusing the high-frequency and low-frequency vectors, new high-frequency and low-frequency coefficient vectors are generated; these coefficient vectors are estimated by the expectation maximization algorithm to obtain feature fusion weights;
[0013] S1.3.3, using inverse wavelet transform, reconstruct a new wavelet tree, where the top leaf node represents the score of the post;
[0014] S1.3.4, sort the posts according to the scores and select the top 10 as high-quality query-code pairs;
[0015] S1.3.5. The queries obtained can be divided into four types: “Debug / Corrective”, “Need-To-Know”, “How-To-Do” and “Seeking-Different-Solution”. Among these, the query of “How-To-Do” is the most relevant for code search task verification.
[0016] S2, obtain the original annotation-code pairs of Java and Python language types from the open source code base to complete unsupervised filtering;
[0017] The wavelet time-frequency transform fusion algorithm WTFF is based on the feature matrix of the social attribute values in the query-code matching pair corpus, and uses the wavelet time-frequency transform algorithm to convert it into a wavelet tree, and generates new high-frequency and low-frequency coefficient vectors by fusing high-frequency and low-frequency vectors. Secondly, the feature fusion weight is obtained by the expectation maximization EM algorithm, and then a new wavelet tree is reconstructed by the inverse wavelet transform operation, in which the top leaf node represents the score of the post, so as to fuse the social attribute values of all query-code matching pair corpora to obtain the quality score of each corpus;
[0018] To identify the most qualified similar queries, the following criteria are required:
[0019] 1. Labeled as 1 by the linear regression “how to do” classifier;
[0020] 2. Start with "how to" and their tags contain the "Java" or "Python" identifier;
[0021] 3. Use manually defined syntactic rules to exclude interrogative sentence rules;
[0022] Finally, 168,780 and 175,410 natural language queries (without pairing codes) were retained from the mined high-quality Java and Python query-code pairs, respectively. Subsequently, they were transformed into declarative sentences by removing the initial “how to” and any question marks to form the guided query corpus.
[0023] As shown in Table 1, these are examples of high-quality guided queries based on Python and Java. The original samples in these examples contain the identifier "python" or "java" label, as well as their corresponding guided queries, excluding strings starting with "how to";
[0024] As shown in Table 1, these are examples of high-quality guided queries based on Python and Java. The original samples in these examples contain the identifier "python" or "java" label, as well as their corresponding guided queries, excluding strings starting with "how to";
[0025] Table 1 Examples of high-quality guided queries based on Python and Java
[0026]
[0027] S2. Obtain the original annotation-code pairs of Java and Python language types from the open source code base to complete unsupervised filtering. The specific steps are as follows:
[0028] S2.1. Delete all comments in Python and Java codes, and pair the first sentence of each comment with the corresponding code to form the original comment-code pair.
[0029] The present invention chooses to use the widely open source CodeSearchNet (CSN) as the experimental dataset, which is collected from GitHub; the CSN dataset contains 2 million data points, which include code, comments and document strings extracted from open source libraries; specifically, comments correspond to the description of top-level functions or methods, while code represents the entire function or method; CSN covers six programming languages, and the present invention focuses on the Java and Python fields, extracts the first sentence of the comment and pairs it with the code, deletes all comments in the code, and generates 406,922 and 454,451 comment-code training datasets respectively, forming the original comment-code pairs;
[0030] S2.2. Establish a set of grammatical rules to detect and process the obtained comment-code pairs, remove the comments whose syntax deviates from the semantic description of the code, and obtain the unsupervised filtered comment-code pairs, wherein the grammatical rules include: URLs, HTML tags, short sentences, question sentences, non-English content, program tags, brackets;
[0031] The URLs may omit comments containing the URL (http: / / ...) to prevent additional explanation of the link to the outside world;
[0032] The HTML tags in the comments are removed while retaining the content to maintain the meaning of the code;
[0033] The short sentence is used to remove comments with less than three words, because short comments may lack sufficient context;
[0034] Said interrogative sentences are used to ignore comments ending in a question mark, as declarative sentences usually provide clearer insights;
[0035] The non-English content is used to filter out non-English comments to ensure the language consistency of the natural language description;
[0036] The program tags are used to remove tags that are irrelevant to the specific details of the language and focus on the semantic content of the annotation;
[0037] The brackets are used to filter out comments containing brackets. The brackets are usually supplementary and not the key part of describing the code;
[0038] S3, loading the guided query corpus into a self-attention-based variational autoencoder model for training and fitting to obtain a model file, and further sampling the unsupervised filtered comment-code pairs according to the model file to refine and purify the comment-code pairs;
[0039] The variational autoencoder model is constructed based on the Transformer network, and the specific construction includes the following:
[0040] Construct a pre-trained word embedding module. The specific steps are as follows:
[0041] Given the i-th annotation in is the jth token, |Q| is the maximum length of the comment token, and provides two types of initialization representations for the vocabulary dictionary: one is a randomly initialized embedding that follows a uniform distribution in the range of [-0.1, 0.1]; the other is a pre-trained stru2vec embedding based on a large-scale code corpus to obtain the embedding vector of each token. The specific expression is as follows:
[0042]
[0043] Where E∈R d×|V| is a randomly initialized word embedding or a pre-trained stru2vec embedding matrix with dimension d, V is the vocabulary size of all annotations, Embedding vector representing each token;
[0044] Build the Transformer attention mechanism encoder (Encoder);
[0045] For the encoder of TVAE, the present invention introduces a self-attention mechanism to enhance its representation learning ability. Specifically, this layer performs attention calculation on the embedded vector for each input x i , and obtain an output sequence Multi-head attention enables the model to consider the relationship between input tokens at the same time, which helps to better capture contextual information; by focusing on different parts of the input sequence, the model can more effectively capture long-distance dependencies during encoding. The specific expression is as follows:
[0046]
[0047] In the formula, Embedding vector representing each token;
[0048] Next, the output sequence is processed by global average pooling to obtain the overall context representation v xi , which is then processed through a fully connected layer to estimate the parameters of the Gaussian distribution, namely the mean vector μ and the variance vector σ 2 , the latent variable z is randomly sampled from this distribution, and the specific expressions are as follows:
[0049]
[0050] Where: w represents the weight matrix of the fully connected layer, b represents the bias vector of the fully connected layer, r is a random vector from the standard normal distribution, and e represents a natural constant;
[0051] The structural changes are designed to improve the representation learning ability of the model, so as to better capture the semantic information of the input sequence; the attention mechanism enhances the model's ability to capture the relationship between tags, thereby improving the model's performance when the encoded information is used for downstream tasks;
[0052] Construct the Transformer attention mechanism decoder (Decoder), the specific steps are as follows:
[0053] For the decoder of TVAE, the decoder reconstructs the input based on the latent variables; during the decoding process, it uses the self-attention mechanism to reconstruct the input and focus on different parts of the input sequence; secondly, the attention output Through the fully connected layer and processed by the argmax function, the final reconstructed annotation result x is obtained. i ', the specific expressions are as follows:
[0054]
[0055] Where: z represents the latent variable obtained by the encoder;
[0056] Construct loss function;
[0057] The present invention uses cross entropy loss to measure the probability of reconstructing the original input. The specific expression of ELBO loss of TVAE is as follows:
[0058]
[0059] Where: n represents the number of samples of the original input, CEloss and D KL Respectively represent the cross entropy loss calculation and KL divergence calculation; D KL The calculation of is shown in the following formula:
[0060]
[0061] Where: μ i is the mean value of the i-th dimension of the latent variable z, σ i is the corresponding standard deviation, N represents the dimension in the latent variable;
[0062] The verification of the refined and purified annotation-code pairs specifically includes the following steps:
[0063] S3.1. Supervised training of the self-attention-based variational autoencoder model on the guided query corpus. Set the initialization parameters to train the parameters of the fitted model. Stop training after 200 iterations and save the model file.
[0064] S3.1.1. Load the guided query corpus into the self-attention based variational autoencoder model;
[0065] S3.1.2. In the model parameter initialization, the maximum length of the annotation is set to 20, the self-attention dimension is 128, the drop rate is 0.1, the number of multi-head attention heads is 1, the exponential decay rate of the first-order moment estimate of the control gradient in the Adam optimizer is 0.9, the exponential decay rate of the second-order moment estimate of the control gradient is 0.999, the decay rate of the learning rate is 0.95, the initial weight of the KL divergence is 0.05, and the starting number of steps for the KL divergence weight increase is 200, such as Figure 3 As shown, the maximum value of KL divergence weight is 0.1, and the rate of increase of KL divergence weight is 0.000002;
[0066] S3.1.3. According to the guided query corpus, the self-attention-based variational autoencoder model is trained with a learning rate of 0.0001 and the model file is obtained;
[0067] S3.2. Based on the variational autoencoder model file obtained through supervised training, further sample the comment-code pairs after unsupervised filtering, remove the comments whose semantics deviate from the semantic description of the code, and obtain the comment-code pairs that are semantically close to the natural language query;
[0068] The trained variational autoencoder (TVAE) model can identify whether an annotation is semantically similar to a query in the corpus. Specifically, when the preliminarily filtered comment-code pairs are input into the TVAE model, their reconstruction loss is measured. The loss value is an anomaly score that measures the distance between the input and the query in the guided query corpus. Annotations with smaller loss values are semantically closer to the query. The annotations are sorted in ascending order according to their reconstruction losses, and the top-ranked annotations are retained. An unsupervised clustering algorithm EM-GMM is used to divide a group of annotations into qualified and unqualified groups based on the reconstruction loss. For each group, a Gaussian mixture model GMM fits a Gaussian probability density function and mixes them together as the distribution of the entire data set. Finally, the expectation maximization EM algorithm is applied to estimate the parameters of the Gaussian probability density of all qualified and unqualified groups, and the mean vector, covariance matrix, and mixing weights are gradually optimized until the parameter combination that maximizes the log-likelihood value, that is, the optimal value, is found.
[0069] S4, the refined and purified comment-code pairs obtained by using the code description embedding neural network for verification;
[0070] The refined comment-code pairs obtained by code description embedding neural network (CoDEnn) network verification are used to set the initialization hyperparameters to train the code search model (DeepCodeSearch). The training is stopped after 100 iterations, and the model file is saved. The model performance is evaluated on the validation set Python (CSNV_python, CosQA) and the validation set Java (CSNV_java, CosBench), and the corresponding MRR, Answered@k (k=10) and UD indicators are obtained.
[0071] In the DeepCS search model training, the maximum length of the query and code is set to 30 and 50 respectively, and the Adam optimizer hyperparameter is 10. -8 , according to the learning rate of 10 -4 To fit the model file of the DeepCodeSearch model;
[0072] To illustrate the experimental effect of the present invention on code search, the model performance is verified on the following three different manually annotated test sets based on the trained DeepCodeSearch model file, including:
[0073] (1) CSNV provides a search benchmark of query-code pairs, where the query is based on Bing search and programmers' function-related annotations, and the code snippets are from CodeSearchNet; 1559 Python and 434 Java related pairs are obtained;
[0074] (2) CodeSearch Question and Answering (CosQA) is a dataset containing code snippets extracted from GitHub functions and queries from Microsoft Bing search logs; it contains 20,604 tags. The present invention uses 500 active instances in the CosQA dataset, and queries are matched with different numbers of target codes;
[0075] (3) CodeSearch Bench (CosBench) is a dataset containing 1,000 Java projects, including 475,783 Java files and 4,199,769 code snippets (i.e., Java methods) collected from GitHub. In addition, CosBench also includes 52 natural language queries from StackOverflow, each of which is accompanied by 10 corresponding matching code snippets;
[0076] The corresponding MRR, Answered@k (k=10) and UD performance evaluation indicators are given and compared with the benchmarks CSNclean, NLQF, and NLQFembed to illustrate the performance of the data refinement and purification framework of the present invention;
[0077] In addition, in order to compare the impact of different filtering modules in this method on the overall performance, the experiment analyzes and quantitatively evaluates the impact of each filtering component on the model in refining the dataset and improving performance in subsequent tasks. To achieve this goal, the present invention conducts an ablation experiment, systematically disabling the filter components one by one, and then trains the model using the dataset generated by the disabled filters, and compares its performance with the baseline;
[0078] A collaborative learning-based annotation code library device, see the attached Figure 4 As shown, it includes: data collection module, guided query library collection module, artificial rule filtering module, self-attention based variational autoencoder model filtering module, feature matrix construction module, normalization processing module, scoring module, pre-trained word embedding module, self-attention based encoder module, self-attention based decoder module, and loss function module;
[0079] The query-code library saved from the data collection module is input into the guided query library collection module. Specifically, the query-code library is input into the feature matrix construction module to obtain the feature matrix, which is then normalized by the normalization processing module. Finally, it is scored and sorted by the scoring module to obtain the guided query corpus. The annotation-code library saved from the data collection module is input into the manual rule filtering module to obtain the preliminary filtered data, which is then input into the self-attention-based variational autoencoder filtering module to obtain the refined and purified data. Specifically, the guided query corpus obtained from the guided query library collection module is used to train the self-attention-based variational autoencoder, which includes a pre-trained word embedding module, a self-attention-based encoder module, a self-attention-based decoder module, and a loss function module.
[0080] The data collection module is used to obtain queries and codes in Python and Java languages, save them as query-code libraries, collect data in the same language, remove comments in the code, and associate the first sentence of the comment with the code, and save them as comment-code libraries;
[0081] The query-code library saved from the data collection module is input into the guided query library collection module. Specifically, the query-code library is input into the feature matrix construction module to obtain the feature matrix, and then the normalization processing module performs normalization processing, and finally the scoring module performs scoring sorting to obtain the guided query corpus; the annotation-code library saved from the data collection module is input into the manual rule filtering module to obtain the preliminary filtered data, and then input into the TVAE model filtering module to obtain the refined and purified data. Specifically, the guided query corpus obtained from the guided query library collection module is used to train the TVAE model, which includes: pre-trained word embedding module, self-attention-based encoder module, self-attention-based decoder module, and loss function module;
[0082] A guided query library collection module is used to extract high-quality queries from the query-code library to construct a query guided corpus, providing an effective data source for the subsequent training of the TVAE model. A guided corpus construction method based on the WTFF algorithm is adopted, including a feature matrix construction module, a normalization processing module, and a scoring module. The feature matrix construction module is used to obtain a social attribute feature matrix, the normalization processing module is used to fill missing values and normalize all features, and the scoring module is used to score the corpus.
[0083] The manual rule filtering module is used to remove the comments with grammatical anomalies in the comment-code base to obtain the preliminary filtered comment-code base, including URLs, HTML tags, short sentences, question sentences, non-English content, program tags, and brackets;
[0084] The TVAE model filtering module is used to remove semantically abnormal annotations in the annotation-code base after preliminary filtering, and obtain an annotation-code base that is semantically close to the natural query.
[0085] Beneficial Effects of the Invention
[0086] 1. The present invention adopts the WTFF algorithm and introduces a Transformer neural network with a self-attention mechanism to construct a semantic filter. The self-attention mechanism shows advantages in processing long-distance dependencies and global semantic associations. By considering all positions of the input sequence at the same time, the self-attention mechanism dynamically adjusts the attention weight of each word, flexibly captures complex relationships based on the context, and maintains high execution efficiency.
[0087] 2. When training the TVAE model, the present invention combines Python structure embeddings pre-trained on a large-scale corpus, so that the method can better understand the semantic relationship between annotations and enhance its performance in data filtering. BRIEF DESCRIPTION OF THE DRAWINGS
[0088] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0089] Figure 1 A flowchart of an implementation of a collaborative learning-based annotation code library purification method provided by the present invention;
[0090] Figure 2 It is the main framework diagram and detailed structure diagram of the model structure of the embodiment of the present invention;
[0091] Figure 3 A comparison chart of training loss values from 0 to 200 training rounds between the embodiment of the present invention and other methods;
[0092] Figure 4 It is a structural block diagram of the annotation code library purification system based on collaborative learning in an embodiment of the present invention. DETAILED DESCRIPTION
[0093] The present invention is further described in detail below in conjunction with specific embodiments.
[0094] Example 1
[0095] See also Figure 1-Figure 2 , a method for purifying annotated code base based on collaborative learning, comprising the following steps:
[0096] S1. Collect query-code matching pairs of Java and Python programming language types through software question-answering websites to form a guided query corpus. The specific steps are as follows:
[0097] S1.1. Use software question-answering websites to crawl query-code matching pairs containing Python and Java programming language types;
[0098] S1.2, extracting the social attribute values according to the query-code matching pairs, constructing a social attribute feature matrix, and filling missing values and normalizing all features;
[0099] S1.3, using the wavelet time-frequency transform fusion algorithm to score the corpus, selecting the top 10 corpora as high-quality query-code pairs according to the score ranking, and obtaining a set of query results starting with "how to do" through the query classifier, which are used to construct the guided query corpus;
[0100] The wavelet time-frequency transform fusion algorithm specifically includes the following steps:
[0101] S1.3.1, the feature matrix of the query-code matching pair corpus is regarded as a signal, and converted into a wavelet tree using wavelet transform;
[0102] S1.3.2, based on the wavelet tree, by fusing the high-frequency and low-frequency vectors, new high-frequency and low-frequency coefficient vectors are generated; these coefficient vectors are estimated by the expectation maximization algorithm to obtain feature fusion weights;
[0103] S1.3.3, using inverse wavelet transform, reconstruct a new wavelet tree, where the top leaf node represents the score of the post;
[0104] S1.3.4, sort the posts according to the scores and select the top 10 as high-quality query-code pairs;
[0105] S1.3.5. The queries obtained can be divided into four types: “Debug / Corrective”, “Need-To-Know”, “How-To-Do” and “Seeking-Different-Solution”. Among these, the query of “How-To-Do” is the most relevant for code search task verification.
[0106] S2, obtain the original annotation-code pairs of Java and Python language types from the open source code base to complete unsupervised filtering;
[0107] The wavelet time-frequency transform fusion algorithm WTFF is based on the feature matrix of the social attribute values in the query-code matching pair corpus, and uses the wavelet time-frequency transform algorithm to convert it into a wavelet tree, and generates new high-frequency and low-frequency coefficient vectors by fusing high-frequency and low-frequency vectors. Secondly, the feature fusion weight is obtained by the expectation maximization EM algorithm, and then a new wavelet tree is reconstructed by the inverse wavelet transform operation, in which the top leaf node represents the score of the post, so as to fuse the social attribute values of all query-code matching pair corpora to obtain the quality score of each corpus;
[0108] To identify the most qualified similar queries, the following criteria are required:
[0109] 1. Labeled as 1 by the linear regression “how to do” classifier;
[0110] 2. Start with "how to" and their tags contain the "Java" or "Python" identifier;
[0111] 3. Use manually defined syntactic rules to exclude interrogative sentence rules;
[0112] Finally, 168,780 and 175,410 natural language queries (without pairing codes) were retained from the mined high-quality Java and Python query-code pairs, respectively. Subsequently, they were transformed into declarative sentences by removing the initial “how to” and any question marks to form the guided query corpus.
[0113] As shown in Table 1, these are examples of high-quality guided queries based on Python and Java. The original samples in these examples contain the identifier "python" or "java" label, as well as their corresponding guided queries, excluding strings starting with "how to";
[0114] As shown in Table 1, these are examples of high-quality guided queries based on Python and Java. The original samples in these examples contain the identifier "python" or "java" label, as well as their corresponding guided queries, excluding strings starting with "how to";
[0115] Table 1 Examples of high-quality guided queries based on Python and Java
[0116]
[0117] S2. Obtain the original annotation-code pairs of Java and Python language types from the open source code base to complete unsupervised filtering. The specific steps are as follows:
[0118] S2.1. Delete all comments in Python and Java codes, and pair the first sentence of each comment with the corresponding code to form the original comment-code pair.
[0119] The present invention chooses to use the widely open source CodeSearchNet (CSN) as the experimental dataset, which is collected from GitHub; the CSN dataset contains 2 million data points, which include code, comments and document strings extracted from open source libraries; specifically, comments correspond to the description of top-level functions or methods, while code represents the entire function or method; CSN covers six programming languages, and the present invention focuses on the Java and Python fields, extracts the first sentence of the comment and pairs it with the code, deletes all comments in the code, and generates 406,922 and 454,451 comment-code training datasets respectively, forming the original comment-code pairs;
[0120] S2.2. Establish a set of grammatical rules to detect and process the obtained comment-code pairs, remove the comments whose syntax deviates from the semantic description of the code, and obtain the unsupervised filtered comment-code pairs, wherein the grammatical rules include: URLs, HTML tags, short sentences, question sentences, non-English content, program tags, brackets;
[0121] The URLs may omit comments containing the URL (http: / / ...) to prevent additional explanation of the link to the outside world;
[0122] The HTML tags in the comments are removed while retaining the content to maintain the meaning of the code;
[0123] The short sentence is used to remove comments with less than three words, because short comments may lack sufficient context;
[0124] Said interrogative sentences are used to ignore comments ending in a question mark, as declarative sentences usually provide clearer insights;
[0125] The non-English content is used to filter out non-English comments to ensure the language consistency of the natural language description;
[0126] The program tags are used to remove tags that are irrelevant to the specific details of the language and focus on the semantic content of the annotation;
[0127] The brackets are used to filter out comments containing brackets. The brackets are usually supplementary and not the key part of describing the code;
[0128] S3, loading the guided query corpus into a self-attention-based variational autoencoder model for training and fitting to obtain a model file, and further sampling the unsupervised filtered comment-code pairs according to the model file to refine and purify the comment-code pairs;
[0129] The variational autoencoder model is constructed based on the Transformer network, and the specific construction includes the following:
[0130] Construct a pre-trained word embedding module. The specific steps are as follows:
[0131] Given the i-th annotation in is the jth token, |Q| is the maximum length of the comment token, and provides two types of initialization representations for the vocabulary dictionary: one is a randomly initialized embedding that follows a uniform distribution in the range of [-0.1, 0.1]; the other is a pre-trained stru2vec embedding based on a large-scale code corpus to obtain the embedding vector of each token. The specific expression is as follows:
[0132]
[0133] Where E∈R d×|V| is a randomly initialized word embedding or a pre-trained stru2vec embedding matrix with dimension d, V is the vocabulary size of all annotations, Embedding vector representing each token;
[0134] Build the Transformer attention mechanism encoder (Encoder);
[0135] For the encoder of TVAE, the present invention introduces a self-attention mechanism to enhance its representation learning ability. Specifically, this layer performs attention calculation on the embedded vector for each input x i , and obtain an output sequence Multi-head attention enables the model to consider the relationship between input tokens at the same time, which helps to better capture contextual information; by focusing on different parts of the input sequence, the model can more effectively capture long-distance dependencies during encoding. The specific expression is as follows:
[0136]
[0137] In the formula, Embedding vector representing each token;
[0138] Next, the output sequence is processed by global average pooling to obtain the overall context representation It is then processed through a fully connected layer to estimate the parameters of the Gaussian distribution, namely the mean vector μ and the variance vector σ 2 , the latent variable z is randomly sampled from this distribution, and the specific expressions are as follows:
[0139]
[0140] Where: w represents the weight matrix of the fully connected layer, b represents the bias vector of the fully connected layer, r is a random vector from the standard normal distribution, and e represents a natural constant;
[0141] The structural changes are designed to improve the representation learning ability of the model, so as to better capture the semantic information of the input sequence; the attention mechanism enhances the model's ability to capture the relationship between tags, thereby improving the model's performance when the encoded information is used for downstream tasks;
[0142] Construct the Transformer attention mechanism decoder (Decoder), the specific steps are as follows:
[0143] For the decoder of TVAE, the decoder reconstructs the input based on the latent variables; during the decoding process, it uses the self-attention mechanism to reconstruct the input and focus on different parts of the input sequence; secondly, the attention output s xi Through the fully connected layer and processed by the argmax function, the final reconstructed annotation result x is obtained. i ', the specific expressions are as follows:
[0144]
[0145] Where: z represents the latent variable obtained by the encoder;
[0146] Construct loss function;
[0147] The present invention uses cross entropy loss to measure the probability of reconstructing the original input. The specific expression of ELBO loss of TVAE is as follows:
[0148]
[0149] Where: n represents the number of samples of the original input, CEloss and D KL Respectively represent the cross entropy loss calculation and KL divergence calculation; D KL The calculation of is shown in the following formula:
[0150]
[0151] Where: μ i is the mean value of the i-th dimension of the latent variable z, σ iis the corresponding standard deviation, N represents the dimension in the latent variable;
[0152] The verification of the refined and purified annotation-code pairs specifically includes the following steps:
[0153] S3.1. Supervised training of the self-attention-based variational autoencoder model on the guided query corpus. Set the initialization parameters to train the parameters of the fitted model. Stop training after 200 iterations and save the model file.
[0154] S3.1.1. Load the guided query corpus into the self-attention based variational autoencoder model;
[0155] S3.1.2. In the model parameter initialization, set the maximum length of the annotation to 20, the self-attention dimension to 128, the drop rate to 0.1, the number of multi-head attention heads to 1, the exponential decay rate of the first-order moment estimate of the control gradient in the Adam optimizer to 0.9, the exponential decay rate of the second-order moment estimate of the control gradient to 0.999, the decay rate of the learning rate to 0.95, the initial weight of the KL divergence to 0.05, the starting number of steps for the KL divergence weight increase to 200, the maximum value of the KL divergence weight to 0.1, and the rate of increase of the KL divergence weight to 0.000002;
[0156] S3.1.3. According to the guided query corpus, the self-attention-based variational autoencoder model is trained with a learning rate of 0.0001 and the model file is obtained;
[0157] S3.2. Based on the variational autoencoder model file obtained through supervised training, further sample the comment-code pairs after unsupervised filtering, remove the comments whose semantics deviate from the semantic description of the code, and obtain the comment-code pairs that are semantically close to the natural language query;
[0158] The trained variational autoencoder (TVAE) model can identify whether an annotation is semantically similar to a query in the corpus. Specifically, when the preliminarily filtered comment-code pairs are input into the TVAE model, their reconstruction loss is measured. The loss value is an anomaly score that measures the distance between the input and the query in the guided query corpus. Annotations with smaller loss values are semantically closer to the query. The annotations are sorted in ascending order according to their reconstruction losses, and the top-ranked annotations are retained. An unsupervised clustering algorithm EM-GMM is used to divide a group of annotations into qualified and unqualified groups based on the reconstruction loss. For each group, a Gaussian mixture model GMM fits a Gaussian probability density function and mixes them together as the distribution of the entire data set. Finally, the expectation maximization EM algorithm is applied to estimate the parameters of the Gaussian probability density of all qualified and unqualified groups, and the mean vector, covariance matrix, and mixing weights are gradually optimized until the parameter combination that maximizes the log-likelihood value, that is, the optimal value, is found.
[0159] S4, the refined and purified comment-code pairs obtained by using the code description embedding neural network for verification;
[0160] The refined comment-code pairs obtained by code description embedding neural network (CoDEnn) network verification are used to set the initialization hyperparameters to train the code search model (DeepCodeSearch). The training is stopped after 100 iterations, and the model file is saved. The model performance is evaluated on the validation set Python (CSNV_python, CosQA) and the validation set Java (CSNV_java, CosBench), and the corresponding MRR, Answered@k (k=10) and UD indicators are obtained.
[0161] In the DeepCS search model training, the maximum length of the query and code is set to 30 and 50 respectively, and the Adam optimizer hyperparameter is 10. -8 , according to the learning rate of 10 -4 To fit the model file of the DeepCodeSearch model;
[0162] To illustrate the experimental effect of the present invention on code search, the model performance is verified on the following three different manually annotated test sets based on the trained DeepCodeSearch model file, including:
[0163] (1) CSNV provides a search benchmark of query-code pairs, where the query is based on Bing search and programmers' function-related annotations, and the code snippets are from CodeSearchNet; 1559 Python and 434 Java related pairs are obtained;
[0164] (2) CodeSearch Question and Answering (CosQA) is a dataset containing code snippets extracted from GitHub functions and queries from Microsoft Bing search logs; it contains 20,604 tags. The present invention uses 500 active instances in the CosQA dataset, and queries are matched with different numbers of target codes;
[0165] (3) CodeSearch Bench (CosBench) is a dataset containing 1,000 Java projects, including 475,783 Java files and 4,199,769 code snippets (i.e., Java methods) collected from GitHub. In addition, CosBench also includes 52 natural language queries from StackOverflow, each of which is accompanied by 10 corresponding matching code snippets;
[0166] The corresponding MRR, Answered@k (k=10) and UD performance evaluation indicators are given and compared with the benchmarks CSNclean, NLQF, and NLQFembed. The experimental results are shown in Tables 2 and 3 to illustrate the performance of the data refinement and purification framework of the present invention, which shows that the present method is more effective in purifying the refined annotation-code dataset;
[0167] Table 2: Indicators of the DeepCS model trained using the Python dataset on the CSNV and CosQA test sets
[0168]
[0169] Table 3: Indicators of the DeepCS model trained using the Java dataset on the CSNV and CosBench test sets
[0170]
[0171] In addition, in order to compare the impact of different filtering modules in this method on the overall performance. The experiment analyzes and quantitatively evaluates the impact of each filtering component on the model in refining the data set and improving performance in subsequent tasks. To achieve this goal, the present invention conducts an ablation experiment to systematically disable filter components one by one, and then trains the model using the data set generated by the disabled filters, and compares its performance with the baseline. The present invention performs a performance evaluation on DeepCS trained under ablation conditions, and lists the results in detail in Tables 4 and 5. The experimental results show that in different programming languages, grammar rules and model filtering modules have different contributions. For Python, both are indispensable, while for Java, only the model filtering module is needed for cross-domain data verification.
[0172] Table 4: Ablation experiment results of DeepCS model trained on different filter modules on Python dataset
[0173]
[0174] Table 5: Ablation experiment results of DeepCS model trained on different filter modules on Java dataset
[0175]
[0176] Example 2
[0177] Based on the same inventive concept, a collaborative learning-based annotation code library device includes: a data collection module, a guided query library collection module, an artificial rule filtering module, a self-attention-based variational autoencoder model filtering module, a feature matrix construction module, a normalization processing module, a scoring module, a pre-trained word embedding module, a self-attention-based encoder module, a self-attention-based decoder module, and a loss function module;
[0178] The query-code library saved from the data collection module is input into the guided query library collection module. Specifically, the query-code library is input into the feature matrix construction module to obtain the feature matrix, which is then normalized by the normalization processing module. Finally, it is scored and sorted by the scoring module to obtain the guided query corpus. The annotation-code library saved from the data collection module is input into the manual rule filtering module to obtain the preliminary filtered data, which is then input into the self-attention-based variational autoencoder filtering module to obtain the refined and purified data. Specifically, the guided query corpus obtained from the guided query library collection module is used to train the self-attention-based variational autoencoder, which includes a pre-trained word embedding module, a self-attention-based encoder module, a self-attention-based decoder module, and a loss function module.
[0179] The data collection module is used to obtain queries and codes in Python and Java languages, save them as query-code libraries, collect data in the same language, remove comments in the code, and associate the first sentence of the comment with the code, and save them as comment-code libraries;
[0180] The query-code library saved from the data collection module is input into the guided query library collection module. Specifically, the query-code library is input into the feature matrix construction module to obtain the feature matrix, and then the normalization processing module performs normalization processing, and finally the scoring module performs scoring sorting to obtain the guided query corpus; the annotation-code library saved from the data collection module is input into the manual rule filtering module to obtain the preliminary filtered data, and then input into the TVAE model filtering module to obtain the refined and purified data. Specifically, the guided query corpus obtained from the guided query library collection module is used to train the TVAE model, which includes: pre-trained word embedding module, self-attention-based encoder module, self-attention-based decoder module, and loss function module;
[0181] A guided query library collection module is used to extract high-quality queries from the query-code library to construct a query guided corpus, providing an effective data source for the subsequent training of the TVAE model. A guided corpus construction method based on the WTFF algorithm is adopted, including a feature matrix construction module, a normalization processing module, and a scoring module. The feature matrix construction module is used to obtain a social attribute feature matrix, the normalization processing module is used to fill missing values and normalize all features, and the scoring module is used to score the corpus.
[0182] The manual rule filtering module is used to remove the comments with grammatical anomalies in the comment-code base to obtain the preliminary filtered comment-code base, including URLs, HTML tags, short sentences, question sentences, non-English content, program tags, and brackets;
[0183] The TVAE model filtering module is used to remove semantically abnormal annotations in the annotation-code base after preliminary filtering, and obtain an annotation-code base that is semantically close to natural queries. A self-attention-based variational autoencoder model is adopted, including a pre-trained word embedding module, a self-attention-based encoder (Encoder) module, a self-attention-based decoder (Decoder) module, and a loss function module; the pre-trained word embedding module is used to obtain Python structure embedding pre-trained on a large-scale corpus, the self-attention Encoder module is used to learn and output the mean and standard deviation of the latent space to obtain a latent vector, the self-attention Decoder module is used to reconstruct from the latent vector according to sampling to generate new annotations, and the loss function module includes a reconstruction term and a regularization term, the former is used to calculate the gap between the reconstructed term and the corresponding annotation, and the latter is used to measure the difference between the distribution learned in the latent space and the pre-selected prior distribution.
[0184] The system introduced in the second embodiment of the present invention is a system of a method for purifying an annotation code library based on collaborative learning in the embodiment of the present invention. Therefore, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems adopted by the method of the first embodiment of the present invention belong to the scope of protection of the present invention.
Claims
1. A method for purifying annotated code base based on collaborative learning, characterized in that: The following steps are involved: S1. Collect query-code matching pairs of Java and Python programming language types through software question-answering websites to form a guided query corpus. S2, obtain the original annotation-code pairs of Java and Python language types from the open source code base to complete unsupervised filtering; S3, loading the guided query corpus into a self-attention-based variational autoencoder model for training and fitting to obtain a model file, and further sampling the unsupervised filtered comment-code pairs according to the model file to refine and purify the comment-code pairs; S4. The refined and purified comment-code pairs obtained by code description embedding neural network verification.
2. According to the collaborative learning-based annotation code library purification method of claim 1, it is characterized in that: The composition of the guided query corpus specifically includes the following steps: S1.
1. Use software question-answering websites to crawl query-code matching pairs containing Python and Java programming language types; S1.2, extracting the social attribute values according to the query-code matching pairs, constructing a social attribute feature matrix, and filling missing values and normalizing all features; S1.
3. The wavelet time-frequency transform fusion algorithm is used to score the corpus. The top 10 corpora are selected as high-quality query-code pairs based on the score ranking. Through the query classifier, a set of query results starting with "how to do" are obtained to build a guided query corpus.
3. The method for purifying an annotation code library based on collaborative learning according to claim 2, characterized in that: The wavelet time-frequency transform fusion algorithm specifically includes the following steps: S1.3.1, the feature matrix of the query-code matching pair corpus is regarded as a signal, and converted into a wavelet tree using wavelet transform; S1.3.2, based on the wavelet tree, by fusing the high-frequency and low-frequency vectors, new high-frequency and low-frequency coefficient vectors are generated; these coefficient vectors are estimated by the expectation maximization algorithm to obtain feature fusion weights; S1.3.3, using inverse wavelet transform, reconstruct a new wavelet tree, in which the top leaf node represents the score of the query-code matching pair corpus; S1.3.4, sort the posts according to the scores and select the top 10 as high-quality query-code pairs; S1.3.
5. Among the queries obtained, they can be divided into four types: "debug / fix", "need to know", "how to do" and "seek different solutions". Among these, the "how to do" queries are the most relevant for code search task verification.
4. The method for purifying an annotation code library based on collaborative learning according to claim 1, characterized in that: The method of obtaining original annotation-code pairs of Java and Python language types from the open source code base to complete unsupervised filtering specifically includes the following steps: S2.
1. Delete all comments in Python and Java codes, and pair the first sentence of each comment with the corresponding code to form the original comment-code pair. S2.
2. Establish a set of grammatical rules to detect and process the obtained comment-code pairs, remove the comments whose syntax deviates from the semantic description of the code, and obtain the unsupervised filtered comment-code pairs, wherein the grammatical rules include: URLs, HTML tags, short sentences, question sentences, non-English content, program tags, and brackets.
5. The method for purifying annotated code base based on collaborative learning according to claim 1, characterized in that: The verification of the refined and purified annotation-code pairs specifically includes the following steps: S3.
1. Supervised training of the self-attention-based variational autoencoder model on the guided query corpus. Set the initialization parameters to train the parameters of the fitted model. Stop training after 200 iterations and save the model file. S3.
2. Based on the variational autoencoder model file obtained through supervised training, the comment-code pairs after unsupervised filtering are further sampled to remove the comments whose semantics deviate from the semantic description of the code, so as to obtain the comment-code pairs that are semantically close to the natural language query.
6. The method for purifying annotated code base based on collaborative learning according to claim 5, characterized in that: The specific steps of supervised training of the self-attention-based variational autoencoder model are as follows: S3.1.
1. Load the guided query corpus into the self-attention based variational autoencoder model; S3.1.
2. In the model parameter initialization, set the maximum length of the annotation to 20, the self-attention dimension to 128, the drop rate to 0.1, the number of multi-head attention heads to 1, the exponential decay rate of the first-order moment estimate of the control gradient in the Adam optimizer to 0.9, the exponential decay rate of the second-order moment estimate of the control gradient to 0.999, the decay rate of the learning rate to 0.95, the initial weight of the KL divergence to 0.05, the starting number of steps for the KL divergence weight increase to 200, the maximum value of the KL divergence weight to 0.1, and the rate of increase of the KL divergence weight to 0.000002; S3.1.
3. According to the guided query corpus, the self-attention-based variational autoencoder model is trained with a learning rate of 0.0001 and the model file is obtained.
7. The method for purifying annotated code base based on collaborative learning according to claim 1, characterized in that: The refined and purified comment-code pairs obtained by using the code description embedding neural network verification specifically include: using the code description embedding neural network to verify the comment-code pairs refined in the step, setting the initialization hyperparameter to train the code search model, stopping the training after 100 rounds of iterations, saving the model file, and evaluating the model performance on the validation set Python and the validation set Java, and obtaining the corresponding average reciprocal ranking, the number of queries answered by the code snippets in the top k results, and the UD index, and then using the average of the sum of the indicator gains of the average reciprocal ranking and the number of queries answered by the code snippets in the top k results multiplied by the gain of the corpus quantity.
8. A collaborative learning-based annotation code library device, characterized in that: include: Data collection module, guided query library collection module, artificial rule filtering module, self-attention-based variational autoencoder model filtering module, feature matrix construction module, normalization processing module, scoring module, pre-trained word embedding module, self-attention-based encoder module, self-attention-based decoder module, loss function module; The data collection module is used to obtain queries and codes in Python and Java languages, save them as query-code libraries, collect data in the same language, remove comments in the codes, and associate the first sentence of the comments with the codes, and save them as comment-code libraries; The query-code library saved from the data collection module is input into the feature matrix construction module to obtain a feature matrix, and then normalized by the normalization processing module, and finally scored and sorted by the scoring module to obtain a guided query corpus, and then input into the guided query library collection module; The annotation-code library saved from the data collection module uses the guided query corpus obtained by the guided query library collection module to train the variational autoencoder based on self-attention, inputs it into the artificial rule filtering module to obtain the preliminary filtered data, and then inputs it into the variational autoencoder filtering module based on self-attention to obtain the refined and purified data; The self-attention-based variational autoencoder includes: a pre-trained word embedding module, a self-attention-based encoder module, a self-attention-based decoder module, and a loss function module.
Citation Information
Cited By
RPA code generation method based on deep learning model
CN119376708A
Attack technique clustering and attribution method for attack mode
CN120110776A
Neural network position coding method and system based on learnable power law attenuation
CN121543673A
