Contrastive learning code search method with reinforced multimodal semantics

Abstract

The present application relates to a code search method based on contrastive learning of reinforced multimodal semantics, and belongs to the field of natural language processing and machine learning. Firstly, the code snippet is represented as token sequence, abstract syntax tree and program expression graph three modalities, and BERT model is used to generate the feature vectors of each modality and splice them into joint code feature vectors. Then, a contrastive loss function is constructed to reduce the distance between the query statement and the corresponding code snippet in the feature space. Finally, the cosine similarity is used to calculate the distance between the query statement feature vector and the joint code feature vector and sort them, and the code search result is output. The present application proposes a code search method based on contrastive learning of reinforced multimodal semantics to solve the problem that the existing method does not fully extract the code structure features and there is a semantic gap between the query statement and the code snippet, thereby improving the accuracy of code search.

Description

Technical Field

[0001] This invention relates to a contrastive learning code search method for enhancing multimodal semantics, belonging to the fields of natural language processing and machine learning. Background Technology

[0002] Early code search research primarily treated code as a sequence of text, transforming the code search problem into a character matching problem between code snippets (special text) and query statements (ordinary text), searching for target code snippets in a codebase using keywords. With technological advancements, current research methods often combine feature classification with deep learning. By introducing Abstract Syntax Tree (AST) structures to enrich the structural semantic information of code, and using various neural networks, such as Long Short-Term Memory (LSTM) networks, the information length limitation is overcome, thereby learning higher-order relationships between code snippets and query statements. However, these methods have some problems in practical applications.

[0003] (1) The semantic gap between query statements and code: There is a difficulty in accurately matching the high-level intent of the query statement with the low-level implementation details of the code. Most existing works extract code features and learn code structure and semantic information by using a single dimension, such as token sequences, AST trees, or a combination of both. These methods analyze code from one or two dimensions (semantics, syntax, or logical structure, etc.), which has a low utilization of code information and cannot accurately and completely describe the code. As a result, the model cannot accurately learn all the code information during training, resulting in inaccurate final code search results.

[0004] (2) The code search process suffers from semantic collapse. Most existing deep learning methods use a Siamese network framework to encode the query statement and code into the feature space, and retrieve code fragments by calculating the cosine similarity between the query statement vector and the code vector. However, in this process, the query statement vector and the code vector have a small range of variation in the feature space, which makes the difference between matching and non-matching query statement-code pairs small, which severely limits their semantic expression and leads to errors in the final code search results.

[0005] In summary, existing code search methods suffer from semantic gaps between query statements and code, as well as semantic collapse in the code search process. Therefore, this invention proposes a contrastive learning code search method that enhances multimodal semantics. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing code search methods in learning the semantic relevance between query statements and code, as well as the small difference between query statement vectors and code vectors in the feature space. This invention proposes a contrastive learning code search method that enhances multimodal semantics.

[0007] The design principle of this invention is as follows: First, the code snippet is represented as three modalities: a token sequence, an AST tree, and a Program Expression Graph (PEG). The query statement generates a query sequence, and byte pair encoding is used. The system first generates a vocabulary for each modality using BERT Encoding (BPE). Then, it performs masking operations on each modality based on defined masking rules, and uses the vocabulary generated in the previous stage to convert each modality before and after masking into a digital sequence representation. Next, it uses the BERT model to vectorize the digital sequences of each modality, and merges the token vector, AST tree vector, and PEG graph vector into a joint code feature vector. Then, it constructs a contrastive learning loss function between the query statement and the code, as well as between different codes. In the calculation of the contrastive learning loss function between the query statement and the code, the query statement and matching code are considered positive samples, and the query statement and non-matching code are considered negative samples. In the calculation of the contrastive learning loss function between different codes, the code before and after masking is considered positive samples, and other codes before and after masking in the same batch are considered negative samples. Finally, it calculates the cosine similarity between the query statement vector and the joint code feature vector, sorts the results, and retrieves the code fragment that best matches the query statement.

[0008] The technical solution of the present invention is achieved through the following steps:

[0009] Step 1: Construct different modal representations of query statements and code snippets, and use BPE to generate vocabulary for the corresponding modalities.

[0010] Step 1.1: For code snippets, convert them into three modalities: token sequence, AST tree, and PEG graph; for query statements, convert them into query sequences.

[0011] Step 1.2: For each modality, use the BPE method to generate its corresponding vocabulary.

[0012] Step 2: Each modality performs a specific masking operation and is converted into a digital sequence representation.

[0013] Step 2.1: The token sequence, AST tree, PEG graph, and query sequence are masked using different masking rules.

[0014] Step 2.2: Using the vocabulary generated in step 1.2, represent each modality before and after the mask as a digitized sequence.

[0015] Step 3: Encode using the BERT model and fuse the three modal representations of the code snippet into a joint code feature vector.

[0016] Step 3.1: Use the BERT model to map the token sequence before and after masking, AST tree, PEG diagram and query sequence into the feature space to generate the corresponding vector representation.

[0017] Step 3.2: Concatenate and merge the token vector, AST tree vector, and PEG graph vector to generate the final joint code feature vector representing the code fragment.

[0018] Step 4: Construct the comparative learning loss function between the query statement and the code, as well as between different code types.

[0019] Step 4.1: In the process of calculating the comparison loss function between the query statement and the code, the query statement vector and the matching joint code feature vector are regarded as positive samples, and the query statement vector and the non-matching joint code feature vector are regarded as negative samples.

[0020] In step 4.2, during the calculation of the comparison loss function between different codes, the joint code feature vector before and after masking is regarded as a positive sample, and the joint code feature vector before and after masking and other joint code feature vectors before and after masking in the same batch are regarded as negative samples.

[0021] Step 5: Calculate and sort the cosine similarity between the query statement vector and the joint code feature vector, and retrieve the code fragment that best matches the query statement.

[0022] Beneficial effects

[0023] Compared to existing code search methods that only use token sequences and AST trees, this invention introduces an additional PEG graph representation to supplement the structural information of the code, enrich the structural and linguistic information representation of the code, and alleviate the semantic gap between query statements and code.

[0024] Compared to existing code search methods, which suffer from limited differences between matching and non-matching query-code pairs, this invention establishes corresponding masking rules for different modalities, constructs positive and negative samples between query statements and codes, and learns the similarity between query statements and codes as well as the differences between different codes. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the principle of the comparative learning code search method for enhancing multimodal semantics in this invention. Detailed Implementation

[0026] To better illustrate the purpose and advantages of the present invention, the implementation methods of the present invention will be further described in detail below with reference to examples.

[0027] The experimental dataset consists of Python and Java code from the CodeSearchNet dataset, comprising 2 million code snippets from open-source libraries. Specifically, code snippets are top-level function or method annotations (e.g., docstrings in Python), representing a complete function or method. The code search experimental data is shown in Table 1.

[0028] Table 1 CodeSearchNet dataset

[0029]

[0030] In the CodeSearchNet dataset, 80% of the data for each corpus is used for training, 10% for validation, and 10% for testing. The first line displays the data name, lines 2-4 display the amount of data used for the Training, Validation, and Test datasets respectively, and line 5 displays the total amount of data.

[0031] During the experiment, the word embedding vocabulary was set to 400,000, with 128 as the word embedding dimension. The code snippet contained a maximum of 200 words, and the natural language query statement contained a maximum of 30 words.

[0032] The experiment used , , as well as As an evaluation indicator.

[0033] The score quantifies the ranking of the target code segment for a given query, and it only focuses on the ranking of the most relevant results. The most relevant code segment should have the highest score, and the lower the ranking, the higher the score. The lower the score, the better. When calculating the test group... When scoring, for each code query pair, other code segments from the same batch act as distractors. The average of all batches is the final score. Fractions, formula as follows As shown.

[0034]

[0035] in This is a batch code query for the quantity. This refers to the rank of the i-th code. If the rank does not exist, then... Approximately infinity.

[0036] The score quantifies whether the ranking position of the target code segment for a given query is within the expected range, by measuring the top-ranking segments in the same batch. The percentage of correct results in each ranking result, the average of all batches, is the final... Fractions, formula as follows As shown:

[0037]

[0038] in This is a batch code query for the quantity. It is a function that returns 1 if the input is true, and 0 otherwise. This indicates the rank of the first code in the results list that matches the query. During the experiment, [the following will be used]. Set them to 1, 5, and 10 respectively.

[0039] The hardware environment used in the experiment is shown in Table 2.

[0040] Table 2 Experimental Hardware Environment

[0041]

[0042] The software environment used in the experiment is shown in Table 3.

[0043] Table 3 Experimental Software Environment

[0044]

[0045] The specific procedure for this experiment is as follows:

[0046] Step 1: Generate different modal representations of query statements and code snippets, and use BPE to generate vocabulary lists for different modalities.

[0047] Step 1.1: For the input code snippet, use Python's parsing expression libraries AST and PE to obtain two modalities: the AST tree and PEG graph of the code snippet. Then, construct a tokenization method to process the code snippet and the natural language query statement to obtain two modalities: the token sequence of the code snippet and the query sequence of the query statement.

[0048] Step 1.2: Use the word segmenter BPE to build the corresponding vocabulary for each modality.

[0049] Step 2: Different modalities are subjected to specific masking operations and converted into digital sequence representations.

[0050] Step 2.1: For the token sequence and query sequence, dynamic masking is used. The dynamic aspect is that the masking operation is repeated in each training round. Specifically, the masking rule is to select K words, replacing 80% with the [MASK] symbol, 10% with random words, and leaving 10% unchanged. For the AST tree, this method uses a dynamic tree masking mechanism, which is reflected in the re-masking in each training round. The tree masking rule is to select K nodes in the AST and replace them with the [MASK] symbol. The closer a node is to the terminal node, the greater the likelihood of it being replaced because 1) the closer a node is to the root node, the more times it is repeated in the path set. If a standard masking strategy is used, many nodes of the same type will be masked. Repeatedly learning the representations of these nodes will degrade the model's performance. 2) Terminal nodes of the AST typically reference user-defined values, which represent identifiers with rich meaning in the code to help the model understand the AST. For the PEG graph, dynamic graph masking is used. The dynamism is reflected in the fact that the masking operation is performed again in each round of training. The specific masking rule is to select K nodes and replace them with the [MASK] symbol. User-defined symbols and variable names are more likely to be selected and replaced because these words contain more code functional information.

[0051] Step 2.2: Using the vocabulary constructed by the word segmenter BPE in step 1.2, represent each modality before and after the mask as a digital sequence.

[0052] Step 3: Encode using the BERT model and fuse the various code modalities into a joint code feature vector representation.

[0053] Step 3.1: Use the BERT model to vectorize the token sequence, AST tree, PEG graph and query sequence before and after masking.

[0054] Step 3.2: Concatenate and merge the token vector, AST tree vector, and PEG graph vector to generate the final joint code feature vector representing the code fragment.

[0055] Step 4: Construct a comparative learning loss function between the query statement and the code, as well as between different code types.

[0056] Step 4.1: In the process of calculating the loss function for contrastive learning between the query statement and the code, the query statement vector and the matching joint code feature vector are considered as positive samples, and the query statement vector and the non-matching joint code feature vector are considered as negative samples. The loss function for contrastive learning is shown in Equation (3):

[0057]

[0058] in, Indicates the current query vector. This represents the joint code feature vector that matches the query vector. This represents the joint code feature vector after matching the query vector with the mask. This represents other query vectors in the same batch. This represents the feature vectors of other joint codes in the same batch. , It is a scalar temperature parameter. and These are weight parameters.

[0059] Step 4.2, in the process of calculating the contrastive loss function between different codes, the joint code feature vector before and after masking is regarded as a positive sample, and the joint code feature vector before and after masking and other joint code feature vectors before and after masking in the same batch are regarded as negative samples. The loss function for contrastive learning is shown in Equation (4):

[0060]

[0061] in, Indicates the current query vector. This represents the joint code feature vector that matches the query vector. This represents the joint code feature vector after matching the query vector with the mask. This represents other query vectors in the same batch. This represents the feature vectors of other joint codes in the same batch. , It is a scalar temperature parameter. and These are weight parameters.

[0062] Step 5: Calculate and sort the cosine similarity between the query statement vector and the joint code feature vector, and retrieve the code fragment that best matches the query statement.

[0063] Table 4 compares the performance of the present invention with existing code search methods. Table 4 shows that, compared to the CoCoSoDa method, the present invention's method performs better on the Python corpus. An increase of 0.158, , , These figures represent increases of 0.212, 0.104, and 0.091 respectively, within the Java corpus. An increase of 0.006, , , They increased by 0.007, 0.008, and 0.002 respectively.

[0064] Compared with the Multi-LCRS method, the method of this invention, in the Python corpus, An increase of 0.029, , , These values ​​increased by 0.042, 0.010, and 0.012 respectively in the Java corpus. An increase of 0.022, , , They increased by 0.008, 0.018, and 0.019 respectively.

[0065] Compared to the GraphSearchNet method, the method of this invention, in the Python corpus, An increase of 0.136, , , These values ​​increased by 0.175, 0.091, and 0.075 respectively in the Java corpus. An increase of 0.074, , , They increased by 0.067, 0.073, and 0.077 respectively.

[0066] Table 4 Experimental Results

[0067]

[0068] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A contrastive learning code search method that enhances multimodal semantics, characterized in that... The method includes the following steps: Step 1: Construct different modal representations of query statements and code snippets. For code snippets, convert them into three modalities: token sequence, AST tree, and PEG graph (Program Expression Graph). For query statements, convert them into query sequences and use BPE to generate the vocabulary of the corresponding modalities. Step 2: The token sequence, AST tree, PEG diagram and query sequence are masked using different masking rules, and the vocabulary generated in Step 1 is used to convert each modality into a digital sequence representation. Different masking rules were constructed for different modalities. For token sequences and query sequences, dynamic masking was used, which is reflected in the fact that the masking operation is repeated in each training round. Specifically, the masking rule is to select K words, of which 80% are replaced with the [MASK] symbol, 10% are replaced with random words, and 10% are left untouched. For AST trees, this method uses a dynamic tree masking mechanism, which is reflected in the fact that the masking is repeated in each training round. The tree masking rule is to select K nodes in the AST and replace them with the [MASK] symbol, where nodes closer to the terminal node are more likely to be replaced. For PEG graphs, dynamic graph masking was used, which is reflected in the fact that the masking operation is repeated in each training round. Specifically, the masking rule is to select K nodes and replace them with the [MASK] symbol, where user-defined symbols and variable names with more code functionality and information are selected and replaced with a higher probability. Step 3: Use the BERT model to vectorize the token sequence, AST tree, PEG graph and query sequence before and after masking, and concatenate and merge the token vector, AST tree vector and PEG graph vector to generate the joint code feature vector that finally represents the code fragment. Step 4: Construct comparative learning loss functions between query statements and code, as well as between different code types; In the process of calculating the contrast loss function between different codes, the joint code feature vector before and after the mask is regarded as a positive sample, and the joint code feature vector before and after the mask and other joint code feature vectors before and after the mask in the same batch are regarded as negative samples. Construct contrast learning loss functions between query statements and codes and between different codes to increase the difference between matching and non-matching query statement-code pairs. Step 5: Calculate and sort the cosine similarity between the query statement vector and the joint code feature vector, and retrieve the code fragment that best matches the query statement.

2. The contrastive learning code search method for enhanced multimodal semantics according to claim 1, characterized in that: Step 1 uses additional modal PEG diagrams to more fully represent code structure and semantic information.

Images