A code search and recommendation method and system based on contrastive learning and pre-training techniques

Abstract

This invention discloses a code search and recommendation method and system based on contrastive learning and pre-training techniques. First, the code snippets undergo structural transformation to obtain multiple equivalent code snippets. Code snippets with the same semantics are used as positive samples, and code snippets with different semantics are used as negative samples. A code encoder is then pre-trained. This is then transferred to a code query model that includes a text encoder, and further optimization is performed using code snippets with code annotations. The code encoder is used to vectorize code snippets, and the text encoder is used to vectorize query statements. Finally, the query statement is input into the text encoder to obtain a query vector. The code encoder is then used to convert all code snippets in the codebase into code vectors. The code snippet corresponding to the code vector with the highest similarity to the query vector is recommended. This invention reduces the reliance on code annotations while improving the accuracy of code search.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing, and in particular to a code search and recommendation method and system based on contrastive learning and pre-training techniques. Background Technology

[0002] In recent years, with the rapid development of open-source software, a large amount of code resources have emerged on the internet. To implement a certain function, programmers often search for the required code snippets and reuse them. Code search methods can effectively help programmers find code snippets relevant to specific programming tasks. Code search has become one of the important research areas in software engineering. Most early code search methods were based on information retrieval techniques. A fundamental problem with information retrieval-based code search is the mismatch between the high-level intent reflected in natural language queries and the low-level implementation details in the source code. Later, deep learning-based code search methods were proposed. However, these methods typically require a large number of code snippets with natural language annotations to train the deep learning model. In reality, only a portion of the code snippets are annotated, and the quality of code annotations is difficult to guarantee; filtering out code snippets with correct annotations often requires considerable effort.

[0003] For example, the "Code Search and Recommendation Device and Method Based on Open Source Knowledge" disclosed in patent document CN112051986A first collects open-source code and information from the open-source community, processes it, and generates a code snippet library. Then, it extracts features from the code snippets, including textual features, structural features, topic features, and development features, and establishes a mapping relationship between each code snippet and its features, generating a code snippet-feature index library. Next, it uses text preprocessing methods to process the query statement and uses it as input for code search and recommendation. Finally, it calculates the similarity between code features and query statements to complete the search ranking of code snippets and recommend them to query users. The drawback of this method is that when calculating the similarity between code features and natural language queries, it does not consider the semantic gap between code snippets and natural language, making it difficult to guarantee the degree of matching between the searched code and the user's needs.

[0004] Furthermore, the "code search method" disclosed in patent document CN115017294A includes the following steps: First, inputting code with descriptions; second, embedding the code and descriptions separately; third, performing interactive calculations on the code features and description features to obtain a fine-grained similarity matrix, and learning the matrix using a pre-constructed CNN network; fourth, constructing fully connected graphs for both the code and descriptions, and updating the node vectors according to certain rules; fifth, performing weighted summation operations on the code and descriptions using an attention mechanism; and sixth, predicting the score of each candidate code segment and re-ranking all candidate code segments using the scores. This method requires a large number of code segments with natural language annotations to train the model, and also places high demands on the quality of the code annotations.

[0005] There are many similar code search solutions, such as: Gu, X., Zhang, H., Kim, S., 2018. Deep code search. In: 2018 IEEE / ACM40th International Conference on Software Engineering (ICSE). IEEE, pp. 933--944. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R., 2020. Momentumcontrast for unsupervised visual representation learning. In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. pp.9729--9738. Lv, F., Zhang, H., Lou, J.-g., Wang, S., Zhang, D., Zhao, J., 2015.Codehow: Effective code search based on api understanding and extendedboolean model (e). In: 2015 30th IEEE / ACM International Conference onAutomated Software Engineering (ASE). IEEE, pp. 260--270. Wan, Y., Shu, J., Sui, Y., Xu, G., Zhao, Z., Wu, J., Yu, P., 2019.Multi-modal attention network learning for semantic source code retrieval.In: 2019 34th IEEE / ACM International Conference on Automated SoftwareEngineering (ASE). IEEE, pp. 13--25. Xu, L., Yang, H., Liu, C., Shuai, J., Yan, M., Lei, Y., Xu, Z., 2021.Two-stage attention-based model for code search with textual and structuralfeatures. In: 2021 IEEE International Conference on Software Analysis,Evolution and Reengineering (SANER). IEEE, pp. 342--353. Wang, H., Wang, X., Xiong, W., Yu, M., Guo, X., Chang, S., Wang, W.~Y., 2019. Self-supervised learning for contextualized extractivesummarization. arXiv preprint arXiv:1906.04466. Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., Khandeparkar, H.,2019. A theoretical analysis of contrastive unsupervised representationlearning. In: International Conference on Machine Learning. PMLR, pp. 5628--5637. Chen, T., Kornblith, S., Norouzi, M., Hinton, G., 2020. A simpleframework for contrastive learning of visual representations. In: International conference on machine learning. PMLR, pp. 1597--1607.

[0006] However, these search solutions have two major drawbacks: 1. They require a large number of annotated code snippets to train the recommendation model; 2. The matching degree between the code snippets matched by the recommendation model and the requirements is not good. Summary of the Invention

[0007] The purpose of this invention is to provide a code search recommendation scheme that is less dependent on code comments and can improve the accuracy of code search, addressing all or part of the problems mentioned above.

[0008] The technical solution adopted in this invention is as follows: This invention provides a code search and recommendation method based on contrastive learning and pre-training techniques, comprising: Each original code segment in the sample set is structurally transformed to generate multiple corresponding equivalent code segments. Using the equivalent code fragments, a code encoder is pre-trained using a contrastive learning method according to the set contrastive learning objective. The code encoder represents the input code fragments as code vectors. Specifically, for an equivalent code fragment, equivalent code fragments corresponding to the same original code fragment are considered as positive samples, and equivalent code fragments corresponding to other original code fragments are considered as negative samples. The pre-trained code encoder is transferred to the code query model, which also includes a text encoder that represents the input code comments as query vectors; the code query model is trained using code snippets with code comments. Input the query into the trained code query model, and the model will retrieve recommended code snippets from the code repository.

[0009] To address the above problems, this invention also provides a code search and recommendation system based on contrastive learning and pre-training techniques, comprising a code preprocessing module, a pre-training module, a model fine-tuning and training module, and a code search module, wherein: The code preprocessing module is configured to perform structural transformation on each original code segment in the sample set to generate multiple corresponding equivalent code segments. The pre-training module is configured to: use the equivalent code snippets generated by the code preprocessing module and the configured contrastive learning objective to pre-train a code encoder using a contrastive learning method, wherein the code encoder represents the input code snippets as code vectors; wherein, for an equivalent code snippet, equivalent code snippets corresponding to the same original code snippet are regarded as positive samples, and equivalent code snippets corresponding to other original code snippets are regarded as negative samples; The model fine-tuning training module is configured to train the code query model using code snippets with code annotations. The code query model includes a text encoder and a code encoder pre-trained by the pre-training module. The text encoder represents the input code annotations as query vectors. The code search module is configured to: receive an input query statement, input the query statement into a trained code query model, and retrieve recommended code snippets from the code library.

[0010] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention trains the code encoder through pre-training and then fine-tunes its model using a small number of annotated code snippets, which greatly reduces the dependence on annotated code snippets.

[0011] 2. This invention generates equivalent code fragments through structural transformation, constructing a large number of positive and negative samples without changing the semantics of the original code. Furthermore, compared to the method of grouping similar semantics through code-annotation pairs, it can quickly achieve the grouping of positive and negative samples.

[0012] 3. This invention uses two encoders to perform unsupervised learning on the code encoder. By introducing negative samples with momentum changes, the randomness and instability of negative sample encoding can be reduced, and the vector representation learning effect in self-supervised learning can be improved.

[0013] 4. The code search framework constructed in this invention combines the code embedding model training part with the similarity score prediction part based on code vectors and natural language query vectors, which has excellent recommendation effect and high code search accuracy. Attached Figure Description

[0014] The present invention will be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the conceptual framework for code search and recommendation.

[0015] Figure 2 This is an example of a code conversion operation.

[0016] Figure 3This is a diagram of the architecture of a pre-trained code encoder.

[0017] Figure 4 This is the architecture diagram of the model fine-tuning and training module. Detailed Implementation

[0018] All features disclosed in this specification, or steps in all methods or processes disclosed herein, may be combined in any way, except for mutually exclusive features and / or steps.

[0019] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0020] Example 1 A code search and recommendation method based on contrastive learning and pre-training techniques, such as Figure 1 As shown, it includes the following parts: Code preprocessing: For sample sets (containing a large number of code snippets) obtained from forums, the internet, or professional databases, these code snippets do not require comments (i.e., they are unmarked). Each original code snippet undergoes structural transformation, and each original code snippet generates multiple corresponding equivalent code snippets.

[0021] Traditional methods involve manually identifying or using machine-assisted methods within a code repository to find code snippets with the same semantics. This approach is costly in terms of manpower and time, and the grouping results are heavily influenced by subjective factors. This invention's program conversion method performs structural transformation on the original code snippets to quickly construct multiple code snippets with the same semantics (referred to as equivalent code snippets). The structural transformation does not change the execution result of the code snippets.

[0022] Specifically, for a given original code snippet, it is first parsed and transformed into a tree-like representation, namely an Abstract Syntax Tree (AST). Then, various transformation operations are performed on this AST to generate multiple variants, all of which share the same semantics. In this embodiment of the invention, these transformation operations are broadly categorized into three types: Code minification: Changing the syntax structure of code. This includes code rearrangement, code beautification, dead code elimination, variable type compatibility, and code constant folding.

[0023] Identifier modification: Changing the names of methods and variables in the code. In this process, method and variable names are often replaced with randomly generated tokens or identifiers to hide some human-readable information in the program while preserving its functionality. This includes variable renaming and identifier obfuscation.

[0024] Code regularization transformations: reduce the number of positive pairs with high text overlap. This includes dead code insertion and subword regularization.

[0025] During the structural transformation of the original code fragment, some or all of the above transformation operations are randomly applied to generate multiple equivalent code fragments.

[0026] like Figure 2 The following are examples of code transformation operations. In the figure, (a) is the original code snippet of JavaScript (a lightweight, interpreted or just-in-time programming language with function priority) that implements the insertion sequence, (b) is the transformation operation of changing variable names, (c) is the transformation operation of swapping two independent statements, and (d) is the transformation operation of inserting dead code that does not affect the function output.

[0027] Comparison with pre-training section: See appendix Figure 3 This part utilizes the equivalent code snippet to pre-train a code encoder using a contrastive learning method based on the set contrastive learning objective. f q This was then transferred to a code query task. The learning objective of this invention is to maximize the consistency of vector representations between code segments with the same semantics and minimize the consistency of vector representations between code segments with different semantics.

[0028] This invention trains a code encoder by matching an encoded query with an encoded key dictionary using a contrastive loss objective. Specifically, this invention trains a code encoder. f q It maps a series of code snippets to code vector embeddings, representing the input code snippets as code vectors. For a code snippet... x q (Considered as query samples), equivalent code snippets of the same original code snippet are considered positive samples. x q , x k+ ), treating equivalent code snippets from different original code snippets as negative samples ( x q , x k- During the comparison pre-training period, x k+ This indicates the key that matches it.x k- Represented as x q Negative samples.

[0029] For the purpose of comparative learning, in some embodiments, the InfoNCE loss function is used to train the code encoder during pre-training. f q The model parameters are optimized, and the similarity between two vectors is measured through the dot product. When querying samples... q The InfoNCE value decreases when a sample is similar to its positive counterpart but significantly different from other samples. Suppose there is a query sample... q and a set of coded samples { k 0 ,k 1 ,k 2 , …}, Equation (1) shows the InfoNCE loss: (1)

[0030] In the formula τ This is a hyperparameter, also known as a temperature parameter. It is usually taken as a small value, such as 0.1 or 0.2, but it cannot be too small, otherwise it will lead to poor model generalization ability or difficulty in convergence. K The number of negative samples minus 1. In this formula, the numerator represents the similarity between normalized positive samples, and the denominator represents the similarity between normalized positive and negative samples. Furthermore, the query sample... q By code encoder f q Calculations show that k i By momentum encoder f k Calculated. Code encoder. f q The network parameters are updated through backpropagation, while the encoder... f k Network parameters are updated using momentum. Formally, θ k express f k Network parameters (weights). θ q express f q The network parameters are expressed by formula (2). θ k Update process: (2)

[0031] In the formula m ∈[0,1) is the momentum coefficient.m The value is usually quite large, typically greater than 0.9 (e.g., ...). m =0.999), which makes θ k Update ratio θ q Smoother.

[0032] An unsupervised representation learning model is used, introducing negative samples with momentum changes into the model to alter the training process of contrastive loss. The code encoder converts the input code snippets into feature vectors, and its weights are updated during training to better capture the features of the code snippets. The momentum encoder's primary role is to encode negative samples; its weights are updated through a momentum term, which is a moving average of the code encoder weights. This introduces more stable negative samples, helping to reduce the randomness and instability of the encoding, thereby enabling the model to better distinguish between positive and negative samples and improving the contrastive learning process.

[0033] Model fine-tuning training: The pre-trained code encoder is transferred to the code query model for initialization. Additionally, the code query model includes a text encoder. The text encoder's role is to represent the input code comments as query vectors. The code query model is trained using a (small number) of code snippets with code comments to optimize the network parameters of the code encoder, such as... Figure 4 As shown.

[0034] As mentioned earlier, a code encoder embeds code snippets into code embeddings. For each code snippet, we extract its tokens (i.e., code keywords / words) and feed them into the code encoder's neural network. Consider code snippets containing... N T A sequence of tokens for 1 word This embodiment uses LSTM (Long Short-Term Memory) as an example to demonstrate how to embed code snippets into vectors. c :

[0035] (3)

[0036] Where t i For markers t i The embedding representation uses the hidden state from the last step. As c The vector representation of .

[0037] For a text encoder, natural language queries are embedded into query vectors, meaning code comments are represented as query vectors. During training, queries are extracted from code comments, while during testing, code comments are treated as the input query statements. Consider the following... N Q Query sequence of words In this embodiment, we still use LSTM to embed code comments into the query vector. q middle.

[0038] (3)

[0039] Where w i Words w i The embedding representation. We use the hidden state from the last step. As q The vector representation of .

[0040] Based on the designed model, code snippets and natural language queries are jointly embedded into a unified vector space. The model's expected goal is that if code snippets and query statements have similar semantics, their embedded vectors should be close to each other; in other words, given an arbitrary code snippet... C And an arbitrary query statement Q We hope that when C Correct match Q When the model calculates a high similarity (probability), the similarity is high; otherwise, the similarity is low.

[0041] During the model training phase, we construct a triplet. <C ,Q + ,Q - As training instances, multiple training instances form a set of triples. For each code snippet C, there is one positive query sample. Q + Negative query samples Q - When trained using a set of triples, the model simultaneously predicts <C ,Q + >and <C ,Q - The similarity of > is used to minimize the ranking loss Φ(*): (4)

[0042] in, θ To query model parameters for the code, sim The similarity between the code snippet and the query sample. β It is a constant residual. c forC The vector representation of , q + and q - They are respectively Q + and Q - The vector representation of , D This is the training dataset.

[0043] In some embodiments, a cosine similarity function is used to calculate the similarity. sim and will β Let's set it to 0.05. Intuitively, the ranking loss increases the similarity between a code snippet and the query it matches, while decreasing the similarity between a code snippet and a query it doesn't match.

[0044] After training the code query model, a test set is used to test its performance. Given a test set, code comments are used as query statements. Q The purpose of the query is based on... Q Similarity to all code snippets C Sort the code snippets. Input each code snippet from the test set into the code encoder to obtain the corresponding code vector. c , query statement Q Input into a text encoder to obtain a query vector q Then calculate the vector q and each vector c The similarity is calculated using cosine similarity as an example in this embodiment: (5)

[0045] The code snippet with the highest similarity score is ranked and used as the recommendation. The model's performance can be evaluated based on the test results, and its parameters can also be optimized.

[0046] Code search section: After training the code query model, it can be used for code searching. The code search process is similar to performance testing, but the query statement is no longer a code comment, but a description of the desired code snippet. The query statement is input into the text encoder to obtain the query vector, and each code snippet in the code repository is input into the code encoder to obtain the corresponding code vector. Alternatively, the code snippets in the code repository can be converted into code vectors using the code encoder and then stored together to save search time. Then, the similarity ranking between the query vector and each code vector is calculated, and the code snippet corresponding to at least one code vector with the highest similarity is recommended.

[0047] For example, the query statement still uses Q This indicates that code snippets in the codebase are used...C This means that each code snippet has been pre-converted into a code vector by a code encoder. c Input the query statement into the text encoder to obtain the query vector. q Then the model applies vectors q and each vector c The similarity is calculated and sorted, and finally the one or several vectors with the highest similarity are selected. c The corresponding code snippet is provided as a recommended code snippet. The use of the cosine similarity function for similarity calculation is merely an example; other similarity functions can also be used to calculate similarity.

[0048] Example 2 This embodiment describes the design scheme of the present invention from the perspective of system architecture.

[0049] The system includes a code preprocessing module, a pre-training module, a model fine-tuning training module, and a code search module, which correspond to the code preprocessing part, the comparison pre-training part, the model fine-tuning training part, and the code search part in Example 1, respectively.

[0050] The code preprocessing module is configured to perform structural transformation on each original code fragment in the sample set to generate multiple corresponding equivalent code fragments.

[0051] The pre-training module is configured to: use the equivalent code snippets generated by the preprocessing module and the configured contrastive learning objective to pre-train a code encoder using a contrastive learning method, wherein the code encoder represents the input code snippets as code vectors; wherein, for an equivalent code snippet, equivalent code snippets corresponding to the same original code snippet are regarded as positive samples, and equivalent code snippets corresponding to other original code snippets are regarded as negative samples; The model fine-tuning training module is configured to train the code query model using code snippets with code annotations. The code query model includes a text encoder and a code encoder pre-trained by the pre-training module, wherein the text encoder represents the input code annotations as query vectors. The code search module is configured to receive an input query statement, input the query statement into a trained code query model, and retrieve recommended code snippets from the code repository.

[0052] The detailed design or preferred implementation of the system's code preprocessing module, pretraining module, model fine-tuning training module, and code search module can be found in the code preprocessing section, comparison pretraining section, model fine-tuning training section, and code search section of Embodiment 1, respectively, and will not be repeated here.

[0053] Example 3 This embodiment discloses a code search recommendation system based on contrastive learning and pre-training technology, which includes a processor and a computer-readable storage medium. The storage medium stores a computer program, and running the computer program executes all or part of the methods in Embodiment 1. The processor is responsible for running the code in the computer program.

[0054] Example 4 This embodiment demonstrates the performance testing of the designed model. Two common metrics are used to measure the effectiveness of code search: SuccessRate@k (SR@k, success rate) and MRR (Mean reciprocal rank), which are also widely used metrics in code search models.

[0055] The proposed scheme theoretically supports training code encoders using networks of any architecture. This embodiment primarily evaluates two architectures: a 2-layer bidirectional LSTM and a 6-layer Transformer (a sequence model based on an attention mechanism). In this embodiment, the dimension of both code embedding (i.e., vector representation) and text embedding is set to 256. The test results are shown in Table 1 below.

[0056]

[0057] From the comparison results in Table 1, the following characteristics can be observed.

[0058] 1) When the learning rate is 0.002, the Bi-LSTM (Bi-directional Long-Short Term Memory) model achieves the best performance on both SR@5 and MRR metrics.

[0059] 2) When the learning rate is 8.5e-5, the Transformer model achieves the best performance on the SR@5 metric, and when the learning rate is 9.5e-5, it achieves the best performance on the MRR metric.

[0060] 3) The performance of the Transformer model is much lower than that of the Bi-LSTM model. We infer that this is because in the deep learning model for code search, the code encoder and text encoder have the same model structure, which facilitates joint embedding of code embedding and text embedding. Therefore, this invention also recommends training the code encoder and text encoder using the same network model architecture.

[0061] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A code search and recommendation method based on contrastive learning and pre-training techniques, characterized in that, include: Each original code segment in the sample set is structurally transformed to generate multiple corresponding equivalent code segments with the same semantics. The original code segments do not have code comments, and the structural transformation does not change the execution result of the code segments. Using the equivalent code fragments, a code encoder is pre-trained using a contrastive learning method according to a set contrastive learning objective. The code encoder represents the input code fragments as code vectors. Specifically, for an equivalent code fragment, equivalent code fragments corresponding to the same original code fragment are considered positive samples, and equivalent code fragments corresponding to other original code fragments are considered negative samples. In the pre-trained code encoder using the contrastive learning method, the query sample q The contrastive learning objective loss is: ,in, k + As a positive sample, k i For negative samples, K The number of negative samples is -1. τ Hyperparameters; query samples q Calculated by the code encoder, the sample k + and k i Calculated by the momentum encoder; the code encoder updates the network parameters through backpropagation, and the momentum encoder updates the network parameters through momentum, where momentum is a moving average of the code encoder weights; The pre-trained code encoder is transferred to the code query model, which also includes a text encoder that represents the input code comments as query vectors; the code query model is trained using code snippets with code comments. Input the query into the trained code query model, and the model will retrieve recommended code snippets from the code repository.

2. The code search and recommendation method based on contrastive learning and pre-training techniques as described in claim 1, characterized in that, The structural transformation performed on each original code segment in the sample set includes: Parse the original code snippet and transform it into an abstract syntax tree; Perform various transformation operations on the abstract syntax tree to generate corresponding equivalent code snippets.

3. The code search and recommendation method based on contrastive learning and pre-training techniques as described in claim 2, characterized in that, The transformation operation includes one or more of the following: code compression, identifier modification, and regularization transformation.

4. The code search and recommendation method based on contrastive learning and pre-training techniques as described in claim 1, characterized in that, The objective of the contrastive learning is to maximize the consistency of vector representations between code segments with the same semantics and minimize the consistency of vector representations between code segments with different semantics.

5. The code search and recommendation method based on contrastive learning and pre-training techniques as described in claim 1, characterized in that, The network parameters of the momentum encoder θ k The update method is as follows: ； in, m ∈[0,1) is the momentum coefficient. θ q These are the network parameters of the code encoder.

6. The code search and recommendation method based on contrastive learning and pre-training techniques as described in claim 1, characterized in that, The step of training the code query model using code snippets with code comments includes: From the training dataset D Middle construction triplet <C ,Q + ,Q - As a training instance, where C This represents a code snippet. Q + 、Q - These represent positive query samples and negative query samples, respectively. Simultaneous prediction using code query model <C ,Q + >and The similarity is considered, and the ranking loss Φ(*) is minimized: in, θ To query model parameters for the code, sim The similarity between the code snippet and the query sample. β It is a constant residual. c for C The vector representation of , q + and q - They are respectively Q + and Q - The vector representation of .

7. The code search and recommendation method based on contrastive learning and pre-training techniques as described in claim 1, characterized in that, The step of inputting the query statement into the trained code query model and retrieving recommended code snippets from the code repository includes: Input the query statement into the text encoder to obtain the query vector, input each code fragment in the code library into the code encoder to obtain the corresponding code vector, or obtain the code vector after the code fragments in the code library have been transformed by the code encoder. Calculate the similarity ranking between the query vector and each code vector, and recommend the code fragment corresponding to at least one code vector with the highest similarity.

8. A code search and recommendation system based on contrastive learning and pre-training techniques, characterized in that, It includes a code preprocessing module, a pre-training module, a model fine-tuning and training module, and a code search module, among which: The code preprocessing module is configured to: perform structural transformation on each original code segment in the sample set to generate multiple corresponding equivalent code segments with the same semantics; the original code segments do not have code comments, and the structural transformation does not change the execution result of the code segments; The pre-training module is configured to: utilize the equivalent code snippets generated by the code preprocessing module and the configured contrastive learning objective to pre-train a code encoder using a contrastive learning method. The code encoder represents the input code snippets as code vectors. Specifically, for an equivalent code snippet, equivalent code snippets corresponding to the same original code snippet are considered positive samples, and equivalent code snippets corresponding to other original code snippets are considered negative samples. In the pre-training of the code encoder using the contrastive learning method, the query sample... q The contrastive learning objective loss is: ,in, k + As a positive sample, k i For negative samples, K The number of negative samples is -1. τ Hyperparameters; query samples q Calculated by the code encoder, the sample k + and k i Calculated by the momentum encoder; the code encoder updates the network parameters through backpropagation, and the momentum encoder updates the network parameters through momentum, where momentum is a moving average of the code encoder weights; The model fine-tuning training module is configured to train the code query model using code snippets with code annotations. The code query model includes a text encoder and a code encoder pre-trained by the pre-training module. The text encoder represents the input code annotations as query vectors. The code search module is configured to: receive an input query statement, input the query statement into a trained code query model, and retrieve recommended code snippets from the code library.

Images