System and Method for Natural Language Code Search

Abstract

An embodiment is directed to converting a natural language query into a programming language code snippet that semantically represents the query. The embodiment includes a cascaded neural network including an encoder network and a classifier network. The encoder network is faster but less accurate than the classifier network. The encoder network is trained to identify code candidates from a large set of code snippets using a contrastive learning framework. The classifier network is trained to identify code snippets that semantically represent the query from the code candidates using a binary classifier.

Description

Technical Field

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 252,393, filed Oct. 5, 2021, and U.S. Non-Provisional Patent Application No. 17 / 587,984, filed Jan. 28, 2022, which are hereby incorporated by reference in their entirety.

[0002] This embodiment generally relates to machine learning systems and natural language processing (NLP), and more specifically, to searching for code snippets using natural language.

Background Art

[0003] Artificial intelligence (AI) models are widely used in a variety of applications. Some AI models may be used to search for and / or generate code snippets in a programming language in response to natural language input. For example, the natural language input may describe a function such as "filter the sales records that occurred at the zip code 94070", and the AI model may generate or search for a code segment (e.g., Python, C#, etc.) that implements this function. Existing code generation systems have focused on either improving the speed of natural language search or improving the accuracy of natural language search. However, these existing natural language search methods are greatly troubled by the trade-off between search efficiency and comprehensiveness. (Registered Trademark)

Brief Description of the Drawings

[0004]

Figure 1

[0005]

Figure 2

[0006] ​

Figure 3

[0007]

Figure 4

[0008] In the figures, elements having the same reference numerals have the same or similar functions. **DETAILED DESCRIPTION OF THE INVENTION**

[0009] Natural language queries are used, for example, to improve searches in various fields such as web searches, database searches, legal searches, etc. There is also interest in using natural language queries to search a large set of code snippets. Organizations with large code repositories can benefit from indexing and searching via code and may reuse code that has been found to function properly. In some recent approaches to natural language search of code and code snippets, a search model from text to code is trained to search for samples of code snippets by leveraging pairs of natural language and source code sequences.

[0010] One approach to training the model involves using a contrastive learning framework. The model may be a neural network of fast encoders, also referred to as fast encoders. In the contrastive learning framework, pairs of semantically matching natural language and program language sequences are pulled together, while pairs that do not semantically match are pushed apart. The fast encoder neural network may use contrastive learning. The fast encoder neural network may be efficient for scenarios that involve searching for a large number of candidate code snippets at the expense of semantic matching accuracy.

[0011] Another approach to training a model is to use a binary classifier. This type of model uses a trained binary classifier that receives natural language and programming language sequences as input and predicts whether the natural language and programming language sequences semantically match. A model that uses a binary classifier may be considered a slow classifier. The slow classifier, while more accurate, can become infeasible when searching for a large number of candidate code snippets due to the amount of time it takes for the model to analyze code snippets against natural language sequences. In other words, a model trained using a contrastive learning framework can be at least 10 times faster, but may be at least 10 times less accurate than a model that uses a binary classifier.

[0012] To improve the natural language search of a large number of code snippets, embodiments are directed to a cascaded neural network model that includes both a fast encoder model and an accurate classifier model. The cascaded neural network model improves the natural language search efficiency of a large set of code snippets. Specifically, the cascaded neural network model is a hybrid approach that combines a fast encoder network and a slow classifier network. First, the encoder network determines the top K code candidates from a set of code snippets based on a natural language query. Second, the top K code candidates pass through a slow classifier network that pairs each code candidate with the natural language query and generates a confidence score for each pair. The code snippet with the highest confidence score may be the code snippet that semantically matches the natural language query.

[0013] The number K may represent a threshold that identifies the number of code candidates that the encoder network can generate. The K threshold is preferably much smaller than the size of the set of code snippets. If the K threshold is too small, there is a high likelihood of missing the correct code snippet, and if the K threshold is too large, it may be infeasible to efficiently execute the second-stage slow classifier.

[0014] In some embodiments, the memory overhead for storing the high-speed encoder network and the low-speed classifier network may be minimized by sharing or partially sharing the weights of the network. For example, the transformer encoders of the high-speed encoder and the low-speed classifier may be shared by training a transformer encoder to be used in both the high-speed encoder network and the low-speed classifier network.

[0015] As used herein, the term "network" may include any hardware or software-based framework including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented in or with it.

[0016] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, the module may be implemented on one or more neural networks.

[0017] FIG. 1 is a simplified diagram of a computing device implementing a code generator according to some embodiments described herein. As shown in FIG. 1, computing device 100 includes a processor 110 coupled to a memory 120. The operation of computing device 100 is controlled by processor 110. Also, although computing device 100 is shown as having only one processor 110, processor 110 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits, graphics processing units (GPUs), etc. within computing device 100. Computing device 100 may be implemented as a stand-alone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0018] Memory 120 may be used to store software executed by computing device 100 and / or one or more data structures used during the operation of computing device 100. Memory 120 may include one or more types of machine-readable media. Some common forms of machine-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAMs, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0019] Processor 110 and / or memory 120 may be arranged in any suitable physical arrangement. In some embodiments, processor 110 and / or memory 120 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, processor 110 and / or memory 120 may include distributed, virtualized, and / or containerized computing resources and multiple processors. Matching such embodiments, processor 110 and / or memory 120 may be located in one or more data centers and / or cloud computing facilities.

[0020] In some examples, memory 120 may include a non-transitory tangible machine-readable medium that includes executable code that, when operated on by one or more processors (e.g., processor 110), may cause the one or more processors to execute methods described in further detail herein. For example, as illustrated, memory 120 may include instructions for a natural language (NL) processing module, such as code generator 130, that may be used to implement and / or emulate systems and models and / or implement any method further described herein. In some examples, code generator 130 may receive input 140, such as natural language text, a query, or computer code, via data interface 115. Data interface 115 may be either a user interface that receives input 140 from a user or a communication interface that receives or retrieves input 140 stored in another memory storage device, such as memory 120 or a database. Code generator 130 may generate output 150, such as a programmable language (PL) sequence, code, or code snippet that is semantically equivalent to the natural language text or query. In some embodiments, code generator 130 may include a cascaded neural network that includes encoder network 132 and classifier network 134 such that the output of encoder network 132 may be input, in part, to classifier network 134.

[0021] Figure 2 is a simplified diagram 200 of a code generator according to some embodiments. As illustrated in FIG. 2, the code generator 130 includes an encoder network 132 and a classifier network 134. The code generator 130 receives a natural language query or a natural language query 202 of text. The natural language query 202 may be the input 140 discussed in FIG. 1. The natural language query 202 may be text written or spoken by a human, such as "filter the sales records that occurred at the zip code 94070", which the code generator 130 can convert into a programming language sequence such as a code snippet. The code generator 130 passes the natural language query 202 through the encoder network 132. The encoder network 132 may generate K code candidates 204A-204K. The code candidates 204A-204K may be code snippets of a programming language that semantically represent the natural language query 202 and / or semantically match the natural language query 128. The classifier network 134 may receive pairs of the natural language query 202 and the code candidates 204A-204K. Each pair of the pairs may include one of the candidates 204A-204K and the natural language query 202. The classifier network 134 may generate a code snippet 206 that is a semantic representation of the natural language query 202.

[0022] In some embodiments, the encoder network 132 is significantly faster, e.g., at least 10 times or more, than the classifier network 134. In fact, due to the speed of the encoder network 132, the encoder network 132 may quickly determine code candidates 204A - 204K from a large set of available code snippets. On the other hand, the classifier network 134 is slower than the encoder network 132 but significantly more accurate, e.g., at least 10 times or more accurate, in identifying code snippets that semantically match the natural language query 202. As shown in FIG. 2, the classifier network 134 receives pairs of code candidates 204A - 204K and the natural language query 202 and identifies the code snippet 206 that is the semantic representation of the natural language query 202. By using a hybrid approach that includes the encoder network 132 and the classifier network 134, the code generator 130 improves the speed and accuracy for determining the code snippet 206 that is the semantic representation of the natural language query 202.

[0023] In some embodiments, the encoder network 132 may be or include a BERT (bidirectional encoder representations from transformers) or a variant of the BERT network. The BERT network or a variant of the BERT network may be pre - trained with programming language sequences of various programming languages to extract code snippets from text inputs. Exemplary pre - trained BERT networks are GraphCodeBERT or CodeBERT. Exemplary programming languages are Ruby, JavaScript (Registered Trademark) , Go, Python (Registered Trademark) , Java (Registered Trademark)It may also be, for example, C, C++, C#, Php, etc. During the training phase, in order to recognize the code candidate 204, the encoder network 132 may further be trained in a contrastive learning framework using a bimodal dataset. In the bimodal dataset, positive pairs of representations of natural language queries and programming language sequences with matching meanings are pulled together. On the other hand, representations of negative pairs, which are randomly paired natural language queries and programming language sequences, are pushed apart. A contrastive loss function, such as the infoNCE loss function, may be used to train the encoder network 132 and is reproduced below. [Number] In the formula, f θ (x i ) is the dense representation of the natural language input x i , y i is the corresponding semantically equivalent programming language sequence, N is the number of training examples in the bimodal dataset, σ is the temperature hyperparameter, and B indicates the current training mini-batch. The encoder network 132 may be trained until the contrastive loss function is minimized.

[0024] Once trained, the encoder network 132 may receive a set of candidate code snippets, shown as code snippet 208 [Number] . The code snippet 208 may include potential code snippets that can correspond to various natural language queries, a universe of code snippets, available code snippets, etc. The code snippet 208 has an index shown as the code snippet index 210 [Number] It may be encoded. The code snippet index 210 may be an index of the encoding of each code snippet within the code snippet 208. The encoder network 132 may, for example, encode a set of code snippets offline before the code generator 130 receives the natural language query 202 that determines the code snippet 206. The code snippet index 210 may be stored anywhere within the encoder network 132 or within the memory 120 described in FIG. 1.

[0025] In some embodiments, after generating the code snippet index 210, the encoder network 132 receives the natural query x i (natural language query 202), computes f θ (x i ), queries the code snippet index 210, and returns a code snippet from C (code snippet 208) corresponding to the nearest neighbor(s) within the code snippet index 210. The neighbor(s) may be computed using a distance metric determined by a similarity function, such as a cosine similarity function. Then, a rank r i assigned to the correct code snippet from the set of code snippets C (code snippet 208) for the natural language query x i is used to compute the MRR (mean reciprocal ranking) metric

Number

[0026] In some embodiments, classifier network 134 may also be or include a variant of BERT (Bidirectional Encoder Representations from Transformers), or a BERT network. The BERT network or a variant of the BERT network may be pre-trained on programming language sequences to extract code snippets from text inputs. Exemplary pre-trained BERT networks may be GraphCodeBERT or CodeBERT, and exemplary programming languages may be Ruby, JavaScript (Registered Trademark) , Go, Python (Registered Trademark) , Java (Registered Trademark) , C, C++, C#, Php, etc.

[0027] Classifier network 134 may receive as input a natural language query x i and a programming language sequence y j (one of code candidates 204A - 204K or another code sequence), and may encode the natural language input x i and the code sequence together and perform binary classification. The binary classification may predict whether the natural language input x i and the code sequence y j semantically match. In some embodiments, classifier network 134 may receive a concatenation of the natural language input x i and the code sequence y j , [x i ;y j , etc.

[0028] The classifier network 134 may be trained for binary classification using training batches. The training batches may include pairs, each pair including a natural language query and a code snippet. The training batches may be for a bimodal dataset, where positive pairs indicate a semantic match between the natural language query and the code snippet, and negative pairs indicate a semantic mismatch. A set of pairs including a natural language query and a semantically programmed language sequence.

number

number

[0029] Training mini-batches of positive pairs

number

number

[0030] Once trained, classifier network 134 may determine code snippet 206 from natural language query 202 and code candidates 204. For example, during inference, classifier network 134 can receive a plurality of pairs as input, each pair being a set of candidate code snippets [Number] (from code candidates 204A - 204K) of natural language sequence x i (e.g., natural language query 202) and code snippet y j (one of code candidates 204A - 204K). Classifier network 134 may generate a confidence score for each pair and rank each code candidate within code candidates 204A - 204K according to that confidence score. The confidence score may be, for example, a probability having a scale from 0 to 1, where a value close to 1 indicates a high probability of match and a value close to 0 indicates a high probability of mismatch. The code snippet y corresponding to the pair with the highest score j (the code candidate among code candidates 204A - 204K) may be a semantic match with natural language sequence x i (natural language query 202).

[0031] As described above, the code generator 130 discussed in this specification includes a cascade of networks such as an encoder network 132 and a classifier network 134, and combines the speed of the fast encoder network 132 and the accuracy of the classifier network 134 in a two-stage process. In the first stage, the encoder network 132 receives a natural language query 202 and generates code candidates 204A - 204K from a set C of code snippets (code snippet 208). The encoder network 132 may determine an encoding of the natural language query 202 and use a distance function to match the encoding with a code snippet index 210 of the code snippets 208. In some embodiments, the encoder network 132 may determine K code candidates 204A - 204K, where K is a configurable candidate threshold that can be a hyperparameter. Typically, the K candidates are the top candidates having the closest distance within the code snippet index 210 to the encoding of the natural language query 202.

[0032] In the second stage, the code candidates 204 are paired with the natural language query 202. Exemplary pairs may be 202 - 204A, 202 - 204B,..., 202 - 204K. The classifier network 134 receives the pairs 202 - 204A, 202 - 204B,..., 202 - 204K. For each pair of 202 - 204A, 202 - 204B,..., 202 - 204K, the classifier network 134 returns a confidence score that the natural language query 202 semantically matches the corresponding one of the code candidates 204A - 204K using a binary classifier. Based on the confidence scores associated with the pairs 202 - 204A, 202 - 204B,..., 202 - 204K, the classifier network 134 selects a code snippet 206 that semantically matches the natural language query 202. In some examples, the code snippet 206 may correspond to the pair having the highest confidence score.

[0033] As described above, although the encoder network 132 is computationally fast, it is less accurate than the classifier network 134 when determining code snippets that semantically match the natural language query. In a scheme where K << |C|, sequentially adding the classifier network 134 together with the encoder network 132 may add a slight computational overhead. The second stage where the classifier network 134 refines the code candidates 204A - 204K improves the retrieval performance when the value of K is set such that the re - invocation of the encoder network 118 is moderately high. In some embodiments, K may be a hyperparameter. Setting a very low K increases the likelihood of overlooking the code snippet 206 within the set of code candidates 204 passed to the classifier network 134. On the other hand, setting a high K makes the scheme infeasible for retrieval by the classifier network 134. However, setting K to a value such as 10 already provides a significant gain in retrieval performance compared to conventional code generation systems that achieve only a slight improvement when K is set to 100 or more.

[0034] In some embodiments, the encoder network 132 and the classifier network 134 may share a portion of the neural network structure. For example, the encoder network 132 and the classifier network 134 may share the weights of the layers within the transformer encoder in the BERT network. By sharing the neural network structure, the memory overhead incurred by the encoder network 132 and the classifier network 134 is minimized. Sharing of the neural network structure by the encoder network 132 and the classifier network 134, for example, the transformer layer, is infoNCE as shown in Equation (1)

Number

Number

[0035] In a shared embodiment, the classifier network 134 may have an additional classification layer or head that determines confidence scores for pairs 202-204A, 202-204B, ..., 202-204K. The classifier network 134 will include a classification head on top of the transformer encoder. Further, the shared neural network structure may receive three inputs, the natural language query 202, the set C of candidate code snippets (code snippet 208), and the pairs 202-204A, 202-204B, ..., 202-204K. In a shared embodiment, two paths are taken through the shared layers of the network, with the natural language query 202 being the input in the first path and the pairs 202-204A, 202-204B, ..., 202-204K being input in the second path.

[0036] FIG. 3 is a schematic diagram of a method 300 for training a code generator according to some embodiments. One or more of the processes 302-304 of the method 300 may be implemented in the form of executable code stored in a non-transitory tangible machine-readable medium that can cause one or more processors to execute one or more of the processes 302-304 when at least partially executed by the one or more processors.

[0037] In process 302, the encoder network is trained. For example, the encoder network 132, which can be a pre-trained BERT network, may be further trained in a contrastive learning framework to identify code snippets that semantically match natural language sequences. The contrastive loss function used to train the encoder network 132 may be a contrastive loss function such as the infoNCE loss function. The training may include batches of negative and positive pairs, where each pair includes a natural language sequence and a programming language sequence. The training may continue iteratively until the infoNCE loss function is minimized.

[0038] In process 304, the classifier network is trained. For example, the classifier network 134, which can be a pre-trained BERT network, may be trained in binary classification to determine the probability score that a code snippet matches a natural language sequence. The cross-entropy objective function may be used to train the classifier network 134. The training may include batches of negative and positive pairs, where each pair includes a natural language sequence and a programming language sequence. The training may continue iteratively until the cross-entropy objective function is minimized.

[0039] FIG. 4 is a schematic diagram of a method 400 for generating a code snippet that is semantically equivalent to a natural language query according to some embodiments. One or more of processes 402-408 of method 400 may be implemented in the form of executable code stored in a non-transitory tangible machine-readable medium that can cause one or more processors to execute one or more of processes 402-408 when executed by the one or more processors.

[0040] In process 402, a code snippet index is generated. For example, encoder network 132 receives a number of natural language queries and code snippets 208 that can semantically correspond to them. Encoder network 132 encodes code snippet 208 and generates a code snippet index 210 corresponding to the encoded code snippet. Process 402 may occur after encoder network 132 has been trained and before encoder network 132 processes natural language query 202.

[0041] In process 404, code candidates for a natural language query are generated. For example, encoder network 132 may receive natural language query 202 and generate an encoding for natural language query 202. Encoder network 132 may use code snippet index 210 to match the encoding of natural language query 202 to the encoding of code snippet 208 and identify code candidates 204A - 204K that can semantically match natural language query 202. As described above, the number of code candidates 204A - 204K may be set using a hyperparameter number K.

[0042] In process 406, pairs including a natural language query and code candidates are generated. For example, code generator 130 may generate pairs 202 - 204A, 202 - 204B,..., 202 - 204K, where each pair includes natural language query 202 and one of code candidates 204A - 204K.

[0043] In process 408, a code snippet is determined. For example, classifier network 104 may receive pairs 202 - 204A, 202 - 204B,..., 202 - 204K and determine a confidence score for each pair. The pair with the highest confidence score may be code snippet 206 that semantically matches natural language query 202.

[0044] Some examples of computing devices, such as computing device 100, may include a non-transitory tangible machine-readable medium that includes executable code that, when operated by one or more processors (e.g., processor 110), may cause the one or more processors to execute the processes of methods 300-400. Some common forms of machine-readable media that may include the processes of methods 300-400 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other media adapted to be read by a processor or computer.

[0045] This description and the accompanying drawings, which illustrate aspects, embodiments, implementations, or applications of the invention, should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of the present disclosure. Similar numerals in two or more figures represent the same or similar elements.

[0046] In this description, specific details are set forth that describe some embodiments consistent with the present disclosure. Numerous details are set forth to provide a complete understanding of the embodiments. It will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative but not limiting. Those skilled in the art may recognize other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless specifically described otherwise or unless one or more of the features render an embodiment non-functional.

[0047] Exemplary embodiments have been shown and described, but a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the following claims, and the claims should be construed broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A method for converting a natural language query into a code snippet of a programming language, comprising: generating a code snippet index from a plurality of code snippets in an encoder network; using the code snippet index and the encoder network to generate a plurality of code candidates for the natural language query; generating pairs including the natural language query and code candidates from the plurality of code candidates for the natural language query; using a classifier network sequentially following the encoder network and the pairs to determine the code snippet of the programming language for the natural language query, wherein the code snippet is a semantic representation of the natural language query.

2. The method according to claim 1, further comprising training the encoder network to determine the code candidates using a contrastive loss function.

3. The method according to claim 1 or 2, further comprising training the classifier network to determine the code snippet from the pairs using a cross-entropy objective function.

4. The method according to claim 1 or 2, wherein the encoder network is one order of magnitude faster and one order of magnitude less accurate than the classifier network.

5. The method according to claim 1 or 2, wherein the encoder network is trained with a loss function different from that of the classifier network.

6. The method according to claim 1 or 2, wherein the encoder network shares a part of the neural network structure with the classifier network.

7. Generating the plurality of code candidates comprises: generating an encoding from the natural language query; using the code snippet index to determine encodings of the plurality of code candidates within a distance determined by a distance function from the encoding of the natural language query.

8. Determining the code snippet comprises: determining a confidence score that each pair of code candidates is a semantic representation of the natural language query; ranking the confidence scores of the pairs. and selecting the pair of code candidates corresponding to the highest confidence score as the code snippet, which is the semantic representation of the natural language query.

9. 1. A system for converting natural language queries into programming language code snippets, comprising: a memory configured to store a cascaded neural network; a processor coupled to the memory, the processor configuring the cascaded neural network to: generating a code snippet index from a plurality of code snippets in an encoder network of the cascaded neural network; generating a plurality of code candidates for the natural language query using the code snippet index and the encoder network; generating, from the natural language query and the plurality of code candidates, pairs including the natural language query and a code candidate from the plurality of code candidates; and using the classifier network and the pair of cascaded neural networks to determine the code snippet in the programming language for the natural language query, the code snippet being a semantic representation of the natural language query.

10. The processor: training the encoder network to determine the code candidates with a contrastive loss function; 10. The system of claim 9, further configured to: train the classifier network to determine the code snippet from the pair using a cross-entropy objective function.

11. 11. The system of claim 9 or 10, wherein the encoder network is an order of magnitude faster and an order of magnitude less accurate than the classifier network.

12. 11. The system of claim 9 or 10, wherein the encoder network shares a portion of a neural network structure with the classifier network.

13. To generate the code candidates, the processor: generating an encoding from the natural language query; The system according to claim 9 or 10, further configured to determine encodings of the plurality of code candidates within a distance determined by a distance function from the encoding of the natural language query using the code snippet index.

14. To determine the code snippet, the processor determines a confidence score that each pair of code candidates is a semantic representation of the natural language query, ranks the confidence scores of the pairs, and selects, as the code snippet that is the semantic representation of the natural language query, the pair of code candidates corresponding to the highest confidence score. The system according to claim 9 or 10, further configured to perform the above.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations for converting a natural language query into a code snippet of a programming language, the operations including generating a code snippet index from a plurality of code snippets in an encoder network, using the code snippet index and the encoder network to generate a plurality of code candidates for the natural language query, generating pairs including the natural language query and code candidates from the natural language query and the plurality of code candidates, and determining, using a classifier network and the pairs, the code snippet of the programming language for the natural language query, where the code snippet is a semantic representation of the natural language query. A non-transitory computer-readable medium including the above.

16. The operations include training the encoder network to determine the code candidates with a contrastive loss function, and training the classifier network to determine the code snippet from the pairs using a cross-entropy objective function. The non-transitory computer-readable medium according to claim 15, further including the above.

17. The encoder network is one order of magnitude faster and one order of magnitude less accurate than the classifier network. The non-transitory computer-readable medium according to claim 15 or 16.

18. The non-transitory computer-readable medium according to claim 15 or 16, wherein the encoder network shares a part of the neural network structure with the classifier network.

19. Generating the code candidates includes generating an encoding from the natural language query, and using the code snippet index to determine encodings of the plurality of code candidates within a distance determined by a distance function from the encoding of the natural language query. The non-transitory computer-readable medium according to claim 15 or 16.

20. Determining the code snippet includes determining a confidence score that each pair of code candidates is a semantic representation of the natural language query, ranking the confidence scores of the pairs, and selecting, as the code snippet that is the semantic representation of the natural language query, the pair of code candidates corresponding to the highest confidence score. The non-transitory computer-readable medium according to claim 15 or 16.

Images