Multilingual code line completion system

Abstract

A code completion tool generates candidate sequences using a neural transformer model to complete lines of source code. The neural transformer model is trained on a large unsupervised dataset of source code programs written in a variety of different programming languages using a conditional language modeling objective. The neural transformer model is used in a beam search that predicts the most likely candidate sequences for a piece of code being developed.

Description

BACKGROUND

[0001] Software development environments are often used to assist software developers (i.e., users, programmers, etc.) in developing program code. A software development environment can include a source code editor and other tools that developers use to write and test their programs. Some software development environments include a code completion feature that can present a list of possible candidates to help the developer based on one or more characters (e.g., letters, symbols, etc.) that the developer types in the source code editor while editing the code. A pop-up menu with several suggested code elements that the developer can use can appear. This help is beneficial because it can speed up development time and reduce common errors, such as spelling errors.

[0002] However, the automatic code completion feature can have problems when the code completion system is unable to recognize code elements that are out of the vocabulary, requires a large amount of memory, takes too long to generate a list of candidates, and / or generates an irrelevant list of candidates. SUMMARY

[0003] This summary is provided to introduce some concepts of the present disclosure in a simplified form that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] A multi-lingual code line completion system is used to generate candidates that are most likely to complete a line of source code during a source code editing session. The predicted string to complete the line of code can include various types of elements, such as local variables, methods, parameters, keywords, and delimiters arranged in an ordered sequence. The system uses a model to predict the ordered sequence that is trained on a large unsupervised dataset of source code programs written in different programming languages (e.g., C, Java, Python, C++), using conditional language modeling objectives.

[0005] Each source code program in the training dataset does not need to be written in the same programming language. The training dataset can be composed of many source code programs, each of which can be written in a different programming language. Each source code program in the training dataset is encoded into a sequence of tokens and / or sub-tokens. Elements that are frequently used in the programming language are encoded into tokens, while elements that do not appear frequently are encoded into combinations of characters called sub-tokens. This reduces the need to store a large vocabulary and provides better accuracy for out-of-vocabulary tokens.

[0006] The multilingual code line completion system is based on a neural transformer model. In one aspect, the neural transformer model is composed of a plurality of decoder blocks. The decoder blocks include a multi-headed self-attention layer coupled to a multi-layer one-dimensional convolutional neural network. Layer normalization is applied before and after the multi-headed self-attention layer to reduce training time of the neural transformer model.

[0007] Beam search is used to generate candidate sequences. Beam search uses the top k sub-tokens / tokens identified from each iteration of the neural transformer model to expand partial candidate sequences of tokens / sub-tokens that can complete a line of source code. Beam search generates a search tree, but only keeps the top k nodes at each level of inference to expand the search. The search ends when the end of line token appears as the most likely prediction.

[0008] These and other features and advantages will be apparent from a reading of the following detailed description and a review of the associated drawings. It is to be understood that both the foregoing general description and the following detailed description are merely explanatory and are not restrictive of claimed aspects. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 An exemplary code completion system is illustrated that has a training phase that generates a neural transformer model and an inference phase that uses the neural transformer model to predict one or more candidate sequences to complete a line of code.

[0010] Figures 2A-2B is a schematic diagram illustrating an exemplary system and method for training a neural transformer model for code line completion.

[0011] Figure 3 is a schematic diagram illustrating an exemplary architecture of the transformer block shown in FIG. 2.

[0012] Figures 4A-4B is a schematic diagram illustrating an exemplary architecture of the inference phase.

[0013] Figures 5A-5B is a flow diagram illustrating an exemplary method for training a neural transformer model for code completion.

[0014] Figures 6A-6B is a flow diagram illustrating an exemplary method for utilizing a neural transformer model in an inference phase.

[0015] Figure 7 is a schematic diagram illustrating an exemplary user interface showing code completion candidates for a line of code in an exemplary source code program.

[0016] Figure 8 is a schematic diagram illustrating an exemplary beam search that generates a search tree of candidate sequences.

[0017] Figure 9is a block diagram illustrating an operating environment. DETAILED DESCRIPTION

[0018] SUMMARY

[0019] The disclosed subject matter relates to the generation of completions of lines of source code in a program development environment. Code completion is a tool that attempts to predict the next string of characters that a developer (e.g., user, end user, programmer, etc.) might type in a source code editor. A line of source code can be composed of various elements (e.g., keywords, delimiters, variables, methods, constants, operators, etc.) that are combined in a particular order according to the syntax of the underlying programming language to form an expression. A line of source code can be a method call, a program statement, a definition, an expression, etc. A line of source code does not include blank lines or comment lines and ends with an end-of-line character.

[0020] A code completion tool uses a neural network machine learning model to predict the next string of code elements to complete a line of source code. A line of source code refers to a physical line of source code that ends with an end-of-line character and does not include blank lines and comment lines. The model is trained on an unsupervised dataset that can contain source code from different programming languages (i.e., multi-lingual). Unsupervised learning derives inferences from a dataset consisting of input data without labeled responses. A vocabulary is formed from these datasets that includes tokens and / or sub-tokens found in source code files. A token is a single element in a programming language syntax such as a keyword, variable name, operator, delimiter, etc.

[0021] Unlike natural languages (e.g., English, etc.), programmers sometimes use arbitrary, complex, and long names to represent variables, functions, or other code elements, which can result in a very large vocabulary for the model when large source code programs are used to train the model. To reduce the size of the vocabulary, tokens that occur infrequently are split into sub-tokens. A sub-token is a portion of a token between a token and a single character. Sub-tokens are used to account for rare or unseen tokens (i.e., out-of-vocabulary tokens) that can occur in a target source code program. The use of sub-tokens allows the model to learn and generate out-of-vocabulary tokens.

[0022] Byte pair encoding is a data compression technique in which pairs of Unicode characters that most frequently co-occur simultaneously across the entire training source code dataset are replaced with out-of-vocabulary characters. When applied recursively, byte pair encoding results in the extraction of tokens / sub-tokens in frequently co-occurring sequences of Unicode characters. In one aspect, byte pair encoding is used to extract ordered sequences of Unicode characters from the syntax representation of source code programs of the training dataset to form tokens and sub-tokens. The ordered sequences represent source code snippets with T tokens / sub-tokens. The ordered sequences of tokens / sub-tokens are converted to token / sub-token embeddings and position embeddings, which are vector representations of the source code snippets.

[0023] In one aspect, the neural network machine learning model is a multi-layer transformer model. A transformer is a neural network architecture that processes dependencies between its inputs and outputs through attention and convolution without using recurrent neural networks (RNNs) (e.g., long short-term memory (LSTM) networks). One disadvantage of RNN-based systems is the sequentiality of RNNs, where each hidden state depends on the previous hidden state. This makes RNN-based systems difficult to parallelize and unable to take advantage of fast computing devices, such as graphics processing units. In addition, RNN-based systems are unable to learn long-range dependencies in long input and output sequences. Transformers overcome these obstacles through attention. Attention is a mechanism that is used to identify which parts of an input sequence are relevant to each token / sub-token in an output sequence. The attention mechanism allows a transformer to access the entire input sequence at once.

[0024] Transformers can act as encoders or decoders, where encoders map input sequences of symbolic representations to sequences of continuous representations, and decoders generate output sequences of symbols from sequences of continuous representations. Encoder-decoder architectures are less suitable for conditional code generation or code completion tasks, and more suitable for machine translation and patch generation types of tasks. A variant of the transformer model is used that consists of decoder blocks with masked self-attention and convolutional layers.

[0025] Beam search is used to generate one or more candidate sequences to complete a line of source code. Beam search uses the probability distribution generated by the neural transformer model to identify the top-k tokens / sub-tokens that are likely to be the next token or sub-token in a candidate sequence. Beam search expands the search by instantiating new partial sequences with each selected token / sub-token identified by the probability distribution of the neural transformer model. The search continues to generate new partial sequences from the top-k tokens / sub-tokens identified by the output distribution of the neural transformer model until the search ends. The search can end when the end-of-line token appears as the most likely next token.

[0026] The task of code line sequence completion is to estimate the conditional probability distribution P (Output | Input) by maximizing the likelihood of the training data, i.e.,t predict the sequence m of response tokens / subtokens given the ordered sequence of tokens / subtokens x = {x0, x1, x2,..., xt,..., xN}t=0...N t

[0027] .

[0028] In one aspect, a multi-headed self-attention multi-layer transformer-decoder neural network model is used to estimate this probability distribution for a source code corpus using unsupervised autoregressive (AR) techniques. The modeling objective is to maximize the likelihood of:

[0029] ,

[0030] where k is the size of the context window, the conditional probability P is modeled using a neural transformer model with parameters The parameters can include the attention length, the number of attention heads, the number of decoder blocks, the embedding dimension, the embedding matrix, and the number of hidden units per layer trained using a stochastic gradient descent optimization procedure.

[0031] The multi-layer transformer decoder produces the output distribution of tokens / subtokens as follows:

[0032] h 0 = U W e +W p ，

[0033] h 1 = transformer block (h t-1 ）,l=1…n，

[0034] ，

[0035] where is the context vector of tokens / subtokens, n is the number of layers, T is the length of the ordered sequence, W e is the tokens / subtokens embedding matrix, W p is the position embedding matrix, e denotes the embedding of a token / subtoken in the vocabulary, p denotes the embedding of a token / subtoken position.

[0036] ​Attention is now directed to further discussion of systems, devices, components, and methods used in code completion systems.

[0037] Machine learning code completion system

[0038] Figure 1 A block diagram of an exemplary code completion system 100 in which various aspects of the present application can be practiced is illustrated. As shown, the system 100 includes a training phase 102 to train a transformer model 122 in a code line completion system and an inference phase 104 to utilize the transformer model 122. The training phase 102 can utilize a source code repository 106, a source code extraction component 108, a parser 112, a token / subtoken sequence extraction component 116, and a model training and validation component 120. Figure 1

[0039] In one aspect, the training phase 102 pre-trains the transformer model from different corpora of unlabeled source code programs. This is referred to as unsupervised learning because the model derives inference results from input data without labeled responses. The source code extraction component 108 extracts selected source code programs 110 from the source code repository 106 to obtain training and validation datasets. The source code repository 106 can be a private or public file archive and web hosting facility that stores a large amount of source code. The source code repository 106 can be structured as a version control system such as GIT, Mercurial, etc. The source code programs residing in the source code repository 106 are different and can be written in different programming languages.

[0040] The source code extraction component 108 obtains several selected source code programs 110 that can be written in the same or different programming languages. Programming languages use context-free grammars that are a set of rules that describe all possible strings in a formal programming language. The selected source code programs 110 can be from different domains such as, but not limited to, scientific computing, web development, dataflow programming, machine learning, etc.

[0041] The parser 112 converts each selected source code program 110 into a concrete syntax tree 114. The concrete syntax tree 114 represents the source code text in parsed form. The concrete syntax tree 114 can also be a parse tree. The parser 112 can be a parser, a part of a front-end compiler, a part of a language compiler, or a part of a compilation tool. The concrete syntax tree 114 represents the syntactic structure of a program in a hierarchical or tree structure. The concrete syntax tree 114 is a tree data structure that represents the syntactic structure of a program. n ​The metatree data structure includes nodes representing the constructions in the grammar of the programming language of the program. The concrete syntax tree 114 includes a root node, multiple internal nodes, and multiple terminal nodes. Terminal nodes represent tokens. Tokens are symbols representing operands or operators. The concrete syntax tree 114 differs from the abstract syntax tree where terminal nodes represent operands.

[0042] The specific syntax tree 114 for the selected source code program 110 is passed to the token / subtoken sequence extraction unit 116. The token / subtoken sequence extraction unit 116 parses the specific syntax tree 114 for each source code program and outputs it. T A sequence of tokens and / or sub-tokens. In one aspect, the token / sub-token sequence extraction component 116 performs byte-pair encoding to extract frequently occurring tokens and extract sub-tokens from less frequently occurring tokens. A sub-token is a part of a token.

[0043] For example, the following line of source code:

[0044] loss=tf.reduce_sum(tf.square(linear_model - y))

[0045] It can be divided into the following sequence of tokens / subtotals, each token / subtotal separated by the character "|":

[0046] loss|=|tf|.|red|uce|_|sum|(|tf|.|squ|are|(|linear|_|model| - |y|)|)|

[0047] In this example, the token "reduce" has been split into sub-tokens "red" and "uce", and the token "square" has been split into sub-tokens "squ" and "are".

[0048] Then T- The ordered token sequence is mapped to a numeric vector, and then to an embedding. An embedding is a learned representation for a text-based token / sub-token, where tokens or sub-tokens with a common meaning share a common representation. There is one embedding and one positional embedding for each token / sub-token in the vocabulary. The token / sub-token embedding represents the learned representation for the token / sub-token. The transformer model does not read each token / sub-token sequentially; therefore, without additional positional information, it does not know the position of the token / sub-token in the sequence. The positional embedding is used to embed positional information about the token / sub-token's position in the sequence into the transformer model. The token / sub-token embedding is input to the model training and validation unit 120.

[0049] The neural transformer model 122 is used in the inference phase 104 of the code completion system. In one or more aspects, the inference phase 104 can be implemented as a function or feature integrated into a source code editor, an integrated development environment (IDE), and / or a standalone application. Code completion can be implemented as a tool or feature that can be an add-on, plug-in, extension, and / or component of a source code editor and / or IDE. In one aspect, the inference phase 104 includes a source code editor 130, a code completion component 142, and a model 122.

[0050] In one aspect, the source code editor 130 can include a user interface 132 and a parser 134. The user interface 132 includes a set of features or functions for developing (e.g., writing, editing, testing) a source code program. The user interface 132 can utilize a pop-up window to present a list 136 of possible candidates for completion, allowing the developer to browse the candidates and select one from the list. Alternatively, as the user types characters in the source code program, the candidates can appear in the line of the current source code line.

[0051] The parser 134 reads the characters entered into the source code program through the source code editor 130 and generates a corresponding concrete syntax tree 140. The parser 134 also updates the concrete syntax tree 140 as the developer creates and edits the source code in the source code editor 130.

[0052] At some point in the editing process, the user interface 132 will request candidates to complete the current line of the source code. The user interface can detect that the user has entered a particular character or string of characters and automatically initiate a request 138 for candidates to complete the line of code. The character is referred to as a marker character. In one aspect, the marker character can be an equals sign “=” or a period “.”. The user interface 132 will then send a request for candidates from the code completion component 142 to present to the developer. Alternatively, the user can request candidates by entering a particular keystroke or sequence of keystrokes, such as a combination of the CTRL key and the space bar.

[0053] In yet another aspect, the system can automatically display a single top candidate at the end of the current source code line in a dim color, regardless of the marker character. Whether or not the user decides to trigger the candidate, the system builds and continuously updates the candidate tree in the background. The candidate is automatically displayed in the user interface after the user has been idle for a period of time. If the user wants to accept the candidate, the user can type a particular keystroke or combination of keystrokes (e.g., CTRL and I) to accept the candidate. In this case, the cursor position will advance to the end of the suggested code sequence and the dim color of the candidate code will change to the normal color of the code. If the user does not want to use the candidate, the candidate disappears as the user continues to type. In this case, the system refines the code sequence according to the prefix filter of the candidate tree based on the newly typed code.

[0054] The code completion component 142 tracks characters entered into the source code editor and serves requests for candidates for completion of a line of source code. The code completion component uses the model 122 to generate candidates based on the current context of the source code in the editor. Multiple candidates are ranked according to their respective probabilities, with the candidate having the highest probability ranked first. A selected number of candidates 136 are then returned to the source code editor 130 and displayed in the user interface 132.

[0055] It should be noted that, Figure 1 Components of a system in one aspect of an environment in which various aspects of the application can be practiced are shown. However, it should be understood that Figure 1 the precise configuration of the components shown in Figure 1 various aspects and variations of the configuration shown in For example, the training phase 102 can be performed in one computing environment and the inference phase 104 can be performed in the same computing environment as the training phase 102 or in a separate computing environment. Various computing environments are described in more detail below.

[0056] Now attention is directed to Figure 2A which illustrates further details of components and processes 200 for training a neural transformer model. Reference is made to Figure 1 and 2A The source code extraction component 108 obtains source code programs to be used as training and validation datasets. Each selected source code file 202 is parsed by the syntax analyzer 112 into a concrete syntax tree 204. The token / subtoken sequence extraction component 116 traverses the concrete syntax tree 204. The token / subtoken sequence extraction component 116 can utilize a tokenizer 206 to extract tokens from each line of source code represented by the concrete syntax tree.

[0057] In one aspect, byte pair encoding is used as the tokenizer 206. Byte pair encoding is used to build a vocabulary of tokens / subtokens. Although its name uses the word “byte,” byte pair encoding operates on Unicode code points rather than byte sequences. This encoding technique divides less frequently occurring tokens into subtokens, while more frequently occurring tokens remain unchanged.

[0058] The tokens and subtokens for each line of source code are then aggregated into an ordered token / subtoken sequence consisting of T tokens / subtokens 208. In one aspect, Tare 1024 tokens, each sequence consisting of 1024 tokens / sub-tokens and representing a specific context of a source code program. Sequences from various source code programs are then input to the model training and validation component 210.

[0059] The neural network is iteratively trained such that the training dataset is passed multiple times before converging to a minimum. One epoch represents one forward and backward pass of the entire training dataset through the neural network. Since the training dataset is very large, it is divided into smaller batches. Training is iterative, with the entire dataset being passed through the neural network multiple iterations. Each training iteration includes forward propagation, loss calculation, backpropagation steps, and then the weights are updated.

[0060] The neural network has multiple layers, so it can learn more detailed relationships in the data and how features interact with each other at a non-linear level. The model architecture, training procedure, data normalization, and vocabulary encoding process are hyperparameters that are tailored to meet specific objectives. The values of the hyperparameters affect how the parameters are learned.

[0061] In one aspect, the hyperparameters can include the following parameters: (1) the dimensions of the token / sub-token and position embedding layers are 30000 x 768 and 1024 x 768, respectively;

[0062] (2) 12 transformer blocks, each consisting of two convolution, masked self-attention, and layer normalization layers; (3) for the training procedure: autoregressive with a cross-entropy loss optimization objective; sequence length of 1024 tokens / sub-tokens; batch size of 8; gradient accumulation step of 8 per weight update; Adam stochastic optimization procedure used to train the neural network; learning rate of 0.0001; (4) data normalization process: normalize all string and numeric literals, retaining the ten most frequent; (5) vocabulary encoding process: extract a joint sub-token vocabulary from the multilingual code corpus using byte pair encoding, retain the ten most frequent string and numeric literals during the byte pair encoding process and encode them as a single token; and introduce special control flow tokens to represent end of line, end of file, (decent), and indentation symbols.

[0063] The training dataset is divided into batches, and each batch of sequences is run throughout the training process. Sequences are initially converted to numerical vectors, followed by embeddings. Embeddings are mappings of discrete categorical variables to continuous numerical vectors. There is a token / sub-token embedding 212 and a position embedding 214 for each sequence. The token / sub-token embedding represents the tokens and / or sub-tokens in the sequence, while the position embedding represents the order of the tokens / sub-tokens in the sequence.

[0064] Initially, random values are used for the initial values of each token / subtoken embedding and position embedding. Thereafter, the neural transformer model 224 learns values for each embedding. Upon completion of the training phase, the embeddings for each token / subtoken and position embedding are saved into respective matrices 218, 220 for later use in the inference phase. There is a token / subtoken embedding matrix W e 218 containing an embedding vector for each token / subtoken C i , i = 0...V and a position embedding matrix W p 220 containing an embedding vector for each position P j , j = 0...T where V is the size of the vocabulary, T is the length of the token / subtoken sequence.

[0065] Each token / subtoken embedding 212 and its corresponding position embedding 214 are combined to form a context tensor 216. A tensor is a mathematical object that has indices and components that follow certain transformation rules. A tensor is a partially defined computation. It is a generalization of vectors and matrices and is represented as an n-dimensional array. The tensor in this instance represents the context of the source code program. The size of the context tensor 216 is T times the size of the embedding vector (e.g., embedding size) where T is the length of the token / subtoken sequence.

[0066] Turning to Figure 2B , the token / subtoken embeddings 212 are learned along with the parameters of the neural transformer model. The output hidden state 228 of the neural transformer model is then multiplied by a linear projection matrix A 230. The linear projection matrix A is defined as . The hidden state vector encodes the information learned by the neural transformer model 224 from the context tensor 216. Finally, the probability distribution 238 for each token / subtoken is predicted 234 by getting unnormalized logits as where is a bias vector, which are then normalized using a softmax function 236 to generate

[0067] Turning back to Figure 2A , the neural transformer model 224 can be composed of one or more transformer blocks 226A, 226B. Referring to Figure 3 , a transformer block 226 can be configured with both encoder and decoder blocks and / or only with decoder blocks. Figure 3An aspect of the neural transformer model 224 configured with multiple decoder blocks 306A, 306N is shown. The decoder blocks 306A, 306N can include a first normalization layer 308 followed by a masked self-attention layer 310 followed by a second layer normalization 312 and two layers of one-dimensional convolutional neural networks 314A, 314B.

[0068] Training of neural networks is a time-consuming task. To reduce the training time, layer normalization is used. Layer normalization normalizes the input across features. The mean and standard deviation are computed across the feature dimension. There is a first layer normalization before the masked self-attention layer 310 and a second layer normalization 312 after the masked self-attention layer 310.

[0069] Each token / subtoken flows through all the decoder blocks 306A, 306N along its own path. The masked self-attention layer 310 allows the neural network to focus on certain features or inputs. Vaswani et al. in “Attention is all you need” in Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017) describe attention as “mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of values, where the weight assigned to each value is computed by a compatibility function of the query and the corresponding key.

[0070] In an aspect of the disclosure, the masked self-attention layer 310 is composed of two or more attention heads 316A, 316B. Multi-headed self-attention heads run multiple scaled dot-product attentions in parallel. Each attention head 316A, 316B operates on a portion of the context tensor 302. Attention head 316A operates on a first segment 318A and attention head 316B operates on a second segment 320A. Each attention head 316A, 316B operates on an input sequence of n x = (x 1 ，…，x n ) operates on and computes a new sequence of the same length z = (z 1 ，…，z n ) Each output element z i is computed as a weighted sum of the linearly transformed input elements:

[0071] .

[0072] Each weight coefficient is computed using a softmax function:

[0073] ,​

[0074] where is the scaled dot-product .

[0075] The input to each attention head 316A, 316B consists of the query matrix 320A, 320B and the key matrix 322A, 322B, both of dimension T x d x where T is the code sequence length, d x is the embedding dimension. The dot-product is over the query matrix 320 and all keys from the key matrix 322 and a softmax function is applied to obtain the weights W 0 …W T 324A, 324B, resulting in the respective value matrices 326A, 326B. The resulting values from the two value matrices are then concatenated 328 and then linearized. The concatenation layer 328 takes T x d v dimensional key matrix to form T x d v dimensional matrix. The linear layer 330 takes the output of the concatenation layer 328 and applies a linear transformation according to the following equation : output = input W T +b where the input is T x d v the matrix, W is the d x ×d v dimensional matrix, b is the T x d x dimensional matrix and the output is T x d x the matrix multiplication and addition

[0076] The layer normalization 312 is then applied to the output of the masked self-attention layer 310. The output of the layer normalization 312 is then applied to the first neural network layer. In one aspect, there are two neural network layers, each consisting of a one-dimensional convolutional neural network. Given an input tensor of dimension (dx, T) and a convolution kernel g , the one-dimensional convolution operation is defined as: output = bias + input where the operation is a sliding dot-product operation.

[0077] Turning to Figure 2B , the output of the neural network at the last time step T is the hidden state vectorh T 228, which encodes information learned by the transducer blocks 226A, B about the tokens / sub-tokens. The output hidden state of the neural transducer model 224 is then multiplied by a linear projection matrix A 230 defined as The hidden state vector 228 encodes information learned by the neural transducer model from the context tensor. Finally, the probability distribution P |V| 238 is generated for each token / sub-token by obtaining non-normalized logits predictions 234 as where b k , k = 0...|V| - 1 is a bias vector, and then normalizing them using a softmax function 236.

[0078] Attention is now turned to a description of the model components used in the inference phase. The inference phase uses beam search to find the most likely candidate sequence. Beam search iteratively generates tokens / sub-tokens by invoking the neural transducer model. The output of the neural transducer model is a matrix of token probabilities for each position in the candidate sequence. Beam search focuses on k the most likely tokens at each iteration to obtain the best path to the most likely candidate sequence. At each iteration, k each of the most likely tokens is concatenated with the tokens in the previous iteration to form a partial candidate sequence.

[0079] Beam search uses a breadth-first search to construct a search tree. The search tree consists of nodes at one or more levels of inference. Each node represents a probability distribution over the tokens / sub-tokens in the model vocabulary generated by the neural transducer model. At each level, only the top k tokens / sub-tokens from the output distribution generated by the neural transducer model are expanded to the next level of inference. The variable k is pre-configured and is also referred to as the beam width. Each of the k sub-tokens / tokens is then expanded into the search which updates the current context sequence with the selected sub-token / token to input into the neural transducer model to generate an additional probability distribution for the next token in the sequence. This process is repeated until the end-of-line token is predicted as the next likely token candidate.

[0080] Turning to Figure 4A , Figure 4A shows components of the inference phase 400. The code snippet 402 is input into a source code editor which is converted into a corresponding concrete syntax tree 404. The token extractor 406 traverses the concrete syntax tree 404 to extract tokens and / or sub-tokens. An ordered sequence of length T is formed and vectorized.

[0081] Beam search 410 uses context vectors 408 to initiate inference process 412 (block 412) using the probability distribution generated from the neural transformer model P 0 …P |V| If the probability distribution indicates that the end-of-line token is the most likely token to follow in the partial candidate sequence (block 416 - YES), the top k candidate sequence is output (block 418). Otherwise, beam search 410 adopts the top k state or token / subtoken identified by the probability distribution generated from the neural transformer model in the inference process (block 420). Using the new token / subtoken in the context vector, new context vectors are generated for k each of the top c 1 、..c k vectors (blocks 422A, 422B). The new context vectors are then input into the inference process (blocks 422A, 422B, 412). Beam search 410 ends when an end-of-line token is selected as the most likely candidate to complete the partial candidate sequence.

[0082] Figure 4B An exemplary inference process 412 is illustrated. The embedding vectors for each token and subtoken in sequence 408 and their corresponding position vectors are obtained from token / subtoken embedding matrix 428 and position embedding matrix 430. The token / subtoken embedding vectors and their corresponding position embedding vectors are combined to form a context tensor 432, which is input into neural transformer model 434.

[0083] The output of neural transformer model 434 is a vector with components h0...h dh 436. The output of the transformer is multiplied by a linear projection layer 438 to generate predicted embedding vectors 440. Token / subtoken embedding vectors 448 are used as an output classification matrix to generate unnormalized predictions or logits V0...V |V| 442. The logits 442 are normalized using a softmax function 444 to generate softmax predictions 446 P0...P |V| .

[0084] Method

[0085] Attention is now directed to the description of various example methods using the systems and devices disclosed herein. The operations of these aspects can be further described with reference to the various example methods. It can be understood that the representative methods need not be performed in the order presented or in any particular order, unless otherwise specified. Additionally, various activities described with respect to the methods can be executed in serial, in parallel, or in any suitable order. In one or more aspects, the methods illustrate the operation of the systems and devices disclosed herein.

[0086] Figures 5A-5B An example method 500 is illustrated that illustrates the use of a neural transformer model for code completion. A set of hyperparameters is randomly selected prior to training the neural transformer model. Hyperparameters are parameters related to the neural network model architecture, training algorithm, and data normalization, and are set prior to the start of model training. Hyperparameters are not learned through deep learning or neural networks. Hyperparameters are randomly selected from a set of categorical values, or for real-valued hyperparameters like learning rate, randomly drawn from a given range. The hyperparameters are adjusted based on the performance of the neural transformer model when tested using a validation dataset.

[0087] Training of the neural transformer model is a compute-intensive task that requires parallel data processing. One or more clusters can be used to train the neural transformer model, where each cluster contains a set of loosely or tightly coupled computers (e.g., processors, processing units, cores) that simultaneously perform the same task under the control of a distributed controller. Each computer uses the same copy of the neural transformer model and uses a distributed data-parallel training algorithm to synchronize processing between the clusters.

[0088] The neural transformer model is trained using batch processing, where the training dataset is divided into batches of a particular size and processed prior to model updates. The size of the batch must be greater than or equal to 1 and less than or equal to the number of samples in the training dataset.

[0089] Reference Figure 1 and 5A One or more source code repositories 106 are searched for source code programs. Each source code program can be written in the same or different programming languages. The source code repositories 106 can be widely used code repositories, such as GitHub, internal code repositories, and / or combinations thereof. A source code extraction component 108 extracts a plurality of source code programs of a type that meet an intended goal, such as frequently accessed source code programs, source code programs that utilize a particular functionality (e.g., database operations, computer graphics programs, asynchronous methods, etc.), and the like. These source code programs are used to generate training and validation datasets (collectively, block 502).

[0090] Each selected source code program 110 is then parsed and / or compiled by a compilation component to produce a concrete syntax tree (block 504).

[0091] The byte pair encoding is used to generate an ordered token / subtoken sequence representing the context of the source code program. The serialized sequence of syntax nodes and tokens is obtained by traversing the concrete syntax tree. In one aspect, the concrete syntax tree is traversed in a depth-first order (i.e., depth-first search, depth-first traversal). The depth-first traversal starts at the root node, traverses the tree in a single path until it reaches a terminal node or leaf node. The traversal then backtracks until it can choose another path to traverse. This process is repeated until all nodes are visited. Next, the token / subtoken sequence is converted to a numerical vector (collectively, block 506).

[0092] One portion of the sequence is used as a training dataset and another portion is used as a validation dataset. The training dataset is divided into epochs, and then the sequences are divided into batches in each epoch. Each sequence in each batch in each epoch (block 510) is then used to train the neural transformer model (block 514). Initial values are generated for the token / sequence and position embeddings for each sequence, which are then used to form a context tensor (block 512).

[0093] Referring now to Figure 5B A first layer normalization is applied to the context tensor (block 522), and then to the masked self-attention (block 524). The output of the masked self-attention is input into a second layer normalization (block 526). The output of the second layer normalization is input into a first one-dimensional convolutional neural network layer (block 528). The output of the first one-dimensional convolutional neural network layer is then input into a second one-dimensional convolutional neural network layer (block 530).

[0094] The neural network is trained iteratively, with multiple passes over the training dataset before converging to a minimum. Each training iteration includes a forward pass (blocks 528-530), a loss calculation (block 532), a backpropagation step (block 534), and then the weights are updated by computing the weight gradients (block 536).

[0095] The loss function estimates the loss or error, which is used to compare how good the prediction is. In one aspect, a categorical cross-entropy loss function is used. Once the loss is calculated, it is backpropagated to the hidden layers that directly contribute to the output. In backpropagation, the partial derivatives of the loss function with respect to the trainable parameters are determined. The weight gradients are computed as the difference between the old and new values of the weights. The weights are adjusted using a gradient descent technique to make the loss as close to zero as possible. In one aspect, a stochastic gradient descent (SGD) method is an optimization algorithm for finding values of the parameters of a supplied function that minimize a given loss function. The weights can be updated using a backpropagation through time (BPTT) algorithm.

[0096] Referring back Figure 5A At the completion of each batch, the parameters of the neural network are updated at a preconfigured frequency represented as N ACCUM is the gradient accumulation frequency, which in one aspect has a value of 8. The parameters include the token / subtoken embeddings stored in the respective embedding matrices, the position embeddings. Other parameters include the parameters of the attention layers and the convolution layers. ACCUM

[0097] Next, the neural transformer model is validated. Prior to training the neural transformer model, a set of hyperparameters are randomly selected and then adjusted to achieve the desired performance. The neural transformer model is tested using a validation dataset to determine the appropriate hyperparameter settings to achieve the intended goals. When the intended goals are not achieved, one or more hyperparameters are adjusted and the training is repeated until the targeted goals are achieved (collectively, block 518).

[0098] Evaluation metrics are used to test the quality of the candidate recommendations. In one aspect, a top k precision approach using Mean Inverse Rank (MRR) is used to perform the evaluation. Top k precision is defined as: ,

[0099] and MMR is defined as: ,

[0100] where N top-k represents the number of relevant recommendations in the top k recommendations, Q represents the total number of test data samples, rank i is the predicted rank of the recommendation.

[0101] Top-1 precision represents the frequency that the first recommendation is correct, while top-5 precision represents the frequency that the first three recommendations in the list contain the candidate that the user is looking for. MRR captures the rank of the results, thereby providing information beyond the top candidates. The larger the MRR value, the smaller the overall rank of the correct recommendation. (collectively, block 518).

[0102] After the model validation is complete, the model is ready to be deployed in the code completion system (block 520). Attention is now turned to a discussion illustrating an exemplary approach for the inference phase using the neural transformer model in the code completion system.

[0103] Figures 6A-6B An exemplary approach 600, 608 is illustrated for code line completion utilizing the neural transformer model. Referring to Figure 1 and Figure 6A ​Code completion is performed in a development environment such as a source code editor 130. The source code editor 130 is configured to interact with a code completion component 142 that performs a beam search that uses a neural transformer model. The source code editor 130 performs a background parsing process that monitors characters input into the source code editor and continuously parses the source code to update a concrete syntax tree representing the source code of the current line of code (block 602).

[0104] The user interface 132 of the source code editor 130 detects a request for a candidate sequence to complete the current line of source code. The request can be initiated by a marker character such as an equals sign "=". The code completion system will provide a candidate sequence after the equals sign to complete the remaining portion of the expression (collectively block 604).

[0105] Alternatively, the user can request a candidate by inputting a specific keystroke or keystroke sequence such as a CTRL key in combination with a space bar. In yet another aspect, the system can automatically display a single top candidate at the end of the current line of source code in a dim color without regard to a marker character. Whether or not the user decides to trigger the candidate, the system builds and continuously updates a candidate tree in the background. The candidate is automatically displayed in the user interface after the user has been idle for a period of time. If the user wants to accept the candidate, the user can type a specific keystroke or keystroke combination (e.g., CTRL and I) to accept the candidate. In this case, the cursor position will advance to the end of the proposed code sequence and the dim color of the candidate code will change to the normal color of the code. If the user does not want to use the candidate, the candidate disappears as the user continues to type. In this case, the system refines the code sequence according to the prefix filter of the candidate tree based on the newly typed code (collectively block 604).

[0106] Upon detecting the request for a candidate sequence, the concrete syntax tree is parsed to extract tokens / sub-tokens from the current code segment. The embeddings are obtained from the token / sub-token embedding matrix and the position matrix. A context tensor is generated from the embeddings (collectively block 606).

[0107] The beam search is then performed until the probability distribution indicates that the next likely token is an end-of-line token (block 608).

[0108] Referring to Figure 6B , the beam search uses a neural transformer model that generates a probability distribution of token / sub-token vocabulary using the context tensor (block 614). If the probability distribution indicates that the next likely token is an end-of-line token, the beam search ends (block 616 - YES) and the top k candidate sequence is output (block 618).

[0109] Otherwise, the top kone token / subtoken (block 620).

[0110] Each selected token / subtoken is then input into the respective context vector and again has a separate data path through the neural transformer model. The context vector utilizes the selected token / subtoken in the current context vector and removes the last token / subtoken. The new context vector will consist of T one token / subtoken, the selected token / subtoken c k being added to the beginning of the sequence and the last token / subtoken removed from the sequence. If the current context vector consists of a sequence of tokens / subtokens including c 0 、c 1 、...、c T the new context vector will consist of c k ，c 0 、c 1 、...、c T-1 . (collectively, block 622).

[0111] Referring back to Figure 6A , the beam search keeps track of the generated sequences in the search tree and returns the preceding candidate sequences to the user interface component for display to the user (block 610). The user can select one of the candidates and then input it into the source code program to complete the line of source code (block 612). Alternatively, the user can ignore the candidate sequences and continue to type. The process is repeated (blocks 602-612) until the user closes the source code program, exits the source code editor, or terminates the code completion tool.

[0112] Code line completion example

[0113] Attention is now directed to an exemplary user interface display of a code completion tool using the techniques described herein. Turning to Figure 7 , Figure 7 shows a source code program being edited in a source code editor. The user interface displays lines 10-36 of the source code program 702. After the user inputs the "=" character, a pop-up window 704 appears on line 36. The pop-up window 704 contains five candidate sequences to complete the line of code on line 36. The five candidates 706-714 are displayed in rank order from highest probability to lowest probability. Each candidate is an ordered sequence of tokens that is likely to complete the expression on line 36.

[0114] Figure 8 is from the pair Figure 7The beam search of the illustrated snippet of source code generates a search tree 800. The search tree 800 tracks all states generated by the neural transformer model in the nodes of the search tree. In this example, the beam width is set to four (4). At the first inference level 802, or execution, of the neural transformer model, the beam search generates a root node 816 with a probability distribution for each token / subtoken in the vocabulary. The top four tokens / subtokens are then selected, which are "tf," "gradient," "gan," and "gd." Each selected token is added to a separate context vector, which is then used in subsequent executions of the neural transformer model. The probability distribution for each token / subtoken in the second inference level 804 is shown resulting from each call 818A-818D of the neural transformer model.

[0115] The top four tokens / subtokens are then selected from each node in the second inference level 804 that generates a new context vector. A third call of the neural transformer model is made using the new nodes generated from each selected token / subtoken in the second inference level 804, which is shown in the third inference level 806. This process is repeated again until the search ends. Figure 8 The search tree 800 resulting from the first seven inference levels 802, 804, 806, 808, 810, 812, 814 is shown.

[0116] As shown, the candidate sequences Figure 8 consist of the tokens / subtokens tf.train.AdamOptimizer(learning_rate inferred in the root node 816 tf , the token / subtoken "." inferred from node 818A in the second inference level 804, the token / subtoken train "tf" inferred from node 820 at the third inference level 806, the token / subtoken AdamOptimizer "gradient" inferred from node 824 at the fifth inference level 810, the token / subtoken （ "gan" inferred from node 826 at the sixth inference level 812, and the token / subtoken learning "gd" inferred from node 828 at the seventh inference level 814.

[0117] Exemplary operating environment

[0118] Attention is now directed to a discussion of an example operating environment. Figure 9An example operating environment 900 is illustrated in which one or more computing devices 902 are used to train a neural transformer model and a second computing device 904 uses the neural transformer model for code completion. However, it should be noted that aspects disclosed herein are not limited to any particular configuration of devices. Any of the computing devices 902, 904 can utilize the neural transformer model in its own code completion system and the computing device 904 can also generate and test the neural transformer model. The computing device 902 can be configured to generate the neural transformer model as a cloud service for other code completion systems as a service. It should be noted that the operating environment is not limited to any particular configuration and other configurations are possible.

[0119] The computing devices 902, 904 can be any type of electronic device, such as, but not limited to, a mobile device, a personal digital assistant, a mobile computing device, a smart phone, a cellular telephone, a handheld computer, a server, a server array or server farm, a Web server, a network server, a blade server, an Internet server, a work station, a minicomputer, a mainframe computer, an supercomputer, a network appliance, a Web appliance, a distributed computing system, multiprocessor systems, or combination thereof. The operating environment 900 can be configured in a network environment, a distributed environment, a multi-processor environment, or an independent computing device that is capable of accessing remote or local storage devices.

[0120] The computing devices 902, 904 can include one or more processors 908, 940, one or more communication interfaces 910, 942, one or more storage devices 912, 944, one or more input / output devices 914, 946, and one or more memory devices 916, 948. The processors 908, 940 can be any commercially available or custom processors and can include dual microprocessors and multiprocessor architectures. The communication interfaces 910, 942 support wired or wireless communication between the computing devices 902, 904 and other devices. The storage devices 912, 944 can be computer-readable media that do not include propagating signals, such as modulated data signals transmitted through a carrier wave. Examples of storage devices 912, 944 include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, all of which are not propagating signals, such as modulated data signals transmitted through a carrier wave. There can be multiple storage devices 912, 944 in the computing devices 902, 904. The input / output devices 914, 946 can include keyboards, mice, pens, voice input devices, touch input devices, displays, speakers, printers, etc., and any combination thereof.

[0121] The memory devices 916, 948 can be any non-transitory computer-readable storage media capable of storing executable programs, applications, and data. Computer-readable storage media does not include propagating signals, such as modulated data signals transmitted through a carrier wave. It can be any type of non-transitory storage device (e.g., random access memory, read only memory, etc.), magnetic storage, volatile storage, non-volatile storage, optical storage, DVD, CD, floppy disk drive, etc. that does not include propagating signals, such as modulated data signals transmitted through a carrier wave. The memory 916, 948 can also include one or more external storage devices or remotely located storage devices that do not include propagating signals, such as modulated data signals transmitted through a carrier wave.

[0122] The computing device 904 can utilize an integrated development environment (IDE) 954 in the computer system that allows a user (e.g., a developer, programmer, designer, coder, etc.) to design, code, compile, test, run, edit, debug, or build a program, a set of programs, a website, a Web application, and a Web service. The software program can include source code files created in one or more source code languages (e.g., Visual Basic, Visual J#, C++, C#, J#, Java Script, APL, COBOL, Pascal, Eiffel, Haskell, ML, Oberon, Perl, Python, Scheme, Smalltalk, etc.). The IDE 954 can provide a native code development environment or can provide a managed code development that runs on a virtual machine or can provide a combination of them. The IDE 954 can provide a managed code development environment using the.NET framework. It is noted that the present operating example is not limited to providing source code development services through an IDE, but other tools can be used instead, such as a standalone source code editor, etc.

[0123] A user can create and / or edit source code program files 952 via a user interface 958 and a source code editor 956 in the IDE 954 according to known software programming techniques and the specific logic and syntax rules associated with a particular source language. Thereafter, the source code program files 952 can be compiled via a compilation component 960, thereby generating data structures representing the syntax and semantic models of the source code.

[0124] The memory device 948 of the computing device 904 can contain instructions, components, and data. A component is a software program that performs a specific function, also referred to as a module, program, and / or application. The memory device 948 can include an operating system 950, one or more source code program files 952, an IDE 954 that can include a source code editor 956, a user interface 958, a compilation component 960, a code completion component 962, and a neural transpiler model 964, and other applications and data 966.

[0125] The memory device 916 of the computing device 902 can include an operating system 918, a source code extraction component 920, a token / subtoken sequence extraction component 922, a parser 924, a model training and testing component 926, a neural transpiler model 928, a source code repository 930, and other applications and data 932.

[0126] The computing devices 902, 904 can be communicatively coupled via a network 909. The network 909 can be configured as an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the PSTN, a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, or any other type of network or combination of networks.

[0127] The network 909 can employ various wired and / or wireless communication protocols and / or technologies. Different generations of communication protocols and / or technologies that the network can employ can include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access 2000 (CDMA-2000), High-Speed Downlink Packet Access (HSDPA), Long Term Evolution (LTE), Universal Mobile Telecommunications System (UMTS), Evolution-Data Optimized (Ev-DO), Worldwide Interoperability for Microwave Access (WiMax), Time Division Multiple Access (TDMA), Orthogonal Frequency Division Multiplexing (OFDM), Ultra-Wide Band (UWB), Wireless Application Protocol (WAP), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), any portion of the Open Systems Interconnection (OSI) model, Session Initiation Protocol / Real-Time Transport Protocol (SIP / RTP), Short Message Service (SMS), Multimedia Messaging Service (MMS), or any other communication protocol and / or technology.

[0128] CONCLUSION

[0129] A system is disclosed comprising one or more processors and memory storing one or more programs configured to be executed by the one or more processors. The one or more programs comprise instructions to: track a sequence of characters entered into a line of a source code program during an editing session; and generate a candidate sequence to complete the line of the source code at a position in the line of the source code program using a neural transformer model, wherein the neural transformer model is trained on an unsupervised dataset of source code programs written in one or more different programming languages.

[0130] The system comprises further instructions which, when executed by the one or more processors: initiate a beam search to build a search tree to generate the candidate sequence, wherein the search tree comprises one or more nodes at one or more reasoning levels, each node representing an output probability distribution of a set of tokens of a vocabulary of the neural transformer model, wherein the output probability distribution is generated by the neural transformer model, each node expands one or more tokens / sub-tokens to a next reasoning level. The beam search iteratively expands the search tree by: calling the neural transformer model to predict a next token given a sequence of tokens representing a partial candidate that completes the line of code. k

[0131] In one aspect, the neural transformer model consists only of decoder blocks. The neural transformer model comprises at least one decoder block having a masked self-attention layer. The neural transformer model comprises at least one one-dimensional convolutional neural network layer.

[0132] The system tracks the sequence of characters entered into the line of the source code program by: obtaining a sequence of tokens / sub-tokens representing a current context of the line of code and finding token / sub-token embedding vectors and position embedding vectors for the sequence of tokens / sub-tokens. The token / sub-token embedding vectors and the position embedding vectors are pre-trained.

[0133] The system comprises instructions to input the token / sub-token embedding vectors and the position embedding vectors into the neural transformer model. The neural transformer model generates a probability distribution of tokens / sub-tokens of a model vocabulary.

[0134] A method is disclosed comprising: monitoring each token entered into a line of code of a source code program during a source code development session; iteratively performing a beam search to generate token candidates to complete the line of code as new tokens are entered into the line of code, wherein the beam search generates the token candidates using token probability matrices generated from a neural transformer model; concatenating the token candidates into a candidate sequence to complete the line of code; and outputting at least one candidate sequence as the token characters entered in the line of code are detected during the source code development session.

[0135] ​The method further includes invoking the neural transformer model to predict the next token given a context vector, the context vector representing a context of the line of code including the new token.

[0136] In one aspect, the neural transformer model includes a self-attention layer and a convolutional neural network. Layer normalization is performed before the self-attention layer and is applied to the output of the self-attention layer. The neural transformer model utilizes token embeddings and position embeddings representing a context of a line of code, where the token embeddings and position embeddings are pre-trained.

[0137] In another aspect, the monitoring of each token input in the source code program further includes: parsing the input into a concrete syntax tree; performing byte pair encoding to extract tokens from the concrete syntax tree; and concatenating the ordered sequence of tokens of length T.

[0138] A device is disclosed that includes at least one processor coupled to a memory device. The at least one processor is configured to: extract one or more ordered sequences of tokens from a plurality of source code programs, where the ordered sequences of tokens represent a context from a selected one of the plurality of source code programs; and train a neural transformer model with the ordered sequences of tokens to predict a next token to complete a partial sequence of tokens, where the partial sequence of tokens is used to generate a candidate sequence of tokens to complete a line of code in a target source code program, where the neural transformer model includes an attention layer and at least one convolutional neural network layer.

[0139] In one aspect, the ordered sequence of tokens includes one or more sub-tokens. The neural transformer block is a transformer that is a decoder only. In some aspects, at least two of the plurality of source code programs are written in different programming languages, and the ordered sequence of tokens is an unsupervised training dataset. In some aspects, the neural transformer model generates a token probability matrix that is used to predict a next token that will be successful in the predicted candidate sequence.

[0140] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A system comprising: One or more processors; as well as A memory storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions, the instructions being: During an editing session, the character sequences input into a partially formed line of a source code program are tracked, wherein the source code program is written in a first programming language; as well as At a location in the partially formed line of the source code program, a neural transformer model with attention-only decoder is used to generate candidate sequences to complete the line of the source code given the tracked character sequence, wherein the neural transformer model with attention-only decoder is trained on an unsupervised dataset of source code programs written in a variety of different programming languages.

2. The system of claim 1, wherein the one or more programs include additional instructions that, when executed by the one or more processors: A bundle search is initiated to construct a search tree to generate the candidate sequence, wherein the search tree includes one or more nodes at one or more inference levels, each node representing a set of output probability distributions of tokens from the vocabulary of the attention-only decoder neural transformer model, wherein the output probability distributions are generated from the attention-only decoder neural transformer model, and each node extends k tokens / sub-tokens to the next inference level.

3. The system of claim 2, wherein the beam search iteratively expands the search tree by invoking the attention-only decoder-only neural transformer model to predict the next token given a sequence of tokens representing partial candidates for completing the partially formed line of code.

4. The system of claim 1, wherein the neural transformer model with attention-only decoder comprises at least one decoder block having a masked self-attention layer and a convolutional neural network layer.

5. The system of claim 1, wherein the neural transformer model with attention-only decoder comprises at least one one-dimensional convolutional neural network layer.

6. The system of claim 1, wherein tracking the character sequence in the partially formed line input to the source code program further comprises: Obtain a sequence of tokens / subtotals representing the current context of the partially formed line of code; as well as Find the token / subtotal embedding vector and the position embedding vector for the sequence of tokens / subtotals, wherein the token / subtotal embedding vector and the position embedding vector are generated by training the neural transformer model with attention-only decoder.

7. The system of claim 6, wherein the one or more programs include additional instructions that, when executed by the one or more processors: The token / subtoken embedding vector and position embedding vector are input into the attention-only decoder neural transformer model, wherein the attention-only decoder neural transformer model generates the probability distribution of the token / subtoken in the model vocabulary.

8. A method comprising: Monitor each token in the partially formed lines of code that are input into the source code program during the source code development session; When a new token is input into the partially formed line of code, a bundle search is iteratively performed to generate multiple token candidates to complete the partially formed line of code, wherein the bundle search uses token probabilities generated from a neural transformer model with attention-only decoder trained on multiple multilingual source code programs to generate token candidates. The multiple token candidates are concatenated into a candidate sequence to complete the partially formed line of code; as well as During the source code development session, at least one candidate sequence is output when a marker character input is detected in the partially formed line of code.

9. The method according to claim 8, further comprising: The neural transformer model of the decoder-only is invoked to predict the next token given a context vector, the context vector representing the context of the partially formed line of code that includes the new token.

10. The method of claim 8, wherein the neural transformer model of the decoder-only includes a self-attention layer and a convolutional neural network.

11. The method of claim 10, wherein layer normalization is performed before the self-attention layer, and the layer normalization is applied to the output of the self-attention layer.

12. The method of claim 8, wherein the neural transformer model with attention-only decoder utilizes token embeddings and position embeddings representing the context of the line of code, wherein the token embeddings and the position embeddings are learned by training the neural transformer model with attention-only decoder.

13. The method of claim 8, wherein monitoring each token input into the source code program further comprises: The characters input into the code line are parsed into a specific syntax tree; Perform byte-pair encoding to extract tokens from the specific syntax tree; as well as An ordered sequence of tokens of length T.

14. An apparatus comprising: At least one processor is coupled to a memory device; The at least one processor is configured to: Extract one or more ordered token sequences from multiple source code programs written in different programming languages, wherein the ordered token sequences represent the context of a source code segment from a selected source code program among the multiple source code programs; as well as The ordered token sequence is used to train an attention-only decoder neural transformer model to predict the next token to complete a partial token sequence, wherein the partial token sequence is used to generate candidate token sequences to complete lines of code in the target source code program, and wherein the attention-only decoder neural transformer model includes an attention layer and at least one convolutional neural network layer.

15. The device of claim 14, wherein the ordered token sequence comprises one or more sub-tokens.

16. The device of claim 14, wherein the ordered token sequence is an unsupervised training dataset.

17. The device of claim 14, wherein the neural transformer model with attention-only decoder generates a token probability matrix for predicting the next token that will succeed in the predicted candidate sequence.

Images