Multilingual code line completion system

Abstract

Embodiments of this disclosure relate to a multilingual code line completion system. A code completion tool uses a neural transformer model to generate candidate sequences to complete lines of source code. The neural transformer model is trained on a large unsupervised dataset using a conditional language modeling objective, which includes source code programs written in multiple different programming languages. The neural transformer model is used for beam search, which predicts the most likely candidate sequences for a small piece of code under development.

Description

Cross-reference to related applications

[0001] This application is a divisional application of Chinese Patent Application No. 202080054713.X, filed on June 11, 2020 and granted on February 27, 2026, and claims the benefit of U.S. Provisional Application No. US62 / 881,736, filed on August 1, 2019, and U.S. Application No. US16 / 680,328, filed on November 11, 2019, the entire contents of which are incorporated herein by reference. Background Technology

[0002] Software development environments (SDAs) are typically used to assist software developers (i.e., users, programmers, etc.) in developing program code. An SDA can include a source code editor and other tools that developers use to write and test their programs. Some SDAs include code completion features, which provide assistance by automatically presenting a list of possible candidates based on one or more characters (e.g., letters, symbols, etc.) that the developer types in the source code editor. A pop-up menu with several suggested code elements that the developer might use may appear. This assistance is beneficial because it can speed up development time and reduce common errors, such as spelling mistakes.

[0003] However, automatic code completion may encounter problems when the code completion system fails to recognize code elements outside the vocabulary, requires a large amount of memory, takes too long to generate a candidate list, and / or generates an irrelevant candidate list. Summary of the Invention

[0004] This summary provides a simplified overview of some concepts that will be further described in the detailed description below. This summary is not intended to identify the principal or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] A multilingual code line completion system is used to generate candidate lines of code most likely to be completed during a source code editing session. The predicted strings used to complete the lines can include various types of elements, such as local variables, methods, parameters, keywords, and delimiters arranged in an ordered sequence. The system uses a model to predict ordered sequences, trained on a large unsupervised dataset with a conditional language modeling objective. This dataset includes source code programs written in different programming languages, such as C, Java, Python, and C++.

[0006] Each source code program in the training dataset does not need to be written in the same programming language. The training dataset may consist of many source code programs, each written in a different programming language. Each source code program in the training dataset is encoded as a sequence of tokens and / or sub-tokens. Frequently used elements in the programming language are encoded as tokens, while less frequently occurring elements are encoded as combinations of characters called sub-tokens. This reduces the need to store a large vocabulary and provides better accuracy for tokens outside the vocabulary.

[0007] The multilingual code line completion system is based on a neural transformer model. In one aspect, the neural transformer model consists of multiple decoder blocks. Each decoder block includes a multi-head self-attention layer coupled to a multi-layer one-dimensional convolutional neural network. Layer normalization is applied before and after the multi-head self-attention layer to reduce the training time of the neural transformer model.

[0008] Beam search is used to generate candidate sequences. Beam search expands the partial candidate sequences of tokens / subtotals that might complete the lines of the source code using the top k subtotals / totals identified from each iteration of the neural transformer model. Beam search generates a search tree but only retains the top k nodes from each inference level to expand the search. The search ends when the end-of-line token appears as the most likely prediction.

[0009] These and other features and advantages will become apparent from the following detailed description and a review of the accompanying drawings. It should be understood that the foregoing general description and the following detailed description are illustrative only and do not limit the claimed aspects. Attached Figure Description

[0010] Figure 1 An exemplary code completion system is illustrated, which has a training phase for generating a neural transformer model and an inference phase for using the neural transformer model to predict one or more candidate sequences to complete lines of code.

[0011] Figure 2A-2B This is a schematic diagram illustrating an exemplary system and method for training a neural transformer model for line completion.

[0012] Figure 3 This is a schematic diagram of an exemplary architecture of the converter block shown in Figure 2.

[0013] Figures 4A-4B This is a schematic diagram of an exemplary architecture for the inference phase.

[0014] Figures 5A-5B This is a flowchart illustrating an exemplary method for training a neural transformer model for code completion.

[0015] Figures 6A-6BThis is a flowchart illustrating an exemplary method for utilizing a neural transformer model during the inference phase.

[0016] Figure 7 This is a schematic diagram illustrating an exemplary user interface, showing code completion candidates for a line of code in an exemplary source code program.

[0017] Figure 8 This is a schematic diagram of an exemplary bundle search that illustrates the search tree for generating candidate sequences.

[0018] Figure 9 This is a block diagram illustrating the operating environment. Detailed Implementation

[0019] Overview

[0020] The disclosed topic relates to the generation of candidate lines for automatic source code completion in a programming development environment. Code completion is a tool that attempts to predict the next string a developer (e.g., user, end-user, programmer, etc.) might type in a source code editor. Lines of source code can consist of various elements (e.g., keywords, delimiters, variables, methods, constants, operators, etc.) that combine in a specific order according to the syntax of the underlying programming language to form expressions. Lines of source code can be method calls, program statements, definitions, expressions, etc. Source code lines do not include blank lines or comment lines and end with a line-ending character.

[0021] Code completion tools use neural network machine learning models to predict the next string of code elements to complete lines of source code. A line of source code refers to a physical line of source code that ends with a line-ending character, excluding blank lines and comment lines. The model is trained on unsupervised datasets, which may contain source code from different programming languages ​​(i.e., multiple languages). Unsupervised learning derives inference from datasets consisting of input data with unlabeled responses. From these datasets, a vocabulary is formed that includes tokens and / or sub-tokens found in the source code files. Tokens are individual elements in the syntax of a programming language, such as keywords, variable names, operators, delimiters, etc.

[0022] Unlike natural languages ​​(such as English), programmers sometimes use arbitrary, complex, and long names to represent variables, functions, or other code elements. This can result in a very large vocabulary for a model when a large amount of source code is used to train the model. To reduce the size of the vocabulary, less frequently occurring tokens are split into sub-tokens. A sub-token is a portion of a token that lies between a main token and a single character. Sub-tokens are used to indicate rare or unseen tokens that might appear in the target source code (i.e., out-of-vocabulary tokens). The use of sub-tokens allows the model to learn and generate out-of-vocabulary tokens.

[0023] Byte-pair encoding is a data compression technique in which the most frequently co-occurring Unicode character pairs in the entire training source code dataset are replaced with out-of-vocabulary characters. When applied recursively, byte-pair encoding results in the extraction of tokens / subtotals from frequently co-occurring sequences of Unicode characters. In one aspect, byte-pair encoding is used to extract ordered sequences of Unicode characters from the syntactic representation of a source code program in the training dataset to form tokens and subtotals. The ordered sequence represents a source code segment with T tokens / subtotals. The ordered sequence of tokens / subtotals is converted into token / subtotal embeddings and positional embeddings, which are vector representations of the source code segment.

[0024] In one aspect, neural network machine learning models are multi-layer transformer models. A transformer is a neural network architecture that handles the dependencies between its inputs and outputs through attention and convolution, without using recurrent neural networks (RNNs) (e.g., Long Short-Term Memory (LSTM) networks). A drawback of RNN-based systems is their sequential nature, where each hidden state depends on previous hidden states. This makes RNN-based systems difficult to parallelize and unable to utilize fast computing devices such as graphics processing units. Furthermore, RNN-based systems cannot learn long-range dependencies in long-term input and output sequences. Transformers overcome these obstacles through attention. Attention is a mechanism used to identify which parts of the input sequence are relevant to each token / sub-token in the output sequence. The attention mechanism allows the transformer to access the entire input sequence at once.

[0025] A transformer can act as either an encoder or a decoder, where an encoder maps an input sequence of symbolic representations to a sequence of continuous representations, and a decoder generates an output sequence of symbols from the sequence of continuous representations. The encoder-decoder architecture is less suitable for conditional code generation or code completion tasks, but better suited for machine translation and patch generation tasks. A variant of the transformer model is used, consisting of decoder blocks with masked self-attention and convolutional layers.

[0026] Beam search is used to generate one or more candidate sequences to complete a line of source code. Beam search uses a probability distribution generated by a neural transformer model to identify the top k tokens / sub-tokens that are likely to be the next token or sub-token in the candidate sequences. Beam search expands the search by instantiating a new partial sequence for each selected token / sub-token identified by the probability distribution of the neural transformer model. The search continues to generate new partial sequences from the top k tokens / sub-tokens identified by the output distribution of the neural transformer model until the search ends. The search can end when the last token of the line appears as the most likely next token.

[0027] The task of code line sequence completion is to estimate the conditional probability distribution P(Output|Input) in the token / subtotal c. t Given an ordered sequence of t=0…T (corresponding to the context of the code fragment C as a product of conditional probabilities), predict the sequence m of response tokens / subtotals. t , t=0…N: . In one aspect, a multi-layer transformer-decoder neural network model with multi-head self-attention was used to estimate this probability distribution for a source code corpus using unsupervised autoregressive (AR) techniques. The modeling objective was to maximize the following possibilities:

[0028] ,

[0029] in k It refers to the size of the context window and the conditional probability. P Use with parameters The model is based on a neural transformer model. Parameters may include attention length, number of attention heads, number of decoder blocks, embedding dimension, embedding matrix, and the number of hidden units per layer trained using a stochastic gradient descent optimization procedure.

[0030] The multi-layer converter decoder generates the following token / sub-token output distribution:

[0031] h 0 = U·W e +W p ，

[0032] h 1 =transformer block(h t-1 ）,l=1…n，

[0033] ，

[0034] in It is the context vector of the token / subtotal. n It's the number of floors. T It is the length of the ordered sequence. W e It is a token / subtoken embedding matrix. W p It is a position embedding matrix. e This indicates the embedding of tokens / subtotals in the vocabulary. p Indicates the embedding of the token / subtotal position.

[0035] Now turn your attention to a further discussion of the systems, devices, components, and methods used in code completion systems.

[0036] Machine learning code completion system

[0037] Figure 1 A block diagram of an exemplary code completion system 100 is illustrated, in which various aspects of the present invention can be practiced. Figure 1 As shown, system 100 includes a training phase 102 for training a transformer model 122 in a line completion system and an inference phase 104 for using the transformer model 122. The training phase 102 may utilize a source code repository 106, a source code extraction component 108, a parser 112, a token / subtotal sequence extraction component 116, and a model training and validation component 120.

[0038] In one aspect, training phase 102 pre-trains the transformer model based on different corpora of unlabeled source code programs. This is called unsupervised learning because the model derives inference results from input data with unlabeled responses. Source code extraction component 108 extracts selected source code programs 110 from source code repository 106 to obtain training and validation datasets. Source code repository 106 can be a private or publicly accessible archive and web hosting facility storing large amounts of source code. Source code repository 106 can be configured as a version control system, such as Git, Mercurial, etc. The source code programs residing in source code repository 106 are different and can be written in different programming languages.

[0039] The source code extraction component 108 obtains several selected source code programs 110, which may be written in the same or different programming languages. The programming languages ​​use context-free grammars, which are a set of rules used in formal programming languages ​​to describe all possible strings. The selected source code programs 110 may come from different fields, such as, but not limited to, scientific computing, web development, dataflow programming, machine learning, etc.

[0040] Parser 112 transforms each selected source code program 110 into a concrete syntax tree 114. The concrete syntax tree 114 represents the source code text in a parsed form. The concrete syntax tree 114 can also be a parse tree. Parser 112 can be part of a parser, a front-end compiler, a language compiler, or a compilation tool. The concrete syntax tree 114 represents the syntactic structure of the program in a hierarchical or tree structure. The concrete syntax tree 114 is... nThe metatree data structure includes nodes representing the constructions in the grammar of the programming language of the program. The concrete syntax tree 114 includes a root node, multiple internal nodes, and multiple terminal nodes. Terminal nodes represent tokens. Tokens are symbols representing operands or operators. The concrete syntax tree 114 differs from the abstract syntax tree where terminal nodes represent operands.

[0041] The specific syntax tree 114 for the selected source code program 110 is passed to the token / subtoken sequence extraction unit 116. The token / subtoken sequence extraction unit 116 parses the specific syntax tree 114 for each source code program and outputs it. T A sequence of tokens and / or sub-tokens. In one aspect, the token / sub-token sequence extraction component 116 performs byte-pair encoding to extract frequently occurring tokens and extract sub-tokens from less frequently occurring tokens. A sub-token is a part of a token.

[0042] For example, the following line of source code:

[0043] loss=tf.reduce_sum(tf.square(linear_model - y))

[0044] It can be divided into the following sequence of tokens / subtotals, each token / subtotal separated by the character "|":

[0045] loss|=|tf|.|red|uce|_|sum|(|tf|.|squ|are|(|linear|_|model| - |y|)|)|

[0046] In this example, the token "reduce" has been split into sub-tokens "red" and "uce", and the token "square" has been split into sub-tokens "squ" and "are".

[0047] Then T- The ordered token sequence is mapped to a numeric vector, and then to an embedding. An embedding is a learned representation for a text-based token / sub-token, where tokens or sub-tokens with a common meaning share a common representation. There is one embedding and one positional embedding for each token / sub-token in the vocabulary. The token / sub-token embedding represents the learned representation for the token / sub-token. The transformer model does not read each token / sub-token sequentially; therefore, without additional positional information, it does not know the position of the token / sub-token in the sequence. The positional embedding is used to embed positional information about the token / sub-token's position in the sequence into the transformer model. The token / sub-token embedding is input to the model training and validation unit 120.

[0048] The neural transformer model 122 is used in the inference phase 104 of the code completion system. In one or more aspects, the inference phase 104 can be implemented as a function or feature integrated into a source code editor, integrated development environment (IDE), and / or standalone application. Code completion can be implemented as a tool or feature that is an add-on, plugin, extension, and / or component of the source code editor and / or IDE. In one aspect, the inference phase 104 includes a source code editor 130, a code completion component 142, and the model 122.

[0049] In one aspect, the source code editor 130 may include a user interface 132 and a parser 134. The user interface 132 includes a set of features or functions for developing (e.g., writing, editing, testing) the source code program. The user interface 132 may utilize pop-up windows to present a list 136 of possible candidates for completion, allowing the developer to browse candidates and select one from the list. Alternatively, candidates may appear in the line of the current source code line as the user types characters in the source code program.

[0050] Parser 134 reads characters input into the source code program through source code editor 130 and generates the corresponding concrete syntax tree 140. When developers create and edit source code in source code editor 130, parser 134 also updates concrete syntax tree 140.

[0051] At certain points during the editing process, the user interface 132 will request candidates to complete the current line of source code. The user interface can detect when the user has entered a specific character or string and automatically initiate a request for candidates to complete the code line. This character is called a marker character. In one aspect, a marker character can be an equal sign "=" or a period ".". The user interface 132 will then send a request 138 for candidates from the code completion component 142 to be presented to the developer. Alternatively, the user can request candidates by entering specific keystrokes or keystroke sequences, such as a combination of the CTRL key and the space bar.

[0052] On another front, the system can automatically display a single top candidate at the end of the current source code line in dark mode, regardless of the marker character. The system continuously builds and updates the candidate tree in the background, regardless of whether the user decides to trigger the candidate. After a period of inactivity, the candidate automatically appears in the user interface. If the user wants to accept the candidate, they can type a specific keystroke or key combination (e.g., CTRL and I). In this case, the cursor position moves to the end of the suggested code sequence, and the dark mode of the candidate code changes to the normal code color. If the user does not want to use the candidate, it disappears as the user continues typing. In this case, the system refines the code sequence based on a prefix filter in the candidate tree based on the newly typed code.

[0053] The code completion component 142 tracks the characters entered into the source code editor and serves requests for candidates to complete lines of source code. The code completion component uses model 122 to generate candidates based on the current context of the source code in the editor. Multiple candidates are ranked according to their respective probabilities, with the candidate with the highest probability listed first. A selected number of candidates 136 are then returned to the source code editor 130 and displayed in the user interface 132.

[0054] It should be noted that Figure 1 The components of a system in one aspect of an environment in which various aspects of the invention can be practiced are shown. However, without departing from the spirit or scope of the invention, it may not be necessary to... Figure 1 The precise configuration of the components shown is used in practice. Figure 1 The configurations shown can vary in various aspects and types, and the types of components may not be required. For example, training phase 102 can be executed in a computing environment and inference phase 104 can be executed in the same computing environment as training phase 102 or in a separate computing environment. The various computing environments are described in more detail below.

[0055] Now pay attention to the turning direction. Figure 2A The illustration shows further details of the components and processes 200 used to train the neural transformer model. (Reference) Figure 1 and 2A The source code extraction component 108 obtains the source code program used as the training and validation dataset. Each selected source code file 202 is parsed into a concrete syntax tree 204 by the parser 112. The token / subtotal sequence extraction component 116 traverses the concrete syntax tree 204. The token / subtotal sequence extraction component 116 can extract tokens from each line of the source code represented by the concrete syntax tree using the tokenizer 206.

[0056] In one aspect, byte-pair encoding is used as a token extractor 206. Byte-pair encoding is used to construct a vocabulary of tokens / subtotals. Although its name uses the word "byte," byte-pair encoding operates on Unicode code points rather than sequences of bytes. This encoding technique divides less frequently occurring tokens into subtotals, while more frequently occurring tokens remain unchanged.

[0057] Then, the tokens and sub-tokens from each line of the source code are aggregated into a single token. T An ordered sequence of 208 tokens / subtotals. In one aspect, TThere are 1024 tokens, and each sequence consists of 1024 tokens / sub-tokens and represents a specific context of the source code program. The sequences from various source code programs are then fed into the model training and validation component 210.

[0058] The neural network is trained iteratively, passing the training dataset multiple times before converging to the minimum. An epoch represents one forward and one backward pass of the entire training dataset through the neural network. Since the training dataset is very large, it is divided into smaller batches. Training is iterative, with the entire dataset passed through the neural network over multiple iterations. Each training iteration includes forward propagation, loss calculation, backpropagation steps, and then updating the weights.

[0059] Neural networks have multiple layers, allowing them to learn more detailed relationships within data and how features interact with each other at a non-linear level. Model architecture, training procedures, data normalization, and vocabulary encoding are hyperparameters tailored to meet specific objectives. The values ​​of these hyperparameters influence how the parameters are learned.

[0060] In one aspect, hyperparameters may include the following parameters: (1) the dimensions of the token / subtoken and the location embedding layer are 30000x768 and 1024x768, respectively; (2) 12 transformer blocks, each block consisting of two convolutional, masked self-attention and layer normalization layers; (3) For the training procedure: autoregressive with cross-entropy loss optimization objective; sequence length of 1024 tokens / subtotals; batch size of 8; gradient accumulation step size of 8 for each weight update; Adam stochastic optimization procedure for training the neural network; learning rate of 0.0001; (4) Data normalization process: normalize all strings and numeric characters, retaining the ten most frequent ones; (5) Vocabulary encoding process: extract the joint subtotal vocabulary from the multilingual code corpus using byte-pair encoding, retain the ten most frequent strings and numeric characters during the byte-pair encoding process and encode them as individual tokens; and introduce special control flow tokens to represent line end, file end, (decent) and indentation symbols.

[0061] The training dataset is divided into batches, and each batch of sequences is run throughout the training process. Sequences are initially converted into numeric vectors, followed by embeddings. An embedding is a mapping from discrete categorical variables to continuous numeric vectors. For each sequence, there is a token / subtoken embedding 212 and a positional embedding 214. The token / subtoken embedding represents the tokens and / or subtokens in the sequence, while the positional embedding represents the order of the tokens / subtokens in the sequence.

[0062] Initially, random values ​​are used as the initial values ​​for each token / sub-token embedding and position embedding. Subsequently, the neural transformer model 224 learns the values ​​for each embedding. After the training phase is complete, the embedding and position embedding for each token / sub-token are saved to corresponding matrices 218 and 220 for later use in the inference phase. There is a token / sub-token embedding matrix. W e 218, which contains each token / sub-token C i , i=0…V The embedding vector and the position embedding matrix W p 220, which contains the embedding vector for each position. P j , j=0…T ,in, V It refers to the size of the vocabulary list. T It is the length of the token / subtotal sequence.

[0063] Each token / subtoken embedding 212 and its corresponding positional embedding 214 are combined to form a context tensor 216. A tensor is a mathematical object with indices and components that follow certain transformation rules. A tensor is a partially defined computation. It is a generalization of vectors and matrices and is represented as an n-dimensional array. In this instance, the tensor represents the context of the source code program. The size of the context tensor 216 is a fraction of the size of the embedding vectors (e.g., the embedding size). T times, of which T It is the length of the token / subtotal sequence.

[0064] Turn Figure 2B The token / subtoken embedding 212 is learned along with the parameters of the neural transformer model. The output hidden state of the neural transformer model 228 is then multiplied by the linear projection matrix A 230. The linear projection matrix A is defined as follows: Hidden state vector The information learned by the neural transformer model 224 from the context tensor 216 is encoded. Finally, the probability distribution of each token / sub-token is calculated. 238 is obtained as The nonnormalized logits predict 234, of which These are bias vectors, which are then normalized using the softmax function 236 to generate the vectors.

[0065] Back Figure 2A The neural transducer model 224 can be composed of one or more transducer blocks 226A and 226B. (See reference) Figure 3 The converter block 226 can be configured with an encoder and a decoder block and / or only a decoder block. Figure 3One aspect of a neural transformer model 224 configured with multiple decoder blocks 306A, 306N is shown. The decoder blocks 306A, 306N may include a first normalization layer 308, followed by a masked self-attention layer 310, followed by a second normalization layer 312, and two one-dimensional convolutional neural networks 314A, 314B.

[0066] Training a neural network is a time-consuming task. To reduce training time, layer normalization is used. Layer normalization normalizes the input across features. The mean and standard deviation are calculated across feature dimensions. There is a first layer normalization 308 before the masked self-attention layer 310 and a second layer normalization 312 after the masked self-attention layer 310.

[0067] Each token / subtoken flows along its own path through all decoder blocks 306A, 306N. The masked self-attention layer 310 allows the neural network to focus on certain features or inputs. Vaswani et al., in their presentation "Attention is All You Need" at the 31st Conference on Neural Information Processing Systems (NIPS 2017), described attention as "mapping a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors. The output is computed as a weighted sum of values, where the weight assigned to each value is calculated by a compatibility function of the query and the corresponding key."

[0068] In one aspect of this disclosure, the masked self-attention layer 310 comprises two or more attention heads 316A, 316B. The multi-head self-attention layers run multiple scaled dot-product attention operations in parallel. Each attention head 316A, 316B operates on a portion of the context tensor 302. Attention head 316A operates on a first segment 318A and attention head 316B operates on a second segment 320A. Each attention head 316A, 316B operates on... n Input sequence of elements x=(x 1 ，…，x n ) Perform the operation and compute a new sequence of the same length. z=(z 1 ，…，z n ) Each output element z i Calculated as a weighted sum of the input elements after linear transformation:

[0069] .

[0070] Calculate each weight coefficient using the softmax function:

[0071] ,

[0072] in It is a scaled dot product .

[0073] The input to each attention head 316A and 316B consists of query matrices 320A and 320B and key matrices 322A and 322B, each with a dimension of 1. T×d x ,in T It is the length of the code sequence. d x This is the embedding dimension. The dot product is obtained by taking all keys from the query matrix 320 and the key matrix 322, and then applying the softmax function to get the weights. W 0 …W T Values ​​324A and 324B are generated by producing corresponding value matrices 326A and 326B. The resulting values ​​from the two value matrices are then concatenated (328) and linearized (330). The concatenation layer 328 obtains values ​​from each attention head. T×d v 3D bond matrix to form T×d v 3D matrix. Linear layer 330 takes the output of connection layer 328 and applies a linear transformation according to the following formula. Output = Input • W T +b The input is T×d v matrix, W yes d x ×d v 3D matrix b yes T×d x A 3D matrix, and the output is obtained by matrix multiplication and addition. T×d x 3D matrix.

[0074] Then, layer normalization 312 is applied to the output of the masked self-attention layer 310. The output of layer normalization 312 is then applied to the first neural network layer. In one aspect, there are two neural network layers, each consisting of a one-dimensional convolutional neural network. Given an input tensor of dimension (dx, T) and a convolutional kernel... g One-dimensional convolution is defined as: Output = Bias + Input, where operations It is a sliding dot product operation.

[0075] Turn Figure 2B The neural network at the last time step T The output is the hidden state vector.h T 228, whose encoding is information related to the token / subtoken learned by transformer blocks 226A and B. Then, the output hidden state of the neural transformer model 224 is multiplied by a factor defined as... The linear projection matrix A is 230. The hidden state vector... 228 pairs of information learned from the context tensor by the neural transformer model are encoded. Finally, the probability distribution P of each token / sub-token is calculated. |V| 238 is obtained as The nonnormalized logits predict 234, where b k k=0…|V|-1 are bias vectors, which are then normalized using the softmax function 236 to generate the vectors.

[0076] Now turn your attention to the description of the model components used in the inference phase. The inference phase uses beam search to find the most likely candidate sequences. Beam search iteratively generates tokens / subtotals by calling a neural transformer model. The output of the neural transformer model is a token probability matrix for each position in the candidate sequence. The beam search focuses on... k The best path to the most likely candidate sequence is obtained by considering the most likely tokens. In each iteration, k Each of the most likely tokens is concatenated with tokens from previous iterations to form a partial candidate sequence.

[0077] The beam search uses breadth-first search to construct the search tree. The search tree consists of nodes at one or more inference levels. Each node represents a probability distribution of tokens / subtotals in the model vocabulary generated by the neural transformer model. At each level, only the nodes with the highest probabilities from the output distribution generated by the neural transformer model are selected. k Individual tokens / subtotals are extended to the next inference level. Variables k It is pre-configured, also known as the beam width. Then... k Each of the sub-tokens / tokens is expanded into a search that updates the current context sequence with the selected sub-tokens / tokens to be fed into the neural transformer model to generate an additional probability distribution for the next token in the sequence. This process is repeated until the end-of-line token is predicted as the next possible token candidate.

[0078] Turn Figure 4A , Figure 4A The components of reasoning stage 400 are shown. Code snippets 402 are input into a source code editor and are converted into a corresponding concrete syntax tree 404. Token extractor 406 traverses the concrete syntax tree 404 to extract tokens and / or sub-tokens. A length of [length missing] is formed. T The ordered sequence is vectorized into 408.

[0079] The beam search 410 uses the context vector 408 to utilize the probability distribution generated from the neural transformer model. P 0 …P |V| To initiate the inference process 412 (box 412). If the probability distribution indicates that the end-of-line token is the most likely token to follow in the partial candidate sequence (box 416 - yes), then output the preceding... k There are 10 candidate sequences (box 418). Otherwise, the beam search 410 uses the probability distribution identifier generated from the neural transformer model during the inference process to identify the previous sequence. k A state or token / subtotal (box 420). Use the new token / subtotal from the context vector, targeting... k each state c 1 、..c k Each of these steps generates a new context vector (boxes 422A and 422B). The new context vector is then fed into the inference process (boxes 422A, 422B, and 412). The beam search 410 ends when the end-of-line token is selected as the most likely candidate to complete the partial candidate sequence.

[0080] Figure 4B An exemplary search process 412 is illustrated. The embedding vector and its corresponding position vector for each token and sub-token in sequence 408 are obtained from the token / sub-token embedding matrix 428 and the position embedding matrix 430. The token / sub-token embedding vectors and their corresponding position embedding vectors are combined to form a context tensor 432, which is input into the neural transformer model 434.

[0081] The output of the neural converter model 434 has components h0…h dh The vector of 436. The output of the transformer is multiplied by the linear projection layer 438 to generate the predicted embedding vector 440. The token / subtotal embedding vector 448 is used as the output classification matrix to generate unnormalized predictions or logits V0…V |V| 442. Normalize logits 442 using the softmax function 444 to generate softmax predictions 446 P0…P |V| .

[0082] method

[0083] Note now turns to the description of various exemplary methods using the systems and apparatus disclosed herein. The operation of these aspects can be further described with reference to the various exemplary methods. It will be understood that, unless otherwise stated, the representative methods need not be performed in the order presented or in any particular order. Furthermore, the various activities described with respect to the methods can be performed in a serial or parallel manner or any combination of serial and parallel operations. In one or more aspects, the method illustrates the operation of the systems and apparatus disclosed herein.

[0084] Figures 5A-5B The illustration shows an exemplary method 500, which illustrates the use of a neural transformer model for code completion. Before training the neural transformer model, a set of hyperparameters is randomly selected. Hyperparameters are parameters related to the neural network model architecture, training algorithm, and data normalization, and are set before model training begins. The hyperparameters are not learned through deep learning or the neural network. The hyperparameters are randomly selected from a set of categorical values, or, for real-valued hyperparameters like the learning rate, randomly sampled from a given range. When testing with a validation dataset, the hyperparameters are tuned based on the performance of the neural transformer model.

[0085] Training a neurotransformer model is a computationally intensive task that requires parallel data processing. One or more clusters can be used to train the neurotransformer model, where each cluster contains a set of loosely or tightly coupled computers (e.g., processors, processing units, cores) that execute the same task simultaneously under the control of a distributed controller. Each computer uses the same copy of the neurotransformer model, and a distributed data-parallel training algorithm is used to synchronize processing across the clusters.

[0086] The neural transformer model is trained using batch processing, where the training dataset is divided into batches of a specific size and processed before model updates. The batch size must be greater than or equal to 1 and less than or equal to the number of samples in the training dataset.

[0087] refer to Figure 1 and 5A One or more source code repositories 106 are searched for source code programs. Each source code program can be written in the same or different programming languages. Source code repository 106 can be a widely used code repository, such as GitHub, an internal code repository, and / or a combination thereof. Source code extraction component 108 extracts multiple source code programs of the type that meet the expected objectives, such as frequently accessed source code programs, source code programs that utilize specific functions (e.g., database operations, computer graphics programs, asynchronous methods, etc.). These source code programs are used to generate training and validation datasets (collectively referred to as box 502).

[0088] Each selected source code program 110 is then parsed and / or compiled by the compilation component 112 to produce a specific syntax tree (box 504).

[0089] Byte-pair encoding is used to generate an ordered sequence of tokens / subtotals representing the context of the source code program. This is achieved by traversing the concrete syntax tree (PTB), resulting in a serialized sequence of syntax nodes and tokens. In one aspect, the PTB is traversed in depth-first order (i.e., depth-first search, depth-first traversal). The depth-first traversal starts at the root node and proceeds along a single path until a terminal or leaf node is reached. The traversal then backtracks until it can choose another path to traverse. This process is repeated until all nodes have been visited. Next, the token / subtotal sequence is converted into a numeric vector (collectively referred to as Box 506).

[0090] A portion of the sequences is used as the training dataset, and another portion is used as the validation dataset. The training dataset is divided into periods, and then the sequences within each period are divided into batches. Each sequence in each batch (box 510) of each period (box 508) is then used to train the neural transformer model (box 514). Initial values ​​are generated for the token / sequence and position embeddings for each sequence, and then used to form the context tensor (box 512).

[0091] Now for reference Figure 5B The first normalization layer is applied to the context tensor (box 522), and then to the masked self-attention layer (box 524). The output of the masked self-attention layer is fed into the second normalization layer (box 526). The output of the second normalization layer is fed into the first one-dimensional convolutional neural network layer (box 528). The output of the first one-dimensional convolutional neural network layer is then fed into the second one-dimensional convolutional neural network layer (box 530).

[0092] The neural network is trained iteratively, passing the training dataset multiple times before converging to the minimum. Each training iteration includes forward propagation (boxes 528-530), loss calculation (box 532), a backpropagation step (box 534), and then updating the weights by calculating the weight gradients (box 536).

[0093] A loss function estimates the loss or error, which is used to compare the quality of predictions. One approach is to use the classification cross-entropy loss function. Once the loss is calculated, it is backpropagated to the hidden layers that directly contribute to the output. During backpropagation, the partial derivatives of the loss function with respect to the trainable parameters are determined. The weight gradient is calculated as the difference between the old and new values ​​of the weights. Gradient descent techniques are used to adjust the weights to make the loss as close to zero as possible. One approach is to use stochastic gradient descent (SGD) as an optimization algorithm to find the values ​​of the function parameters that minimize the loss function. Backpropagation in time (BPTT) can be used to update the weights.

[0094] Return to reference Figure 5A At the end of each batch, the parameters of the neural network are represented as N. ACCUM Pre-configured frequency updates (box 516). N ACCUM This is the gradient accumulation frequency, with a value of 8 in one aspect. Parameters include token / sub-token embeddings and positional embeddings stored in the corresponding embedding matrix. Other parameters include those of the attention layer and convolutional layer.

[0095] Next, the neural transformer model is validated. Before training the neural transformer model, a set of hyperparameters is randomly selected and then tuned to achieve the desired performance. The neural transformer model is tested using a validation dataset to determine the appropriate hyperparameter settings to achieve the expected goal. When the expected goal is not achieved, one or more hyperparameters are tuned and training is repeated until the target goal (collectively referred to as box 518) is reached.

[0096] Evaluation metrics are used to test the quality of candidate recommendations. One aspect used is the mean reciprocal ranking (MRR) of the top-ranked candidates. k A precision method for performing the evaluation. (Previous) k A precision is defined as: ,

[0097] And MMR is defined as: ,

[0098] Where N top-k Indicates the preceding k The number of relevant recommendations in each suggestion. Q This represents the total number of test data samples. rank i This is a recommended predicted ranking.

[0099] The precision in the top-1 (top-1) indicates how often the top recommendation is correct, while the precision in the top 5 indicates how often the top three recommendations in the list contain the candidate the user is looking for. MRR captures the ranking of the results, thus providing information beyond the top candidates. The higher the MRR value, the lower the overall ranking of the correct recommendation. (Collectively referred to as Box 518).

[0100] After model validation is complete, the model is ready to be deployed in the code completion system (Box 520). Now turn to the discussion to illustrate an exemplary approach to the inference phase of using a neural transformer model in a code completion system.

[0101] Figures 6A-6B Exemplary methods 600 and 608 for code line completion using a neural transformer model are illustrated. (Reference) Figure 1 and Figure 6ACode completion is performed in a development environment such as a source code editor 130. The source code editor 130 is configured to interact with a code completion component 142, which performs a bundle search using a neural transformer model. The source code editor 130 performs a background parsing process that monitors characters input into the source code editor and continuously parses the source code to update the specific syntax tree (box 602) representing the current line of code.

[0102] The user interface 132 of the source code editor 130 detects requests for candidate sequences to complete the current line of source code. These requests can be initiated by marker characters, such as the equals sign "=", where the code completion system will provide candidate sequences after the equals sign to complete the remainder of the expression. (Collectively referred to as box 604).

[0103] Alternatively, users can request candidates by entering specific keystrokes or keystroke sequences, such as a combination of the CTRL key and the spacebar. On another front, the system can automatically display a single top candidate at the end of the current source code line in dark mode, regardless of the marker character. The system builds and continuously updates the candidate tree in the background, regardless of whether the user decides to trigger the candidate. This candidate automatically appears in the user interface after a period of inactivity. If the user wants to accept the candidate, they can type a specific keystroke or keystroke combination (e.g., CTRL and I) to accept it. In this case, the cursor position moves to the end of the suggested code sequence, and the dark mode of the candidate code changes to the normal color of the code. If the user does not want to use the candidate, it disappears as the user continues typing. In this case, the system refines the code sequence based on a prefix filter in the candidate tree based on the newly typed code. (Collectively referred to as Box 604).

[0104] Upon detecting a request for a candidate sequence, the concrete syntax tree is parsed to extract tokens / subtotals from the current code segment. Embeddings are obtained from the token / subtotal embedding matrix and the position matrix. A context tensor is generated from the embeddings (collectively referred to as box 606).

[0105] Then perform a beam search until the probability distribution indicates that the next possible token is the end-of-line token (box 608).

[0106] refer to Figure 6B The beam search uses a neural transformer model that leverages the context tensor to generate a probability distribution of token / sub-token vocabulary (box 614). If the probability distribution indicates that the next possible token is a line-end token, the beam search ends (box 616 - yes) and the previous token is output. k There are 10 candidate sequences (box 618).

[0107] Otherwise, choose the first part of the completed sequence. kIndividual tokens / subtotals (box 620).

[0108] Each selected token / sub-token is then fed into the corresponding context vector, again having a separate data path through the neural transformer model. The context vector utilizes the selected tokens / sub-tokens in the current context vector, and the last token / sub-token is removed. The new context vector will be generated by... T Composed of individual tokens / subtotals, the selected token / subtotal c k The last token / subtotal is added to the beginning of the sequence, while the last token / subtotal is removed from the sequence. If the current context vector contains... c 0 、c 1 、...、c T If the token / subtotal sequence consists of a sequence of tokens, then the new context vector will be composed of... c k ，c 0 、c 1 、...、c T-1 Composition. (Collectively referred to as frame 622).

[0109] Return to reference Figure 6A The search beam keeps track of the sequence generated in the search tree and returns the preceding candidate sequences to the user interface widget for display to the user (box 610). The user can select one of the candidates and then enter it into the source code program to complete the line of source code (box 612). Alternatively, the user can ignore the candidate sequence and continue typing. This process (boxes 602-612) is repeated until the user closes the source code program, exits the source code editor, or terminates the code completion tool.

[0110] Code line completion example

[0111] Now turn your attention to an exemplary user interface display of a code completion tool that uses the techniques described in this article. Figure 7 , Figure 7 This shows a source code program being edited in a source code editor. The user interface displays lines 10-36 of source code program 702. After the user enters the "=" character, pop-up window 704 appears on line 36. Pop-up window 704 contains five candidate sequences to complete the code line 36. The five candidates 706-714 are displayed in ranking order from highest to lowest probability. Each candidate is an ordered sequence of tokens that is very likely to complete the expression on line 36.

[0112] Figure 8 From the Figure 7A diagram illustrating the search tree 800 generated by the beam search of the source code fragment shown. Search tree 800 tracks all states generated by the neurotransformer model in its nodes. In this example, the beam width is set to four (4). At the first inference level or execution of the neurotransformer model 802, the beam search generates a root node 816 with a probability distribution for each token / sub-token in the vocabulary. The first four tokens / sub-tokens are then selected: “tf”, “gradient”, “gan”, and “gd”. Each selected token is added to a separate context vector, which is then used in subsequent executions of the neurotransformer model. The probability distribution for each token / sub-token in the second inference level 804, generated by each call to the neurotransformer model 818A-818D, is shown.

[0113] Then, the first four tokens / sub-tokens are selected from each node in the second inference level 804, which generates a new context vector. The third call to the neural transformer model is performed using new nodes generated from each selected token / sub-token in the second inference level 804, as shown in the third inference level 806. This process is repeated again until the search ends. Figure 8 The search tree 800 is shown, which is generated by the first seven reasoning levels 802, 804, 806, 808, 810, 812, and 814.

[0114] like Figure 8 As shown, candidate sequences tf.train.AdamOptimizer(learning_rate It consists of the following: the token / sub-token obtained by reasoning in root node 816. tf The token / sub-token "." obtained from node 818A in the second inference level 804, and the token / sub-token obtained from node 820 in the third inference level 806. train The token / sub-token "." derived from node 822 at level 4 (808), and the token / sub-token derived from node 824 at level 5 (810). AdamOptimizer The token / sub-token obtained from node 826 at the sixth reasoning level 812 （ ", and the token / sub-token derived from node 828 at the seventh reasoning level 814." learning .

[0115] Exemplary operating environment

[0116] Now turn your attention to the discussion of the exemplary operating environment. Figure 9An exemplary operating environment 900 is illustrated, in which one or more computing devices 902 are used to train a neural transformer model and a second computing device 904 uses the neural transformer model for code completion. However, it should be noted that the aspects disclosed herein are not limited to any particular configuration of the devices. Either computing device 902, 904 can utilize the neural transformer model in its own code completion system, and computing device 904 can also generate and test the neural transformer model. Computing device 902 can be configured to generate neural transformer models as a cloud service for use with other code completion systems. It should be noted that the operating environment is not limited to any particular configuration and other configurations are possible.

[0117] Computing devices 902 and 904 can be any type of electronic device, such as, but not limited to, mobile devices, personal digital assistants, mobile computing devices, smartphones, cellular phones, handheld computers, servers, server arrays or server clusters, web servers, network servers, blade servers, internet servers, workstations, minicomputers, mainframes, supercomputers, network devices, web devices, distributed computing systems, multiprocessor systems, or combinations thereof. The operating environment 900 can be configured in a network environment, a distributed environment, a multiprocessor environment, or in a standalone computing device capable of accessing remote or local storage devices.

[0118] Computing devices 902 and 904 may include one or more processors 908 and 940, one or more communication interfaces 910 and 942, one or more storage devices 912 and 944, one or more input / output devices 914 and 949, and one or more memory devices 919 and 948. Processors 908 and 940 may be any commercially available or custom processor and may include dual-microprocessor and multiprocessor architectures. Communication interfaces 910 and 942 support wired or wireless communication between computing devices 902 and 904 and other devices. Storage devices 912 and 944 may be computer-readable media that do not contain propagated signals such as modulated data signals transmitted via a carrier wave. Examples of storage devices 912 and 944 include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, and disk storage, all of which do not contain propagated signals such as modulated data signals transmitted via a carrier wave. Multiple storage devices 912 and 944 may be present in computing devices 902 and 904. Input / output devices 914, 946 may include keyboards, mice, pens, voice input devices, touch input devices, displays, speakers, printers, and any combination thereof.

[0119] Memory devices 916 and 948 can be any non-transitory computer-readable storage medium capable of storing executable programs, applications, and data. Computer-readable storage media are not propagating signals, such as modulated data signals transmitted via a carrier wave. It can be any type of non-transitory storage device (e.g., random access memory, read-only memory, etc.), magnetic storage, volatile storage, non-volatile storage, optical storage, DVD, CD, floppy disk drive, etc., which are not propagating signals, such as modulated data signals transmitted via a carrier wave. Memory devices 916 and 948 may also include one or more external storage devices or remotely located storage devices, which are not propagating signals, such as modulated data signals transmitted via a carrier wave.

[0120] Computing device 904 can utilize an Integrated Development Environment (IDE) 954 within the computer system, which allows users (e.g., developers, programmers, designers, coders, etc.) to design, code, compile, test, run, edit, debug, or build programs, sets of programs, websites, web applications, and web services. Software programs may include source code files created in one or more source code languages ​​(e.g., Visual Basic, Visual J#, C++, C#, J#, JavaScript, APL, COBOL, Pascal, Eiffel, Haskell, ML, Oberon, Perl, Python, Scheme, Smalltalk, etc.). IDE 954 can provide a local code development environment, managed code development running on a virtual machine, or a combination thereof. IDE 954 can provide a managed code development environment using the .NET Framework. It should be noted that this embodiment is not limited to providing source code development services through an IDE; other tools, such as a standalone source code editor, can also be used instead.

[0121] Users can create and / or edit source code files 952 using the user interface 958 and source code editor 956 in the IDE 954, based on known software programming techniques and specific logic and syntax rules associated with a specific source language. The source code files 952 can then be compiled by the compilation unit 960 to generate data structures representing the syntactic structure and semantic model of the source code.

[0122] The memory device 948 of the computing device 904 may contain instructions, components, and data. A component is a software program that performs a specific function, also known as a module, program, and / or application. The memory device 948 may include an operating system 950, one or more source code program files 952, an IDE 954 that may include a source code editor 956, a user interface 958, a compilation component 960, a code completion component 962, and a neural converter model 964, as well as other applications and data 966.

[0123] The memory device 916 of the computing device 902 may include an operating system 918, a source code extraction component 920, a token / subtotal sequence extraction component 922, a parser 924, a model training and testing component 926, a neural transformer model 928, a source code repository 930, and other applications and data 932.

[0124] Computing devices 902 and 904 can be communicatively coupled via network 909. Network 909 can be configured as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless local area network (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, part of the public switched telephone network (PSTN), ordinary old-style telephone service (POTS) network, wireless network, WiFi® network, or any other type of network or combination of networks.

[0125] Network 909 can employ a variety of wired and / or wireless communication protocols and / or technologies. The different generations of communication protocols and / or technologies that the network can employ may include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access 2000 (CDMA-2000), High-Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), Universal Mobile Telecommunications System (UMTS), Evolved Data Optimized (Ev-DO), WiMax, Time Division Multiple Access (TDMA), Orthogonal Frequency Division Multiplexing (OFDM), Ultra Wideband (UWB), Wireless Application Protocol (WAP), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), any part of the Open Systems Interconnection (OSI) model protocol, Session Initiation Protocol / Real-Time Transport Protocol (SIP / RTP), Short Message Service (SMS), Multimedia Messaging Service (MMS), or any other communication protocol and / or technology.

[0126] in conclusion

[0127] A system is disclosed, comprising one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions to: track a sequence of characters input into a line of source code program during an editing session; and at a location within the line of source code program, generate candidate sequences using a neural transformer model to complete the line of source code, wherein the neural transformer model is trained on an unsupervised dataset of source code programs written in one or more different programming languages.

[0128] The system includes other instructions, when executed by one or more processors: initiating a beam search to construct a search tree to generate candidate sequences, wherein the search tree comprises one or more nodes at one or more inference levels, each node representing an output probability distribution of a set of tokens from the vocabulary of a neural transformer model, wherein the output probability distribution is generated by the neural transformer model, and each node will... k Each token / subtotal is expanded to the next inference level. The beam search iteratively expands the search tree by invoking a neural transformer model to predict the next token given a partial candidate token sequence that completes a line of code.

[0129] In one aspect, the neural transformer model consists only of decoder blocks. The neural transformer model includes at least one decoder block with a masked self-attention layer. The neural transformer model includes at least one one-dimensional convolutional neural network layer.

[0130] The system tracks the character sequences input into source code lines by: obtaining a sequence of tokens / subtotals representing the current context of the code line, and finding the token / subtotal embedding vector and positional embedding vector for the sequence of tokens / subtotals. The token / subtotal embedding vector and positional embedding vector are pre-trained.

[0131] The system includes instructions to input token / sub-token embedding vectors and position embedding vectors into a neural transformer model. The neural transformer model generates a probability distribution of tokens / sub-tokens in the model vocabulary.

[0132] A method is disclosed, comprising: monitoring each token input into a line of code of a source code program during a source code development session; iteratively performing a bundle search to generate token candidates to complete the line of code as a new token is input, wherein the bundle search generates token candidates using a token probability matrix generated from a neural transformer model; concatenating the token candidates into a candidate sequence to complete the line of code; and outputting at least one candidate sequence when a token character input into the line of code is detected during the source code development session.

[0133] The method further includes invoking a neural transformer model to predict the next token given a context vector, which represents the context of the line of code that includes the new token.

[0134] In one aspect, the neural transformer model comprises a self-attention layer and a convolutional neural network. Layer normalization is performed before the self-attention layer, and this normalization is applied to the output of the self-attention layer. The neural transformer model utilizes token embeddings and positional embeddings representing the code line context, where the token embeddings and positional embeddings are pre-trained.

[0135] On the other hand, monitoring each token input into the source code program further includes: parsing the input into a concrete syntax tree; performing byte-pair encoding to extract tokens from the concrete syntax tree; and concatenating an ordered sequence of tokens of length T.

[0136] A device is disclosed, comprising at least one processor coupled to a memory device. The at least one processor is configured to: extract one or more ordered token sequences from a plurality of source code programs, wherein the ordered token sequences represent the context of a selected source code segment from the plurality of source code programs; and train a neural transformer model using the ordered token sequences to predict the next token to complete a partial sequence of tokens, wherein the partial sequence of tokens is used to generate candidate token sequences for completing lines of code in a target source code program, wherein the neural transformer model includes an attention layer and at least one convolutional neural network layer.

[0137] In one aspect, the ordered sequence of tokens comprises one or more sub-tokens. The neural transformer block is a decoder-only transformer. In some aspects, at least two of the multiple source code programs are written in different programming languages, and the ordered sequence of tokens is an unsupervised training dataset. In some aspects, the neural transformer model generates a token probability matrix, which is used to predict the next token that will succeed in the predicted candidate sequence.

[0138] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A computer-implemented method, comprising: Extract multiple ordered token sequences from multiple source code programs, wherein the ordered token sequence represents the context of a source code segment from one of the multiple source code programs selected by the source code programs; as well as The multiple ordered token sequences are used to train a decoder-only neural transformer model with an attention mechanism to learn to predict the next token to complete a partial token sequence, wherein the partial token sequence completes a line of code in a target source code program, and wherein the decoder-only neural transformer model with an attention mechanism includes one or more decoder blocks, each decoder block including an attention layer and a neural network layer.

2. The method of claim 1, wherein the attention layer comprises a masked self-attention layer having a plurality of attention heads.

3. The method according to claim 1, wherein the plurality of source code programs belong to the same programming language.

4. The method of claim 1, wherein the plurality of source code programs includes at least two source code programs written in two different programming languages.

5. The method of claim 1, wherein the plurality of source code programs come from two or more domains.

6. The method according to claim 1, further comprising: Each of the plurality of source code programs is parsed into a specific syntax tree; as well as Each specific syntax tree is traversed to form the ordered token sequence.

7. The method according to claim 6, further comprising: The ordered token sequence is mapped to a token embedding sequence and a position embedding sequence.

8. A system comprising: At least one processor and memory; The memory stores instructions, which, when executed on the at least one processor, perform the following actions: Obtain unsupervised training datasets from multiple source code programs; Each of the plurality of source code programs is converted into an ordered token sequence of a predetermined length, wherein the ordered token sequence represents one or more lines of source code; as well as The ordered token sequence is used to train a decoder-only neural transformer model with an attention mechanism to learn to predict the token sequence used to complete a line of code given the context of the line of code, wherein the context includes the token sequence in the line of code, and wherein the decoder-only neural transformer model with an attention mechanism includes one or more decoder blocks, each decoder block including an attention layer and a neural network layer.

9. The system of claim 8, wherein the plurality of source code programs includes a first set of source code programs in a first programming language and a second set of source code programs in a second programming language.

10. The system of claim 8, wherein the plurality of source code programs belong to the same programming language.

11. The system of claim 8, wherein the memory stores instructions that, when executed on the at least one processor, perform the following actions: The ordered token sequence is mapped to a token embedding sequence and a position embedding sequence.

12. The system of claim 11, wherein the token embedding sequence and the position embedding sequence are input to train the decoder-only neural transformer model with attention mechanism.

Images