Code adaptation by deep learning

Through deep learning models, especially attention-friendly neural transformer models, the undefined variable names in the source code snippets are automatically detected and replaced, and the problem that copying and pasting source code snippets cannot directly adapt to pre-existing source code is achieved. Correct program generation of syntax and semantics improves development efficiency.

CN118922810APending Publication Date: 2024-11-08MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280092971.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-24
Filing Date
2022-12-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

During source code development, copy-pasteed source code snippets often cannot be directly adapted to pre-existing source code, resulting in developers having to manually change to adapt to semantic and syntax-correct programs.

Method used

Using deep learning models, especially attention-based neural transformer models, the most likely variable names are predicted by automatically detecting and replacing undefined variable names in pasted source code snippets, using the context of pre-existing source code programs to predict the most likely variable names. Code adaptation.

Benefits of technology

It realizes automatic adaptation of source code snippets, generates syntactic and semantic correct programs, reduces user intervention and improves development efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922810A_ABST
    Figure CN118922810A_ABST
Patent Text Reader

Abstract

And the code adaptation mechanism automatically integrates the variable names of the pasted source code fragments into variable names defined in a pre-existing partial source code program. The variable name from the pasted source code fragment is replaced with an anonymous value. A deep learning model predicts the most probable variable name from a pre-existing partial source code program to replace each anonymous value. A deep learning model is trained over a number of variable usage patterns from various source code programs to learn a most probable mapping that predicts undefined variable names from pasted source code fragments to variable names in pre-existing portions of source code programs to generate syntactically and semantically correct programs.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Code completion is a tool that predicts the next string of characters a developer (e.g., user, end-user, programmer, etc.) is likely to type into a source code development tool (such as a source code editor, integrated development environment, etc.). Source code may consist of various code elements (e.g., keywords, delimiters, variables, methods, constants, operators, etc.) that are combined in a specific order to form expressions according to the syntax of the underlying programming language. Code completion is used to complete partially formed source code snippets, such as lines of source code, method calls, method signatures, or method bodies.

[0002] During source code development, developers often copy source code snippets from external sources (such as GitHub or Stack Overflow) and paste them into the source code program. However, the copy-pasted source code snippets are often not adaptable to fit into the pre-existing source code, resulting in the developer manually changing the copy-pasted source code snippets to adapt the copy-pasted source code snippets to form a semantically and syntactically correct program. Summary of the invention

[0003] This summary is intended to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] The code adaptation mechanism automatically integrates the variable names of a source code snippet copied and pasted into a pre-existing partial source code program with the variable names defined in the pre-existing partial source code program. The variable names from the pasted source code snippet are replaced with anonymous values. The deep learning model predicts the most likely variable names from the pre-existing partial source code program to replace the anonymous values.

[0005] These and other features and advantages will be apparent from a reading of the following detailed description and a review of the associated drawings.It is to be understood that both the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic diagram illustrating an exemplary system for adapting source code through deep learning.

[0007] Figure 2 is a schematic diagram showing an exemplary configuration of a neural transformer model with attention.

[0008] Figure 3A is a schematic diagram showing an exemplary architecture of a neural transformer model in the first embodiment, and Figure 3B is a schematic diagram showing an exemplary architecture of a neural transformer model in the second embodiment.

[0009] Figure 4 is a flow chart illustrating an exemplary method for self-supervised training and deployment of a deep learning model.

[0010] Figure 5 is a flow chart illustrating a first exemplary method for code adaptation.

[0011] Figure 6 is a flow chart illustrating a second exemplary method for code adaptation.

[0012] Figure 7 is a block diagram illustrating an exemplary operating environment. DETAILED DESCRIPTION

[0013] Overview

[0014] Aspects of the present disclosure relate to automatically adapting external source code snippets to pre-existing partial source code programs to form syntactically and semantically correct programs. In one aspect, the technology adapts variable names found in the external source code snippet to conform to the variable usage of the pre-existing partial source code program. A deep learning model is used to predict the most likely variable names of the pre-existing partial source code program to associate with undefined variable names of the external source code snippet.

[0015] In one aspect, the automatic adaptation technique is used for code completion. A code completion component of a source code development environment detects the insertion of a source code snippet copied from an external source and pasted into a pre-existing partial source code program, and automatically generates a mapping of undefined variable names in the pasted source code snippet to defined variable names used in the pre-existing partial source code program. The mapping is based on a probability distribution generated by a deep learning model, which is used to determine the most likely variable names of the defined pre-existing partial source code program to replace the undefined variable names in the pasted source code snippet.

[0016] Attention is now turned to a more detailed description of the systems, components, methods, and apparatus used in various aspects of the source code adaptation technique.

[0017] system

[0018] Figure 1 1 shows a block diagram of an exemplary system 100 in which various aspects of the present invention may be practiced. Figure 1As shown in , system 100 includes a pre-existing or original partial source program 102 , a static code analyzer 104 , an extended context extractor 106 , a variable name anonymizer 108 , a deep learning model 110 , and a variable name mapper 112 .

[0019] The pre-existing partial source code program 102 may be embodied in a software development environment 102 such as a source code editor, an integrated development environment (IDE), or an application in which source code is developed. Pasted source code snippet 115 is copied from an external source 114 into the pre-existing partial source code program 102. The external source has a vocabulary scope outside of the pre-existing partial source code program. The external source includes, but is not limited to, an external website (e.g., GitHub, Stack Overflow), an external file, or other external source.

[0020] The static code analyzer 104 parses the source code program into a concrete syntax tree 116, from which other static code analysis is performed. The concrete syntax tree 116 represents the grammatical structure of the program in a hierarchical structure or a tree structure. The concrete syntax tree 116 is an n-ary tree data structure that includes nodes representing constructs in the syntax of the programming language of the program. The concrete syntax tree 116 includes a root node, multiple internal nodes, and multiple terminal nodes. The terminal nodes represent source code tokens. Source code tokens are symbols that represent operands or operators.

[0021] An extended context of a pre-existing partial source code program is generated by an extended context extractor 106 using a concrete syntax tree 116. The deep learning model 110 operates on sequence data of fixed dimension. The deep learning model 110 is constrained to sequence data with a limited number of source code tokens that are included in the input sequence. The size of the input sequence affects the accuracy of the deep learning model. A larger size provides more context from which the model can understand the structure of the source code in order to generate more accurate results. However, increasing the size of the input sequence is problematic because it requires a huge amount of computing resources to train the model and make predictions that may be impractical.

[0022] To remedy this problem, the model is trained using the extended context of the source code program 120 based on a grammatical hierarchy. The grammatical hierarchy prioritizes certain code elements over other elements to represent the context of the pasted source code snippet in the pre-existing partial source code program. A code element is a series of consecutive source code tokens that correspond to a set of specific syntax tree nodes. The grammatical hierarchy is represented by a priority list that indicates the order in which the grammatical elements in the source code program are extracted and used as the extended context. The priority list of code elements represents those code elements that are more closely associated with the context of the pasted source code snippet and are not limited to the source code immediately preceding the pasted source code snippet. The priority list of code elements includes code elements defined outside the method that embodies the pasted source code snippet. In this way, the model is given a more effective view back into the source code program for the model to learn prediction patterns for making predictions.

[0023] The term scope or lexical scope, as used in computer science, refers to the portion of a source code program that defines the binding of names to elements (variables, methods, constants, etc.). Local scope refers to when an element is defined inside a method or function that uses it, and global scope refers to when an element is defined outside of a method that uses it. The syntactic hierarchy of precedence lists places certain elements in a program above others, and can include elements in a source code program that are part of the local scope of another method in the program. Syntactic elements of other scopes, such as methods or classes defined outside of the method that embodies the pasted source code snippet, are included as part of the extended context.

[0024] In one aspect, an exemplary syntax hierarchy includes the following syntax elements in order of priority: (1) the pasted source code snippet; (2) the method signature of the focus method (i.e., the method that embodies the pasted source code snippet), the documentation string of the focus method (if any), and the class name of the focus method; (3) global import statements; (4) assigned values, but not assigned expressions; (5) class attributes; (6) peer class method signatures, which are method signatures of the same class as the focus method; (7) the class documentation string (if any) is a modification of the class of the focus method; (8) the peer class method documentation string (if any) is the documentation string of a method of the same class as the focus method; (9) global expressions; and (10) the source code body of the peer class method of the focus method.

[0025] The static code analyzer 104 also generates a control flow graph to perform data flow analysis on a pre-existing partial source code program using the pasted source code snippet. The data flow analysis 118 analyzes the data flow of the program to generate information about the definition and use of variables in the program. The data flow analysis 118 generates def-use and use-def chains. The def-use chain lists the uses of a specific variable name that are reachable from the definition of the variable, while the use-def chain indicates all uses of the variable name that are reachable from the definition of the variable.

[0026] Variable name anonymizer 108 uses data flow analysis 118 to identify variable names from pasted source code snippet 115. Variable names used in the pasted source code snippet are replaced with masked or anonymous values ​​123.

[0027] The extended context 122 with anonymous values ​​is input into the deep learning model 110, which outputs a probability distribution 124 indicating the most likely label to replace each mask value. The probability distribution 124 is used to map each mask value or anonymous value to a variable name defined in a pre-existing partial source code program. The variable name mapper 112 replaces each mask value with a predicted variable name, thereby generating a modified source code program 126.

[0028] like Figure 1 As shown in FIG. 1 , a developer (i.e., a user, developer, customer, etc.) copies three lines of source code of the method body of ranking_loss from an external source 114 into a pre-existing partial source code program 102. The pasted source code snippet 115 includes variable names partial_losses, y_pred, loss, and y_true. All variables of the pasted source code snippet are anonymized with mask values ​​(e.g., _v1, _v2, _v3, _v4) 123. The extended context 122 with the mask values ​​is input into the deep learning model 110.

[0029] The deep learning model 110 outputs a probability distribution 124 indicating the most likely variable name to replace each mask value. The variable name mapper 112 replaces each mask value with the corresponding predicted variable name. As shown in the modified source code program 126, line 128 is changed to include predicted_classes instead of y_pred, and line 130 is changed to include target_classes instead of y_true. Both predicted_classes and target_classes are defined in the pre-existing partial source code program.

[0030] Attention is now turned to the description of various embodiments of the deep learning model.

[0031] Neural Transformer Model

[0032] The Neural Transformer model with attention is a unique type of machine learning model. Machine learning involves the use and development of computer systems that can learn and adapt without following explicit instructions by using algorithms and statistical models to analyze patterns in data and draw inferences from them. Machine learning uses different types of statistical methods to learn from data and predict future decisions. Traditional machine learning includes classification models, data mining, Bayesian networks, Markov models, clustering, and visual data mapping.

[0033] Deep learning differs from traditional machine learning because it uses multi-stage data processing through many hidden layers of a neural network to learn and interpret features and the relationships between features. Deep learning embodies neural networks, which is different from traditional machine learning techniques that do not use neural networks. The Neural Transformer model is a type of deep learning that utilizes an attention mechanism. Attention guides the neural network to focus on a subset of features or tokens in the input sequence, thereby learning different representations from different positions of the tokens in the input sequence. The Neural Transformer model utilizes attention instead of using recurrent neural networks (RNNs) (e.g., long short-term memory (LSTM) networks) and convolutional neural networks (CNNs) to handle the dependencies between its input and output.

[0034] It should be noted that the terms neural transformer model and neural transformer model with attention are used interchangeably. It should also be noted that aspects disclosed herein are described with respect to attention-based neural transformer models. However, these techniques are not limited to these types of neural networks and can be applied to other types of deep learning models that utilize neural networks with fixed-size context windows.

[0035] Figure 2 An exemplary structure of a neural transformer model in an encoder-decoder configuration is shown. The neural transformer model 200 includes one or more encoder blocks 202A-202B ("202") and one or more decoder blocks 204A-204B ("204"). The input layer 206 of the first encoder block 202A includes an input embedding layer 210, a position embedding layer 208, and a context tensor 212.

[0036] The initial input to the first encoder block 202A is the input embedding 210 of the input sequence with the extended context 256 of the anonymous value. In order to preserve the order of the tokens in the input sequence, the position embedding 208 is added to the input embedding 210, thereby forming a context tensor 212. The initial input to the first decoder block 204A is <start>tagged, and at each subsequent time step, the input is a sequence of output embeddings 226 shifted from the previous time step, to which the position embedding 228 is added, forming a context tensor 230.

[0037] The encoder block 202 consists of two layers. The first layer includes a multi-head self-attention component 214, followed by a layer normalization component 216. The second layer includes a feedforward neural network 218, followed by a layer normalization component 220. The context tensor 212 is input into the multi-head self-attention layer 214 of the encoder block 202 and establishes a residual connection with the layer normalization 216. The output of the layer normalization 216 is input into the feedforward neural network 218 and establishes another residual connection with the layer normalization 220. The output of the encoder block 202 is a set of hidden representations 217. The set of hidden representations 217 is then sent through an additional encoder block if there are multiple encoder blocks, or to the decoder 222.

[0038] Attention is used to decide which parts of the input sequence are important for each token, especially when decoding long sequences because the encoder is limited to encoding fixed-size vectors. The attention mechanism collects information about the relevant context of a given token and then encodes that context into a vector representing that token. It is used to identify relationships between tokens in long sequences while ignoring other tokens that do not have much influence on a given prediction.

[0039] The multi-head self-attention component 214 takes the context tensor 212 and weights the relevance between each token represented in the context tensor by generating an attention weight for each token in the input embedding 210. In one aspect, the attention function is a scaled dot product attention, which is mathematically described as follows:

[0040]

[0041] The input consists of dimensions d k The query Q and key K and dimension d v Q is a matrix containing the query or vector representation of one token in the sequence, K is the vector representation of all tokens in the sequence, and V is the vector representation of all tokens in the sequence.

[0042] Query, keys and values ​​with d v The output values ​​are linearly projected h times in parallel and these output values ​​are concatenated into the final value:

[0043] MultiHead(Q,K,V)=Concat(head1,…,head h )W o ,

[0044] Among them head i =Attention(QW i Q ,KW i K ,VW i V ),

[0045] Parameter Matrix and

[0046] To reduce the training time of the neural transformer, layer normalization is used between layers. The layer normalization component normalizes the input across features. The mean and standard deviation are calculated across feature dimensions. There is a first layer normalization 216 before the feedforward neural network 218, and there is a second layer normalization 220 after the feedforward neural network 218.

[0047] The feed-forward neural network 218 processes each output encoding separately. The output of the top encoder block is a set 222 of attention vectors K and V, which are used by the encoder-decoder multi-head self-attention layer 236 of the decoder block 204.

[0048] The decoder block 204 takes all previously generated target labels t1, ...t i-1 Predict each token t in the target language for the condition one by one i . The decoder block 204 consists of three layers. The first layer includes a masked multi-head self-attention component 232, followed by a layer normalization component 234. The output of the layer normalization component 234 is input into the encoder-decoder multi-head attention component 236 and establishes a residual connection with the layer normalization component 238. The second layer includes an encoder-decoder multi-head self-attention component 236, followed by a layer normalization component 238. The output of the layer normalization component 238 is input into the feedforward neural network 240 and establishes a residual connection with the layer normalization component 242. The third layer includes a feedforward neural network 240, followed by a layer normalization component 242.

[0049] The masked multi-head self-attention component 232 receives the output embedding of the previous time step. The masked multi-head self-attention component 232 masks the output embedding from the future time step. The encoder-decoder multi-head self-attention layer 236 receives the query from the previous decoder layer 224 and the memory key and value 222 from the output of the encoder block 202. In this way, the decoder block 204 can pay attention to each position of the input sequence. The feedforward neural network 240 processes each output encoding separately. Layer normalization components 234, 238, 242 are used between layers to normalize the input across features.

[0050] The output layer 246 includes a linear layer 248 and a softmax layer 250. The linear layer 248 projects the vector produced by the decoder stack into a logits vector. The softmax layer 250 then transforms the scores of the logits vector into output probabilities for each token in the vocabulary V, which are positive and normalized.

[0051] Training is the process of learning the parameters (i.e., embeddings, weights, biases) of the model from a training dataset. During training, the first encoder block receives a self-supervised training dataset of input sequences. The input sequences include sequences of input embeddings representing source code tokens with extended contexts of anonymous or masked values. In this way, the model learns the grammatical structure of the programming language of the source code snippets and the semantics of variable usage from the training samples.

[0052] Inference is the process by which a model makes predictions given an input sequence of data. During inference, the first encoder block of the model receives an extended context 256 of the program with anonymous values.

[0053] During training, the first decoder block receives a signal indicating the start of <start>258. Thereafter, at each subsequent time step, the input embedding 226 is the output embedding of 258 shifted by one token. During inference, the input sequence is based on a specific embodiment of the decoder block described below.

[0054] Model configuration

[0055] Go to Figure 3A , a first embodiment of an encoder-decoder neural transformer model 300 is shown. In this embodiment, the encoder-decoder neural transformer model is configured with a single encoder unit 304 and multiple decoder units 306A-306N. The encoder unit 304 includes multiple stacked encoder blocks, and the decoder units 306A-306N include multiple stacked decoder blocks. There are different decoder units for each anonymous value 306A-306N. The encoder output is transmitted to each decoder unit 322A-322N simultaneously. Each decoder unit is responsible for decoding a specific anonymous value 306A-306N. Each decoder unit 322A-322N is trained in parallel using the same training data and learns the same parameter values ​​(i.e., embedding, attention weights, biases). The probability distribution output from each decoder unit 310A-310N indicates the probability of each token in the vocabulary of the model as a variable name that replaces a specific anonymous value associated with the decoder 310A-310N.

[0056] During the training phase of the model, the training dataset includes input sequences, where each input sequence represents an extended context with anonymous values ​​and anonymous value mapping 302. During the inference phase of the model, the input sequence of the first encoder block represents the extended context with anonymous values ​​302. The first input sequence of the first decoder block is the anonymous value as the start symbol, and thereafter the input sequence of the first decoder block is the shifted output embedding sequence 308A-308N.

[0057] Go to Figure 3B In a second embodiment, an encoder-decoder neural transformer model is configured with a single encoder unit 314 and a single decoder unit 316. The model is trained on a supervised training dataset comprising an input sequence consisting of a sequence of token embeddings representing an extended context 312 with anonymous values. The decoder outputs a probability distribution 318 over the vocabulary of the model. The vocabulary of the model consists of mappings, where each mapping includes an anonymous value and its corresponding variable name in a pre-existing partial source code program.

[0058] method

[0059] Attention is now directed to a more detailed description of the method for constraint decoding used in the system for source code generation and translation. It will be appreciated that the representative method does not necessarily have to be performed in the order presented or in any particular order unless otherwise indicated. In addition, the various activities described with respect to the method can be performed in a serial or parallel manner, or in any combination of serial and parallel operations. In one or more aspects, the method illustrates the operation of the system and device disclosed herein.

[0060] train

[0061] Figure 4 An exemplary method 400 of a code adaptation process is shown. A neural transformer model is trained to learn variable usage patterns.

[0062] Training of the model is performed by applying a training dataset of input sequences to the model. The training dataset is extracted from source code files of various source code repositories, which may be public or private. For each source code file, the pasted source code snippets are extracted by randomly selecting pastable blocks in these files, removing them from the original source code file, and pasting them back into the file. This training method is considered self-supervised training. The training dataset is split into a training dataset and a validation dataset. (collectively referred to as block 402).

[0063] The training dataset includes an extended context with anonymous values ​​and anonymous value mappings. The model learns the correct actions the model should take from the loss calculation and learns to generalize its actions to take actions in situations that are not present in the training dataset. (collectively referred to as block 402).

[0064] Each sample of the training data set is parsed into a concrete syntax tree (block 604). A priority list is obtained and used to extract a sequence of tokens representing each code element of the syntax hierarchy from the concrete syntax tree in the order indicated in the priority list. The sequence of tokens is tokenized into sub-tokens using byte pair encoding tokenization and transformed into embeddings, thereby forming an embedded input sequence. The terms "token" and "sub-token" are used interchangeably. (collectively referred to as block 402).

[0065] The training dataset consists of thousands of samples that are applied to the model. The Neural Transformer model is trained iteratively, performing multiple rounds on the training dataset before converging to a minimum. An epoch represents one pass of the entire training dataset forward and backward through the Neural Transformer block. Since the training dataset is very large, it is divided into smaller batches. Training is iterative, and the entire dataset is passed through the Neural Transformer in multiple iterations. Each training iteration includes a forward propagation, a loss calculation, a backpropagation step, followed by updating the weights. The training dataset is divided into batches, where the input sequence of each batch from the context window runs through the entire training process. (collectively referred to as block 402).

[0066] The neural transformer model has multiple blocks and layers, allowing for learning of more detailed relationships within the data and how features interact with each other at a non-linear level. The model architecture, training process, data normalization, and vocabulary encoding process are hyperparameters that are tailored to meet specific goals. The values ​​of the hyperparameters affect how the parameters are learned. (collectively referred to as block 402).

[0067] For each input sequence of each batch in each epoch, the T sorted sub-token sequence is then mapped into a numeric vector and then mapped into a corresponding sub-token embedding and position embedding. Embedding is a text-based learned representation for sub-tokens, where sub-tokens with common meanings have common representations. Embedding is a mapping of discrete categorical variables to continuous numeric vectors. There is an embedding and a corresponding position embedding for each sub-token in the vocabulary. The sub-token embedding represents the learned representation for the sub-token. The neural transformer model does not read each sub-token sequentially, and therefore, without additional position information, it does not know the position of the sub-token in the sequence. Position embedding is used to embed information about the position of the sub-token in the sequence into the neural transformer model. (collectively referred to as block 402).

[0068] Initial values ​​are generated for the sub-token embedding and position embedding for each sequence, and then these initial values ​​are used to form the context tensor. Thereafter, the neural transformer model learns the value of each embedding. At the completion of the training phase, the embedding and position embedding for each sub-token are saved into corresponding matrices for later use. There is a sub-token embedding matrix We, which contains the embeddings for each sub-token t i , i = 0…V, and there is a position embedding matrix Wp, which contains the embedding vector P for each position j , j = 0 ... T, where V is the size of the vocabulary, and T is the length of the sub-token sequence. (collectively referred to as block 402).

[0069] The first encoder block of the neural transformer model takes the context tensor as input and passes it through multiple layers of multi-head self-attention, layer normalization, and feed-forward neural networks to ultimately produce a set of hidden representations. If there are additional encoder blocks, the output of each encoder block is passed to the next encoder block, where the output of the last encoder block produces a set of hidden representations. The set of hidden representations is passed to each decoder block. (collectively referred to as block 402).

[0070] The decoder block of the neural transformer model takes as input the sequence of shifted output embeddings. The mask in the masked multi-head attention layer is used to prevent positions from paying attention to subsequent positions in the future. The mask combined with the output embedding shifted by one position ensures that the prediction for position T depends only on the known outputs at positions less than T. Starting from the first token of the output sequence, the sub-token is passed through the self-attention layer and the normalization layer and into the encoder-decoder attention layer to be used as a query for the encoder-decoder attention, where the key and value pairs for the attention are the outputs of the encoder. The encoder output is calculated using the entire sequence of input embeddings. (collectively referred to as block 402).

[0071] The feedforward neural network in the encoder block and the decoder block is trained iteratively, so that multiple rounds are performed on the training data set before converging to the minimum value. Each training iteration includes forward propagation, loss calculation, back propagation steps, followed by updating the weights by calculating the weight gradient. The loss function estimates the loss or error, which is used to compare the quality of the prediction results. In one aspect, a categorical cross entropy loss function is used. Once the loss is calculated, it is back-propagated to the hidden layer that directly contributes to the output. In back propagation, the partial derivative of the loss function with respect to the trainable parameters is determined. The weight gradient is calculated as the difference between the old value and the new value of the weight. The weights are adjusted using a gradient descent technique to minimize the loss as much as possible. In one aspect, the stochastic gradient descent (SGD) method is an optimization algorithm for finding the parameter values ​​of a function that minimizes the loss function. The weights can be updated using a back propagation time (BPTT) algorithm. (collectively referred to as block 402).

[0072] When training of the model is complete, the model is deployed in a source code development system (block 404).

[0073] reasoning

[0074] Figure 5 One aspect of an exemplary method 500 for code adaptation using a deep learning model is shown. A developer copies a source code snippet from an external source and pastes it into a pre-existing partial source code program. The pre-existing partial source code program is partially formed in a software development environment. The pasted source code snippet includes variable names that may not be defined in the pre-existing partial source code program, and may include variable names defined in the pasted source code snippet. (collectively, block 502).

[0075] Undefined variable names from the pasted source code snippet are identified by data flow analysis of a pre-existing partial source code program that contains the pasted source code snippet (block 504). Undefined variable names in the pasted source code snippet (except for API calls, keywords, and punctuation marks) are anonymized with mask values ​​(block 506). Each variable name in the pre-existing partial source code program is replaced with a specific mask value (block 506).

[0076] The deep learning model is given the context of the pre-existing partial source code program to predict the variable names defined in the pre-existing partial source code program to replace the undefined variable names in the pasted source code snippet. The deep learning model is trained to learn semantic variable usage patterns in order to predict the most likely variable names in the pre-existing partial source code program to replace the anonymous values, and thus replace the undefined variable names in the pasted source code snippet. (collectively referred to as block 508).

[0077] The deep learning model makes its predictions based on the context of a pre-existing partial source code program that includes the pasted source code snippet. In one aspect, the context includes an extended context that spans beyond the local context that includes the pasted source code snippet. (collectively, block 508).

[0078] The undefined variable names in the pasted source code snippet are then replaced with the predicted variable names with the highest probabilities defined in the pre-existing partial source code program (block 510).

[0079] Figure 6 Another aspect of an exemplary method 600 for code adaptation using a deep learning model is shown. In this aspect, code adaptation techniques are used for source code development. A pre-existing partial source code program can be developed using a source code editor, IDE, or other type of development application. A developer can copy a source code snippet from an external source and paste it into a source code editor. The source code editor detects the insertion (block 602) and initiates the code adaptation technique.

[0080] The pre-existing program with the pasted source code snippet is parsed into a concrete syntax tree, and data flow analysis is performed using the concrete syntax tree (block 604).

[0081] An extension context is generated from the pre-existing partial source code program using the concrete syntax tree and the prioritized list of code elements (block 606). Variables used in the pasted source code snippet are anonymized with mask values ​​(block 608).

[0082] The extended context with anonymous or masked values ​​is then input into the neural transformer model. The input sequence containing the extended context with anonymous values ​​is tokenized and then transformed into an input sequence with token embeddings of corresponding position embeddings. (collectively referred to as block 610).

[0083] The inference phase uses beam search to find the most likely variable name to replace the mask value. A variable name can consist of several sub-labels / labels that are concatenated to form a predicted variable name. The beam search iteratively generates labels / sub-labels at each time step by calling the neural transformer model. The output of the neural transformer model at each time step is a matrix of label probabilities. The beam search focuses on the k most likely labels in each iteration to obtain the best path to the most likely variable name. In each iteration, each of the k most likely labels is concatenated with the label in the previous iteration to form a partial variable name. (Collectively referred to as block 610).

[0084] Beam search uses breadth-first search to build a search tree. The search tree consists of nodes at one or more inference levels. Each node represents a probability distribution generated by the neural transformer model for a token / sub-token in the model vocabulary. At each level, only the top k tokens / sub-tokens with the highest probability from the output distribution generated by the neural transformer model are extended to the next inference level. The variable k is preconfigured and is also called the beam width. Each of the k sub-tokens / tokens is then extended to the search, which updates the current context sequence with the selected sub-token / token to input into the neural transformer model, thereby generating an additional probability distribution for the next token in the sequence. This process is repeated until the tag at the end of the predictor variable is the next possible tag candidate. (collectively referred to as block 610).

[0085] for Figure 3A In the model configuration shown in , each decoder outputs the probability of each token / sub-token in the model's vocabulary at each time step to replace the mask value associated with the decoder. The model's vocabulary consists of a fixed number of sub-tokens / tokens learned from the training dataset. (collectively referred to as block 610).

[0086] for Figure 3B In the model configuration shown in , at each time step, the decoder outputs the probability of each mapping in the vocabulary of the model. A mapping is a pair of values, where each pair includes a mask value and a corresponding token / sub-token. (collectively referred to as block 610).

[0087] The variable name mapper receives the probability distribution output by the model and modifies each mask value with the predicted variable name with the highest probability or the predicted mapping with the highest probability (block 612).

[0088] Exemplary Operating Environment

[0089] Attention now turns to a discussion of exemplary operating environment 700 . Figure 7 An exemplary operating environment 700 is shown in which one or more computing devices 702 are used to train a neural transformer model and to perform code adaptation using the neural transformer model to develop source code. However, it should be noted that the aspects disclosed herein are not limited to any particular device configuration. The computing device 700 can be configured as a source code development environment that utilizes a code adaptation system. It should be noted that the operating environment is not limited to any particular configuration and other configurations are possible.

[0090] The computing device 702 can be any type of electronic device, such as, but not limited to, a mobile device, a personal digital assistant, a mobile computing device, a smart phone, a cellular phone, a handheld computer, a server, a server array or server farm, a web server, a network server, a blade server, an Internet server, a workstation, a microcomputer, a mainframe computer, a supercomputer, a network appliance, a web appliance, a distributed computing system, a multiprocessor system, or a combination thereof. The operating environment 700 can be configured in a network environment, a distributed environment, a multiprocessor environment, or a stand-alone computing device that can access a remote or local storage device.

[0091] The computing device 702 may include one or more processors 704, one or more communication interfaces 706, one or more storage devices 708, one or more memory devices or storage 710, and one or more input / output devices 712. The processor 704 may be any commercially available or custom processor, and may include dual microprocessors and multi-processor architectures. The communication interface 706 facilitates wired or wireless communication between the computing device 702 and other devices. The storage device 708 may be a computer-readable medium that does not contain a propagation signal (such as a modulated data signal transmitted by a carrier wave). Examples of the storage device 708 include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, cassettes, tapes, disk storage, all of which do not contain a propagation signal, such as a modulated data signal transmitted by a carrier wave. There may be multiple storage devices 708 in the computing device 702. The input / output device 710 may include a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, a speaker, a printer, etc., and any combination thereof.

[0092] The memory device or memory 710 can be any non-transitory computer-readable storage medium that can store executable programs, applications, and data. The computer-readable storage medium is not a propagated signal, such as a modulated data signal transmitted by a carrier wave. It can be any type of non-transitory storage device (e.g., random access memory, read-only memory, etc.), magnetic storage, volatile memory, non-volatile memory, optical storage, DVD, CD, floppy disk drive, etc., which is not a propagated signal, such as a modulated data signal transmitted by a carrier wave. The memory 610 can also include one or more external storage devices or remote storage devices, which are not propagated signals, such as modulated data signals transmitted by a carrier wave.

[0093] Storage device 710 may contain instructions, components, and data. A component is a software program that performs a specific function and is otherwise referred to as a module, program, and / or application. Storage device 710 may include an operating system 714, one or more software development tools 716, a static code analyzer 718, an extended context extractor 720, a variable name analyzer 722, a deep learning model 724, a variable name mapper 726, and other applications and data 728.

[0094] The computing device 702 may be communicatively coupled via a network 730. The network 730 may be configured as an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a wireless network, network or any other type of network or combination of networks.

[0095] The network 730 may employ various wired and / or wireless communication protocols and / or technologies. The different generations of communication protocols and / or technologies that may be employed by the network may include, but are not limited to, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access 2000 (CDMA-2000), High Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), Universal Mobile Telecommunications System (UMTS), Evolution Data Optimized (Ev-DO), Worldwide Interoperability for Microwave Access (WiMax), Time Division Multiple Access (TDMA), Orthogonal Frequency Division Multiplexing (OFDM), Ultra-Wideband (UWB), Wireless Application Protocol (WAP), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), any part of the Open Systems Interconnection (OSI) model protocol, Session Initiation Protocol / Real-time Transport Protocol (SIP / RTP), Short Message Service (SMS), Multimedia Messaging Service (MMS), or any other communication protocol and / or technology.

[0096] Technical Effects

[0097] Aspects of the subject matter disclosed herein relate to the technical problem of automatically adapting the variable usage of a copy-pasted source code snippet to the variable usage of a pre-existing partial source code program. A technical feature associated with solving the problem is to configure a deep learning model, such as a neural transformer model with attention, to predict defined variable names in a pre-existing partial source code program to replace undefined variable names from the copy-pasted source code snippet. The deep learning model is trained with samples of a training data set to learn variable usage patterns. The technical effect achieved is to automatically adapt the pasted code snippet to a pre-existing partial source code program without user intervention to generate a syntactically and semantically correct program.

[0098] in conclusion

[0099] A computer-implemented method is disclosed, comprising: detecting an insertion of a pasted source code snippet into a pre-existing partial source code program, wherein the pasted source code snippet includes the pasted source code snippet and variable names that are undefined in the pre-existing partial source code program, wherein the pasted source code snippet is from a source external to the pre-existing partial source code program; predicting defined variable names from the pre-existing partial source code program to replace the undefined variable names based on a probability distribution output from a deep learning model, the deep learning model being given a context of the pre-existing partial source code program; and changing the undefined variable names with the predicted defined variable names in the pre-existing partial source code program.

[0100] In one aspect, the method further includes generating an extended context of the pre-existing partial source code program with the inserted pasted source code snippet, wherein the extended context includes code elements from the pre-existing partial source code program outside of a local scope of the pasted source code snippet.

[0101] In one aspect, the extended context is based on a prioritized list of code elements. In one aspect, the method further comprises: transforming the undefined variable name into an anonymous value in the extended context; and providing the extended context to the deep learning model. In one aspect, the deep learning model comprises a plurality of decoder units, wherein each decoder unit is associated with selecting the anonymous value, wherein each of the decoder units outputs a probability of selecting the anonymous value as a selected defined variable name from a pre-existing portion of the source code program.

[0102] In one aspect, the probability distribution is based on a plurality of mappings, each mapping comprising anonymous values ​​and corresponding variable names from a pre-existing partial source code program. In one aspect, the deep learning model comprises a single decoder unit. In one aspect, the deep learning model is a neural transformer model with attention.

[0103] A system is disclosed, comprising: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing the following actions: accessing a pre-existing partial source code program with a pasted source code snippet from an external source, wherein the pasted source code snippet includes one or more variable names that are not defined in the pre-existing partial source code program; replacing each of the one or more variable names from the pasted source code snippet with a mask value; obtaining a predicted variable name from a deep learning model to replace each mask value, wherein the deep learning model is given a selected portion of the pre-existing source code program, wherein each predicted variable name is defined in the pre-existing partial source code program; and changing each of the one or more variable names that are not defined in the pre-existing source code program with a corresponding predicted variable name.

[0104] In one aspect, the selected portion of the pre-existing partial source code program includes code elements from the pre-existing source code program that are outside the local scope of the pasted source code snippet. In one aspect, the one or more programs include instructions for: extracting code elements from a concrete syntax tree representation of the pre-existing partial source code program according to a priority list, wherein the priority list includes code elements outside the local scope of the pasted source code snippet; and generating an expansion context from the extracted code elements.

[0105] In one aspect, the one or more programs include instructions for performing the following actions: performing data flow analysis on the concrete syntax tree representation to identify one or more variable names from the pasted source code snippet. In one aspect, the deep learning model includes a plurality of decoder units, wherein each decoder unit of the plurality of decoder units is configured to predict a variable name to replace a particular mask value.

[0106] In one aspect, the deep learning model includes a single encoder unit coupled to each of a plurality of decoder units. In one aspect, the deep learning model includes a decoder unit, wherein the decoder unit is configured to output a probability for each of a plurality of mappings, wherein the mapping includes a particular mask value and a variable name defined in a pre-existing partial source code program to replace the particular mask value. In one aspect, the deep learning model is a neural transformer model with attention.

[0107] A computer-implemented method is disclosed, comprising: constructing a training dataset of multiple samples of source code programs containing a portion of source code with undefined variable names; extracting an extended context for each of the multiple samples; anonymizing each of the undefined variable names in each extended context with a selected value from a plurality of anonymous values; training a deep learning model to learn variable usage patterns given the extended context for each of the multiple samples; and applying the trained deep learning model to predict defined variable names from a target source code program to replace the undefined variable names in an external source code snippet inserted into the target source code program.

[0108] In one aspect, a deep learning model includes a plurality of decoder units, each of the plurality of decoder units is associated with a selected one of a plurality of anonymous values, a training data set is applied to each of the plurality of decoder units, and each of the plurality of decoder units outputs a probability distribution of the selected one of the anonymous values ​​relative to each of the defined variable names in a target source code program.

[0109] In one aspect, the deep learning model predicts a mapping of each of the plurality of anonymous values ​​to each of the defined variable names. In one aspect, the deep learning model is a neural transformer model with attention.

[0110] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0111] It can be appreciated that the representative methods described herein do not necessarily have to be executed in the order presented or in any particular order unless otherwise indicated. In addition, various activities described with respect to the methods can be executed in serial or parallel manner, or in any combination of serial and parallel operations.< / start> < / start>

Claims

1. A computer-implemented method comprising: detecting insertion of a pasted source code snippet into a pre-existing partial source code program, wherein the pasted source code snippet includes the pasted source code snippet and a variable name that is not defined in the pre-existing partial source code program, wherein the pasted source code snippet is from a source external to the pre-existing partial source code program; predicting defined variable names from the pre-existing partial source code program to replace the undefined variable names based on a probability distribution output from a deep learning model, the deep learning model being given a context of the pre-existing partial source code program; as well as The undefined variable names are changed with the predicted defined variable names in the pre-existing partial source code program.

2. The computer-implemented method of claim 1 , further comprising: An extended context of the pre-existing partial source code program with the pasted source code snippet inserted is generated, wherein the extended context includes code elements from the pre-existing partial source code program outside of a local scope of the pasted source code snippet. 3 . The computer-implemented method of claim 2 , wherein the expansion context is based on a priority list of the code elements.

4. The computer-implemented method of claim 2, further comprising: Transforming the undefined variable name into an anonymous value in the extension context; as well as The extended context is provided to the deep learning model.

5. The computer-implemented method of claim 1, The deep learning model comprises a plurality of decoder units, wherein each decoder unit is associated with a selected anonymous value, wherein each of the decoder units outputs a probability that the selected anonymous value is a selected defined variable name from the pre-existing partial source code program.

6. The computer-implemented method of claim 1, The probability distribution is based on a plurality of mappings, each mapping comprising anonymous values ​​and corresponding variable names from the pre-existing partial source code program.

7. The computer-implemented method of claim 1, wherein the deep learning model is a neural transformer model with attention.

8. A system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following actions: accessing, from an external source, a pre-existing partial source code program having a pasted source code snippet, wherein the pasted source code snippet includes one or more variable names that are not defined in the pre-existing partial source code program; replacing each of the one or more variable names from the pasted source code snippet with a mask value; Obtaining a predicted variable name from a deep learning model to replace each mask value, wherein the deep learning model is given a selected portion of the pre-existing source code program, wherein each predicted variable name is defined in the pre-existing portion of the source code program; as well as Each of the one or more variable names that are not defined in the pre-existing source code program is replaced with a corresponding predicted variable name.

9. The system of claim 8, wherein the selected portion of the pre-existing partial source code program includes code elements from the pre-existing source code program that are outside the local scope of the pasted source code snippet.

10. The system of claim 9, wherein the one or more programs include instructions for performing the following actions: extracting the code elements from the concrete syntax tree representation of the pre-existing partial source code program according to a priority list, wherein the priority list includes code elements outside the local scope of the pasted source code snippet; and The extended context is generated from the extracted code elements.

11. The system of claim 10, wherein the one or more programs include instructions for performing the following actions: Data flow analysis is performed on the concrete syntax tree representation to identify the one or more variable names from the pasted source code snippet.

12. The system of claim 8, wherein the deep learning model comprises a plurality of decoder units, wherein each of the plurality of decoder units is configured to predict a variable name to replace a specific mask value.

13. The system of claim 12, wherein the deep learning model comprises a single encoder unit coupled to each of the plurality of decoder units.

14. The system of claim 8, wherein the deep learning model comprises a decoder unit, wherein the decoder unit is configured to output a probability for each of a plurality of mappings, wherein a mapping comprises a specific mask value and a variable name defined in the pre-existing partial source code program for replacing the specific mask value.

15. The system of claim 8, wherein the deep learning model is a neural transformer model with attention.