Code enhancement for training language models related to computer programming

Abstract

To provide a method of code enrichment for training language models relating to computer programming.SOLUTION: A method comprises: receiving source code data; determining code blocks from a computer-executable code; extracting a set of features related to components of the source code data from the code blocks, where the extraction is performed by parsing the code blocks using Abstract Syntax Tree (AST) data of the code blocks; revising the AST data; updating the source code data based on the revised AST data; generating a dataset of NL and abstracted code features as training data based on the updated source code data; and training a language model on a sequence-to-sequence generation task based on the generated dataset.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] [Cross - Reference to Related Applications / Incorporation by Reference] This application claims priority to U.S. Provisional Patent Application No. 63 / 261,602, filed on September 24, 2021, entitled "Library Corpus for Large - Scale Language Models and Code Retrieval Models Using Augmented Code", the entire content of which is hereby incorporated by reference into this specification.

[0002] [Technical Field] The embodiments discussed in this disclosure relate to code augmentation for training language models related to computer programming.

Background Art

[0003] With the progress of machine learning, various language models have been developed for different machine programming tasks such as code retrieval, code cloning, or code synthesis. A language model is a statistical representation of a probability distribution over sequences of words, aiming to find relationships between different words by processing a large corpus of text. Some language models aim to learn general - purpose representations that support downstream natural - language - programming - language (NL - PL) applications such as natural - language code retrieval, code synthesis, and code documentation generation. However, processing only source code through relationships between words and other words is not sufficient to understand source code in different programming languages. Without improvements in language model learning, the performance of language models may be inferior to expectations, which may ultimately affect the performance of downstream tasks such as code synthesis.

[0004] The subject matter claimed herein is not limited to embodiments that resolve any shortcomings or operate only in the aforementioned environment. Rather, this background art is provided solely to illustrate one exemplary technical field in which some embodiments described herein may be carried out. [Overview of the project]

[0005] According to one embodiment, the operation may include receiving source code data, which includes computer executable code and natural language (NL) text associated with the computer executable code. The operation may further include determining blocks of code from the computer executable code and extracting a set of features from the blocks of code that relate to components of the source code data. The extraction is performed by parsing the blocks of code using abstract syntax tree (AST) data of the blocks of code. The operation may further include revising the AST data by abstracting the nodes of the AST data. The nodes may be abstracted based on the extracted set of features and the level of abstraction achieved for each node. The operation may further include updating the source code data based on the revised AST data and generating a dataset of NL and abstracted code features as training data based on the updated source code data. The operation may then include training a language model on a sequence-to-sequence generation task based on the generated dataset.

[0006] The objectives and advantages of this embodiment are realized and achieved by at least the elements, features, and combinations specifically indicated in the claims.

[0007] It should be understood that both the general description above and the detailed description below are merely examples, explanatory, and not intended to limit the claimed invention. [Brief explanation of the drawing]

[0008] Exemplary embodiments are described and illustrated with further specificity and detail through the use of the accompanying drawings. [Figure 1] This diagram illustrates an exemplary environment related to code enhancement for training language models on computer programming-related tasks. [Figure 2] This is a block diagram of a code enhancement system for training language models on tasks related to computer programming. [Figure 3] This figure shows example source code data for code augmentation used to train language models on computer programming-related tasks. [Figure 4] This flowchart illustrates an exemplary method of code augmentation for training language models on computer programming-related tasks. [Figure 5] A flowchart illustrating an exemplary method for extracting the first portion of a feature set for revision of AST data is shown. [Figure 6] A flowchart illustrating an exemplary method for extracting a second portion of the feature set for revision of AST data is shown. [Figure 7] A flowchart illustrating an exemplary method for extracting a third portion of the feature set for revision of AST data is shown. [Figure 8] A flowchart illustrating an exemplary method for extracting a fourth part of the feature set for revision of AST data is shown. [Figure 9] This figure illustrates exemplary code enhancement behavior for training language models on tasks related to computer programming. [Figure 10] This figure illustrates an exemplary scenario for the application of a language model trained on computer programming-related tasks.

[0009] All of these are subject to at least one embodiment described herein. [Modes for carrying out the invention]

[0010] Advances in machine learning and artificial intelligence have led to the development of language models for various machine programming tasks. Language models are typically probabilistic models that provide a statistical representation of the probability distribution of a sequence of words, aiming to find relationships between different words by processing a large corpus of text (e.g., executable code). For example, a language model may be able to predict the probability that the word "pandas" appears after "import" in a sequence like "from nbconvert.preprocessors import ExecutePreprocessor". Specifically, given a sequence of length "m", a language model can predict the probability P(w1, w2, ... w) of the entire sequence. m ) can be assigned.

[0011] Language models are used in a variety of sequence-to-sequence generation tasks or machine programming tasks, such as code extraction, code annotation, code cloning, and code synthesis. Code extraction tasks involve extracting code snippets from a codebase that relate to a given natural language query. Code annotation tasks involve annotating code snippets with natural language descriptions, and code synthesis tasks involve generating source code based on natural language queries.

[0012] For each machine programming task, the language model needs to be trained on a dataset prepared for that specific task. Current solutions for training language models use only source code and its corresponding natural language description. Training these language models, which use only source code through the natural language description of words, may not be sufficient to understand the source code of different programming languages.

[0013] This disclosure describes how to train or fine-tune a language model using abstract syntax tree (AST) data in conjunction with source code. Abstract syntax tree (AST) data may include ASTs, each of which may be a tree representation of the abstract syntactic structure of text (often source code) and may be written in a formal language (such as a programming language). Each node in the tree in the AST may represent a construct that may occur within the text. Furthermore, the AST data may include data associated with each word in the source code (i.e., variables, functions) and connections to other words through the tree structure (i.e., input parameters of functions, types of input parameters). Typically, the AST of source code must be validated by a compiler / interpreter before the source code is executed.

[0014] This disclosure further describes training a language model using a source code representation that integrates the AST data of the source code with other important information. Further information from the AST may allow the language model to provide more detailed insights into the source code than simply examining words as tokens and their statistical order. Therefore, one objective of this disclosure is to generate a new type of source code presentation for language models that embeds Abstract Syntax Tree (AST) information into the original source code. Improvements to this language model could directly impact the performance of downstream tasks in code synthesis. Furthermore, this disclosure provides an extended language model that can be used to generate new code based on existing code and may be more efficient and robust compared to existing language models. This extended language model may also be used in software development.

[0015] In contrast to existing solutions, language models generated using AST data may be more generalized, efficient, and accurate compared to language models generated without incorporating AST data. Due to the addition of AST data in training language models, language models may have detailed information about the source code. For example, based on AST data, language models may be able to predict the names of functions (i.e., built-in functions and user-defined functions). Along with the improved accuracy of language models, this disclosure can extend the resource computation utilization overseeing deep learning models for pre-training language models because each token contains detailed information about the programming code (i.e., function type or package name) rather than just a simple word. For example, language models can learn better in a corpus when the functions "x.to_csv()" and "y.to_csv()" are revised to "pd.DataFrame.x.to_csv()" and "pd.DataFrame.y.to_csv()".

[0016] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0017] Figure 1 shows an exemplary environment related to code enrichment for training a language model on a computer programming-related task, arranged according to at least one embodiment described herein. Referring to Figure 1, an exemplary environment 100 is shown. The exemplary environment 100 shows a system 102 and one or more data sources 104. Further shown are a language model 106, a user device 108, a communication network 110, source code data 112, and a dataset 114. The system 102, one or more data sources 104, and the user device 108 may be communicatively coupled to each other via the communication network 110. Further shown is a user 116 that may be associated with the user device 108. Examples of the user device 108 may include, but are not limited to, a mobile device, a desktop computer, a laptop, or a computer workstation. In one or more embodiments, the user device 108 may include a user end terminal device and a server communicatively coupled to the user end terminal device. Examples of user end terminal devices include, but are not limited to, mobile devices, desktop computers, laptops, or computer workstations.

[0018] One or more data sources 104 may include appropriate logic, circuitry, and interfaces that can be configured to store source code data 112. Source code data 112 may include computer executable code and natural language (NL) text that may be associated with the computer executable code. In one embodiment, the computer executable code may be associated with an open-source software (OSS) project. Examples, but not limited to, one or more data sources 104 may include web-based code hosting servers, database servers, file servers, web servers, RSS (Really Simple Syndication) feeds, and servers hosting websites and web applications associated with repositories (for storing software code).

[0019] In one embodiment, one or more data sources 104 may be implemented as a server, which may include storage distributed across one or more availability zones (e.g., data centers). In one embodiment, each data source may include a front-end system and a back-end system. The front-end system may be configured to provide an interface (such as a web page or a client-side interface of a web application) for viewing information associated with the source code data 112. The back-end system may store databases, logics, and instructions for displaying content on the interface provided by the front-end system.

[0020] The language model 106 may be a probability model that can be trained to generate a probability distribution over sequences on a set of code tokens. The language model 106 may be one of a statistical language model or a neural language model. A statistical language model can learn the probability distribution of words using statistical methods. These statistical methods may include, but are not limited to, unigram methods, N-gram methods, hidden Markov models (HMMs), and other language rules. Details regarding the implementation of the above statistical methods are known in the art. Therefore, a detailed description of the above statistical methods is omitted for the sake of brevity.

[0021] A neural language model can learn the probability distribution of words using one or more neural networks. In one embodiment, each of the one or more neural networks included in the neural language model may be a computational network or system of artificial neurons (also called nodes) that may consist of multiple layers. The multiple layers of the neural network may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (or, for example, artificial neurons represented by circles). The outputs of all nodes in the input layer may be coupled to at least one node in the hidden layer. Similarly, the input of each hidden layer may be coupled to the output of at least one node in the other layers of the neural network. The output of each hidden layer may be coupled to the input of at least one node in the other layers of the neural network. Nodes in the final layer may receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from the hyperparameters of the neural network. Such hyperparameters may be set before training, during training, or after training the neural network on the training dataset.

[0022] Each node in a neural network can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that can be adjusted during network training. The parameter set may include, for example, weight parameters and regularization parameters. Each node can use the mathematical function to compute an output based on one or more inputs from nodes in other layers of the neural network (e.g., previous layers). All or some nodes in a neural network may correspond to the same or different mathematical functions.

[0023] During neural network training, one or more parameters of each node in the neural network may be updated based on whether the output of the final layer for a given input (from a training dataset) matches the correct result based on the neural network's loss function. This process may be repeated for the same or different inputs until the minimum value of the loss function is achieved and the training error is minimized. Several methods for training are known in the field, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristics.

[0024] The neural language model may include electronic data, which may be implemented, for example, as a software component of an application executable on system 102. The neural language model may depend on libraries, external scripts, or other logic / instructions for execution by a processing device such as a processor. The neural language model may include code and routines configured to enable a computing device such as a processor to perform one or more actions to generate one or more lines of computer-executable code for natural language queries as input to the neural language model. Furthermore, or alternatively, the neural language model may be implemented using hardware including, but not limited to, a processor, a microprocessor (e.g., for performing or controlling one or more actions), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural language model may be implemented using a combination of hardware and software.

[0025] Each example of one or more neural networks may include, but is not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), CNN-recurrent neural networks (CNN-RNNs), artificial neural networks (ANNs), long-short-term memory (LSTM) network-based RNNs, LSTM+ANNs, gated recurrent unit (GRU)-based RNNs, fully connected neural networks, connectionist temporal classification (CTC)-based RNNs, deep Bayesian neural networks, and / or combinations of such networks. In a particular embodiment, each of the one or more neural networks may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0026] In one embodiment, the language model 106 may correspond to a DNN using an encoder-decoder architecture. The DNN may be trained to generate one or more lines of computer-executable code for natural language queries as input to a language. Specifically, such a language model may include an encoder neural network and a decoder neural network. Examples of such DNNs, but not limited to, include long-short-term memory (LSTM) networks, gated recurrent unit (GRU) networks, transformer models, or variations of transformer models, such as bidirectional encoder representations from transformers (BERT) or CodeBERT.

[0027] In operation, system 102 may be configured to receive source code data 112. As discussed, source code data 112 may include computer executable code and natural language (NL) text that may be associated with the computer executable code. In one embodiment, the NL text associated with the computer executable code may correspond to metadata associated with the computer executable code. In another embodiment, the NL text may correspond to a description of the computer executable code or comments (or user comments) added to the computer executable code. In yet another embodiment, the NL text may correspond to a docstring that may be included in the computer executable code.

[0028] In one embodiment, system 102 may be configured to determine the structure of the computer executable code that may be contained in the received source code data 112. For example, system 102 may determine the complexity of the computer executable code according to a top-down approach. Specifically, system 102 may be configured to determine the import of one or more libraries in the computer executable code. After determining such libraries, it may be possible to determine one or more instances of classes and each function of such classes. Details regarding the structure of the computer executable code are provided, for example, in Figure 3. System 102 may further be configured to determine blocks of code from the computer executable code. Each code block may correspond to a subset of the computer executable code and may include at least one of a class, a function, a nested block of lines of code, a single line of code, or a loop from the computer executable code. Details regarding blocks of code are provided, for example, in Figure 4.

[0029] Based on the determination of a block of code, system 102 may be configured to further extract a set of features from the block of code. Such features may relate to components of source code data 112. Components of source code data 112 may include, for example, libraries, classes, user-defined functions (or methods), programming language built-in functions (or methods), variables (or arguments), etc. In one embodiment, the extraction of a set of features can be performed by parsing the block of code using abstract syntax tree (AST) data of the block of code. The AST data may correspond to a tree representation of the abstract syntactic structure of a block of code in a particular programming language such as Python®, Java®, or JavaScript®. Each node in the tree may represent a construct that may occur within the block of code. Further details regarding the extraction of a set of features are provided, for example, in Figures 5, 6, 7, and 8.

[0030] System 102 may further be configured to revise the AST data by abstracting the nodes of the AST data. The nodes of the AST data can be abstracted based on the set of extracted features and the level of abstraction achieved for each node. This level of abstraction may be performed by selecting nodes of the AST data and further determining the type of the selected nodes. System 102 may select the level of abstraction based on the type of the node and the abstract grammar of the programming language used in the source code data 112. The system may further filter the attributes of the selected nodes based on the selected level of abstraction. Further details regarding the revision of the AST data are provided, for example, in Figures 4 and 9.

[0031] Based on revisions to the AST data, system 102 may be configured to further update source code data 112. Specifically, the updated source code data may correspond to the abstracted source code data 112. In one embodiment, source code data 112 may be updated based on the removal of one or more lines of code from the source code data. This one or more lines of code may be associated with a node processed using the AST.

[0032] After revising the source code data 112, the system 102 can generate a dataset 114 of NL and abstracted code features as training data based on the updated source code data. The abstracted code features may be generated based on the updated source code. Based on the generated dataset 114, the system 102 can train the language model 106 on a sequence-to-sequence generation task. The sequence-to-sequence generation task may include, for example, a code synthesis task, a code generation task, a code enhancement task, a code cloning task, a code extraction task, and so on. Details regarding the dataset and the training of the language model 106 are provided, for example, in Figure 10.

[0033] It should be noted that communication between system 102, one or more data sources 104, language model 106, and user device 108 may occur via a communication network 110. The communication network 110 may include a communication medium that enables system 102 to communicate with one or more data sources 104, language model 106, user device 108, and / or other devices (not shown). Examples of the communication network 110 include, but are not limited to, the Internet, cloud networks, cellular networks (such as fourth-generation Long-Term Evolution (LTE®) or fifth-generation New Radio (NR)), Wireless Fidelity (Wi-Fi) networks, personal area networks (PANs), local area networks (LANs), and / or metropolitan area networks (MANs). Various devices in the exemplary environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of the following, or any combination thereof: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee®, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and / or Bluetooth® (BT) Communication Protocol.

[0034] System 102 can be modified, added to, or omitted without departing from the scope of this disclosure. For example, in some embodiments, System 102 may include any number of other components that are not expressly illustrated or described.

[0035] Figure 2 is a block diagram of a code enhancement system for training a language model on computer programming-related tasks, arranged according to at least one embodiment described herein. Figure 2 is described in relation to the elements of Figure 1. Referring to Figure 2, a block diagram 200 of system 102 of Figure 1 is shown. Block diagram 200 may further include a processor 202, memory 204, persistent data storage 206, I / O block 208, network interface 210, and language model 106.

[0036] The processor 202 may include appropriate logic, circuitry, and / or interfaces that can be configured to execute program instructions associated with different operations performed by the system 102. The processor 202 may include any appropriate dedicated or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored in any applicable computer-readable storage medium. For example, the processor 202 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Although shown as a single processor in Figure 2, the processor 202 may include any number of processors configured to individually or collectively perform or direct any number of operations of the system 102 described herein.

[0037] In some embodiments, the processor 202 may be configured to interpret and / or execute program instructions stored in memory 204 and / or persistent data storage 206, and / or process data. In some embodiments, the processor 202 may fetch program instructions from persistent data storage 206 and load them into memory 204. After the program instructions are loaded into memory 204, the processor 202 may execute the program instructions. Some examples of the processor 202 may be a central processing unit (CPU), a reduced instruction set computer (RISC) processor, an ASIC processor, a composite instruction set computer (CISC) processor, a graphical processing unit (GPU), a coprocessor, and / or a combination thereof.

[0038] Memory 204 may include appropriate logic, circuitry, and / or interfaces that can be configured to store program instructions executable by the processor 202. In certain embodiments, memory 204 may be configured to store source code data 112, updated source code data, AST data, and revised AST data. In certain embodiments, memory 204 may be configured to store a trained language model 106. Memory 204 may include a computer-readable storage medium that carries or has stored computer-executable instructions or data structures. Such a computer-readable storage medium may include any available medium that can be accessed by a general-purpose or dedicated computer, such as the processor 202.

[0039] Such computer-readable storage media may include, but are not limited to, tangible or non-temporary computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices), or any other storage media that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and that are accessible by a general-purpose or dedicated computer. The above combinations may also be included in the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor 202 to execute a specific operation or group of operations associated with system 102.

[0040] The persistent data storage 206 may include appropriate logic, circuitry, and / or interfaces that can be configured to store program instructions executable by the processor 202. The persistent data storage 206 may also include a computer-readable storage medium that carries or holds the stored computer-executable instructions or data structures. Such a computer-readable storage medium may include any available medium that can be accessed by a general-purpose or dedicated computer, such as the processor 202.

[0041] Such computer-readable storage media may include, but are not limited to, tangible or non-temporary computer-readable storage media, including optical disk storage devices, magnetic disk storage devices or other magnetic storage devices (e.g., hard disk drives (HDDs)), flash memory devices (e.g., solid-state drives (SSDs), secure digital (SD) cards, and other solid-state memory devices), or any other storage media that can be used to carry or store specific program code in the form of computer-executable instructions or data structures and that are accessible by a general-purpose or dedicated computer. The above combinations may also be included in the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause the processor 202 to execute a specific operation or group of operations associated with system 102.

[0042] The I / O device 208 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to receive one or more user inputs. The I / O device 208 may also be configured to provide outputs in response to one or more user inputs. The I / O device 208 may include various input and output devices, which may be configured to communicate with the processor 202 and other components, such as the network interface 210. Examples of input devices include, but are not limited to, a touchscreen, keyboard, mouse, joystick, and / or microphone. Examples of output devices include, but are not limited to, a display device and a speaker.

[0043] The network interface 210 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to establish communication between the system 102, one or more data sources 104, a language model 106, and a user device 108 over the communication network 110. The network interface 210 may be implemented using various known techniques to support wired or wireless communication of the system 102 over the communication network 110. The network interface 210 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, and / or a local buffer.

[0044] The network interface 210 can communicate wirelessly with networks such as the Internet and intranets, and / or wireless networks such as cellular telephone networks and wireless local area networks (LANs), and / or metropolitan area networks (MANs). Wireless communication may use any of the multiple communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), wideband code division multiple access (W-CDMA), Long-Term Evolution (LTE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n, etc.), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), or Wi-MAX.

[0045] The functions or operations performed by the system 102 shown in Figure 1 may also be performed by the processor 202. The operations performed by the processor 202 are described in detail, for example, in Figures 3, 4, 5, 6, 7, 8, 9, and 10.

[0046] Figure 3 shows exemplary source code data for code enhancement to train a language model on a computer programming-related task, according to at least one embodiment described herein. Figure 3 is described in relation to the elements of Figures 1 and 2. Referring to Figure 3, Diagram 300 is shown. Diagram 300 includes a file of source code data 302 and an abstracted code feature 308 (also called a Foundation of Block of Code, FBloC) generated based on the source code data 302.

[0047] As shown in the figure, for example, source code data 302 includes computer executable code 304 and natural language (NL) text 306. The NL text 306 can be associated with the computer executable code 304. The computer executable code 304 may be written in a specific programming language such as Python, C, C++, C#, Swift, JavaScript, Go, Java®, or R, but is not limited to these. The computer executable code 304 in Figure 3 is written in Python.

[0048] NL text 306 can correspond to a description of computer executable code 304. In one embodiment, NL text 306 can correspond to one or more comments associated with computer executable code 304. In another embodiment, NL text 306 may correspond to a dockstring that may be associated with computer executable code 304. In one embodiment, NL text 306 may be contained between lines or within an internal block (e.g., a function or class) of computer executable code 304 and can describe computer executable code 304 or a component of computer executable code 304. For example, without limitation, the NL text for the function "ci_to_errsize" that converts an interval to an error argument may be "Converts an interval to an error argument for plot height".

[0049] Referring to Figure 3, the abstract code features 308 that can be generated based on the source code data 302 are further shown. The abstract code features 308 include the first abstract code feature (y1) 308A, the second abstract code feature (y2) 308B, the third abstract code feature (y3) 308C, and the N abstract code features (y N )308N ​​may be included. Specifically, system 102 may be configured to transform X into Y, where X=(D,f,A) and Y=(y1,y2,y3,......,y N ) is as follows: "f" can correspond to the name of a function associated with the corresponding code block, "A" can correspond to a list of parameters associated with the function, and "D" can correspond to NL text associated with the function. Abstracted code feature 308 is sometimes called the code block foundation (FBloC). Referring to Figure 3, the foundation (or core) components of the received computer executable code include "np.atleast_2d(cis).reshape", "np.atleast_1d(X)", "loop", and "np.asarray". Further details regarding the abstracted source code features are provided, for example, in Figure 4.

[0050] Figure 4 shows a flowchart of an exemplary method for code enhancement to train a language model on a computer programming-related task, according to at least one embodiment described herein. Figure 4 is described in relation to the elements of Figures 1, 2, and 3. Referring to Figure 4, flowchart 400 is shown. The method shown in flowchart 400 can be started at 402 and can be performed by any suitable system, apparatus, or device, for example, by system 102 in Figure 1 or 2.

[0051] In 402, source code data 302 may be received from one or more data sources 104. For example, source code data 302 may be received from a user device 108 or from a web-based source such as a source code repository. Source code data 302 may include computer executable code 304 and NL text 306 associated with the computer executable code 304. An exemplary page of source code data 302 is shown in Figure 3.

[0052] In 404, a block of code may be determined from the computer executable code 304 contained in the source code data 302. A block of code may include, but is not limited to, at least one of a class, a function (or method), a nested block of code lines, a single line of code, or a loop from the computer executable code 304. In one embodiment, a nested block of code lines may be separated by a newline escape sequence (\n).

[0053] In one embodiment, before determining a block of code, the system 102 may be configured to determine the structure of the computer executable code 304 contained in the received source code data 302. For example, the system 102 can determine the complexity of the computer executable code 304 following a top-down approach. Specifically, the system 102 may be configured to import one or more libraries used in the computer executable code 304. After importing one or more libraries, it can determine one or more instances of classes and one or more functions of each of these one or more classes. Based on the determination of one or more functions, the system 102 may be configured to determine package metadata, a list of classes, and a list of functions used in the computer executable code. The system 102 may further be configured to determine a block of code based on the package metadata, the list of classes, and the list of functions.

[0054] In 406, a set of features related to the components of the source code data 302 from the block of code can be extracted. To extract the set of features, system 102 can be configured to generate abstract syntax tree (AST) data for the block of code. The AST data may include at least one tree representation of the abstract syntactic structure of the block of code written in a particular programming language. Each node in the tree may represent a construct that may occur within the block of code. Specifically, system 102 may be configured to generate AST data for each block of code.

[0055] In one embodiment, the extraction of a set of features may be performed by parsing a block of code using the AST data of the block of code. The extracted set of features can be associated with functions and classes in source code data 302, a set of methods associated with source code data 302, a set of arguments associated with source code data 302, a list of classes associated with source code data 302, and objects associated with source code data 302. The extracted set of features may include at least one comment for each function and class in the received source code data 302. The set of features may further include a set of name conversions associated with the set of methods, a set of pseudo-names associated with the set of arguments, a dictionary associated with the list of classes, and object identifiers associated with objects. Further details regarding the extraction of the set of features are provided, for example, in Figures 5, 6, 7, and 8.

[0056] In 408, the AST data may be revised. The AST data can be revised by abstracting the nodes of the AST data. The nodes may be abstracted based on the set of extracted features and the level of abstraction achieved for each node of the AST data. In one embodiment, the abstraction of the nodes of the AST data may be performed by iteratively selecting nodes of the AST data and determining the type of the selected nodes. The level of abstraction can be selected based on the type of the node and the abstract grammar of the programming language (e.g., Python) used in the source code data 302. The attributes of the selected nodes may be filtered based on the selected level of abstraction. Further details regarding the revision of the AST data are provided, for example, in Figure 9.

[0057] In 410, the source code data 302 may be updated based on the revised AST data. To update the source code data 302, the system 102 may be configured to compare the AST data of blocks of code with the revised AST data based on matching criteria. The matching criteria are intended to generalize the source code data 302.

[0058] Based on the comparison, system 102 can determine a set of changes that may be applicable to the portion of the received source code data 302. The source code data 302 may be updated based on the set of changes. Details regarding the updating of the source code data 302 are provided, for example, in Figure 9.

[0059] In 412, based on the updated source code data 302, datasets of NL and abstracted code features can be generated as training data. In one embodiment, NL features can be extracted from natural language (NL) text 306 associated with computer executable code 304, and abstracted code features can be generated from computer executable code 304. According to one embodiment, abstracted code features 308 may be called the basic blocks of code. Details regarding the generation of datasets are provided in Figure 10.

[0060] In 414, the language model 106 may be trained on a sequence-to-sequence generation task based on the generated dataset. The sequence-to-sequence generation task may be one of the following: a code synthesis task, a code generation task, a code enhancement task, a code cloning task, or a code extraction task. The language model can be trained to generate one or more lines of computer-executable code for natural language queries as input to the language model. In one embodiment, the language model may be implemented using a deep neural network with an encoder-decoder architecture. If a pre-trained language model exists, the system 102 may fine-tune the pre-trained language model based on the generated dataset. In fine-tuning, parameters such as the weights of the pre-trained language model may be updated using an example of the training dataset.

[0061] Control may proceed to termination. Flowchart 400 is shown as individual operations such as 402, 404, 406, 408, 410, 412, and 414, however, in a particular embodiment, such individual operations may be further divided into further operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0062] Figure 5 shows a flowchart of an exemplary method for extracting a first portion of a set of features for revising AST data, according to at least one embodiment described herein. Figure 5 is described in relation to the elements of Figures 1, 2, 3, and 4. Referring to Figure 5, flowchart 500 is shown. The method shown in flowchart 500 can be started at 502 and can be performed by any suitable system, apparatus, or device, for example, by system 102 in Figure 1 or 2.

[0063] In 502, the database may be loaded into memory 204. System 102 may be configured to load the database into memory 204 based on the determination of a block of code. The database may contain package metadata associated with the source code of multiple code packages. Such packages can be identified by parsing the AST data associated with the block of code. Multiple code packages may be contained in the computer executable code 304. Each of the multiple code packages may correspond to a module that can be added to the computer executable code 304 to incorporate further options, features, or functions that may be provided by the corresponding code package.

[0064] In one embodiment, package metadata may include, but is not limited to, a description of the corresponding code package, the name of the corresponding code package, a short description of the corresponding code package, a classifier for the corresponding code package, and the names of one or more modules contained in the corresponding code package. In one embodiment, the database may further include a set of classes that may be used in the source code of multiple code packages, a set of subclasses that may be used in the source code, a set of functions that may be used in the source code, and natural language text that may be used in the source code. The natural language text in the source code may include, for example, a dockstring associated with the corresponding source code, or a comment associated with the corresponding source code.

[0065] In 504, functions and classes can be determined. System 102 can be configured to determine functions and classes that may be used in the received source code data 302. To determine functions and classes, system 102 may use AST data. In one embodiment, the classes used in the received source code may include one or more nested classes that may be used in the received source code. As an example, but not limited to, the received source code data 302 may be given as follows:

number

[0066] In 506, the usage of each function and class can be determined. System 102 can be configured to determine the usage of each function and class. In one embodiment, the usage of each function and class may be determined based on a match between functions and classes using database entries. In one embodiment, the usage of each function or class may correspond to the purpose of each function or class in the received source code data 302. For example, but not limited to, if the function is "np.random.seed(seed)", the usage of the function may be to initialize the seed value of the random number generator in NumPy®. In one embodiment, System 102 may further be configured to add functions and classes to a function usage stack.

[0067] In 508, comments may be generated for the determined usage. System 102 can be configured to generate comments for each usage of a function and a class. In one embodiment, System 102 may be configured to generate comments for each usage of a function and a class added to the function usage stack. The generated comments may be natural language text that can describe the usage of the corresponding function or class. The comments may be included as a first part of a set of features that can be extracted to revise the AST data (described later). For example, without limitation, if the object of a first function in the function usage stack is adding two integers, the generated comment corresponding to the first function may be "function that adds two integers" or "function that adds integer A and integer B". It should be noted that comments may be generated for each function and a class.

[0068] In step 510, the AST data of a block of code may be revised. System 102 can further revise the AST data by abstracting the nodes of the AST data. The nodes of the AST data may be abstracted based on the generated comments and the level of abstraction achieved for each node. System 102 further updates the source code data 302 based on the revised AST data. System 102 can further generate datasets of NL and abstracted code features as training data based on the updated source code data 302, and can further train a language model on a sequence-to-sequence generation task based on the generated datasets. Details regarding the generation of datasets and the training of language models are provided, for example, in Figure 10.

[0069] Control may proceed to termination. Flowchart 500 is shown as individual operations such as 502, 504, 506, 508, and 510, however, in a particular embodiment, such individual operations may be further divided into further operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0070] Figure 6 shows a flowchart of an exemplary method for extracting a second portion of a set of features for revising AST data, according to at least one embodiment described herein. Figure 6 is described in relation to the elements of Figures 1, 2, 3, 4, and 5. Referring to Figure 6, flowchart 600 is shown. The method shown in flowchart 600 can be started at 602 and can be performed by any suitable system, apparatus, or device, for example, by system 102 in Figure 1 or 2.

[0071] In 602, a list of code packages may be collected. System 102 can be configured to collect a list of code packages. Such packages may be imported into the incoming source data 302 using statements such as "import package A". The list of code packages may also be collected using AST data. As discussed, each code package may contain one or more modules, a set of methods, and one or more aliases for the set of methods. Details regarding the collection of the list of code packages are provided, for example, in Figure 5.

[0072] In 604, a set of methods may be collected from the AST data associated with a block of code. System 102 can be configured to collect a set of methods used in a list of code packages.

[0073] In 606, a set of name translations may be generated. System 102 can be configured to generate a set of name translations by associating a set of methods or one or more aliases of a set of methods with the names of each module that may be contained in each code package in the list of code packages. The list of name translations may be contained in a set of extracted features. For example, System 102 may be configured to associate a set of methods or one or more aliases with module.method. In one embodiment, System 102 may be configured to determine whether an alias exists for each module contained in the source code data 302. Based on the determination, System 102 may be configured to replace node.id in the AST with module.method. An example of such associations is provided below.

[0074] In 608, the AST data of a block of code may be revised. System 102 can be configured to revise the AST data by replacing the names of methods or one or more aliases in the AST data with a set of name translations. An exemplary portion of the received source code data 302 may be given as follows:

number

[0075] For the received source code data 302, the revised AST data may be updated based on the following changes (i.e., name changes):

number

[0076] Referring to the example above, system 102 can revise the node corresponding to "abc" in the AST data using a new node that can correspond to "import.abc". Similarly, the node corresponding to "sys" can be revised using a new node that can correspond to "import.sys".

[0077] System 102 may further be configured to revise the AST data by abstracting the nodes of the AST data. The nodes of the AST data can be abstracted based on the set of name translations generated and the level of abstraction achieved for each node. System 102 further updates the source code data 302 based on the revised AST data. System 102 can further generate datasets of NL and abstracted code features as training data based on the updated source code data 302, and can train a language model on a sequence-to-sequence generation task based on the generated datasets. Details regarding dataset generation and language model training are provided, for example, in Figure 10.

[0078] The control may proceed to termination. Although flowchart 600 is shown as separate operations such as 602, 604, 606, and 608, in a particular embodiment, such separate operations may be further divided into further operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0079] Figure 7 shows a flowchart of an exemplary method for extracting a third portion of a set of features for revising AST data, according to at least one embodiment described herein. Figure 7 is described in relation to the elements of Figures 1, 2, 3, 4, 5, and 6. Referring to Figure 7, flowchart 700 is shown. The method shown in flowchart 700 can be started at 702 and can be performed by any suitable system, apparatus, or device, for example, by system 102 in Figure 1 or 2.

[0080] In 702, a list of classes and a list of methods may be collected. System 102 can be configured to collect a list of classes and a list of methods associated with source code data 302 by using AST data. Specifically, system 102 may be configured to collect each of the lists of classes and methods contained in source code data 302 based on AST data associated with blocks of code.

[0081] In 704, a dictionary may be generated. System 102 can be configured to generate a dictionary based on a list of collected classes and a list of collected methods. The generated dictionary may include a list of classes as keys and a list of methods as values ​​corresponding to the keys. The dictionary may be included as a third part of the set of extracted features.

[0082] In 706, built-in objects can be identified. System 102 can be configured to identify built-in objects and types of built-in objects that may be used in source code data. In one embodiment, built-in objects and types of built-in objects may be identified based on a generated dictionary. In one embodiment, built-in objects may correspond to built-in functions (or methods). The types of built-in functions may differ in different programming languages. The types of built-in objects may correspond to the data types of built-in objects, and information about the types of built-in objects may provide further information for training a language model.

[0083] In 708, an object identifier can be generated. System 102 can be configured to generate an object identifier by appending the respective type of the identified type and a built-in keyword to the name of each built-in object. In one embodiment, the built-in keyword may correspond to the identifier (id) of the corresponding built-in object. The object identifier may also be a third part of a set of features that can be extracted to revise the AST data. In one embodiment, each object identifier is “Obj->“builtins. <type> . <func method obj>It may also be in a specific format such as "[...]. Here, "type" can correspond to the data type of the built-in function, and "fucn / method / obj" can correspond to the name of the built-in function or the object of the built-in function.

[0084] In 710, the AST data may be revised. In one embodiment, system 102 may be configured to revise the AST data. The AST data may be revised by updating the identifiers of built-in objects in the AST data with the generated object identifiers. System 102 further updates the source code data based on the revised AST data. The updated source code data may be used to generate a dataset of NL and abstracted code features as training data. The language model may be for a sequence-to-sequence generation task based on the generated dataset. Details regarding the generation of the dataset and training of the language model are provided, for example, in Figure 10.

[0085] The control may proceed to termination. Although flowchart 700 is shown as individual operations such as 702, 704, 706, 708, and 710, in a particular embodiment such individual operations may be further divided into further operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0086] Figure 8 shows a flowchart of an exemplary method for extracting a fourth portion of a set of features for revising AST data, according to at least one embodiment described herein. Figure 8 is described in relation to the elements of Figures 1, 2, 3, 4, 5, 6, and 7. Referring to Figure 8, flowchart 800 is shown. The method shown in flowchart 800 can be started at 802 and can be performed by any suitable system, apparatus, or device, for example, by system 102 in Figure 1 or 2.

[0087] In 802, a set of arguments can be identified. System 102 can be configured to identify a set of arguments in the AST data. For example, and not limited to, the received source code data may be given as follows:

number

[0088] In 804, a set of pseudonames may be generated. System 102 can be configured to generate a set of pseudonames by associating identifiers included in the set of arguments with information relating to how identifiers are used in the AST data. The generated set of pseudonames may also be part of the set of extracted features. In one embodiment, the set of pseudonames may correspond to a fourth set of features in the set of extracted features. For example, the pseudoname for the "path" argument may be "func._fspath.arg.path".

[0089] In 806, the AST data of a block of code may be revised. System 102 can be configured to revise the AST data by replacing each identifier in the AST data with the corresponding pseudo-name from a set of pseudo-names. For example, the "path" argument in the AST data may be revised to "func._fspath.arg.path".

[0090] System 102 can update source code data based on revised AST data and generate datasets of NL and abstracted code features as training data based on the updated source code data. The language model can be trained on a sequence-to-sequence generation task based on the generated dataset. Details regarding dataset generation and language model training are provided, for example, in Figure 10.

[0091] The control may proceed to termination. Although flowchart 800 is shown as separate operations such as 802, 804, and 806, in a particular embodiment, such separate operations may be further divided into further operations, combined into fewer operations, or eliminated, depending on the particular implementation, without impairing the essence of the disclosed embodiment.

[0092] Figure 9 shows an exemplary operation for code enhancement to train a language model on a task related to computer programming, according to one embodiment of the present disclosure. Figure 9 is described in relation to elements of Figures 1, 2, 3, 4, 5, 6, 7, and 8. Referring to Figure 9, a block diagram 900 is shown illustrating exemplary operations 902A to 902H described herein. The exemplary operations shown in block diagram 900 can be initiated in 902A and can be executed by any computing system, apparatus, or device, for example, by system 102 in Figure 1 or processor 202 in Figure 2. Although shown in individual blocks, exemplary operations associated with one or more blocks in block diagram 900 may be divided into further blocks, combined into fewer blocks, or erased, depending on the particular implementation.

[0093] In 902A, source code data may be received from one or more data sources 104. The source code data may include computer executable code and NL text associated with the computer executable code. As a first example, not limited to, the received source code data may include code fragments given as follows:

number

[0094] In 902B, a block of code can be determined from the source code data. A block of code can include at least one of the following: a class, a function, a nested block of code lines, a single line of code, or a loop. Details regarding the determination of a block of code are provided in Figure 4.

[0095] In 902C, AST data may be generated for the determined block of code. As discussed, the AST data may include an AST904 which may contain a tree representation of the abstract syntactic structure of the source code. Each node in the tree represents a construct that occurs within the determined block of code. As a second example, not limited to this, the AST904 may contain five nodes such as "A", "B", "C", "D", and "E".

[0096] In 902D, a set of features may be extracted. The extracted set of features may be associated with components of source code data from a block of code. The extracted set of features may be associated with functions and classes of source code data 302, a set of methods associated with source code data 302, a set of arguments associated with source code data 302, a list of classes associated with source code data 302, and objects associated with source code data 302. The extracted set of features may include at least one comment for each function and class of the received source code data 302. The set of features may further include a set of name conversions associated with the set of methods, a set of pseudo-names associated with the set of arguments, a dictionary associated with the list of classes, and object identifiers associated with objects. Further details regarding the extraction of the set of features are provided, for example, in Figures 5, 6, 7, and 8.

[0097] In 902E, AST data revision may be performed. In the AST data revision operation, system 102 may be configured to revise the AST data. Specifically, system 102 may be configured to revise AST904 contained in the AST data. Revision can be performed by abstracting the nodes of AST904 in the AST data. In one embodiment, the abstraction of nodes in the AST data may correspond to the removal of nodes from AST904. Abstraction can be performed based on the set of extracted features (as described in Figures 5, 6, 7, and 8) and the level of abstraction achieved for each node. The level of abstraction may be associated with the grammar type associated with the programming language of the source code data 302. This specification provides, as an example, an iterative process for abstracting nodes.

[0098] In one embodiment, system 102 may be configured to abstract nodes in AST data by selecting nodes in the AST data. Based on the node selection, system 102 may be configured to determine the type of the selected nodes. System 102 may further be configured to select a level of abstraction based on the node type and the abstract grammar of a programming language that may be used in the source code data. System 102 may be configured to filter the attributes of the selected nodes based on the selected level of abstraction.

[0099] In another embodiment, system 102 may be configured to select nodes in the AST data. In one embodiment, the AST data can be revised by dropping selected nodes from the AST data. In such an embodiment, the dropping may be performed based on the determination that there are no function calls in the selected nodes.

[0100] Referring to the second example, system 102 can select node "E" from AST904. System 102 can further determine that there are no function calls in the selected node "E". Based on this determination, system 102 can be configured to drop the selected node "E" to generate a revised AST906 that may be included in the revised AST data.

[0101] In 902F, source code data may be updated. System 102 can be configured to update source code data based on revised AST data. In one embodiment, system 102 may be configured to compare the AST data of blocks of code with revised AST data based on matching criteria. Each criterion of the matching criteria may aim to transfer the AST data to text and / or generalize the source code data by highlighting information associated with components of the source code. In one embodiment, the matching criteria may be associated with, but are not limited to, the names of imported packages, argument names, built-in methods / functions / types, function calls, for loop minimization, and return value extraction.

[0102] System 102 may further be configured to determine, based on the comparison, a set of changes that may be applicable to the portion of the received source code data. System 102 may further be configured to update the source code data based on the set of changes. Note that the source code data may be updated based on the revised AST data, since abstractions for each node may be generated based on the revised AST. Abstractions for each node can ultimately generate abstractions for the corresponding data blocks.

[0103] In 902G, a dataset generation operation may be performed. In the dataset generation operation, system 102 may be configured to generate datasets of NL and abstracted code features as training data. The dataset may be generated based on updated source code data. Referring to the first example, the abstracted code features may include "Type(Input1).__fspath__(Input1)".

[0104] In 902H, a model training operation may be performed. In the model training operation, system 102 may be configured to train a language model. The language model may be trained on a sequence-to-sequence generation task based on the generated dataset. Details regarding the language model are provided in Figure 10.

[0105] Figure 10 shows an exemplary scenario for the application of a language model trained on a task related to computer programming, according to an exemplary embodiment. Figure 10 is described in relation to elements of Figures 1, 2, 3, 4, 5, 6, 7, 8, and 9. Referring to Figure 10, an exemplary scenario 1000 is shown. Figure 10 shows a system 1002 which may include a language model 1004. Furthermore, it shows the first training sample 1006, input 1008, and output 1010 from among several training samples included in the dataset.

[0106] In one embodiment, system 102 can operate in two phases: a setup phase and a prediction phase. System 102 may operate in the prediction phase after one or more operations in the setup phase have been performed. In the setup phase, system 102 may be configured to train a language model 1004. To train the language model, system 102 may be configured to generate a dataset of NL and abstracted code features as training data based on updated source code data. The dataset may include multiple training samples. Each training sample in the dataset may include an NL feature and a corresponding abstracted code feature. For example, the first training sample 1006 of the multiple training samples may include a first NL feature 1006A and a first abstracted code feature 1006B. Based on the generated dataset, the language model 1004 may be trained on a sequence-to-sequence generation task. The sequence-to-sequence generation task may be one of a code synthesis task, a code generation task, a code enhancement task, a code cloning task, or a code extraction task.

[0107] The language model 1004 may be a deep neural network that may use an encoder-decoder architecture. The language model 1004 can be trained to generate one or more lines of computer-executable code for natural language queries as input to the language model 1004. In one embodiment, during the training of the language model 1004, the system 1002 may provide a first NL feature 1006A as input to the encoder network of the deep neural network, and a first abstraction code feature 1006B to the decoder network. The language model 1004 can be trained to reconstruct abstraction code features (such as the first abstraction code feature 1006B) from NL features (such as the first NL feature 1006A).

[0108] In the prediction phase, system 1002 may be configured to receive input 1008. Input 1008 can be received from user 116 via user device 108 and may contain a natural language query. In one embodiment, the natural language query may be "create a TensorFlow session". Based on the received input 1008, system 1002 may be configured to apply a trained language model 1004 to the received input 1008. System 1002 may further be configured to apply the language model 1004 to input 1008 to produce output 1010. The produced output may contain first computer executable code that can be associated with the natural language query, for example, as shown in Figure 10.

[0109] Various embodiments of this disclosure can provide one or more non-temporary computer-readable storage media configured to store instructions causing a system (such as system 102) to perform an action in response to being executed. An action may include receiving source code data. The source code data may include computer-executable code and natural language (NL) text associated with the computer-executable code. An action may further include determining a block of code from the computer-executable code. An action may further include extracting a set of features from the block of code that are related to components of the source code data. The extraction can be performed by parsing the block of code using abstract syntax tree (AST) data of the block of code. An action may further include revising the AST data by abstracting the nodes of the AST data. The nodes may be abstracted based on the extracted set of features and the level of abstraction achieved for each node. An action may further include updating the source code data based on the revised AST data. An action may further include generating a dataset of NL and abstracted code features as training data based on the updated source code data. The operation may further include training a language model (such as language model 106) on a sequence-to-sequence generation task based on the generated dataset.

[0110] As described above, the embodiments described herein may include the use of a dedicated or general-purpose computer (e.g., processor 202 in Figure 2) containing various computer hardware or software modules, as will be discussed in more detail below. Furthermore, as described above, the embodiments described herein may be implemented using a computer-readable medium (e.g., memory 204 or persistent data storage 206 in Figure 2) that carries or has stored computer-executable instructions or data structures.

[0111] When used in this disclosure, the terms “module” or “component” may refer to a specific hardware implementation configured to perform the actions of a module or component, and / or a software object or software routine that can be stored and / or executed on general-purpose hardware of a computing system (e.g., computer-readable media, processing devices, or any other hardware). In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as being implemented in software (stored and / or executed on general-purpose hardware), specific hardware implementations, or combinations of software and specific hardware implementations, are also possible and intended. In this description, “computation entity” may be any computing system as previously defined in this disclosure, or any combination of modules or modulates that operate on a computing system.

[0112] In accordance with common practice, various features shown in the drawings may not be depicted to scale. The examples presented in this disclosure are not intended to be actual diagrams of any particular apparatus (e.g., a device, system, etc.) or method, but are merely idealized representations used to illustrate various embodiments of this disclosure. Accordingly, the size of various features may be arbitrarily enlarged or reduced for clarity. Furthermore, some of the drawings may be simplified for clarity. Accordingly, the drawings may not show all components of a given apparatus (e.g., a device) or all operations of a particular method.

[0113] In this disclosure, the terms used in particular in the claims set forth separately (for example, in the text of the claims set forth separately) are generally intended to be "open" terms (for example, the term "includes" should be interpreted as "includes but not limited to", the term "possesses" should be interpreted as "at least possesses", and the term "contains" should be interpreted as "includes but not limited to", etc.).

[0114] Furthermore, if a specific number of introduced claims are intended, such intention is explicitly stated in the claim; if there is no such statement, such intention does not exist. For example, for the sake of understanding, the claims listed below may include the use of the introductory phrases "at least one" and "one or more" to introduce the claims.

[0115] Furthermore, even when a claim description introducing a specific number is explicitly stated, a person skilled in the art will recognize that such description should be interpreted to mean at least the number stated (for example, the statement “two descriptions” without other modifiers means at least two descriptions, or two or more descriptions). Moreover, in examples where a similar provision is used, such as “at least one of A, B, and C, etc.” or “one or more of A, B, and C, etc.”, such structure is generally intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.

[0116] Furthermore, any separate word or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of containing one of the terms, either one of the terms, or both. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0117] However, the use of such phrases should not be seen as implicitly indicating that the introduction of a claim description with the indefinite article "a" or "an" limits any particular claim containing such introduced claim description to embodiments containing only one such description, even when the same claim contains the introduction phrase "one or more" or "at least one" and an indefinite article such as "a" (for example, "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"). The same applies to the use of definite articles used to introduce claim descriptions.

[0118] Furthermore, the use of terms such as "first," "second," and "third" in this specification does not necessarily mean a specific order or number of elements. Generally, terms such as "first," "second," and "third" are used as generic identifiers to distinguish different elements. Unless there is an indication that terms such as "first," "second," and "third" mean a specific order, these terms should not be understood to mean a specific order. Furthermore, unless there is an indication that terms such as "first," "second," and "third" mean a specific number of elements, these terms should not be understood to mean a specific number of elements. For example, a first widget may be described as having a first side, and a second widget may be described as having a second side. The use of the term "second side" in relation to a second widget may be intended to distinguish such a side of the second widget from the "first side" of the first widget, and may not mean that the second widget has two sides.

[0119] All examples and conditional language described herein are intended for educational purposes to assist readers in understanding the present invention and the concepts to which the inventors have contributed to advancing the art, and should be considered as not being limited to the examples and conditions described herein. Although embodiments of this disclosure have been described in detail, it should be understood that various changes, substitutions, and modifications may be made to them without departing from the spirit and scope of this disclosure.

[0120] The following note is left regarding the above embodiment. (Note 1) A method executed by a processor, The steps include receiving source code data which includes computer executable code and natural language (NL) text associated with the computer executable code, The steps include determining a block of code from the aforementioned computer executable code, The step of extracting a set of features related to the components of the source code data from the block of code, wherein the extraction is performed by parsing the block of code using the abstract syntax tree (AST) data of the block of code. A step of revising the AST data by abstracting the nodes of the AST data, wherein the nodes are abstracted based on the set of extracted features and the level of abstraction achieved for each of the nodes, The steps include updating the source code data based on the revised AST data, The steps include generating datasets of NL and abstract code features as training data based on the updated source code data, The steps include training a language model on a sequence-to-sequence generation task based on the generated dataset, A method that includes this. (Note 2) The method according to Appendix 1, wherein the block of code includes at least one of a class, a function, a nested block of lines of code, a single line of code, or a loop. (Note 3) Package metadata associated with the source code of multiple code packages, The set of classes used in the aforementioned source code, The set of subclasses used in the aforementioned source code, The set of functions used in the aforementioned source code, The natural language text used in the aforementioned source code, The steps include loading a database containing and The steps include determining the functions and classes used in the received source code data by using the aforementioned AST data, A step of determining the usage of each of the functions and classes based on the matching between the functions and classes using the entries in the database, A step of generating comments relating to the usage of each of the functions and classes, wherein the comments are included as part of the set of features extracted to revise the AST data, The method described in Appendix 1, further including the method described in Appendix 1. (Note 4) The steps include: using the aforementioned AST data to collect a list of code packages imported into the received source code data; The steps include: collecting a set of methods used in the aforementioned list of code packages, A step of generating a set of name transformations by associating the set of methods or one or more aliases of the set of methods with the names of each module contained in each code package in the list of code packages, wherein the list of name transformations is contained in the set of features, The method described in Appendix 1, further including the method described in Appendix 1. (Note 5) The method according to Appendix 4, wherein the AST data is further modified by replacing the names of the methods or one or more aliases in the AST data with the set of name translations. (Note 6) The steps include identifying the set of arguments in the AST data, A step of generating a set of pseudo-names by associating identifiers included in the set of arguments with information related to how the identifiers are used in the AST data, wherein the generated set of pseudo-names is part of the set of extracted features, The method described in Appendix 1, further including the method described in Appendix 1. (Note 7) The method according to Appendix 6, wherein the AST data is further revised by replacing each of the identifiers in the AST data with the respective pseudo-names of the set of pseudo-names. (Note 8) The abstraction of the nodes in the AST data is as follows: Selecting a node in the aforementioned AST data, To determine the type of the selected node, Selecting the level of abstraction based on the type of the node and the abstract grammar of the programming language used in the source code data, and Filter the attributes of the selected node based on the selected level of abstraction. The method described in Appendix 1, which is performed by [the specified method]. (Note 9) The steps include collecting a list of classes and a list of methods using the aforementioned AST data, A step of generating a dictionary containing a list of classes as keys and a list of methods as values ​​corresponding to the keys, wherein the generated dictionary is part of the set of extracted features, The method described in Appendix 1, further including the method described in Appendix 1. (Note 10) The steps include identifying the built-in objects used in the source code data and the types of the built-in objects based on the dictionary, The step of generating an object identifier by appending the respective type and built-in keyword of the identified type to the name of each of the built-in objects, and further revising the AST data by updating the identifier of the built-in object in the AST data with the generated object identifier. The method described in Appendix 9, which further includes the method described in Appendix 9. (Note 11) The method according to Appendix 10, wherein the object identifier is part of the set of features extracted to revise the AST data. (Note 12) The method according to Appendix 1, further comprising the step of selecting nodes in the AST data, wherein the AST data is further revised by dropping the selected nodes from the AST data, the dropping being performed on the basis of a determination that the selected nodes do not have function calls. (Note 13) A step of comparing the AST data of the code block with the revised AST data based on matching criteria, The steps include determining, based on the comparison, a set of changes applicable to the portion of the received source code data, and updating the source code data based on the set of changes; The method described in Appendix 1, further including the method described in Appendix 1. (Note 14) The method according to Appendix 1, wherein the language model is a deep neural network using an encoder-decoder architecture, and the language model is trained to generate one or more lines of computer-executable code for natural language queries as input to the language model. (Note 15) The method according to Appendix 1, wherein the sequence-to-sequence generation task is one of a code synthesis task, a code generation task, a code enhancement task, a code cloning task, or a code extraction task. (Note 16) A non-temporary computer-readable storage medium configured to store instructions that cause the system to perform an action in response to being executed, wherein the action is: Receiving source code data including computer executable code and natural language (NL) text associated with said computer executable code, Determining a block of code from the aforementioned computer executable code, The process involves extracting a set of features related to the components of the source code data from the aforementioned block of code, and this extraction is performed by parsing the block of code using the abstract syntax tree (AST) data of the block of code. The AST data is revised by abstracting the nodes of the AST data, wherein the nodes are abstracted based on the set of extracted features and the level of abstraction achieved for each of the nodes. Updating the source code data based on the revised AST data, Based on the updated source code data, a dataset of NL and abstract code features is generated as training data, The process involves training a language model on a sequence-to-sequence generation task based on the generated dataset, Non-temporary computer-readable storage media, including [specific data / information]. (Note 17) The block of code comprises at least one of a class, a function, a nested block of code lines, a single line of code, or a loop, as described in Appendix 16 of the non-temporary computer-readable storage medium. (Note 18) The language model is a deep neural network using an encoder-decoder architecture, and the language model is trained to generate one or more lines of computer-executable code for natural language queries as input to the language model, in the non-temporary computer-readable storage medium as described in Appendix 16. (Note 19) The sequence-to-sequence generation task is one of a code synthesis task, a code generation task, a code enhancement task, a code cloning task, or a code extraction task, as described in Appendix 16, for a non-temporary computer-readable storage medium. (Note 20) It is a system, Includes a processor, the processor is The system receives source code data, which includes computer executable code and natural language (NL) text associated with the computer executable code. A block of code is determined from the aforementioned computer executable code, A set of features related to the components of the source code data is extracted from the aforementioned block of code, and this extraction is performed by parsing the block of code using the abstract syntax tree (AST) data of the block of code. The AST data is revised by abstracting the nodes of the AST data, wherein the nodes are abstracted based on the set of extracted features and the level of abstraction achieved for each of the nodes. The source code data is updated based on the revised AST data. Based on the updated source code data, a dataset of NL and abstracted code features is generated as training data. A language model is trained on a sequence-to-sequence generation task based on the generated dataset. A system configured in such a way.< / func> < / type>

Claims

1. A method executed by a processor, The steps include receiving source code data which includes computer executable code and natural language (NL) text associated with the computer executable code, The steps include determining a block of code from the aforementioned computer executable code, The step of extracting a set of features related to the components of the source code data from the block of code, wherein the extraction is performed by parsing the block of code using the abstract syntax tree (AST) data of the block of code. A step of revising the AST data by abstracting the nodes of the AST data, wherein the nodes are abstracted based on the set of extracted features and the level of abstraction achieved for each of the nodes, The steps include updating the source code data based on the revised AST data, The steps include generating a dataset of NL and abstract code features as training data based on the updated source code data, The steps include training a language model on a sequence-to-sequence generation task based on the generated dataset, A method that includes this.

2. The method according to claim 1, wherein the block of code includes at least one of a class, a function, a nested block of lines of code, a single line of code, or a loop.

3. Package metadata associated with the source code of multiple code packages, The set of classes used in the aforementioned source code, The set of subclasses used in the aforementioned source code, The set of functions used in the aforementioned source code, The natural language text used in the aforementioned source code, The steps include loading a database containing and The steps include determining the functions and classes used in the received source code data by using the AST data, A step of determining the usage of each of the functions and classes based on the matching between the functions and classes using the entries in the database, A step of generating comments relating to the usage of each of the functions and classes, wherein the comments are included as part of the set of features extracted to revise the AST data, The method according to claim 1, further comprising:

4. The steps include: using the AST data to collect a list of code packages imported into the received source code data; The steps include: collecting a set of methods used in the aforementioned list of code packages, A step of generating a set of name transformations by associating the set of methods or one or more aliases of the set of methods with the names of each module contained in each code package in the list of code packages, wherein the list of name transformations is contained in the set of features, The method according to claim 1, further comprising:

5. The method according to claim 4, wherein the AST data is further modified by replacing the names of the methods or one or more aliases in the AST data with the set of name conversions.

6. The steps include identifying the set of arguments in the AST data, A step of generating a set of pseudo-names by associating identifiers included in the set of arguments with information relating to how the identifiers are used in the AST data, wherein the generated set of pseudo-names is part of the set of extracted features, The method according to claim 1, further comprising:

7. The method according to claim 6, wherein the AST data is further modified by replacing each of the identifiers in the AST data with each of the pseudo-names in the set of pseudo-names.

8. The abstraction of the nodes in the AST data is Selecting the node of the aforementioned AST data, To determine the type of the selected node, Selecting the level of abstraction based on the type of the node and the abstract grammar of the programming language used in the source code data, and Filter the attributes of the selected node based on the selected level of abstraction. The method according to claim 1, as performed by...

9. The steps include collecting a list of classes and a list of methods using the aforementioned AST data, A step of generating a dictionary containing a list of classes as keys and a list of methods as values ​​corresponding to the keys, wherein the generated dictionary is part of the set of extracted features, The method according to claim 1, further comprising:

10. The steps include identifying the built-in objects used in the source code data and the types of the built-in objects based on the dictionary, The step of generating an object identifier by appending the respective type and built-in keyword of the identified type to the name of each of the built-in objects, and further revising the AST data by updating the identifier of the built-in object in the AST data with the generated object identifier. The method according to claim 9, further comprising:

11. The method according to claim 10, wherein the object identifier is part of the set of features extracted to revise the AST data.

12. The method according to claim 1, further comprising the step of selecting nodes in the AST data, wherein the AST data is further revised by dropping the selected nodes from the AST data, the dropping being performed on the basis of a determination that there are no function calls in the selected nodes.

13. A step of comparing the AST data of the code block with the revised AST data based on matching criteria, The steps include determining, based on the comparison, a set of changes applicable to the portion of the received source code data, and updating the source code data based on the set of changes; The method according to claim 1, further comprising:

14. The method according to claim 1, wherein the language model is a deep neural network using an encoder-decoder architecture, and the language model is trained to generate one or more lines of computer-executable code for natural language queries as input to the language model.

15. The method according to claim 1, wherein the sequence-to-sequence generation task is one of a code synthesis task, a code generation task, a code enhancement task, a code cloning task, or a code extraction task.

16. A computer program that causes a system to perform an action, the action being: Receiving source code data including computer executable code and natural language (NL) text associated with said computer executable code, Determining a block of code from the aforementioned computer executable code, The process involves extracting a set of features related to the components of the source code data from the aforementioned block of code, and this extraction is performed by parsing the block of code using the abstract syntax tree (AST) data of the block of code. The AST data is revised by abstracting the nodes of the AST data, wherein the nodes are abstracted based on the set of extracted features and the level of abstraction achieved for each of the nodes. The source code data is updated based on the revised AST data, Based on the updated source code data, a dataset of NL and abstract code features is generated as training data. Training a language model on a sequence-to-sequence generation task based on the generated dataset, A computer program that includes [this].

17. The computer program according to claim 16, wherein the block of code includes at least one of a class, a function, a nested block of lines of code, a single line of code, or a loop.

18. The computer program according to claim 16, wherein the language model is a deep neural network using an encoder-decoder architecture, and the language model is trained to generate one or more lines of computer-executable code for natural language queries as input to the language model.

19. The computer program according to claim 16, wherein the sequence-to-sequence generation task is one of a code synthesis task, a code generation task, a code enhancement task, a code cloning task, or a code extraction task.

20. It is a system, Includes a processor, the processor is The system receives source code data, which includes computer executable code and natural language (NL) text associated with the computer executable code. A block of code is determined from the aforementioned computer executable code, A set of features related to the components of the source code data is extracted from the aforementioned block of code, and this extraction is performed by parsing the block of code using the abstract syntax tree (AST) data of the block of code. The AST data is revised by abstracting the nodes of the AST data, wherein the nodes are abstracted based on the set of extracted features and the level of abstraction achieved for each of the nodes. The source code data is updated based on the revised AST data. Based on the updated source code data, a dataset of NL and abstracted code features is generated as training data. A language model is trained on a sequence-to-sequence generation task based on the generated dataset. A system configured in such a way.

Images