Deep parameter learning for code synthesis

Abstract

To provide a method of training a language model on a sequence-to-sequence generation task, a storage medium, and a system.SOLUTION: Methods may include: receiving (402) a source code file; generating (410) an abstract syntax tree (AST) of a code included in the source code file; determining (412) a set of functions from the code; extracting (414) metadata associated with the set of functions from the code; selecting (416) a subset of functions for which descriptions in the extracted metadata satisfy filtering criteria; updating (418) the code; generating (424) a dataset of code features and respective metadata features based on the updated code and the metadata; and training (426) a language model on a sequence-to-sequence generation task, based on the dataset.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 261,602, entitled "Library Corpus for Large-Scale Language Models and Code Retrieval Models Using Augmented Code", filed on September 24, 2021, the entire content of which is incorporated herein by reference.

[0002] Embodiments discussed in the present disclosure relate to deep parameter learning for code synthesis. Background Art

[0003] With advances in machine learning, various types of language models have been developed for different machine programming tasks such as code synthesis and code retrieval. A language model is a statistical representation of the probability distribution of word sequences, which aims to find relationships between different words by processing a large corpus. Some language models aim to learn general-purpose representations that support downstream natural language-programming language (NL-PL) applications such as code synthesis. Code synthesis corresponds to the task where a machine (such as a computer) aims to generate computer-executable code in response to a query provided as input. To perform code synthesis using a language model, the language model must first be trained on a dataset containing training samples. The quality of the training samples included in the dataset significantly affects the overall performance of the language model. For training language models, many state-of-the-art technologies mainly focus on the quantity of training samples rather than the quality of training samples.

[0004] The subject matter of the claims in this disclosure is not limited to embodiments that solve any defects or that operate only in the environment described above. Rather, this background art is provided solely to illustrate one exemplary technique in which several embodiments described herein may be carried out. [Overview of the project]

[0005] According to one embodiment, the operation may include receiving a source code file associated with a software package and generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the source code file. This operation may further include, based on the AST, determining a set of functions / procedures of all defined classes from the computer-executable code and extracting metadata associated with the set of functions from the computer-executable code. This operation may further include selecting a subset of functions from the set of functions whose descriptions in the extracted metadata satisfy filtering criteria. This operation may further include updating the computer-executable code by filtering the lines of code (LoC) corresponding to the subset of functions / procedures from the computer-executable code. This operation may further include generating a training dataset of code features and their respective metadata features based on the updated computer-executable code and metadata. This operation may then include training a language model on a sequence-to-sequence generation task based on the training dataset.

[0006] The objectives and advantages of this embodiment are realized and achieved by at least the elements, features, and combinations specifically indicated in the claims.

[0007] Both the general description above and the embodiments for carrying out the invention described below are merely illustrative and descriptive, and do not limit the invention as described in the claims. [Brief explanation of the drawing]

[0008] Exemplary embodiments are described and illustrated with additional specificity and detail through the use of the accompanying drawings.

[0009] [Figure 1] This diagram illustrates an exemplary environment related to deep parameter learning for code synthesis.

[0010] [Figure 2] This is a block diagram of a system for deep parameter learning for code synthesis.

[0011] [Figure 3] This is a diagram illustrating exemplary source code for deep parameter learning.

[0012] [Figure 4] A flowchart illustrating an example of a deep parameter learning method is provided.

[0013] [Figure 5] A flowchart illustrating an exemplary method for updating computer-executable code for code synthesis is provided.

[0014] [Figure 6A] Draw an exemplary Abstract Syntax Tree (AST) for a subset of functions.

[0015] [Figure 6B] A revised AST is shown as an example of a subset of functions obtained from the exemplary AST in Figure 6A.

[0016] [Figure 7] This diagram depicts exemplary computer-executable code and an updated version of the exemplary computer-executable code.

[0017] [Figure 8]It is a diagram illustrating an example scenario for training a language model for code synthesis based on deep parameter learning.

[0018] All are in accordance with at least one embodiment described in the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS

[0019] Machine learning has led to the development of language models for various machine programming tasks. A language model is a probabilistic model that provides a statistical representation of the probability distribution of word sequences, aiming to find relationships between different words by processing a large corpus. For example, a language model may be able to predict the probability of the word "pandas" appearing after "import", such as in "from nbconvert.preprocessors import ExecutePreprocessor". Specifically, given a sequence of length "m", the language model assigns a probability P(w1,w2,...w m ) to the entire sequence.

[0020] Language models are used in various sequence-to-sequence generation tasks such as code synthesis tasks, code search tasks, and software package analysis tasks. A code synthesis task corresponds to a task of generating source code based on a natural language query. A code search task corresponds to a task of searching for code snippets related to a given natural language query from a code base. A software package analysis task corresponds to a task of analyzing a software package for relevant information.

[0021] To perform the above sequence-to-sequence generation task, the language model must be trained on dataset examples. For example, in the case of a code synthesis task, the language model must be trained on a dataset including pairs of code snippets and natural language queries. A code snippet may include natural language text such as docstrings and comments that are associated with the code snippet and can be semantically similar to the corresponding natural language query. However, some code snippets do not include any natural language text. In such scenarios, the language model determines the relationship between the code snippet and the natural language query based on keywords present in the code snippet.

[0022] Current state-of-the-art techniques for training language models use only natural language queries that are independent of the existence of code snippets and natural language text within the code snippets. Training a language model using such a dataset results in a language model that is not sufficiently generalized and lacks accuracy. Therefore, there is a need to generate a filtered dataset such that when a language model is trained on such a filtered dataset (as compared to a language model trained on an unfiltered dataset), the trained language model is generalized and accurate.

[0023] This disclosure uses code snippets containing function metadata and detailed information related to existing natural language descriptions (comments, dockstrings, etc.) for training language models. Specifically, this disclosure uses metadata associated with code snippets in a training dataset for training language models. This disclosure provides a rich dataset that can be used to train language models using function parameters, return values, and corresponding natural language text associated with code snippets. This disclosure also provides a method for filtering out and removing code snippets for which natural language text does not exist. Furthermore, this disclosure provides a method for pruning code snippets to generate a training dataset.

[0024] In contrast to current state-of-the-art approaches, this disclosure provides an approach to obtaining datasets that help achieve more generalized language models compared to language models obtained using state-of-the-art approaches. Furthermore, the disclosed language models may help software engineers, data scientists, or developers generate more detailed code compared to state-of-the-art methods. Based on experimental data, it has been observed that language models trained on filtered datasets may perform better than language models trained on current state-of-the-art approaches.

[0025] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0026] Figure 1 is a diagram representing an exemplary environment for deep parameter learning for code synthesis arranged according to at least one embodiment described herein. Referring to Figure 1, an exemplary environment 100 is shown. In the exemplary environment 100, a system 102 and a data source 104 are shown. A language model 106, a user device 108, a communication network 110, a source code file 112, and a dataset 114 are further shown. The system 102, the data source 104, and the user device 108 may be communicatively coupled to each other via the communication network 110.

[0027] Further examples of users 116 that may be associated with user device 108 are provided. Examples of user device 108 may include, but are not limited to, a mobile device, a desktop computer, a laptop, or a computer workstation. In one or more embodiments, user device 108 may include a user terminal device and a server communicatively coupled to the user terminal device. Examples of user terminal devices may include, but are not limited to, a mobile device, a desktop computer, a laptop, or a computer workstation.

[0028] The data source 104 may include appropriate logic, circuitry, and interfaces that can be configured to store a source code file 112. The source code file 112 may be associated with a software package and may include computer-executable code 112A. In one embodiment, the data source 104 may store metadata associated with the software package. Examples of the data source 104 may include, but are not limited to, a web-based code hosting server, a database server, a file server, a web server, an RSS (Really Simple Syndication) feed, and a server hosting web applications related to websites and packages.

[0029] In one embodiment, the data source 104 may be implemented as multiple servers, which may include storage distributed across one or more availability zones (e.g., data centers). In one embodiment, the data source 104 may include a front-end system and a back-end system. The front-end system may be configured to provide an interface (e.g., a web page or a client-side interface for a web application) for viewing information associated with a package. The back-end system may store a database, logic, and instructions for displaying content on the interface provided by the front-end system.

[0030] The language model 106 may be a probabilistic model that can be trained to generate a probability distribution on a sequence of tokens on an alphabet. The language model 106 may be either a statistical language model or a neuro-linguistic model. A statistical language model may use statistical techniques to learn the probability distribution. These statistical techniques may include, for example, the unigram technique, the N-gram technique, hidden Markov models (HMMs), and other linguistic rules. Details of the implementation of the above-mentioned statistical techniques are known in the art. Therefore, for simplicity, a detailed explanation of the above-mentioned statistical techniques is omitted.

[0031] A neuro-linguistic model may use one or more neural networks to learn the probability distribution of words. In one embodiment, each of the one or more neural networks included in the neuro-linguistic model may be a system of computational networks or artificial neurons arranged in multiple layers as nodes. The multiple layers of the neural network may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (i.e., artificial neurons). The outputs of all nodes in the input layer may be coupled to at least one node in the hidden layers. Similarly, the input of each hidden layer may be coupled to the output of at least one node in the other layers of the neural network. The output of each hidden layer may be coupled to the input of at least one node in the other layers of the neural network. Nodes in the final layer may receive input from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer may be determined from the hyperparameters of the neural network. Such hyperparameters may be set before or after training the neural network on the dataset 114.

[0032] Each node in a neural network may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that can be tuned during network training. These parameters may include, for example, weight parameters and normalization parameters. Each node may use the mathematical function to compute an output based on one or more inputs from nodes in other layers of the neural network (e.g., previous layers). All or some nodes in a neural network may correspond to the same or different mathematical functions.

[0033] In training a neural network, one or more parameters of each node in the neural network may be updated based on whether the output of the final layer for a given input (from dataset 114) matches the correct result based on the loss function for the neural network. The above process may be repeated for the same or different inputs until the minimum loss function is achieved and the training error is minimized. Several methods for training, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristics, are known in the art.

[0034] The neural language model may include electronic data that can be implemented, for example, as a software component of an application executable on system 102. The neural language model may depend on libraries, external scripts, or other logic / instructions for execution by a processing device such as a processor. The neural language model may include code and routines configured to enable a computer device such as a processor to perform one or more actions to generate a line of computer-executable code 112A for a natural language query as input to the neural language model. Additionally or alternatively, the neural language module may be implemented using hardware including a processor, a microprocessor (for example, to perform or control one or more actions), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural language module may be implemented using a combination of hardware and software.

[0035] Each example of one or more neural networks includes, but is not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), CNN-recurrent neural networks (CNN-RNNs), artificial neural networks (ANNs), LSTM (Long Short-Term Memory) network-based RNNs, LSTM+ANNs, gated recurrent unit (CRU)-based RNNs, fully connected neural networks, CTC (Connectionist Temporal Classification)-based RNNs, deep Bayes neural networks, and / or combinations of such networks. In certain embodiments, each of the one or more neural networks may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0036] In one embodiment, the language model 106 may correspond to a DNN using an encoder-decoder architecture. The DNN may be trained to generate lines of computer-executable code 112A in response to natural language queries as input to the language model 106. Specifically, such a language model may include an encoder neural network and a decoder neural network. Examples of such DNNs may include, but are not limited to, LSTM (Long Short-Term Memory) networks, gated recurrent unit (GRU) networks, transformer models, or variations of transformer models such as BERT (Bidirectional Encoder Representations from Transformers) models and CodeBERT models.

[0037] In operation, system 102 may be configured to receive a source code file 112 from a data source 104. The source code file 112 may be associated with a software package and may contain computer-executable code 112A. In one embodiment, the source code file 112 may contain resource files associated with the software package. Details regarding the reception of the source code file 112 are provided, for example, in Figures 3 and 4.

[0038] Upon receiving, system 102 may be further configured to generate an abstract syntax tree (AST) of at least a portion of the computer-executable code 112A contained in source code file 112. The AST may correspond to a tree representation of the abstract syntax structure of the computer-executable code 112A in a particular programming language such as Python, Java®, or JavaScript. Each node in the tree may represent an configuration that may occur within the computer-executable code 112A. More specifically, the AST may be a tree representation of the computer-executable code 112A that can be primarily used by a compiler (or interpreter) to read the computer-executable code 112A and generate a target binary. Further details regarding the generation of the AST are provided, for example, in Figure 4.

[0039] System 102 may be further configured to determine a set of functions (also called a set of procedures) from computer-executable code 112A. The set of functions may be determined based on the generated AST. Each function in the set of functions may be a self-contained module of computer-executable code 112A capable of performing a specific task (e.g., adding two digits or concatenating two strings). System 102 may be further configured to extract metadata 112B that may be associated with the set of functions from the computer-executable code 112A. In one embodiment, the metadata 112B may be extracted from dockstrings and comments that may be associated with the set of functions.

[0040] System 102 may be further configured to select a subset of functions whose descriptions in the extracted metadata 112B satisfy filtering criteria. The subset of functions may be extracted from a set of functions. The filtering criteria may specify rules for selecting functions from the set of functions. According to one embodiment, the rules may include a first rule for determining whether a description exists in the metadata 112B, and a second rule for determining whether the description in the metadata 112B includes descriptions of one or more parameters associated with the function, and descriptions of one or more return values ​​associated with the function. Details regarding the filtering criteria are provided, for example, in Figure 4.

[0041] System 102 may be configured to update computer-executable code 112A by filtering out lines of code (LoCs) corresponding to a subset of functions from the computer-executable code 112A. System 102 may be further configured to generate a dataset 114 of code features and their respective metadata features based on the updated computer-executable code 112A and metadata 112B. Details regarding the code features and their respective metadata features are provided, for example, in Figure 7. System 102 may be further configured to train a language model 106 for a sequence-to-sequence generation task based on the dataset 114. A sequence-to-sequence generation task may, for example, be a code synthesis task, a code retrieval task, or a software package analysis task. Details regarding the training of the language model 106 are provided, for example, in Figure 7.

[0042] Communication between system 102, data source 104, language model 106, and user device 108 may be performed via a communication network 110. The communication network 110 may include a communication medium from which system 102 can communicate with data source 104, language model 106, user device 108, and / or other devices (not shown). Examples of the communication network 110 may include, but are not limited to, the internet, cloud networks, cellular networks (such as 4th generation LTE (Long-Term Evolution) or 5th generation NR (New Radio)), Wi-Fi (Wireless Fidelity) networks, PAN (Personal Area Network), LAN (Local Area Network), and / or MAN (Metropolitan Area Network). Various devices in the exemplary environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: TCP / IP (Transmission Control Protocol and Internet Protocol), UDP (User Datagram Protocol), HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), ZigBee, EDGE, IEEE 802.11, Li-Fi (light fidelity), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and / or Bluetooth® (BT) communication protocols, or any combination thereof.

[0043] System 102 can be modified, added to, or omitted without departing from the scope of this disclosure. For example, in some embodiments, System 102 may include any number of other components that are not expressly exemplified or described.

[0044] Figure 2 is a block diagram of a system for deep parameter learning for code synthesis arranged according to at least one embodiment described in this disclosure. Figure 2 is described in conjunction with the elements from Figure 1. Referring to Figure 2, a block diagram 200 of system 102 of Figure 1 is shown. The block diagram 200 may further include a processor 202, memory 204, persistent data storage 206, I / O block 208, network interface 210, and language model 106.

[0045] Processor 202 may include suitable logic, circuitry, and / or interfaces that can be configured to execute program instructions associated with different operations performed by System 102. Processor 202 may include any suitable special-purpose or general-purpose computer, computing entity, or processing device, including various computer hardware or software modules, and may be configured to execute instructions stored in any applicable computer-readable storage medium. For example, Processor 202 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Although shown as a single processor in Figure 2, Processor 202 may include any number of processors configured individually or collectively to execute or direct any number of operations of System 102 as described herein.

[0046] In some embodiments, the processor 202 may be configured to interpret and / or execute program instructions stored in memory 204 and / or persistent data storage 206, and / or process data. In some embodiments, the processor 202 may fetch program instructions from persistent data storage 206 and load them into memory 204. After the program instructions are loaded into memory 204, the processor 202 may execute the program instructions. Some examples of the processor 202 may be a central processing unit (CPU), a reduced instruction set computer (RISC) processor, an ASIC processor, a composite instruction set computer (CISC) processor, a graphics processing unit (GPU), a coprocessor, and / or a combination thereof.

[0047] Memory 204 may include preferred logic, circuitry, and / or interfaces that can be configured to store program instructions executable by the processor 202. In a given embodiment, memory 204 may be configured to store received source code files 112, extracted metadata 112B, a set of functions, metadata features, a list of software packages, resource files, and a generated dataset 114. In a given embodiment, memory 204 may be configured to store a language model 106. Memory 204 may include a computer-readable storage medium for carrying or having computer-executable instructions or data structures stored therein. Such a computer-readable storage medium may include any available medium that can be accessed by a general-purpose or dedicated computer such as the processor 202.

[0048] For example, but not limited to, such computer-readable storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage, flash memory devices (e.g., solid-state memory devices), or any other storage media that can be used to carry or store specific program code in the form of computer-executable instructions or data structures, and that can be accessed by a general-purpose or special-purpose computer. The above combinations may also fall within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause a processor 202 to perform a specific operation or group of operations associated with system 102.

[0049] The persistent data storage 206 may include preferred logic, circuitry, and / or interfaces that can be configured to store program instructions executable by the processor 202. The persistent data storage 206 may include a computer-readable storage medium for carrying or having computer-executable instructions or data structures stored therein. Such a computer-readable storage medium may include any available medium that can be accessed by a general-purpose or dedicated computer, such as the processor 202.

[0050] For example, and not limited to, such computer-readable storage media may include tangible or non-temporary computer-readable storage media that are optical disk storage, magnetic disk storage, or other magnetic storage devices (e.g., HDD (Hard-Disk Drive)), flash memory devices (e.g., SSD (Solid State Drive), SD (Secure Digital) card, or other solid-state memory devices), or any other storage media that are used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause a processor 202 to perform a specific operation or group of operations associated with system 102.

[0051] I / O device 208 may include preferred logic, circuitry, interfaces, and / or code that can be configured to receive one or more user inputs. I / O device 208 may be further configured to provide outputs in response to one or more user inputs. I / O device 208 may include various I / O devices that can be configured to communicate with other components such as processor 202 and network interface 210. Examples of input devices may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, and / or microphone. Examples of output devices may include, but are not limited to, a display device and a speaker.

[0052] The network interface 210 may include preferred logic, circuitry, interfaces, and / or code that can be configured to establish communication between system 102, data source 104, language model 106, and user device 108 via the communication network 110. The network interface 210 may be implemented using a variety of known techniques to support wired or wireless communication of system 102 via the communication network 110. The network interface 210 may include, but is not limited to, an antenna, radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, and / or a local buffer.

[0053] The network interface 210 can communicate wirelessly with wireless networks such as the Internet, intranets, and / or cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs). Wireless communication may use any of several communication standards, protocols, and technologies, such as GSM (Global System for Mobile Communications), EDGE (Enhanced Data GSM Environment), W-CDMA (wideband code division multiple access), LTE (Long Term Evolution), CDMA (code division multiple access), TDMA (time division multiple access), Bluetooth, Wi-Fi (Wireless Fidelity) (IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n, etc.), VoIP, Li-Fi (light fidelity), or Wi-MAX.

[0054] As shown in Figure 1, functions or operations performed by system 102 may be performed by processor 202. Operations performed by processor 202 are described in detail, for example, in Figures 3, 4, 5, 6, 6A, 6B, 7, and 8.

[0055] Figure 3 is a diagram illustrating exemplary source code for deep parameter learning for code synthesis according to at least one embodiment described in this disclosure. Figure 3 is described in conjunction with the elements of Figures 1 and 2. Referring to Figure 3, an electronic user interface (UI) 300 is shown. The electronic UI 300 may be displayed on a user device 108. Within the electronic UI 300, a source code file 302 is further shown. The source code file 302 may include computer-executable code 304 and metadata 306.

[0056] In one embodiment, system 102 may be configured to receive a source code file 302 associated with a software package. The source code file 302 may be received from a data source 104. The source code file 302 may be associated with a software package, such as an open-source Python package. The source code file 302 may contain computer-executable code 304. The computer-executable code 304 may be written in a programming language such as Python, C, C++, C#, Swift, JavaScript, Go, Java®, or R, but is not limited to these. Referring to Figure 3, the computer-executable code 304 is written in Python.

[0057] In one embodiment, the computer-executable code 304 may include a set of functions. Specifically, the computer-executable code 304 may include one or more of the following: a class, methods contained within a class, and functions which may be different from the methods. In one embodiment, the set of functions may include methods and functions. For example, the set of functions in the computer-executable code 304 may include "list()", "_get_cache_or_reload()", "sys.path.insert()", "os.path.join()", and "sys.path.remove()".

[0058] In addition to the computer-executable code 304, the source code file 302 may include metadata 306 associated with at least one of the set of functions. The metadata 306 may include at least one of the following: the function name, the count of parameters associated with the function, the count of one or more default parameters associated with the function, the default value of one or more default parameters, the type of each parameter, a description of the function, one or more decorators associated with the function, etc. The description may include at least one of the following: a description of each parameter, a description of the return value associated with the function, a description of one or more default parameters associated with the function, or a description of the function's purpose. For example, as shown, the metadata 306 associated with the "list()" function is:

number

[0059] Figure 4 illustrates a flowchart of an exemplary method of deep parameter learning for code synthesis according to at least one embodiment described herein. Figure 4 is described in conjunction with the elements of Figures 1, 2 and 3. Referring to Figure 4, flowchart 400 is shown. The method shown in flowchart 400 may begin from 402 and may be performed by any preferred system, apparatus, or device, such as system 102 in Figure 1 or 2.

[0060] In 402, the repository address of the package dataset is received. In one embodiment, system 102 may be configured to receive the repository address of the package dataset as input from user 116 via user device 108. The repository address may be a URL (uniform resource locator) associated with a web page of the package dataset that may be hosted on data source 104. The package dataset may include a list of software packages. Examples of datasets may include, but are not limited to, CodeSearchNet and PyTorrent.

[0061] In 404, a list of software packages may be extracted. In one embodiment, system 102 may be configured to extract a list of software packages based on the received repository address. The list of software packages may be extracted from data source 104, and such packages may be associated with a package dataset associated with the repository address. Each software package in the list of software packages may contain resource files. As an example, these resource files may contain information about resources such as definitions, configurations, setups, requirements, and distributions associated with the software package.

[0062] In 406, a software package may be selected from the extracted list of software packages. In one embodiment, the software package may be selected based on user input via the user device 108. In another embodiment, the system 102 may be configured to select software packages sequentially (or alphabetically).

[0063] In 408, resource files associated with the selected software package may be retrieved. In one embodiment, system 102 may be configured to scrape resource files associated with the selected software package from data source 104. For example, system 102 may be configured to scrape resource files using a web crawler or web scraper.

[0064] The retrieved resource files may include source code files 302 and package metadata associated with the selected software package. The source code files 302 may include computer-executable code 304 that can be separated into elements such as functions, methods, classes, import statements, and loop statements. In one embodiment, the retrieved resource files may include, but are not limited to, files that provide information about resources such as definitions, configurations, setups, requirements, and distributions associated with the selected software package.

[0065] In 410, an abstract syntax tree (AST) may be generated for at least a portion of the computer-executable code 304. The AST may correspond to a tree representation of the abstract syntax structure of a portion of the computer-executable code 304 in a specific programming language such as Python, Java, or JavaScript. Each node in the tree may represent an configuration that could occur within the portion of the computer-executable code 304. More specifically, the AST may be a tree representation of a portion of the computer-executable code 304 that can be primarily used by a compiler (or interpreter) to read the portion of the computer-executable code 304 and generate a target binary. Further details regarding the AST are provided in Figure 6A.

[0066] In 412, a set of functions may be determined. In one embodiment, the set of functions may be determined from the AST of at least a portion of the computer-executable code 304. The set of functions may be contained in at least a portion of the computer-executable code 304. Each function in the determined set of functions may have a name, one or more parameters, and at least one return value. In addition, the determined set of functions may include one or more default parameters and one or more decorators. As a first example, a function may be defined as follows:

number

[0067] In 414, metadata 306 associated with the determined set of functions may be extracted from computer-executable code 304. In one embodiment, metadata 306 may be included as a dockstring or comment in computer-executable code 304 contained in source code file 302. Metadata 306 may include, for example, the function name, the count of parameters associated with the function, the count of one or more default parameters associated with the function, the default value of one or more default parameters, the type of each parameter, a description of the function, or one or more decorators associated with the function. The function description may include at least one of the following: a description of each parameter, a description of the return value associated with the function, a description of one or more default parameters associated with the function, or a description of the purpose of the function.

[0068] Referring to the function "list()" in the first example, the function name could be "list". The number of parameters associated with the function could be three (i.e., "abc", "force_reload", and "skip_validation"). The number of default parameters associated with the function could be two (i.e., "force_reload" and "skip_validation"). The default value for the default parameters could be "False". The type of each parameter could correspond to the data type associated with that parameter. As in the first example, Python uses dynamic typing, so the type of each parameter may be dynamically selected at runtime based on the value associated with the corresponding parameter.

[0069] In 416, a subset of functions may be extracted from a set of functions. System 102 may be configured to select a subset of functions from the set of functions that satisfy filtering criteria. The filtering criteria may specify rules for selecting functions from the set of functions. In one embodiment, the rules may include a first rule and a second rule. The first rule may determine whether a description of a corresponding function exists in metadata 306, and the second rule may determine whether the description of a corresponding function in the metadata includes descriptions of one or more parameters associated with the corresponding function and descriptions of one or more return values ​​associated with the corresponding function.

[0070] In one embodiment, system 102 may be configured to determine one or more parameters associated with each of the set of functions. System 102 may also determine one or more return values ​​associated with each of the set of functions. Based on the one or more parameters and the one or more return values, system 102 may detect the presence of the one or more parameters and the one or more return values ​​in the description within the metadata 306 of the corresponding function. Based on the detection, system 102 may be configured to select a subset of functions.

[0071] In one embodiment, system 102 may be configured to detect the presence of one or more parameters and one or more return values ​​in the description of a corresponding function within metadata 306. Presence may be detected based on a match between one or more parameters and one or more return values ​​in the function definition and one or more parameters and one or more return values ​​in the description of the corresponding function. In another embodiment, metadata 306 may be formatted in a structured format such as NumPy Doc Format, Google® Doc Format, or Java Doc Format, but is not limited to these. In such a case, system 102 may be configured to detect the presence of parameters or return values ​​in fields corresponding to parameters or return values ​​in the structured format.

[0072] In 418, the computer-executable code 304 may be updated. The computer-executable code 304 may be updated by filtering out lines of code (LoCs) corresponding to a subset of functions from the computer-executable code 304. In one embodiment, the computer-executable code 304 may be updated based on the AST of the computer-executable code 304. Details relating to the updating of the computer-executable code 304 are provided, for example, in Figures 5 and 6.

[0073] In 420, it may be determined whether more source code files exist in the selected software package. If more source code files are available within the selected software package, control may be passed to 408. Otherwise, control may be passed to 422.

[0074] At 422, it may be determined whether there are more software packages available in the package dataset. If there are more software packages available in the package dataset, control may be passed to 406. Otherwise, control may be passed to 424.

[0075] In 424, a dataset of code features and their respective metadata features may be generated. In one embodiment, system 102 may be configured to generate a dataset 114 of code features and their respective metadata features. The dataset 114 may be generated based on updated computer-executable code and metadata 306. The metadata features include the respective natural language (NL) text portion of the description corresponding to the subset of functions, the class name corresponding to the subset of functions, the function name corresponding to the subset of functions, and the path to the source code file 302. Details regarding the dataset 114 are provided, for example, in Figure 7.

[0076] In one embodiment, the system 102 may be configured to determine the return value corresponding to each function in a subset of functions from updated computer-executable code in order to generate a dataset 114.

[0077] In one embodiment, the generated dataset 114 may be a dictionary in key-value format. The keys of the dictionary may correspond to metadata features including the natural language (NL) text portion of the description corresponding to the subset of functions, the class name corresponding to the subset of functions, the function name corresponding to the subset of functions, and the path to the source code file. The keys may also further include code features which may further include return values ​​and abstracted code data. Details regarding the abstracted code data are provided, for example, in Figures 6A and 6B.

[0078] In 426, the language model 106 may be trained for a sequence-to-sequence generation task based on the generated dataset 114. The sequence-to-sequence generation task may be one of a code synthesis task, a code retrieval task, or a software package analysis task. Specifically, the language model 106 may be trained to generate computer-executable lines of code in response to natural language queries as input to the language model 106. In one embodiment, the language model 106 may be implemented using a deep neural network with an encoder-decoder architecture. If a pre-trained language model exists, the system 102 may fine-tune the pre-trained language model based on the generated dataset 114. In fine-tuning, examples of the dataset 114 may be used to update parameters such as the weights of the pre-trained language model. Details regarding the training of the language model 106 are provided, for example, in Figure 7.

[0079] Control may be passed to termination. Flowchart 400 is illustrated with individual operations such as 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, 424, and 426. However, in certain embodiments, such individual operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular embodiment, without impairing the essence of the disclosed embodiments.

[0080] Figure 5 illustrates a flowchart of an exemplary method for updating computer-executable code for code synthesis, according to at least one embodiment described in this disclosure. Figure 5 is described in conjunction with the elements of Figures 1, 2, 3, and 4. Referring to Figure 5, flowchart 500 is shown. The method shown in flowchart 500 may begin at 502 and may be performed by any preferred system, apparatus, or device, such as system 102 in Figure 1 or 2.

[0081] In 502, nodes corresponding to a subset of functions in the AST can be determined. In one embodiment, system 102 may be configured to determine nodes corresponding to a subset of functions in the AST. The AST may be a tree in which each node of the tree may represent an configuration that could occur in computer-executable code 304. In one embodiment, system 102 may be configured to determine the AST corresponding to a subset of functions based on the configuration associated with the subset of functions. Details regarding the subset of functions are provided, for example, in Figure 4.

[0082] In 504, a first layer of the AST node may be selected. In one embodiment, system 102 may be configured to select a first layer of the AST node. The first layer of the AST may correspond to the top layer of the AST corresponding to a subset of functions. In one embodiment, system 102 may be configured to loop through all layers of the AST. Details regarding multiple layers are provided, for example, in Figures 6A and 6B.

[0083] In 506, a first node may be selected from the selected first layer of nodes in the AST. In one embodiment, the system 102 may be configured to select a node from the nodes determined (in 502). In one embodiment, the first node of the first layer of nodes in the AST may correspond to the root node of the AST. Each node in the AST may correspond to the configuration of computer-executable code 304.

[0084] In 508, selected nodes may be marked for pruning operations. In one embodiment, system 102 may be configured to mark selected nodes for pruning operations based on a determination of whether the selected node corresponds to a code component different from a function call having one or more parameters.

[0085] Specifically, system 102 may determine the parameters associated with a function that may have been used in a function call. If each of the one or more parameters associated with the function is used in the function call, the selected node may be marked for pruning. Otherwise, if at least one parameter associated with the function is not used in the function call, the selected node may not be marked for pruning.

[0086] In one embodiment, the pruning operation may correspond to a pruning function that can be executed on a selected node. For example, the pruning function may be a function called "_is_call(node)". When executed, the pruning function may extract a segment of the function definition from the AST and process the extracted segment to check whether the function contains one or more parameters and a function call. If the selected node contains both a function call and one or more parameters, the pruning function may return True; otherwise, the pruning function may return False.

[0087] In another embodiment, the pruning function may return False if there are no function calls in one or more child nodes of the function associated with the selected node. If the pruning function returns False, system 102 may be configured to prune the node selected from the AST. Otherwise, the selected node may not be pruned.

[0088] In one embodiment, a pruning function may be executed to find function calls that may contain one or more parameters as input. By collecting such sequences of function calls, system 102 may have detailed information about the computer-executable code from which noise (irrelevant lines of the computer-executable code) has been filtered out. The sequence of function calls may be used to determine the order of the main functions in the computer-executable code 304.

[0089] In one embodiment, pruning operations may be performed on nodes associated with user-defined functions and methods. In another embodiment, pruning operations may be performed on nodes associated with user-defined functions and methods, as well as built-in functions. Such execution may be based on a configuration that can be performed by the administrator of system 102.

[0090] In 510, the identifier associated with the selected node and the corresponding return value of the pruning operation may be stored. In one embodiment, the identifier of the selected node and the corresponding return value of the pruning operation may be stored in the memory 204 of the system 102 in key-value format.

[0091] In step 512, it may be determined whether more nodes are present in the selected layer of the AST. If more nodes are present in the selected layer, control may be returned to step 506, and the next node in the selected layer may be selected. Otherwise, control may be transferred to step 514.

[0092] At 514, it may be determined whether the selected layer is the last layer of the AST. If the selected layer is not the last layer of the AST, control may be returned to 504, and the next layer may be selected. Otherwise, control may be transferred to 516.

[0093] In 516, a revised AST can be obtained. In one embodiment, the revised AST can be obtained based on the return value associated with the stored identifier and the corresponding node. Specifically, the revised AST can be obtained by performing a pruning operation to remove marked nodes from the AST. Further details regarding the revised AST are provided, for example, in Figures 6A and 6B.

[0094] In 518, the computer-executable code may be updated. In one embodiment, the system 102 may be configured to update the computer-executable code 304 by extracting a sequence of function calls. The extracted sequence of function calls may correspond to each function in a subset of functions. In one embodiment, the sequence of function calls may be extracted by using a revised AST.

[0095] System 102 may be further configured to generate abstracted code data by abstracting a sequence of function calls. In one embodiment, the abstraction may correspond to the removal of function calls from a sequence of function calls. In another embodiment, the abstraction may correspond to the modification of a computer executable based on the addition (or modification of one or more lines of code) of one or more lines of code present in the computer executable code to negate the effect of the removal of a sequence of function calls. Further details about the abstracted code data are provided in Figures 6A and 6B.

[0096] Figure 6A shows an exemplary Abstract Syntax Tree (AST) of a subset of functions according to at least one embodiment described in this disclosure. Figure 6A is described in conjunction with the elements of Figures 1, 2, 3, 4, and 5. Referring to Figure 6A, an exemplary Abstract Syntax Tree (AST) 600A associated with the exemplary subset of functions is shown. Further shown are several nodes which may include a first node 602, a second node 604, a third node 606, a fourth node 608, a fifth node 610, a sixth node 612, a seventh node 614, and an eighth node 616.

[0097] System 102 may be configured to receive source code files associated with a software package. Upon receipt, System 102 may generate an exemplary AST600 of exemplary computer-executable code that may be present in the source code file. The exemplary AST600 may correspond to a tree representation of the abstract syntactic structure of computer-executable code in a particular programming language such as Python, Java, C++, or JavaScript. Each node in the tree may represent an configuration that may occur within the computer-executable code 112A.

[0098] System 102 may be further configured to determine a set of functions from computer-executable code based on AST 600. The set of functions may correspond to methods and functions contained within code components of the computer-executable code. A code component may include one or more of the following: classes, methods contained within classes, and functions which may differ from methods. System 102 may be further configured to extract metadata associated with the set of functions from the computer-executable code. System 102 may be further configured to select a subset of functions whose descriptions within the extracted metadata satisfy filtering criteria. The subset of functions may be extracted from the set of functions.

[0099] In one embodiment, system 102 may be configured to determine a node corresponding to a subset of functions in an exemplary AST600. In one embodiment, the determined node corresponding to a subset of functions in an exemplary AST600 may include a first node 602, a second node 604, a third node 606, a fourth node 608, a fifth node 610, a sixth node 612, a seventh node 614, and an eighth node 616. The first node 602 may be the root node of the exemplary AST600 and may be in the first layer (or first level) of the node. The second node 604, the third node 606, and the fourth node 608 may be child nodes of the first node 602 and may be in the second layer (or second level) of the node. The fifth node 610 and the sixth node 612 may be child nodes of the third node 606 and may be in the third layer (or third level) of the node. Similarly, the seventh node 614 and the eighth node 616 may be child nodes of the fourth node 608 and may be in the third layer (or third level) of the node. The first layer of the node may be part of the top layer of the node in AST600, and the third layer of the node may be the last layer of the node in the exemplary AST600.

[0100] Figure 6B shows an exemplary revised AST of a subset of functions obtained from the exemplary AST of Figure 6A, according to at least one embodiment described in this disclosure. Figure 6B is described in conjunction with elements of Figures 1, 2, 3, 4, 5 and 6A. Referring to Figure 6A, an exemplary revised AST600B associated with the exemplary subset of functions is shown.

[0101] In one embodiment, system 102 may be configured to select a first node 602 from the nodes determined in 502. Subsequently, based on the determination that the selected node corresponds to a code component different from a function call having one or more parameters, the selected node may be marked for pruning operations.

[0102] The execution of a pruning operation may correspond to the execution of a pruning function for the corresponding node (i.e., the first node 602). For example, the pruning function may be "_is_call(node)". When executed, the pruning function may extract a segment of the function definition from the AST and process the extracted segment to check whether the function contains both one or more parameters and a function call. If the selected node contains both a function call and one or more parameters, the pruning function may return True. Otherwise, the pruning function may return False. In another embodiment, the pruning function may return False if one or more child nodes of the function associated with the selected node do not have a function call.

[0103] When executed on the first node 602, the pruning operation may return True because the first node 602 has at least one child. When executed on the second node 604, the pruning operation may return False because the second node 604 does not have at least one child. As an example and not an limitation, the return values ​​of the pruning function for each determined node in AST600 are shown in Table 1 below. [Table 1] Furthermore, when executed on the sixth node 612 and the eighth node 616, the return value of the pruning operation may be "True" because the corresponding nodes may contain parameters of the parent node and function calls to the parent nodes (i.e., the third node 606 and the fourth node 608), respectively.

[0104] System 102 may be further configured to remove all nodes whose return value is "False" and, based on such removal, obtain an exemplary revised AST600B. In one embodiment, System 102 may be further configured to extract a sequence of function calls corresponding to each function in a subset of functions by using the exemplary revised AST600B. The sequence of function calls may be extracted from computer-executable code. System 102 may be further configured to generate abstracted code data by abstracting the sequence of function calls. For example, the abstracted code data may include "Import.Ab(X)" and "Import.Ab(X)".

[0105] Figure 7 shows exemplary computer-executable code and updated versions of exemplary computer-executable code according to at least one embodiment described in this disclosure. Figure 7 is described in conjunction with elements of Figures 1, 2, 3, 4, 5, 6A, and 6B. Referring to Figure 7, Figure 700 is shown. System 702, computer-executable code 704, and updated computer-executable code 706 are further shown.

[0106] In one embodiment, system 702 may be configured to receive source code files associated with a software package. The source code files may be retrieved from a data source and may contain computer-executable code 704. As shown, for example, computer-executable code 704 may contain a function "data_filter". The function "data_filter" may have two parameters "jdata" and "selected_user_notebooks" and may return an array as output.

[0107] System 702 may be further configured to generate updated computer-executable code 706 based on filtering criteria. Specifically, system 702 may be configured to update computer-executable code 704 by filtering out lines of code (LoCs) corresponding to a subset of functions from the computer-executable code 704. Details regarding filtering lines of code (LoCs) are provided, for example, in Figure 4.

[0108] For example, lines of code such as "output=[]", "print(db_user, db_notebook)", "found_rec = None", "flag = false", "break", "if found_rec is not None:", and "break" can be filtered out from the computer-executable code 704, as shown. Based on the filtering of the Line of Code (LoC), the system 102 may be configured to obtain the updated computer-executable code 706. In one embodiment, the system 702 may be configured to add one or more lines of code to the computer-executable code 704 so that the purpose of the function does not change based on the filtering of the LoC. For example, the system 702 may add the updated computer-executable code 706 "for index, (db_user, db_notebook) in enumerate(selected_user_notebooks):". Further details related to updating the computer-executable code 704 are provided, for example, in Figures 5, 6A, and 6B.

[0109] Figure 8 illustrates an exemplary scenario for training a language model for code synthesis based on deep parameter learning, according to an exemplary embodiment. Figure 8 is described in conjunction with elements of Figures 1, 2, 3, 4, 5, 6A, 6B, and 7. Referring to Figure 8, an exemplary scenario 800 is shown. In Figure 8, a system 802 including a language model 804 is shown. The first training sample 806, input 808, and output 810 of several training samples included in the dataset are further shown.

[0110] System 102 may operate in two phases: a setup phase and a prediction phase. System 102 may operate in the prediction phase after one or more operations in the setup phase have been performed.

[0111] In the setup phase, system 802 may be configured to train a language model 804 on a sequence-to-sequence generation task. To train the language model 804, system 802 may be configured to generate a dataset of code features and their respective metadata features as training data, based on updated computer-executable code and metadata. In one embodiment, the generated dataset may be a dictionary in key-value format. The generated dataset may include, as keys, metadata features containing the natural language (NL) text portion of the description corresponding to a subset of functions, the class name corresponding to a subset of functions, the function name corresponding to a subset of functions, the path to the source code file, and the corresponding code feature.

[0112] In one embodiment, a single code feature and its associated metadata feature may be collectively referred to as a training sample. The generated dataset may contain multiple training samples. Each training sample in the multiple training samples within the dataset may contain a code feature and its associated metadata feature. For example, the first training sample 806 of the multiple training samples may contain a first code feature 806A and a first metadata feature 806B.

[0113] The language model 804 may be trained on a sequence-to-sequence generation task based on a generated dataset. The sequence-to-sequence generation task may be one of the following: a code synthesis task, a code retrieval task, or a software package analysis task. In one embodiment, the language model 804 may be a deep neural network that may use an encoder-decoder architecture. In one embodiment, the language model 804 may be trained to generate computer-executable lines of code in response to natural language queries as input to the language model 106.

[0114] In one embodiment, system 802 may be configured to extract code features and their respective metadata features from the generated dataset. System 802 may be further configured to use the generated embeddings to generate embeddings of the extracted code features and their respective metadata features for training a language model 804 on a sequence-to-sequence generation task. In one embodiment, the embeddings of the extracted code features and their respective metadata features may correspond to a concatenated vector representation of the extracted code features and their respective metadata features.

[0115] In the prediction phase, the language model 804 can be considered a trained model. The system 802 may be configured to receive an input 808. The input 808 may be received from a user 116 via a user device 108 and may contain a natural language query. For example, the natural language query may contain the text "Get video from URL". Based on the received input 808, the system 802 may be configured to apply the trained language model 804 to the received input 808. The system 802 may be further configured to control the language model 804 to generate an output 810 based on the application of the language model 804 to the received input 808. The generated output may contain, for example, a computer-executable line of code associated with the natural language query, as shown in Figure 8.

[0116] In one embodiment, system 802 may be configured to fine-tune a pre-trained language model. Fine-tuning a pre-trained language model may correspond to adjusting the pre-trained language model to achieve a desired output or performance. System 802 may fine-tune the pre-trained language model using the generated dataset. Specifically, system 802 may update parameters such as the weights of the pre-trained language model using the generated dataset.

[0117] Various embodiments of the present disclosure may provide one or more non-temporary computer-readable storage media configured to store instructions causing a system (such as system 102) to perform an action in response to being executed. This action may include receiving a source code file associated with a software package. This action may further include generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the source code file. This action may further include determining a set of functions from the computer-executable code based on the AST. This action may further include extracting metadata associated with the set of functions from the computer-executable code. This action may further include selecting a subset of functions from the set of functions whose descriptions in the extracted metadata satisfy filtering criteria. This action may further include updating the computer-executable code by filtering out lines of code (LoCs) corresponding to the subset of functions / procedures from the computer-executable code. This action may further include generating a dataset of code features and their respective metadata features based on the updated computer-executable code and metadata, and training a language model for a sequence-to-sequence generation task based on the dataset.

[0118] As described above, embodiments described in this disclosure may involve the use of a special-purpose or general-purpose computer (e.g., processor 202 in Figure 2) including various computer hardware or software modules, as will be discussed in more detail below. Furthermore, as described above, embodiments described in this disclosure may be implemented using a computer-readable medium (e.g., memory 204 or persistent data storage 206 in Figure 2) that carries or stores computer-executable instructions or data structures.

[0119] Where used in this disclosure, the terms “module” or “component” may refer to a software object or software routine that can be stored and / or executed in a particular hardware implementation and / or general-purpose hardware of a computing system (e.g., computer-readable media, processing devices, or some other hardware) configured to perform the actions of a module or component. In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as being implemented in software (stored and / or executed by general-purpose hardware), specific hardware implementations or combinations of software and specific hardware implementations are also possible and intended. In this description, “computation entity” may be any computing system as previously defined in this disclosure, or any combination of modules or modulators operating on a computing system.

[0120] In accordance with general practice, various features illustrated in the drawings may not be depicted to scale. The examples presented in this disclosure are not intended to be actual drawings of any particular apparatus (e.g., a device, system, etc.) or method, but are merely idealized representations used to describe various embodiments of this disclosure. Accordingly, the dimensions of various features may be enlarged or reduced as appropriate for clarity. In addition, some of the drawings may be simplified for clarity. Accordingly, the drawings may not depict all components of a given apparatus (e.g., a device) or all operations of a particular method.

[0121] The terms used in this disclosure, particularly in the appended claims (e.g., the text of the appended claims), are generally intended to be “open” terms (for example, the term “contains” should be interpreted as “contains, but is not limited to,” the term “has” should be interpreted as “has at least,” and the term “includes” should be interpreted as “contains, but is not limited to”).

[0122] Additionally, if a specific number of introduced claims are intended, such intent is explicitly stated in the claims; if there is no such provision, such intent does not exist. For example, to aid understanding, the following appended claims may include the use of the introductory phrases “at least one” and “one or more” to introduce the provisions of the claims.

[0123] Additionally, even if a specific number of provisions in an introduced claim is explicitly specified, a person skilled in the art will recognize that such a provision should be interpreted as meaning at least the specified number (for example, the mere provision "two provisions," unless otherwise specified, means at least two provisions, or two or more provisions). Furthermore, where conventions similar to "at least one of A, B, and C" or "one or more of A, B, and C, etc." are used, such structures are generally intended to include A alone, B alone, C alone, A and B, A and C, B and C, or A, B and C, etc.

[0124] Furthermore, any disjunct word or phrase presenting two or more alternative terms, whether in the specification, claims, or drawings, should be understood to include the possibility of including one of those terms, either of those terms, or both of those terms. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0125] However, the use of such phrases should not be interpreted as suggesting that the introduction of a claim provision by the indefinite article "a" or "an" limits any particular claim containing such introduced provision to only one embodiment containing such provision, and this is also true when the same claim contains an indefinite article such as the introductory phrase "one or more" or "at least one" and "a" or "an" (for example, "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"), and the same is true in the case of the use of an indefinite article used to introduce a claim provision.

[0126] Additionally, the use of terms such as “first,” “second,” and “third” is not necessarily used in this specification to imply a particular order or number of elements. Generally, terms such as “first,” “second,” and “third” are used as general identifiers to distinguish different elements. Terms such as “first,” “second,” and “third” should not be understood to imply a particular order without indicating that they do. Furthermore, terms such as “first,” “second,” and “third” should not be understood to imply a particular number of elements without indicating that they do. For example, a first widget may be described as having a first aspect, and a second widget may be described as having a second aspect. The use of the term “second aspect” with respect to a second widget may distinguish such an aspect of the second widget from the “first aspect” of the first widget and may not imply that the second widget has two aspects.

[0127] All examples and conditional language set forth in this disclosure are intended for educational purposes to assist the reader in understanding the concepts to which the disclosure and the inventors have contributed to advancing the art, and should be construed as not being limited to the examples and conditions thus specifically set forth. While embodiments of this disclosure are described in detail, it should be understood that various modifications, substitutions, and exchanges may be made thereto without departing from the spirit and scope of this disclosure.

[0128] This disclosure includes the following inventions. (Note 1) A method that is executed by a processor, Receiving source code files associated with a software package, The process involves generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the aforementioned source code file, Based on the aforementioned AST, determine a set of functions from the computer-executable code, Extracting metadata associated with the set of functions from the computer-executable code, From the set of functions, select a subset of functions whose descriptions in the extracted metadata satisfy the filtering criteria, Updating the computer-executable code by filtering out lines of code (LoC) corresponding to a subset of the function from the computer-executable code, Based on the updated computer-executable code and the metadata, a dataset of code features and their respective metadata features is generated. A method comprising training a language model for a sequence-to-sequence generation task based on the aforementioned dataset. (Note 2) The input will be the repository address of the package dataset, Using the aforementioned repository address, extract a list of software packages from the data source, Selecting the software package from the extracted list of software packages, The method according to Appendix 1, further comprising searching for resource files associated with the selected software package from the data source, wherein the resource files to be searched include the source code files. (Note 3) The use of the AST further includes identifying code components from the computer-executable code, The code component includes one or more of the following: a class, a method included in the class, and a function different from the method. The set of functions is the method described in Appendix 1, corresponding to the methods and functions included in the code component. (Note 4) The method according to Appendix 1, wherein the metadata includes at least one of the following: a function name, a count of parameters associated with the function, a count of one or more default parameters associated with the function, a default value for the one or more default parameters, the type of each of the parameters, a description of the function, or one or more decorators associated with the function. (Note 5) The method according to Appendix 4, wherein the description includes at least one of the following: a description of each of the parameters, a description of the return value associated with the function, a description of the one or more default parameters associated with the function, or a description of the purpose of the function. (Note 6) The filtering criteria specify a rule for selecting a function from the set of functions, The rules include a first rule for determining whether the description is present in the metadata, and the description in the metadata is A description of one or more parameters associated with the function, and The method according to Appendix 1, comprising a second rule for determining whether to include a description of one or more return values ​​associated with the function. (Note 7) Determining one or more parameters associated with each of the aforementioned set of functions, Determine one or more return values ​​associated with each of the aforementioned set of functions, The method according to Appendix 1, further comprising detecting the presence of one or more parameters and one or more return values ​​in the description of a corresponding function in the metadata, wherein a subset of the functions is selected based on the detection. (Note 8) Determining the nodes corresponding to the subset of the function in the AST, Selecting a node from the previously determined nodes, Based on the determination of whether the selected node corresponds to a code component different from a function call with one or more parameters, the selected node is marked for pruning operations, The method according to Appendix 1, further comprising obtaining a revised AST by performing the pruning operation to remove the marked nodes from the AST. (Note 9) The aforementioned computer-executable code is: From the computer-executable code, by using the revised AST, extract the sequence of function calls corresponding to each function in the subset of the functions, and The method described in Appendix 8, further updated by abstracting the sequence of function calls and generating abstracted code data. (Note 10) The method according to Appendix 9, further comprising determining the return value corresponding to each function in the subset of the functions from the updated computer-executable code. (Note 11) The generated dataset is a dictionary of key-value formats, and the dataset is, The code feature includes the return value and the abstracted code data, The method according to Appendix 10, comprising: the natural language (NL) text portion of the description corresponding to the subset of the function; the class name corresponding to the subset of the function; the function name corresponding to the subset of the function; and the metadata feature including the path to the source code file. (Note 12) The method according to Appendix 1, wherein the sequence-to-sequence generation task is one of a code synthesis task, a code retrieval task, or a software package analysis task. (Note 13) Receiving input associated with natural language queries, The method according to Appendix 1, further comprising generating computer-executable lines of code based on the application of the trained language model to the received input. (Note 14) One or more non-temporary computer-readable storage media configured to store instructions, wherein the instructions cause the system to perform an action in response to being executed, and the action is: Receiving source code files associated with a software package, The process involves generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the aforementioned source code file, Based on the aforementioned AST, determine a set of functions from the computer-executable code, Extracting metadata associated with the set of functions from the computer-executable code, From the set of functions, select a subset of functions whose descriptions in the extracted metadata satisfy the filtering criteria, Updating the computer-executable code by filtering out lines of code (LoC) corresponding to a subset of the function from the computer-executable code, Based on the updated computer-executable code and the metadata, a dataset of code features and their respective metadata features is generated. A non-temporary, computer-readable storage medium, comprising training a language model for a sequence-to-sequence generation task based on the aforementioned dataset. (Note 15) The aforementioned operation is, The input will be the repository address of the package dataset, Using the aforementioned repository address, extract a list of software packages from the data source, Selecting the software package from the extracted list of software packages, A non-temporary computer-readable storage medium as described in Appendix 14, further comprising searching for resource files associated with the selected software package from the data source, wherein the resource files to be searched include the source code files. (Note 16) The operation further includes identifying code components from the computer-executable code by using the AST, The code component includes one or more of the following: a class, a method included in the class, and a function different from the method. The set of functions is a non-temporary computer-readable storage medium as described in Appendix 14, corresponding to the methods and functions included in the code component. (Note 17) The metadata includes at least one of the following: a function name, a count of parameters associated with the function, a count of one or more default parameters associated with the function, a default value for the one or more default parameters, the type of each of the parameters, a description of the function, or one or more decorators associated with the function, as described in Appendix 14, on a non-temporary computer-readable storage medium. (Note 18) The non-temporary computer-readable storage medium described in Appendix 17, wherein the description includes at least one of the following: a description of each of the parameters, a description of the return value associated with the function, a description of the one or more default parameters associated with the function, or a description of the purpose of the function. (Note 19) The aforementioned operation is, Determining one or more parameters associated with each of the aforementioned set of functions, Determine one or more return values ​​associated with each of the aforementioned set of functions, A non-temporary computer-readable storage medium as described in Appendix 14, further comprising detecting the presence of one or more parameters and one or more return values ​​in the description of a corresponding function in the metadata, wherein a subset of the functions is selected based on the detection. (Note 20) It is a system, Includes a processor, the processor is Receiving source code files associated with a software package, The process involves generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the aforementioned source code file, Based on the aforementioned AST, determine a set of functions from the computer-executable code, Extracting metadata associated with the set of functions from the computer-executable code, From the set of functions, select a subset of functions whose extracted metadata descriptions satisfy the filtering criteria, Updating the computer-executable code by filtering out lines of code (LoC) corresponding to a subset of the function from the computer-executable code, Based on the updated computer-executable code and the metadata, a dataset of code features and their respective metadata features is generated. A system configured to train a language model for a sequence-to-sequence generation task based on the aforementioned dataset. [Explanation of Symbols]

[0129] 104 Data Sources 110 Communication Network 108 User Devices 102 System 114 datasets 106 Language Models 112 Source code files

Claims

1. A method that is executed by a processor, Receiving source code files associated with a software package, The process involves generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the source code file, Based on the AST, determine a set of functions from the computer-executable code, Extracting metadata associated with the set of functions from the computer-executable code, From the set of functions, select a subset of functions whose descriptions in the extracted metadata satisfy the filtering criteria, Updating the computer-executable code by filtering out the lines of code (LoC) corresponding to a subset of the functions from the computer-executable code, Based on the updated computer-executable code and the metadata, a dataset of code features and their respective metadata features is generated. This includes training a language model for a sequence-to-sequence generation task based on the aforementioned dataset, Determining the node corresponding to the subset of the function in the AST, Selecting a node from the previously determined nodes, Based on the determination of whether the selected node corresponds to a code component different from a function call with one or more parameters, the selected node is marked for pruning operations, A method further comprising obtaining a revised AST by performing the pruning operation to remove the marked nodes from the AST.

2. The input will be the repository address of the package dataset, Using the aforementioned repository address, extract a list of software packages from the data source, Selecting the software package from the extracted list of software packages, The method according to claim 1, further comprising searching the data source for resource files associated with the selected software package, wherein the resource files to be searched include the source code files.

3. The use of the AST further includes identifying code components from the computer-executable code, The code component includes one or more of the following: a class, a method included in the class, and a function different from the method. The method according to claim 1, wherein the set of functions corresponds to the methods and functions included in the code component.

4. The method according to claim 1, wherein the metadata includes at least one of the following: a function name, a count of parameters associated with the function, a count of one or more default parameters associated with the function, a default value for the one or more default parameters, the type of each of the parameters, a description of the function, or one or more decorators associated with the function.

5. The method according to claim 4, wherein the description includes at least one of the following: a description of each of the parameters, a description of the return value associated with the function, a description of the one or more default parameters associated with the function, or a description of the purpose of the function.

6. The filtering criteria specify a rule for selecting a function from the set of functions, The rules include a first rule for determining whether the description is present in the metadata, and the description in the metadata is A description of one or more parameters associated with the function, and The method according to claim 1, comprising a second rule for determining whether to include a description of one or more return values ​​associated with the function.

7. Determining one or more parameters associated with each of the aforementioned set of functions, Determine one or more return values ​​associated with each of the aforementioned set of functions, The method according to claim 1, further comprising detecting the presence of one or more parameters and one or more return values ​​in the description of a corresponding function in the metadata, wherein a subset of the functions is selected based on the detection.

8. The aforementioned computer-executable code is: From the computer-executable code, by using the revised AST, extract the sequence of function calls corresponding to each function in the subset of the functions, and The method according to claim 1, further updated by abstracting the sequence of function calls and generating abstracted code data.

9. The method according to claim 8, further comprising determining the return value corresponding to each function in the subset of the functions from the updated computer-executable code.

10. The generated dataset is a dictionary of key-value formats, and the dataset is, The code feature includes the return value and the abstracted code data, The method according to claim 9, comprising: a metadata feature including a natural language (NL) text portion of the description corresponding to a subset of the functions, a class name corresponding to a subset of the functions, a function name corresponding to a subset of the functions, and a path to the source code file.

11. The method according to claim 1, wherein the sequence-to-sequence generation task is one of a code synthesis task, a code search task, or a software package analysis task.

12. Receiving input associated with natural language queries, The method according to claim 1, further comprising generating computer-executable lines of code based on the application of the trained language model to the received input.

13. One or more non-temporary computer-readable storage media configured to store instructions, wherein the instructions cause the system to perform an action in response to being executed, and the action is: Receiving source code files associated with a software package, The process involves generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the source code file, Based on the AST, determine a set of functions from the computer-executable code, Extracting metadata associated with the set of functions from the computer-executable code, From the set of functions, select a subset of functions whose descriptions in the extracted metadata satisfy the filtering criteria, Updating the computer-executable code by filtering out the lines of code (LoC) corresponding to a subset of the functions from the computer-executable code, Based on the updated computer-executable code and the metadata, a dataset of code features and their respective metadata features is generated. The operation includes training a language model for a sequence-to-sequence generation task based on the aforementioned dataset, and the operation is as follows: Determining the node corresponding to the subset of the function in the AST, Selecting a node from the previously determined nodes, Based on the determination of whether the selected node corresponds to a code component different from a function call with one or more parameters, the selected node is marked for pruning operations, A non-temporary computer-readable storage medium, further comprising obtaining a revised AST by performing the pruning operation to remove the marked nodes from the AST.

14. It is a system, Includes a processor, the processor is Receiving source code files associated with a software package, The process involves generating an abstract syntax tree (AST) of at least a portion of the computer-executable code contained in the source code file, Based on the AST, determine a set of functions from the computer-executable code, Extracting metadata associated with the set of functions from the computer-executable code, From the set of functions, select a subset of functions whose extracted metadata descriptions satisfy the filtering criteria, Updating the computer-executable code by filtering out the lines of code (LoC) corresponding to a subset of the functions from the computer-executable code, Based on the updated computer-executable code and the metadata, a dataset of code features and their respective metadata features is generated. Based on the aforementioned dataset, the system is configured to train a language model for a sequence-to-sequence generation task, and the processor is configured to perform the following: Determining the node corresponding to the subset of the function in the AST, Selecting a node from the previously determined nodes, Based on the determination of whether the selected node corresponds to a code component different from a function call with one or more parameters, the selected node is marked for pruning operations, A system further configured to obtain a revised AST by performing the pruning operation to remove the marked nodes from the AST.

Images