Code enrichment using metadata for code synthesis

Abstract

To provide operations for code enrichment through metadata for code synthesis.SOLUTION: Operations include acquiring package data that include source code files and package metadata. The operations further include extracting additional metadata associated with software package and preparing metadata features based on the package metadata and the additional metadata. The operations further include identifying a set of target portions of a source code included in the source code files and updating one or more source code files using the metadata features. Such files are updated by performing at least one of a revision of existing code comments, and an addition of new code comments for the target portions. The operations further include generating a dataset of natural language (NL) text features and respective code features and training a language model on a sequence-to-sequence generation task.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] [Cross - Reference to Related Applications / Incorporation by Reference] This application claims priority to U.S. Provisional Patent Application No. 63 / 261,602, filed on September 24, 2021, under the title "Library Corpus for Large - Scale Language Models and Code Retrieval Models Using Augmented Code". The entire content of the above - mentioned U.S. Provisional Patent Application is incorporated herein by reference.

[0002] [Field] The embodiments discussed in this disclosure relate to code enrichment with metadata for code synthesis.

Background Art

[0003] With the progress of machine learning, various types of language models have been developed for different machine programming tasks, such as code synthesis or code retrieval. A language model is a statistical representation of the probability distribution over sequences of words, which aims to find relationships between different words by processing large corpora. Some language models aim to learn general - purpose representations that support downstream natural language - programming language (NL - PL) applications such as code synthesis. Code synthesis corresponds to the task of aiming for a machine (e.g., a computer) to generate source code for a given query as input. To perform code synthesis using a language model, the language model needs to be trained first. To train a language model, many state - of - the - art techniques ignore the vast amount of non - code information present in source code.

[0004] The scope claimed herein is not limited to embodiments that operate only in the environments described above or embodiments that eliminate any shortcomings. Rather, this background is provided solely to illustrate an example of the technical scope in which some of the embodiments described herein may be implemented. [Overview of the Initiative]

[0005] Depending on the embodiment, the operation may include retrieving package data associated with a software package from a data source. The package data may include source code files and package metadata associated with the software package. The operation may further include extracting additional metadata associated with the software package from the source code files and preparing metadata features based on the package metadata and additional metadata. The operation may further include identifying a set of target portions of source code contained in the source code files and updating one or more source code files in the source code files using the metadata features. One or more source code files may be updated by performing at least one of the following: modifying existing code comments that may be associated with the set of target portions, and adding new code comments to the set of target portions. The operation may include generating a dataset of natural language (NL) text features and each code feature using the updated one or more source code files. Subsequently, the operation may include training a language model on a sequence-to-sequence generation task based on the generated dataset.

[0006] The objectives and advantages of the embodiments are realized and achieved by at least the elements, features, and combinations specifically indicated in the claims.

[0007] Both the above summary and the following detailed description are illustrative and descriptive, and do not limit the claimed invention.

[0008] Exemplary embodiments are described and explained with further identification and detail through the use of the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram illustrates an exemplary environment for code enrichment using metadata for code synthesis. [Figure 2] This is a block diagram of a system for code enrichment using metadata for code synthesis. [Figure 3A] This diagram illustrates exemplary package data for code enrichment using metadata for code synthesis. [Figure 3B] This diagram illustrates exemplary additional metadata associated with a software package. [Figure 4] This flowchart illustrates an exemplary method for code enrichment using metadata for code synthesis. [Figure 5] This represents an exemplary hierarchical model for code enrichment using metadata for code synthesis. [Figure 6] This illustrates an exemplary scenario for updating one or more source code files for code enrichment using metadata for code synthesis. [Figure 7] This diagram illustrates an exemplary scenario for training a language model for code synthesis. [Modes for carrying out the invention]

[0010] All figures are represented in accordance with at least one embodiment described in this disclosure.

[0011] Machine learning has led to the development of language models for various machine programming tasks. A language model is a probabilistic model that provides a statistical representation of the probability distribution of a sequence of words, and it aims to find relationships between different words by processing a large corpus. For example, a language model can predict the probability that the word "Deliver" appears after "Leverages," as in "ABC Leverages World's Fastest Supercomputer 'XYZ' and AI to Deliver Real-Time Tsunami Prediction in Joint Project." Specifically, given a sequence of length m, a language model can predict the probability P(w1,w2,···,w m ) can be assigned to the entire sequence.

[0012] Language models are used in a variety of sequence-to-sequence generation tasks, such as code synthesis tasks, code retrieval tasks, or software package analysis tasks. Code synthesis tasks involve generating source code based on natural language queries. Code retrieval tasks involve searching a codebase for code snippets related to a given natural language query. Software package analysis tasks involve analyzing software packages for relevant information.

[0013] To perform the sequence-to-sequence generation task described above, the language model needs to be trained on an example dataset. For example, in the case of a code synthesis task, the language model needs to be trained on a dataset containing pairs of code snippets and natural language queries. Current state-of-the-art techniques for training language models only use code snippets and natural language queries. Training with such datasets results in the generation of language models that are unlikely to generalize and may lack accuracy. For example, if the natural language queries are "Library A save a csv" and "Library B save a csv", the trained language model may not be able to distinguish between the two queries and may generate the same code snippet for both queries. This output may be irrelevant or undesirable. Therefore, there is a need to generate a generalized language model trained on such sequence-to-sequence generation tasks.

[0014] This disclosure utilizes metadata associated with code snippets in a training dataset. Specifically, the disclosed invention trains a language model using code snippets and metadata features associated with those code snippets. Such metadata features may include software package information, installation requirements information, metadata version information, license information, supported programming languages, entry points, descriptions, platform information, and the like. The use of such metadata features can enhance and enrich the functionality of the language model for sequence-to-sequence generation tasks.

[0015] In contrast to the latest solutions, the disclosed language models may be more generalizable compared to language models trained using the latest approaches. Furthermore, the disclosed language models may help engineers generate code with greater detail compared to the latest methods. Based on experimental data, it has been confirmed that language models trained on metadata features, code, and natural language queries may perform better than language models trained using the latest approaches.

[0016] Embodiments of this disclosure will be described with reference to the accompanying drawings.

[0017] Figure 1 is a diagram illustrating an example of an environment related to metadata-based code enrichment for code synthesis, configured according to at least one embodiment described herein. Referring to Figure 1, an example environment 100 is shown. The example environment 100 includes a system 102 and a data source 104. Furthermore, a language model 106, a user device 108, a communication network 110, package data 112, and a dataset 114 are also shown. The system 102, the data source 104, and the user device 108 may be coupled to communicate with each other via the communication network 110.

[0018] Furthermore, a user 116 that may be associated with the user device 108 is shown. Examples of the user device 108 may include, but are not limited to, a mobile device, a desktop computer, a laptop, or a computer workstation. In one or more embodiments, the user device 108 may include a user end terminal device and a server that is communicably coupled to the user end terminal device. Examples of the user end terminal device may include, but are not limited to, a mobile device, a desktop computer, a laptop, or a computer workstation.

[0019] The data source 104 may include suitable logic, circuitry, and interfaces configured to store package data 112. The package data 112 may be associated with a software package and may include source code files 112A and package metadata 112B associated with the software package. In an embodiment, the source code file 112A may include source code (i.e., computer-executable code), and the package metadata 112B may include metadata regarding the software package in the form of natural language text. In an embodiment, the source code file 112A may also include additional metadata regarding the software package. Examples of the data source 104 may include, but are not limited to, web-based code host servers, database servers, file servers, web servers, Really Simple Syndication (RSS) feeds, servers hosting websites and web applications related to packages.

[0020] In an embodiment, the data source 104 may be implemented as a plurality of servers that may include storage distributed across one or more availability zones (e.g., data centers). In an embodiment, the data source may include a front-end system and a back-end system. The front-end system may be configured to provide an interface (e.g., a client-side interface of a web page or web application) for viewing information related to the package data 112. The back-end system may store databases, logic, and instructions for displaying content on the interface provided by the front-end system.

[0021] The language model 106 may be a probability model that can be trained to generate a probability distribution over a sequence on the alphabet of tokens. The language model 106 may be one of a statistical language model or a neural language model. The statistical language model may use statistical techniques to learn the probability distribution. Such statistical techniques include, for example, unigram techniques, N-gram, Hidden Markov Models (HMMs), and other language models. Details of the implementation of the above statistical techniques are known in the art. Therefore, a detailed description of the above statistical techniques is omitted for brevity.

[0022] The neural language model may use one or more neural networks to learn the probability distribution of words. In an embodiment, each of the one or more neural networks included in the neural language model may be a system or computational network of artificial neurons arranged as nodes in a plurality of layers. The plurality of layers of the neural network may include an input layer, one or more hidden layers, and an output layer. Each layer of the plurality of layers may include one or more nodes (i.e., artificial neurons). The outputs of all the nodes in the input layer may be coupled to at least one node in the hidden layer. Similarly, the inputs of each hidden layer may be coupled to the outputs of at least one node in other layers of the neural network. The output of each hidden layer may be coupled to the input of at least one node in other layers of the neural network. The nodes in the final layer may receive inputs from at least one hidden layer and output results. The number of layers and the number of nodes in each layer may be determined from the hyperparameters of the neural network. Such hyperparameters may be set before or after training the neural network with respect to the dataset 114.

[0023] Each node in a neural network may correspond to a mathematical function (e.g., a sigmoid function or a normalized linear unit) with a set of parameters that can be tuned during network training. These parameter sets may include, for example, weight parameters, regularization parameters, etc. Each node may use a mathematical function to compute an output based on one or more inputs from nodes in other layers of the neural network (e.g., previous layers). All or some nodes in a neural network may correspond to the same or different mathematical functions.

[0024] In training a neural network, one or more parameters of each node in the neural network may be updated based on whether the output of the final layer for a given input (from dataset 114) matches the exact result based on the neural network's loss function. The above process may be repeated for the same or different inputs until the minimum value of the loss function is achieved and the training error is minimized. Several training methods are known in this technique, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristics.

[0025] The neural language model may include electronic data that can be implemented, for example, as a software component of an application executable on system 102. The neural language model may depend on libraries, external scripts, or other logic / instructions for execution by a processing device such as a processor. The neural language model may include code and routines configured to enable a computing device such as a processor to perform one or more actions for generating lines of computer-executable code in response to natural language queries as input to the neural language model. Additionally or alternatively, the neural language model may be implemented using hardware including a processor, a microprocessor (e.g., for performing or controlling one or more actions), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural language model may be implemented using a combination of hardware and software.

[0026] Examples of one or more neural networks may include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), CNN-recurrent neural networks (CNN-RNNs), artificial neural networks (ANNs), RNNs based on long-shorter-term memory (LSTM) networks, LSTM+ANNs, RNNs based on gated recurrent units (GRUs), fully connected neural networks, RNNs based on Connectionist Temporal Classification (CTC), deep Bayesian neural networks, and / or combinations of such networks. In certain embodiments, each of the one or more neural networks may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0027] In an embodiment, the language model 106 may correspond to a DNN using an encoder-decoder architecture. The DNN may be trained to generate one or more lines of computer-executable code in response to natural language queries as input to the language model. Specifically, such a language model may include an encoder neural network and a decoder neural network. Examples of such DNNs may include, but are not limited to, long short-term memory (LSTM) networks, gated recurrent unit (GRU) networks, transformer models, or variations of transformer models, such as Bidirectional Encoder Representations from Transformers (BERT) models or CodeBERT models.

[0028] During operation, system 102 may be configured to retrieve package data 112 associated with a software package from data source 104. The package metadata may include source code files 112A and package metadata 112B. In an embodiment, package metadata 112B may include at least one of the following: the name of the software package, one or more classes used in the software package, a description of the software package, a summary of the software package, the programming language associated with the software package, the author of the software package, and a set of classifiers. Details regarding retrieving the package data 112 are given, for example, in Figure 4.

[0029] Upon receiving, system 102 may be configured to further extract additional metadata related to the software package from source code file 112A. In one embodiment, system 102 may be configured to parse source code file 112A and extract additional metadata related to the software package. The content of the additional metadata may differ from the package metadata 112B. Details regarding the additional metadata are given, for example, in Figure 3B.

[0030] System 102 may further be configured to prepare metadata features based on package metadata 112B and extracted additional metadata. In embodiments, the preparation may include parsing the package metadata 112B and additional metadata into metadata features. Each prepared metadata feature may be expressed in key-value format and may include at least one (but not limited to) of software package information, license information, supported programming languages, entry points, descriptions, or platform information. Further details regarding metadata features are given, for example, in Figures 4 and 5.

[0031] Based on the preparation of metadata features, system 102 may be configured to identify sets of target portions of source code contained in source code file 112A. These sets of target portions may correspond to functions or classes that may be used in the source code. The identification may be performed to limit the scope of updates to the target portions within the source code file, as described herein.

[0032] System 102 may further be configured to update one or more source code files of source code file 112A by using metadata features. Such files may be updated by performing at least one of the following: modifying existing code comments related to a set of target parts, and adding new code comments to a set of target parts. An example of updating one or more source code files is given, for example, in Figure 6.

[0033] System 102 may further be configured to generate a dataset 114 of natural language (NL) text features and their respective code features by using one or more updated source code files. In an embodiment, System 102 may be configured to control a user device 108 to display the generated dataset 114 on the user device 108. Based on the generated dataset 114, System 102 may further be configured to train a language model 106. The language model 106 may be trained for sequence-to-sequence generation tasks such as code synthesis tasks, code retrieval tasks, or software package analysis tasks, for example, but without limitation. Details regarding the training of the language model 106 are given, for example, in Figure 7.

[0034] It should be noted that communication between system 102, data source 104, language model 106, and user device 108 may be performed via a communication network 110. The communication network 110 may include a communication medium through which system 102 can communicate with data source 104, language model 106, user device 108, and / or other devices (not shown). Examples of the communication network 110 may include, but are not limited to, the Internet, cloud networks, cellular networks (e.g., 4th generation Long-Term Evolution (LTE) or 5th generation New Radio (NR)), Wireless Fidelity (Wi-Fi) networks, personal area networks (PANs), local area networks (LANs), and / or metropolitan area networks (MANs). Various devices in the example environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and / or Bluetooth® (BT) Communication Protocol, or a combination thereof.

[0035] Without departing from the scope of this disclosure, modifications, additions, or omissions may be made to System 102. For example, in some embodiments, System 102 may include any number of other components that are not expressly exemplified or described.

[0036] Figure 2 is a block diagram of a system for code enrichment by metadata for code synthesis, arranged according to at least one embodiment described herein. Figure 2 will be described in relation to elements from Figure 1. Referring to Figure 2, a block diagram 200 of system 102 of Figure 1 is shown. Block diagram 200 may further include a processor 202, memory 204, persistent data storage 206, I / O block 208, network interface 210, and language model 106.

[0037] The processor 202 may include appropriate logic, circuitry, and / or interfaces that can be configured to execute program instructions related to various operations performed by the system 102. The processor 202 may also include any appropriate dedicated or general-purpose computer, computing entity, or processing device, including various hardware or software modules, which may be configured to execute instructions stored in any applicable computer-readable storage medium. For example, the processor 202 may include a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any other digital or analog circuitry configured to interpret and / or execute program instructions and / or process data. Although represented as a single processor in Figure 2, the processor 202 may include any number of processors configured to individually or collectively perform or direct any number of operations of the system 102, as described in this disclosure.

[0038] In some embodiments, the processor 202 may be configured to interpret and / or execute program instructions stored in memory 204 and / or persistent data storage 206 and / or process the stored data. In some embodiments, the processor 202 may fetch program instructions from persistent data storage 206 and load the program instructions into memory 204. After the program instructions are loaded into memory 204, the processor 202 may execute the program instructions. Some examples of the processor 202 may be a central processing unit (CPU), a reduced instruction set computer (RISC) processor, an ASIC processor, a multiple instruction set computer (CISC) processor, a graphical processing unit (GPU), a coprocessor, and / or a combination thereof.

[0039] Memory 204 may include appropriate logic, circuitry, and / or interfaces that can be configured to store program instructions executable by the processor 202. In certain embodiments, memory 204 may be configured to store acquired package data 112, extracted additional metadata, prepared metadata features, identified target portion sets, updated source code files, and generated dataset 114. In certain embodiments, memory 204 may be configured to store a language model 106. Memory 204 may include a computer-readable storage medium that carries or stores computer-executable instructions or data structures. Such a computer-readable storage medium may include any available medium that can be accessed by a general-purpose or dedicated computer, such as the processor 202.

[0040] For example, and not as an limitation, such computer-readable storage media may include tangible or non-temporary computer-readable storage media, including random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk-read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash devices (e.g., solid-state memory devices), or any other storage media that may be used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or dedicated computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause a processor 202 to perform a specific operation or group of operations related to system 102.

[0041] The persistent data storage 206 may include appropriate logic, circuitry, and / or interfaces that can be configured to store program instructions executable by the processor 202. The persistent data storage 206 may include a computer-readable storage medium that carries or stores computer-executable instructions. Such a computer-readable storage medium may include any available medium that can be accessed by a general-purpose or dedicated computer, such as the processor 202.

[0042] For example, and not as an limitation, such computer-readable storage media may include tangible or non-temporary computer-readable storage media, including optical disk storage, magnetic disk storage or other magnetic storage devices (e.g., hard disk drives (HDDs)), flash devices (e.g., solid-state drives (SSDs), secure digital (SD) cards, other solid-state memory devices), or any other storage media that may be used to carry or store specific program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or dedicated computer. Combinations of the above may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause a processor 202 to perform a specific operation or group of operations related to system 102.

[0043] The I / O device 208 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to receive one or more user inputs. The I / O device 208 may further be configured to supply outputs in response to one or more user inputs. The I / O device 208 may include various input and output devices that can be configured to communicate with the processor 202 and other components, such as the network interface 210. Examples of input devices may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, and / or microphone. Examples of output devices may include, but are not limited to, a display device and a speaker.

[0044] The network interface 210 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to establish communication between the system 102, the data source 104, the language model 106, and the user device 108 via the communication network 110. The network interface 210 may be implemented by using various known techniques to support wired or wireless communication of the system 102 via the communication network 110. The network interface 210 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, and / or a local buffer.

[0045] The network interface 210 may communicate wirelessly with the Internet, intranet, and / or wireless networks, such as cellular telephone networks, wireless local area networks (LANs), and / or metropolitan area networks (MANs). Wireless communication may use any of several communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced DATA GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long-Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, etc.

[0046] The functions or operations performed by system 102, as described in Figure 1, may be performed by processor 202. The operations performed by processor 202 are described in detail, for example, in Figures 3A, 3B, 4, 5, 6, 7, and 8.

[0047] Figure 3A is a diagram representing exemplary package data for code enrichment by metadata for code synthesis, according to at least one embodiment described herein. Figure 3A is described in relation to elements from Figures 1 and 2. Referring to Figure 3A, an electronic user interface (UI) 300A is shown. The electronic UI 300A may be displayed on a user device 108. Package data 302 is further shown within the electronic UI 300A. The package data 302 may include source code files 304 and package metadata 306.

[0048] In an embodiment, system 102 may be configured to retrieve package data 302 from data source 104. Package data 302 may relate to a software package, such as an open-source package in Python. Package data 302 may include source code files 304 and package metadata 306. Each source code file may contain source code written in a programming language such as C, C++, C#, Swift, JavaScript, Go, Java®, or R, for example, but without limitation. Depending on the embodiment, source code files 304 may include resource files related to the software package. These resource files may contain information about resources such as definitions, configurations, setups, requirements, and distributions related to the software package, for example, but without limitation. In other embodiments, source code files 304 may include folders and / or subfolders containing additional source code files.

[0049] The package metadata 306 associated with the software package may include, for example, the name of the software package, one or more classes used in the software package, a description of the software package, a summary of the software package, the programming language associated with the software package, the author of the software package, a set of classifiers 308, and so on. The set of classifiers 308 may include, for example, the license associated with the software package, the operating system dependencies associated with the software package, the topics associated with the software package, and so on.

[0050] As shown in Figure 3A, for example, the source code file 304 for the 'ABC' package may include the “_init_.py” file, the “setup.py” file, the “versioneer.py” file, the “version.py” file, the “PKG-INFO” file, and the “config” folder. The “_init_.py” file, the “setup.py” file, and the “versioneer.py” or “version.py” file may contain the version of the software package's source code, and the “PKG-INFO” file may contain information about resources related to the software package. For example, the information in the PKG-INFO file may include package characteristics along with information that helps control the installation of the package. The “config” folder may contain one or more subfolders or additional source code files. Furthermore, package metadata 306 may indicate the name of the software package as "ABC", the description of the software package as "ABC is a Python package that privides fast, flexible, ...", the programming language associated with the software package as "Python", the license associated with the software package as "OSI Approved", the operating system dependency associated with the software package as "OS Independent", and the topic associated with the software package as "Science / Research".

[0051] Figure 3B is a diagram representing additional metadata associated with a software package according to at least one embodiment described in this disclosure. Figure 3B is described in relation to elements from Figures 1, 2, and 3A. Referring to Figure 3B, an electronic user interface (UI) 300B is shown. The electronic UI 300B may be displayed on a user device 108. Within the electronic UI 300B, a patch 310 of the first source code file (i.e., the PKG-INFO file) and additional metadata 312 contained in the first source code file are shown.

[0052] In one or more embodiments, system 102 may be configured to extract additional metadata 312 associated with the software package from source code files 304. For extraction, system 102 may be configured to parse each of the source code files 304 associated with the source code package. Such additional metadata may include, for example, a metadata version associated with the software package, author contact details associated with the software package, one or more uniform resource locators (URLs) associated with the software package, programming language requirements associated with the software package, description type associated with the software package, or background information associated with the software package. In one or more embodiments, the additional metadata 312 may include one or more components of package metadata 306.

[0053] For example, the additional metadata 312 included in the first source code file “PKG-INFO” may be in key-value format, and may include, for example, “Metadata-Version”, “Name”, “Version”, “Summary”, “Home-page”, “Author”, “Author-email”, “License”, “Project-URL”, “Project-URL”, “Project-URL”, “Platform”, “Classifier:Development Status”, “Classifier:Envoironment”, “Classifier:Intended Audience”, “Classifier:License”, “Classifier:Operating The values ​​may include keys such as "System" (Classifier: Operating System), "Classifier: Programming Language", "Classifier: Topic", "Requires-Python", "Description-Content-Type", "Provides-Extra", "Version", and "License-File".

[0054] Figure 4 shows a flowchart of an exemplary method of code enrichment with metadata for code synthesis, according to at least one embodiment described herein. Figure 4 is described in relation to elements from Figures 1, 2, and 3. Referring to Figure 4, flowchart 400 is shown. The method represented in flowchart 400 may begin at 402 and may be performed by any suitable system, apparatus, or device, for example, by system 102 in Figure 1 or 2.

[0055] A 402 error may be received, in which case the repository address may be received. In one embodiment, the system 102 may be configured to receive the repository address from the user 116 via the user device 108. The repository address may be a URL associated with a web page of a repository that may be hosted on the data source 104. Examples of repositories for the Python programming language may include, but are not limited to, Anaconda and PyPI. An example of a repository for the JavaScript programming language is npm.

[0056] In a 404 error, a list of software packages may be extracted. In one embodiment, system 102 may be configured to extract a list of software packages based on the received repository address. The list of software packages may be extracted from data source 104, and such packages may be associated with repositories linked to the repository address.

[0057] In 406, data relating to a list of software packages may be scraped or extracted. In an embodiment, system 102 may be configured to scrape data relating to an extracted list of software packages from a data source. For scraping, system 102 may use a web crawler or web scraper to scrape data relating to each of the extracted lists of software packages. Data scraping may correspond to a process in which elements of a web-based resource are discovered and parsed to select data that matches a defined set of rules for data collection.

[0058] In step 408, package data 302 may be obtained. In an embodiment, system 102 may be configured to obtain package data associated with a software package in a list of software packages. Package data 302 may be obtained from data source 104. Specifically, package data 302 may be obtained from data scraped from the data source using a repository address.

[0059] The package data 302 may include source code files 304 and package metadata 306 associated with the software package. In one embodiment, each of the source code files 304 may contain source code that can be executed to achieve the purpose of the software package. In another embodiment, the source code files 304 may include resource files associated with the software package. Such resource files may contain information about resources such as definitions, configurations, setups, requirements, and distributions associated with the software package, for example, but not limited to these. The package metadata 306 associated with the software package may include at least one (but not limited to) of the following: the name of the software package, one or more classes used in the software package, a description of the software package, a summary of the software package, a programming language associated with the software package, the author of the software package, or a set of classifiers. An example of a page in the package data 302 is given, for example, in Figure 3A.

[0060] Additional metadata may be extracted in step 410. In one embodiment, system 102 may be configured to extract additional metadata related to the software package from the source code file 304. Details regarding the additional metadata and the extraction of the additional metadata are given, for example, in Figure 3B.

[0061] Metadata features may be prepared in 412. In an embodiment, system 102 may be configured to prepare metadata features. Metadata features may be prepared based on package metadata 306 and additional metadata. In an embodiment, preparing metadata features may include parsing package metadata 306 and additional metadata into metadata features. Each prepared metadata feature may be expressed in key-value (i.e., key:value) format. In an embodiment, a prepared metadata feature may include at least one of the following: software package information, installation requirements information, metadata version information, license information, supported programming language information, entry point information, description information, or platform information. Software package information may relate to a software package and may, without limitation, include information about the name of the software package, a URL related to the software package, an alias for the software package, or a version of the software package. Installation requirements information may include information about one or more software or hardware resources that a computer may need to install and / or run the source code related to the software package. Metadata version information may include information related to the version of package metadata 306. License information may include information related to the type of license related to the software package. Support programming language information may include information related to one or more programming languages ​​that may be used in preparing the source code of the software package. Keyword information may include information related to one or more keywords used in preparing the source code of the software package. Entry point information may include information related to one or more entry points in the source code of the software package. Description information may, without limitation, include information about the software package summary, the software package project description, and a short description of the software package.Platform information may, without restriction, include information about the platform (or operating system) required to run the source code associated with the software package.

[0062] As a first example, if the name of the software package is "ABC", the corresponding metadata feature may be expressed as ("package_name", "ABC"). As another example, if the metadata version associated with the software package "ABC" is "1.0", the corresponding metadata feature may be expressed as ("metadata_version", "1.0").

[0063] In a 404 error, a hierarchical model may be generated. The hierarchical model may be generated by including source code files and metadata features. The source code files and metadata features may be included in order of priority. In some embodiments, the priority order may be predefined or pre-set based on rules and criteria. Details regarding the hierarchical model and its generation are given, for example, in Figure 5.

[0064] In 416, a set of target parts of the source code contained in the source code file 304 may be identified. Each part of the set of target parts of the source code may be identified from the source code file using a hierarchical model. In an embodiment, the set of target parts may correspond to a function or class that may be used in the source code. Details regarding the identification of the set of target parts are given, for example, in Figure 6.

[0065] In 418, one or more source code files in the source code file may be updated. In an embodiment, system 102 may be configured to update one or more source code files in the source code file by using metadata features. As will be discussed, metadata features may be expressed as key-value pairs. System 102 may be configured to update one or more source code files by performing at least one of the following: modifying existing code comments that may be related to the target part set, and adding new code comments to the target part set.

[0066] In an embodiment, system 102 may be configured to look up metadata feature keys in one or more source code files. The lookup may be performed within the content of a set of target parts. Specifically, the lookup may be performed within content that may be within the scope of the set of target parts.

[0067] Based on the search, system 102 may be configured to determine content fragments containing keywords that match at least a subset of the keys in the metadata features. The determined content fragments may correspond to existing code comments. System 102 may be configured to replace the keywords in the existing code comments with values ​​corresponding to a subset of the keys in the metadata features. This replacement of keywords in existing code comments with values ​​corresponding to a subset of the keys in the metadata features may correspond to modifications of existing code comments.

[0068] In accordance with the embodiment, system 102 may be configured to update one or more source code files by adding new code comments to a set of target parts. Each new code comment may include a key from a subset of keys and a value corresponding to that key in a metadata feature. Each new code comment may be included near each target part in the set of target parts of the source code. Referring to Figure 3A and Example 1, at least one of the source code files may be updated by adding the new comment “ABC is a Python package that provides fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open-source data analysis / manipulation tool available in any language. It is already well on its way towards this goal.”Details regarding the updating of one or more source code files are given, for example, in Figure 6.

[0069] In step 420, datasets of natural language (NL) text features and their respective coded features may be generated. In an embodiment, system 102 may be configured to generate datasets of NL text features and coded features. The datasets may be generated by using one or more updated source code files. Details regarding the generation of the datasets are given, for example, in Figure 7.

[0070] In 422, the language model 106 may be trained for a sequence-to-sequence generation task based on the generated dataset. The sequence-to-sequence generation task may be a code synthesis task, a code retrieval task, or a software package analysis task. The language model may be trained to generate computer executable code in response to natural language queries as input to the language model 106. In embodiments, the language model may be implemented using a deep neural network with an encoder-decoder architecture. If a pre-trained language model exists, the system 102 may fine-tune the pre-trained language model based on the generated dataset. In fine-tuning, the example dataset 114 may be used to update parameters such as the weights of the pre-trained language model. Details regarding the training of the language model 106 are given, for example, in Figure 7.

[0071] Control may be passed to termination. Flowchart 400 is represented as separate operations such as 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, and 422. However, in certain embodiments, such separate operations may be further divided into additional operations, combined into fewer operations, or deleted, depending on the particular implementation, without deviating from the essence of the disclosed embodiments.

[0072] Figure 5 represents an exemplary hierarchy model for metadata-based code enrichment for code synthesis, according to at least one embodiment described herein. Figure 5 is described in relation to elements from Figures 1, 2, 3, 3A, 3B, and 4. Referring to Figure 5, the hierarchy model 500 is shown. Furthermore, package data 502, source code files 504, and metadata features 506 are shown.

[0073] In an embodiment, system 102 may be configured to retrieve package data 502 associated with a software package from a data source. The package data 502 may include source code files 504 and package metadata associated with the software package. For example, and not as an limitation, source code files 504 may include a “PKG-INFO” file, a “Setup.py” file, an “_init_” file, and “*.py” files. In an embodiment, the asterisk in “*.py” may indicate that all files with the “.py” extension may be considered part of the source code files.

[0074] System 102 may be further configured to extract additional metadata related to the software package from the source code file 504. Based on the package metadata and the extracted additional metadata, System 102 may be configured to prepare metadata features (Ai) 506. System 102 may be further configured to generate a hierarchy model 500. The hierarchy model 500 may include the source code file 504 and the metadata features 506. The source code file 504 and the metadata features 506 may be arranged in order of priority. In embodiments, the order of priority may be predetermined or pre-set based on rules and criteria. For example, the “PKG-INFO” file may have the highest priority and therefore may be placed at the top of the hierarchy model 500. “Setup.py” may have the next highest priority after “PKG-INFO” and therefore may be placed below the “PKG-INFO” file in the section related to the source code file 504 within the hierarchy model 500. From metadata feature 506, the key "packages" and its corresponding value from the "Setup.py" file may have the highest priority among the metadata features, and therefore such a metadata feature may be placed at the top of the hierarchy model 500. As another example, the metadata feature with the key "install-requires" and its corresponding value from the "Setup.py" file may have the second highest priority within metadata feature 506. Therefore, such a feature may be placed immediately below the metadata feature with the key "packages".

[0075] In an embodiment, the metadata feature 506 may be represented by Ai, and may include, for example, a first metadata feature (A1), a second metadata feature (A2), a third metadata feature (A3), a fourth metadata feature (A4), a fifth metadata feature (A5), a sixth metadata feature (A6), a seventh metadata feature (A7), an eighth metadata feature (A8), and a ninth metadata feature (A9). Mathematically, the metadata feature may be represented by the following equation (1): Ai=[A1,A2,A3,A4,A5,A6,A7,A8,A9] (1)

[0076] In an embodiment, metadata feature 506 may include at least one of the following: software package information, installation requirements information, metadata version information, license information, supported programming language information, entry point information, description information, or platform information. Specifically, the first metadata feature (A1) may include software package information, the second metadata feature (A2) may include installation requirements information, the third metadata feature (A3) may include metadata version information, the fourth metadata feature (A4) may include license information, the fifth metadata feature (A5) may include supported programming language information, the sixth metadata feature (A6) may include keyword information, the seventh metadata feature (A7) may include entry point information, the eighth metadata feature (A8) may include description information, and the ninth metadata feature (A9) may include platform information.

[0077] In this embodiment, software package information related to the first metadata feature (A1) may be extracted from the "Setup.py" file, installation requirements information related to the second metadata feature (A2) may be extracted from the "Setup.py" file, metadata version information related to the third metadata feature (A3) may be extracted from the "PKG-INFO" file, license information related to the fourth metadata feature (A4) may be extracted from the "PKG-INFO" file, supported programming language information related to the fifth metadata feature (A5) may be extracted from the "PKG-INFO" file, keyword information related to the sixth metadata feature (A6) may be extracted from the "Setup.py" file, entry point information related to the seventh metadata feature (A7) may be extracted from the "Setup.py" file, description information related to the eighth metadata feature (A8) may be extracted from the "PKG-INFO" file, and platform information related to the ninth metadata feature (A9) may be extracted from the "PKG-INFO" file.

[0078] System 102 may be further configured to identify sets of target parts of the source code contained in the source code file 504 by using the generated hierarchy model 500. The sets of target parts may correspond to functions or classes that may be used in the source code. The identified sets of target parts may need to be updated using metadata features 506. Details regarding the updating of the sets of target parts are given, for example, in Figure 6.

[0079] In an embodiment, system 102 may be configured to generate an index list based on a set of identified target parts. Specifically, system 102 may be configured to generate an index list of one or more classes in a set of identified target parts based on the existence of one or more functions (or methods) associated with one or more classes in the set of identified target parts.

[0080] By using an index list, system 102 may be configured to look up keys of metadata feature 506 in one or more source code files. Specifically, the lookup may be performed within the content of a set of target parts. In an embodiment, the content of a set of target parts may include comments or documentation strings (written in one or more source code files) associated with the corresponding set of target parts. An example of target part content including comments and documentation strings is given, for example, in Figure 6. System 102 may be configured to determine content fragments containing keywords that match at least a subset of the keys of metadata feature 506. In an embodiment, the determined content fragments may correspond to existing code comments. For example, content fragments corresponding to existing code comments may be given as follows: “Parameters ---------- arrays:Iterator[np.ndarray] num_items:int Return -------- np.ndarray[unit64] Should be the same as CPython’s tupleobject.c”

[0081] System 102 may be configured to update one or more source code files of source code file 504 by using metadata feature 506. One or more source code files may be updated by modifying existing code comments associated with a set of target parts. Modification of existing code comments may be performed by replacing keywords in the existing code comments with values ​​corresponding to a subset of keys in metadata feature 506.

[0082] In other embodiments, the system 102 may be configured to update one or more source code files of source code file 504 by using metadata feature 506 and by adding new code comments to a set of target parts. Each new code comment may contain a key in a subset of keys in metadata feature 506 and a value corresponding to that key. In embodiments, each new code comment may be included near each target part in the set of target parts of source code. For example, the new comment may be included immediately before or after the source code of each target part. Details regarding the addition of new comments are given, for example, in Figure 6.

[0083] Figure 6 illustrates an exemplary scenario in which one or more source code files are updated for metadata-based code enrichment for code synthesis, according to at least one embodiment described herein. Figure 6 is described in relation to elements from Figures 1, 2, 3A, 3B, 4, and 5. Referring to Figure 6, scenario 600 is shown. Furthermore, system 602, a first source code file 604, and an updated first source code file 606 are shown. System 602 may be an exemplary implementation of system 102 in Figure 1 or 2.

[0084] In an embodiment, system 602 may be configured to obtain package data 302 associated with a software package from a data source 104. The package data 302 may include a source code file 304 and package metadata 306 associated with the software package. The source code file 304 may include a first source code file 604. In an embodiment, the first source code file 604 may include a first source code 604A. System 602 may further be configured to extract additional metadata 312 that may be associated with the software package from the source code file 304. System 602 may further be configured to prepare metadata features based on the package metadata 306 and the additional metadata. System 602 may further be configured to identify a set of target portions of the source code contained in the source code file 304. The set of target portions may correspond to a function or class that may be used in the first source code 604A.

[0085] Based on the identification of the target part set, system 602 may be configured to further update the first source code file 604 using metadata features to generate an updated first source code file 606. The first source code file 604 may be updated by adding a new code comment 608 to the target part set. The new code comment 608 may contain a key from a subset of keys in the metadata feature and the value corresponding to that key, as described in 500. For example, if the metadata feature is ("package_name", "ABC"), the new code comment 608 may be "ABC is a Python package that provides fast, flexible, and expressive data structures...".

[0086] As another example, if the package metadata associated with a software package includes “Read a table of fixed-width formatted lines into DataFrame”, the source code file associated with the corresponding software package may include the following: “z=ZipFile(io.BytesIO(content),'r') sg=z.read('19SG_DESC.txt').decode('latin-1') dx=z.read('19DX_DESC.txt').decode('latin-1') sg=pd.read_fwf( io.Stringto(sg), widths=[5,200], names=['icd_prcdr_cd','desc'], dtype={'icd_prcdr_cd':'str'}) dx=pd.read_fwf( io.StringIo(dx), widths=[5,200], names=['icd_dgns_cd','desc'], dtype={'icd_dgns_cd':'str'})”

[0087] System 602 may be configured to update the above source code based on metadata features that can be prepared from package metadata. Specifically, System 602 may prepare metadata features that may include the name of a function, i.e., “pd.read_fwf”, and may identify sets of target parts that may contain the function “pd.read_fwf”. System 602 may add new comments near each target part (for example, immediately before its start) to update the code. The updated code may be given as follows: “z=ZipFile(io.BytesIO(content),'r') sg=z.read('19SG_DESC.txt').decode('latin-1') dx=z.read('19DX_DESC.txt').decode('latin-1') #Read a table of fixed-width formatted lines into DataFrame. sg=pd.read_fwf( io.Stringto(sg), widths=[5,200], names=['icd_prcdr_cd','desc'], dtype={'icd_prcdr_cd':'str'}) #Read a table of fixed-width formatted lines into DataFrame. dx=pd.read_fwf( io.StringIo(dx), widths=[5,200], names=['icd_dgns_cd','desc'], dtype={'icd_dgns_cd':'str'})”

[0088] Figure 7 is a diagram illustrating an exemplary scenario for training a language model for code synthesis, according to an exemplary embodiment. Figure 7 is described in relation to elements from Figures 1, 2, 3A, 3B, 4, 5, and 6. Referring to Figure 7, an exemplary scenario 700 is shown. Figure 7 shows a system 702 which may include a language model 704. System 702 may be an exemplary implementation of system 102 in Figure 1 or 2. A first training sample 706 from a set of training samples included in the dataset, an input 708, and an output 710 are further shown.

[0089] In one embodiment, the system 702 may operate in two phases, namely a setup phase and a prediction phase. The system 702 may operate in the prediction phase after one or more operations of the setup phase have been performed.

[0090] In the setup phase, system 702 may be configured to train a language model 704 with respect to a sequence-to-sequence generation task. To train the language model 704, system 702 may be configured to generate a dataset of NL text features and each code feature as training data by using one or more updated source code files. The dataset may contain multiple training samples. Each training sample in the multiple training samples in the dataset may contain an NL text feature and its respective code feature. For example, the first training sample 706 in the multiple training samples may contain a first NL text feature 706A and a first code feature 706B.

[0091] The language model 704 may be trained on a sequence-to-sequence generation task based on a generated dataset. The sequence-to-sequence generation task may be one of a code synthesis task, a code retrieval task, or a software package analysis task. In an embodiment, the language model 704 may be a deep neural network that can use an encoder-decoder architecture. In an embodiment, the language model 704 may be trained to generate lines of computer executable code in response to natural language queries as input to the language model 704.

[0092] In an embodiment, system 702 may be configured to extract NL text features and their respective code features from a dataset. System 702 may further be configured to generate embeddings of the extracted NL text features and their respective code features, and to train a language model 704 with respect to a sequence-to-sequence generation task using the generated embeddings. In an embodiment, system 702 may generate multiple tokens from the text features and their respective code features based on the generated multiple tokens. The embeddings of the extracted NL text features and their respective code features may correspond to concatenated vector representations of the extracted NL text features and their respective code features.

[0093] In the prediction phase, system 702 may be configured to receive input 708. Input 708 may be received from user 116 via user device 108 and may contain a natural language query. For example, but not limited to, the natural language query may contain text, e.g., “hashing a content for ABC data frame”. Upon receipt, system 702 may be configured to apply a trained language model 704 to the received input 708 and generate an output 710 based on the application of the language model 704 to the received input 708. The generated output may contain lines of computer executable code related to the natural language query, for example, as shown in Figure 7.

[0094] In some embodiments, system 702 may be configured to fine-tune a pre-trained language model. Fine-tuning a pre-trained language model may correspond to adjusting the pre-trained language model to achieve a desired output or performance. System 702 may fine-tune the pre-trained language model using a generated dataset. Specifically, system 702 may update parameters such as the weights of the pre-trained language model using a generated dataset.

[0095] In one embodiment, the received input 708 may correspond to a license associated with package data or a category of computer executable code (e.g., web development, application development, mobile application development). For example, if the received input 708 corresponds to a category of computer executable code, the generated output 710 may include all code associated with the corresponding category.

[0096] It should be noted that enriching one or more source code files with metadata features and then training a language model from them can improve the performance of the language model in downstream tasks such as code synthesis or code retrieval. If a source code file lacks user comments, the disclosure provides a way to automatically add new comments from metadata associated with the corresponding package.

[0097] Various embodiments of this disclosure may provide one or more non-temporary computer-readable storage media configured to store instructions causing a system (e.g., system 102) to perform an action in response to being executed. The action may include retrieving package data associated with a software package from a data source. The package data may include source code files and package metadata associated with the software package. The action may further include extracting additional metadata associated with the software package from the source code files. The action may further include preparing metadata features based on the package metadata and additional metadata. The action may further include identifying a set of target portions of source code contained in the source code files. The action may further include updating one or more source code files of source code by using the metadata features. One or more source code files may be updated by performing at least one of the following: modifying existing code comments associated with the set of target portions, and adding new code comments to the set of target portions. The action may further include generating datasets of natural language (NL) text features and their respective code features by using the updated one or more source code files. The operation may further include training a language model on a sequence-to-sequence generation task based on the generated dataset.

[0098] As described above, embodiments described herein may involve the use of a dedicated or general-purpose computer (e.g., processor 202 in Figure 2) including various computer hardware or software modules, as will be discussed in detail below. Furthermore, as described above, embodiments described herein may be implemented using a computer-readable medium (e.g., memory 204 or persistent data storage 206 in Figure 2) that carries or stores computer executable instructions or data structures.

[0099] As used in this disclosure, the terms “module” or “component” may refer to a specific hardware implementation configured to perform the operation of a module or component, and / or a software object or software routine stored in and / or executed by general-purpose hardware of a computing system (e.g., computer-readable media, processing devices, or other hardware). In some embodiments, different components, modules, engines, and services described in this disclosure may be implemented as objects or processes that run on a computing system (e.g., as separate threads). While some of the systems and methods described in this disclosure are generally described as being implemented in software (stored in and / or executed by general-purpose hardware), specific hardware implementations, or combinations of software and specific hardware implementations, are also possible and conceivable. As used herein, “computation entity” may be any computing system as previously defined in this disclosure, or any module or combination of modules that run on a computing system.

[0100] In accordance with common practices, various features depicted in the drawings may not be to actual size. The examples presented in this disclosure are not intended to be actual drawings of any particular apparatus (e.g., devices, systems, etc.) but are merely idealized representations used to illustrate various embodiments of the disclosure. Accordingly, the dimensions of various features may be enlarged or reduced as appropriate for clarity. Furthermore, some drawings may be simplified for clarity. Thus, the drawings may not represent all components of a given apparatus (e.g., a device) or all operations of a particular method.

[0101] In this disclosure, the terms used in particular in the attached claims (e.g., the text of the attached claims) are generally intended to be “open” terms (for example, the word “including” should be interpreted as “including but not limited to,” the word “having” should be interpreted as “having at least,” and the word “includes” should be interpreted as “including but not limited to,” etc.).

[0102] Furthermore, if a specific number is intended in the introduced claim recitation, that intention must be clearly stated in the claim; if there is no such statement, then no such intention exists. For example, to facilitate understanding, subsequent appended claims may use introductory phrases such as "at least one" and "one or more" to introduce the claim recitation.

[0103] Furthermore, even if a specific number is explicitly stated in the introduced claim description, it will be understood by those skilled in the art that such description should generally be interpreted to mean at least the number stated (for example, if there is a description of only "two descriptions" without any other modifiers, this description means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, such a structure is generally intended to include A only, B only, C only, both A and B, both A and C, both B and C, and / or all of A, B, and C, etc.

[0104] Furthermore, any disjunctions and / or disjunctions representing two or more selectable terms, whether in the specification, claims, or drawings, should be understood as intended to include the possibility of including one of those terms, either of those terms, or both of those terms. For example, the phrase "A or B" should be understood to include the possibility of "A or B" or "A and B".

[0105] However, the use of such phrases does not mean that, when introducing a claim description with an indefinite article such as "a" or "an," even if the same claim contains both an introductory phrase such as "one or more" or "at least one" and an indefinite article such as "a" or "an," it should be interpreted that the particular claim containing the introduced claim description is limited to examples that contain only one of the described items (for example, "a" and / or "an" should be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim description using a definite article.

[0106] Furthermore, the use of terms such as "first," "second," and "third" is not necessarily used herein to imply a specific order or number of elements. Generally, terms such as "first," "second," and "third" are used to distinguish different elements as general identifiers. Unless it is indicated that terms such as "first," "second," and "third" imply a specific order, these terms should not be understood to mean a specific order. Furthermore, unless it is indicated that terms such as "first," "second," and "third" imply a specific number of elements, these terms should not be understood to mean a specific number of elements. For example, a first widget may be described as having a first side, and a second widget may be described as having a second side. The use of the term "second side" in relation to the second widget is to distinguish such a side of the second widget from the "first side" of the first widget, and does not mean that the second widget has two sides.

[0107] All examples and conditional language cited herein are intended for educational purposes to help readers understand the concepts and inventions to which the inventors have contributed to the advancement of the art, and should be construed as not being limited to such specifically cited examples and conditions. While embodiments of this disclosure have been described in detail, various modifications, substitutions, and alternatives may be made without departing from the spirit and scope of this disclosure.

[0108] In addition to the embodiments described above, the following additional information is disclosed. (Note 1) The way in which a processor executes This involves obtaining package data related to a software package from a data source, wherein the package data includes source code files and package metadata related to the software package. Extracting additional metadata related to the software package from the source code file, Prepare metadata features based on the package metadata and the additional metadata, Identifying the target portion of the source code contained in the aforementioned source code file, The process involves updating one or more source code files among the source code files by using the metadata features, and the one or more source code files are: Correction of existing code comments related to the aforementioned target section set, and Adding new code comments to the aforementioned target section set It is updated by performing at least one of the following: The process involves generating a dataset of natural language (NL) text features and their respective code features by using one or more of the updated source code files, Based on the generated dataset, a language model is trained for a sequence-to-sequence generation task. A method of having. (Note 2) Extracting a list of software packages from the data source by using the repository address, This involves scraping data relating to the list of software packages from the data source by using a web crawler, and the package data related to the software packages is obtained from the scraped data. The method described in Appendix 1, further comprising the above. (Note 3) The package metadata associated with the software package is: The name of the aforementioned software package, One or more classes used in the aforementioned software package, Description of the aforementioned software package, Summary of the aforementioned software package, Programming language related to the aforementioned software package, The author of the aforementioned software package, or Classifiers Having at least one of the following, The method described in Appendix 1. (Note 4) The preparation includes parsing the package metadata and the additional metadata into metadata features, each of which is represented in key-value format. The method described in Appendix 1. (Note 5) The aforementioned metadata features are Software package information, Installation requirements information, Metadata version information, License information, Supported programming language information, Keyword information, Entry point information, Descriptive information, or Platform Information Having at least one of the following, The method described in Appendix 1. (Note 6) The method further comprises generating a hierarchical model by including the source code files and metadata features in order of priority. The method described in Appendix 1. (Note 7) Each of the set of target portions of the source code is identified from the source code file using the hierarchical model. The method described in Appendix 6. (Note 8) The set of target parts corresponds to a function or class used in the source code. The method described in Appendix 1. (Note 9) The search involves looking up the key of the metadata feature in one or more source code files, and the search is performed within the content of the target portion set. Based on the search, determine a fragment of the content containing keywords that match at least a subset of the keys of the metadata features, wherein the determined fragment of the content corresponds to the existing code comments. The method described in Appendix 1, further comprising the above. (Note 10) The modification of the existing code comment is performed by replacing the keyword in the existing code comment with a value corresponding to the subset of the key of the metadata feature. The method described in Appendix 9. (Note 11) Each of the aforementioned new code comments includes a key from the subset of the aforementioned key and a value corresponding to that key in the metadata feature, Each of the aforementioned new code comments is included near each target portion in the set of target portions of the source code, The method described in Appendix 9. (Note 12) Extracting NL text features and their respective code features from the aforementioned dataset, The process involves generating embeddings of the extracted NL text features and each of the code features, and the language model is trained with respect to the sequence-to-sequence generation task using the generated embeddings. The method described in Appendix 1, further comprising the above. (Note 13) The sequence-to-sequence generation task is a code synthesis task, a code search task, or a software package analysis task. The method described in Appendix 1. (Note 14) Receiving input related to unobserved software packages, To generate lines of computer executable code based on the application of the trained language model to the received input. The method described in Appendix 1, further comprising the above. (Note 15) A non-temporary computer-readable storage medium configured to store instructions, In response to the execution of the aforementioned instruction, the system will: This involves obtaining package data related to a software package from a data source, wherein the package data includes source code files and package metadata related to the software package. Extracting additional metadata related to the software package from the source code file, Prepare metadata features based on the package metadata and the additional metadata, Identifying the target portion of the source code contained in the aforementioned source code file, The process involves updating one or more source code files among the source code files by using the metadata features, and the one or more source code files are: Correction of existing code comments related to the aforementioned target section set, and Adding new code comments to the aforementioned target section set It is updated by performing at least one of the following: The process involves generating a dataset of natural language (NL) text features and their respective code features by using one or more of the updated source code files, Based on the generated dataset, a language model is trained for a sequence-to-sequence generation task. A non-temporary computer-readable storage medium that enables the execution of an operation having the following characteristics. (Note 16) The aforementioned operation is, Extracting a list of software packages from the data source by using the repository address, This involves scraping data relating to the list of software packages from the data source by using a web crawler, and the package data related to the software packages is obtained from the scraped data. It further has, A non-temporary computer-readable storage medium as described in Appendix 15. (Note 17) The package metadata associated with the software package is: The name of the aforementioned software package, One or more classes used in the aforementioned software package, Description of the aforementioned software package, Summary of the aforementioned software package, Programming language related to the aforementioned software package, The author of the aforementioned software package, or Classifiers Having at least one of the following, A non-temporary computer-readable storage medium as described in Appendix 15. (Note 18) The operation further comprises generating a hierarchical model by including the source code files and metadata features in order of priority. A non-temporary computer-readable storage medium as described in Appendix 15. (Note 19) Each of the set of target portions of the source code is identified from the source code file using the hierarchical model. A non-temporary computer-readable storage medium as described in Appendix 18. (Note 20) This involves obtaining package data related to a software package from a data source, wherein the package data includes source code files and package metadata related to the software package. Extracting additional metadata related to the software package from the source code file, Prepare metadata features based on the package metadata and the additional metadata, Identifying the target portion of the source code contained in the aforementioned source code file, The process involves updating one or more source code files among the source code files by using the metadata features, and the one or more source code files are: Correction of existing code comments related to the aforementioned target section set, and Adding new code comments to the aforementioned target section set It is updated by performing at least one of the following: The process involves generating a dataset of natural language (NL) text features and their respective code features by using one or more of the updated source code files, Based on the generated dataset, a language model is trained for a sequence-to-sequence generation task. A system having a processor configured to perform the following. [Explanation of symbols]

[0109] 100 Environment 102,602,702 System 104 Data Sources 106,704 language models 108 User Devices 110 Communication Network 112,302,502 Package Data 112A,304,504 Source code files 112B,306 Package Metadata 114 datasets 116 users 202 processors 204 memory 206 Persistent Data Storage 208 I / O blocks 210 Network Interfaces 300A, 300B Electronic User Interface (UI) 312 Additional metadata 500 Hierarchical Model 506 Metadata Features

Claims

1. The way in which a processor executes This involves obtaining package data related to a software package from a data source, wherein the package data includes source code files and package metadata related to the software package. Extracting additional metadata related to the software package from the source code file, Prepare metadata features based on the package metadata and the additional metadata, Identifying the target portion of the source code contained in the aforementioned source code file, The process involves updating one or more source code files among the source code files by using the metadata features, and the one or more source code files are: Correction of existing code comments related to the aforementioned target section set, and Adding new code comments to the aforementioned target section set It is updated by performing at least one of the following actions, The process involves generating a dataset of natural language (NL) text features and their respective code features using one or more of the updated source code files, The language model is trained for a sequence-to-sequence generation task based on the generated dataset. A method of having.

2. Extracting a list of software packages from the data source by using the repository address, This involves scraping data relating to the list of software packages from the data source by using a web crawler, and the package data related to the software packages is obtained from the scraped data. The method according to claim 1, further comprising:

3. The package metadata associated with the software package is: The name of the aforementioned software package, One or more classes used in the aforementioned software package, Description of the aforementioned software package, Summary of the aforementioned software package, Programming language related to the aforementioned software package, The author of the aforementioned software package, or Classifiers Having at least one of the following, The method according to claim 1.

4. The preparation includes parsing the package metadata and the additional metadata into metadata features, each of which is represented in key-value format. The method according to claim 1.

5. The aforementioned metadata features are Software package information, Installation requirements information, Metadata version information, License information, Supported programming language information, Keyword information, Entry point information, Descriptive information, or Platform Information Having at least one of the following, The method according to claim 1.

6. The method further comprises generating a hierarchical model by including the source code files and metadata features in order of priority. The method according to claim 1.

7. Each of the set of target portions of the source code is identified from the source code file using the hierarchical model. The method according to claim 6.

8. The set of target parts corresponds to a function or class used in the source code. The method according to claim 1.

9. The search involves looking up the key of the metadata feature in one or more source code files, and the search is performed within the content of the target portion set. Based on the search, determine a fragment of the content containing keywords that match at least a subset of the keys of the metadata features, wherein the determined fragment of the content corresponds to the existing code comments. The method according to claim 1, further comprising:

10. The modification of the existing code comment is performed by replacing the keyword in the existing code comment with a value corresponding to the subset of the key of the metadata feature. The method according to claim 9.

11. Each of the aforementioned new code comments includes a key from the subset of the aforementioned key and a value corresponding to that key in the metadata feature, Each of the aforementioned new code comments is included near each target portion in the set of target portions of the source code, The method according to claim 9.

12. Extracting NL text features and their respective code features from the aforementioned dataset, The process involves generating embeddings of the extracted NL text features and each of the code features, and the language model is trained with respect to the sequence-to-sequence generation task using the generated embeddings. The method according to claim 1, further comprising:

13. The sequence-to-sequence generation task is a code synthesis task, a code search task, or a software package analysis task. The method according to claim 1.

14. Receiving input including a natural language query, To generate lines of computer executable code based on the application of the trained language model to the received input. The method according to claim 1, further comprising:

15. A non-temporary computer-readable storage medium configured to store instructions, In response to the execution of the aforementioned instruction, the system will: This involves obtaining package data related to a software package from a data source, wherein the package data includes source code files and package metadata related to the software package. Extracting additional metadata related to the software package from the source code file, Prepare metadata features based on the package metadata and the additional metadata, Identifying the target portion of the source code contained in the aforementioned source code file, The process involves updating one or more source code files among the source code files by using the metadata features, and the one or more source code files are: Correction of existing code comments related to the aforementioned target section set, and Adding new code comments to the aforementioned target section set It is updated by performing at least one of the following actions, The process involves generating a dataset of natural language (NL) text features and their respective code features using one or more of the updated source code files, The language model is trained for a sequence-to-sequence generation task based on the generated dataset. A non-temporary computer-readable storage medium that enables the execution of an operation having the following characteristics.

16. The aforementioned operation is, Extracting a list of software packages from the data source by using the repository address, This involves scraping data relating to the list of software packages from the data source by using a web crawler, and the package data related to the software packages is obtained from the scraped data. It further has, A non-temporary computer-readable storage medium according to claim 15.

17. The package metadata associated with the software package is: The name of the aforementioned software package, One or more classes used in the aforementioned software package, Description of the aforementioned software package, Summary of the aforementioned software package, Programming language related to the aforementioned software package, The author of the aforementioned software package, or Classifiers Having at least one of the following, A non-temporary computer-readable storage medium according to claim 15.

18. The operation further comprises generating a hierarchical model by including the source code files and metadata features in order of priority. A non-temporary computer-readable storage medium according to claim 15.

19. Each of the set of target portions of the source code is identified from the source code file using the hierarchical model. A non-temporary computer-readable storage medium according to claim 18.

20. This involves obtaining package data related to a software package from a data source, wherein the package data includes source code files and package metadata related to the software package. Extracting additional metadata related to the software package from the source code file, Prepare metadata features based on the package metadata and the additional metadata, Identifying the target portion of the source code contained in the aforementioned source code file, The process involves updating one or more source code files among the source code files by using the metadata features, and the one or more source code files are: Correction of existing code comments related to the aforementioned target section set, and Adding new code comments to the aforementioned target section set It is updated by performing at least one of the following actions, The process involves generating a dataset of natural language (NL) text features and their respective code features using one or more of the updated source code files, The language model is trained for a sequence-to-sequence generation task based on the generated dataset. A system having a processor configured to perform the following.

Images