System and method for generating code using language model trained from computer code

Abstract

Methods, systems, and computer-readable media for generating computer code based on natural language input are disclosed herein. In an embodiment, a method may include one or more of: receiving a document string representing natural language text, the natural language text specifying a digital programming result; generating a computer code sample configured to generate a corresponding candidate result using a trained machine learning model and based on the document string; causing the computer code sample to be executed; based on the execution, identifying a computer code sample configured to produce a particular candidate result associated with the digital programming result; at least one of: outputting the identified computer code sample via the user interface; compiling the identified computer code sample; transmitting the identified computer code sample to a recipient device; storing the identified computer code sample; and / or re-executing the identified computer code sample.

Description

[0001] Cross-references to related applications This disclosure claims priority to U.S. Patent Application No. 18 / 321,921, filed May 23, 2023, entitled “SYSTEMS AND METHODS FOR GENERATING CODE USING LANGUAGE MODELS TRAINED ON COMPUTER CODE” (Attorney’s Case No. 16273.0006-01000), which is a continuation of and claims priority to U.S. Patent Application No. 18 / 321,852, filed May 23, 2023, entitled “SYSTEMS AND METHODS FOR GENERATING CODE USING LANGUAGE MODELS TRAINED ON COMPUTER CODE” (Attorney’s Case No. 16273.0006-00000), which in turn claims priority to U.S. Patent Application No. 18 / 321,852, filed July 14, 2022, entitled “SYSTEMS AND METHODS FOR EVALUATING LANGUAGE MODELS TRAINED ON COMPUTER CODE”. Priority is claimed in U.S. Provisional Patent Application No. 63 / 389,326, "Computercode" (Agent's Case No. 16273.6000-00000). The entire contents of the foregoing application are incorporated herein by reference for all purposes. Technical Field

[0002] The disclosed embodiments generally relate to systems, apparatus, methods, and computer-readable media for generating computer code based on natural language input or generating natural language based on computer code input. Background Technology

[0003] Existing methods and systems for writing or interpreting computer code fail to improve efficiency and accuracy due to a lack of integration with natural language processing models. Conventional methods and systems also require extensive knowledge of the various programming languages ​​and syntaxes to be used appropriately and effectively. Furthermore, conventional methods and systems lack the ability to understand the context of a programming task in order to generate code suitable for a given context or environment. Additionally, conventional methods and systems lack the ability to continuously learn and improve through user interaction, feedback, and / or self-generated data.

[0004] The inventors have recognized several technical problems with such conventional methods and systems. These technical problems include the time-consuming and repetitive process associated with writing or understanding code, particularly with complex programming tasks, potential syntactic errors and other defects within the code, complex and time-consuming memory management tasks, the challenges associated with developing programs that work across multiple programming languages ​​and platforms, and the advanced skills and knowledge required to write code, especially highly complex code. Summary of the Invention

[0005] The embodiments of this disclosure propose technical improvements as solutions to one or more of the aforementioned technical problems recognized by the inventors in conventional systems. For example, in an embodiment, a method for generating computer code based on natural language input may include receiving a document string representing natural language text specifying a numerical programming result; using a trained machine learning model and based on the document string, generating one or more computer code samples configured to produce corresponding candidate results; causing each of the one or more computer code samples to be executed; based on the execution, identifying at least one of the computer code samples configured to produce a specific candidate result associated with the numerical programming result; and / or performing at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code sample, or re-executing at least one identified computer code sample.

[0006] According to some disclosed embodiments, the method may further include verifying each of one or more executed computer code samples, wherein verification includes calculating a functional correctness score for each of the one or more executed computer code samples, wherein identifying at least one of the computer code samples is based on the functional correctness score.

[0007] Consistent with some publicly available embodiments, trained machine learning models can be fine-tuned based on validated samples of computer code.

[0008] According to some disclosed embodiments, the method may further include verifying each of one or more generated computer code samples, wherein verification includes evaluating each of the one or more generated computer code samples based on at least one unit test. In some embodiments, identifying at least one of the computer code samples may also be based on verification. In some embodiments, a trained machine learning model may be fine-tuned based on the verified computer code samples. In some embodiments, verification may further include evaluating each of the one or more generated computer code samples based on a threshold associated with at least one unit test.

[0009] Consistent with some disclosed embodiments, each of one or more generated computer code samples may be associated with at least one text token. In some embodiments, each of one or more generated computer code samples may also be associated with at least one whitespace token.

[0010] According to some disclosed embodiments, the method may also include outputting specific candidate results of at least one identified computer code sample via a user interface.

[0011] According to some publicly available embodiments, a trained machine learning model can be fine-tuned based on at least one public web source or software repository. In some embodiments, the trained machine learning model can be fine-tuned based on a set of training questions constructed from examples within at least one public web source or software repository.

[0012] Consistent with some disclosed embodiments, identifying at least one of the computer code samples can also be based on the average log probability.

[0013] In some embodiments, at least a portion of one or more computer code samples may be executed in a sandboxed computing environment.

[0014] According to some disclosed embodiments, the method may also include outputting definitions of functions, methods, classes, or modules associated with at least one identified sample of computer code via a user interface.

[0015] In some embodiments, a trained machine learning model can be developed by applying training data comprising annotated computer code to a precursor model, the precursor model including a machine learning model trained based on natural language prompts. Consistent with some disclosed embodiments, the trained machine learning model can also be trained using the execution results that cause each of one or more computer code samples to be executed. In some embodiments, the trained machine learning model can include multiple layers, at least one of which has a transformer-decoder architecture.

[0016] According to some disclosed embodiments, a system for generating computer code based on natural language input may include at least one memory storing instructions and at least one processor configured to execute the instructions to perform operations. In some embodiments, operations may include receiving a document string representing natural language text specifying a numerical programming result; using a trained machine learning model and based on the document string, generating one or more computer code samples configured to produce corresponding candidate results; causing each of the one or more computer code samples to be executed; based on the execution, identifying at least one of the computer code samples configured to produce a specific candidate result associated with the numerical programming result; and / or performing at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code sample, or re-executing at least one identified computer code sample.

[0017] According to some disclosed embodiments, a computer-readable medium, such as a non-transitory computer-readable medium, may include instructions executable by one or more processors to perform operations. In some embodiments, operations may include receiving a document string representing natural language text specifying a numerical programming result; generating one or more computer code samples configured to produce corresponding candidate results using a trained machine learning model and based on the document string; causing each of the one or more computer code samples to be executed; identifying at least one of the computer code samples configured to produce a specific candidate result associated with the numerical programming result based on the execution; and / or performing at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code sample, or re-executing at least one identified computer code sample.

[0018] According to some disclosed embodiments, a method for generating natural language based on computer code input may include accessing a document string generation model configured to generate document strings from computer code; receiving one or more computer code samples; using the document string generation model and, based on the received one or more computer code samples, generating one or more candidate document strings representing natural language text, each of the one or more candidate document strings being associated with at least a portion of the one or more computer code samples; identifying at least one of the one or more candidate document strings that represents an intent to provide at least a portion of the one or more computer code samples; and / or outputting at least one identified document string and at least a portion of the one or more computer code samples via a user interface.

[0019] Consistent with some disclosed embodiments, the docstring generation model can also generate similarities between intent and accompanying natural language text. In some embodiments, the docstring generation model can also be trained using at least one identified docstring of output associated with at least a portion of one or more computer code samples.

[0020] According to some publicly available embodiments, a docstring generation model can be trained using concatenated strings, each concatenated string comprising at least two of a function signature, a reference solution, or a docstring. In some embodiments, the docstring generation model can be further trained by minimizing the negative log-likelihood associated with the docstring in each concatenated string.

[0021] Consistent with some disclosed embodiments, identifying at least one of one or more candidate document strings can be based on a correctness score calculated for each candidate document string.

[0022] According to some disclosed embodiments, the method may further include validating each of one or more candidate document strings, wherein the validation includes determining a correctness score for each of the one or more candidate document strings, and wherein identifying at least one of the one or more candidate document strings is based on the determined correctness score. In some embodiments, the document string generation model may be fine-tuned based on the validated candidate document strings.

[0023] According to some disclosed embodiments, the method may further include ranking one or more candidate document strings based on the determined correctness score, wherein identifying one of the one or more candidate document strings is based on selecting the top k candidate document strings.

[0024] Consistent with some disclosed embodiments, the document string generation model can be a trained machine learning model. In some embodiments, the trained machine learning model can have between 10 billion and 14 billion parameters. In some embodiments, the trained machine learning model can include multiple layers, at least one of which has a transformer-decoder architecture. In some embodiments, the transformer-decoder architecture can include at least one of a masked self-attention head or a feedforward network.

[0025] According to some publicly available embodiments, a docstring generation model can be fine-tuned based on at least one of public web sources or software repositories. In some embodiments, the docstring generation model can be fine-tuned based on a training dataset constructed from examples within at least one public web source or software repository.

[0026] According to some publicly available embodiments, identifying at least one of one or more candidate document strings can also be based on the average log probability.

[0027] In some embodiments, a docstring generation model can be developed by applying training data including annotated computer code to a precursor model, the precursor model comprising a machine learning model trained based on natural language prompts.

[0028] In some embodiments, the method may further include training a machine learning model for generating computer code based on natural language input using training data, said training data including at least one identified document string of output associated with at least a portion of one or more computer code samples.

[0029] According to some disclosed embodiments, a system for generating natural language based on computer code input may include at least one memory storing instructions and at least one processor configured to execute the instructions to perform operations. In some embodiments, operations may include accessing a document string generation model configured to generate document strings based on computer code, receiving one or more computer code samples, using the document string generation model and, based on the received one or more computer code samples, generating one or more candidate document strings representing natural language text, each of the one or more candidate document strings being associated with at least a portion of the one or more computer code samples, identifying at least one of the one or more candidate document strings that identify the intent to provide at least a portion of the one or more computer code samples, and / or outputting at least one identified document string and at least a portion of the one or more computer code samples via a user interface.

[0030] According to some disclosed embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) may include instructions executable by one or more processors to perform operations. In some embodiments, operations may include accessing a document string generation model configured to generate document strings based on computer code, receiving one or more computer code samples, using the document string generation model and, based on the received one or more computer code samples, generating one or more candidate document strings representing natural language text, each of the one or more candidate document strings being associated with at least a portion of the one or more computer code samples, identifying at least one of the one or more candidate document strings that identify an intent to provide at least a portion of the one or more computer code samples, and / or outputting at least one identified document string and at least a portion of the one or more computer code samples via a user interface.

[0031] Other systems, methods, and computer-readable media are also discussed in this paper. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate several embodiments and, together with this description, serve to explain the disclosed principles. In the drawings: Figure 1 This is a flowchart illustrating exemplary methods according to some embodiments of the present disclosure.

[0033] Figure 2 The flowchart illustrates another exemplary method according to some embodiments of the present disclosure.

[0034] Figure 3 This is a block diagram illustrating an exemplary system for generating computer code based on natural language input, according to some embodiments of the present disclosure.

[0035] Figure 4 This is a block diagram illustrating an exemplary system for generating natural language from computer code input, according to some embodiments of the present disclosure.

[0036] Figure 5 The flowchart illustrates an exemplary method for training a machine learning model, based on some disclosed embodiments.

[0037] Figure 6 The flowchart illustrates an exemplary method for training a document string generation model, based on some disclosed embodiments.

[0038] Figure 7 This is a block diagram illustrating an exemplary operating environment for implementing various aspects of this disclosure, according to some embodiments of the present disclosure.

[0039] Figure 8This is a block diagram illustrating an exemplary machine learning platform for implementing various aspects of this disclosure, according to some embodiments of the present disclosure. Detailed Implementation

[0040] Exemplary embodiments are described with reference to the accompanying drawings. In the drawings, the leftmost number(s) of the reference numerals indicate the drawing in which that reference numeral first appears. Where convenient, the same reference numerals are used throughout the drawings to refer to the same or similar parts. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosed exemplary embodiments. However, those skilled in the art will understand that the principles of the exemplary embodiments can be practiced without every particular detail. Well-known methods, processes, and components are not described in detail so as not to obscure the principles of the exemplary embodiments. Unless expressly stated otherwise, the exemplary methods and processes described herein are not bound to a particular order or sequence, nor to a particular system configuration. Additionally, some of the described embodiments or elements thereof can occur or be performed (e.g., executed) simultaneously, at the same point in time, or concurrently. Reference will now be made in detail to the disclosed embodiments, examples of which are illustrated in the accompanying drawings.

[0041] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory descriptions, and are not intended to limit the scope of this disclosure. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate several exemplary embodiments and, together with this description, serve to outline the principles of the exemplary embodiments.

[0042] This disclosure can be described in the general context of custom hardware capable of executing custom preloaded instructions, such as, for example, computer-executable instructions for executing program modules. Program modules may include one or more of routines, programs, objects, variables, commands, scripts, functions, applications, components, data structures, etc., which may perform specific tasks or implement specific abstract data types. The disclosed embodiments can also be practiced in a distributed computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules may reside in local and / or remote computer storage media, including memory storage devices.

[0043] The embodiments discussed herein relate to or relate to artificial intelligence (AI). AI may involve using computerized tools and techniques (e.g., machine learning) to perceive, synthesize, reason, predict, and / or generate information. For example, an AI system may use a combination of hardware and software as the basis for rapidly performing complex operations to perceive, synthesize, reason, predict, and / or generate information. An AI system may use one or more models, which may have specific configurations (e.g., model parameters and relationships between these parameters, as discussed below). While a model may have an initial configuration, this configuration can change over time as the model learns from input data (e.g., training input data), allowing the model to improve its capabilities. For example, a dataset may be the input to the model, which may produce outputs based on the dataset and the model's own configuration. Then, based on additional information (e.g., additional input datasets, validation data, reference data, feedback data), the model may deduce and automatically and electronically implement changes to its configuration that will result in improved outputs.

[0044] A powerful combination of model parameters and sufficiently large datasets, along with high-performance hardware, can produce complex and ingenious models. These models enable AI systems to interpret vast amounts of information based on the model being used, something that would otherwise be impractical, if not impossible, for the human mind. The results across a wide range of applications, including the embodiments discussed herein, are astonishing. For example, AI systems can be configured to automatically navigate vehicles, automatically identify objects, generate natural language on the fly, understand human speech, and generate artistic images.

[0045] The methods, systems, and media disclosed herein provide technological improvements to the fields of artificial intelligence and natural language processing. For example, embodiments of this disclosure improve the efficiency and accuracy of methods for synthesizing computer programming code and individual computer code functions, as well as methods for building entire computer programs, based on input containing natural language text. For example, in response to user input describing a problem to be solved in natural language, a machine learning model consistent with the disclosed embodiments can output computer code. As a result, it may not require the user to have any programming knowledge or experience in order to create executable computer code that provides a solution to the problem. In some embodiments, the machine learning model can be trained using extremely large datasets of code and / or natural language from entirely different sources, enabling the model to learn to generate, and generate functionally accurate code in response to natural language input. This disclosure also provides methods for automatically evaluating the correctness of synthesized code, for example, via unit testing or heuristic grading instead of manual evaluation. This disclosure also provides improved methods for generating natural language descriptions for computer code. For example, in response to receiving input containing programming code, a machine learning model consistent with the disclosed embodiments can output natural language text. As a result, it may not require the user to have any programming knowledge or experience in order to understand the purpose or functionality associated with the computer code. This disclosure further improves the performance of machine learning models by providing methods and systems for fine-tuning trained machine learning models based on known, correctly implemented functions, known associations, and other known data. This disclosure also improves the performance of machine learning models by providing methods and systems for generating and utilizing unit tests to enhance the decision-making capabilities of machine learning models.

[0046] Practical applications of this disclosure can include translating comments into computer code, providing predictive code suggestions based on user comments, automatically populating computer code (e.g., repetitive code, routine coding tasks), proposing alternative code based on user comments, and identifying redundant or unnecessary code to produce faster and more efficient code that requires less memory and resources. Other practical applications include generating frameworks (e.g., front-end user interfaces or UX frameworks) that match user preferences or coding styles based on user comments, providing and executing terminal commands based on user comments (e.g., natural language input), providing automated descriptions of user-selected computer code, and providing intelligent templates for building machine learning models and / or unit tests. Overall, this disclosure can assist programmers in their work, allowing them to generate code snippets, functions, and even entire programs based on natural language descriptions in several programming languages, including Python, Java, C++, Ruby, and JavaScript.

[0047] Some of the disclosed embodiments automate many programming tasks, many of which are prone to human error and / or extremely time-intensive, allowing developers to focus on higher-level tasks and improve productivity. For example, developers can rapidly prototype and experiment with different programming concepts, generating code snippets and functions in approximately seconds, allowing them to quickly test and refine their ideas. This disclosure can also help non-programmers learn programming by allowing them to describe the purpose or functionality of specific code using natural language.

[0048] Illustrative embodiments of this disclosure are described below. In one embodiment, a method for generating computer code based on natural language input may include receiving a document string representing natural language text that specifies the result. As used herein, a document string may refer to any text, including comments, documentation, sentences, paragraphs, words, or any natural language phrase. In some embodiments, a document string may be generated based on natural language input. In some embodiments, a combination of computer code and at least one document string may be generated based on natural language input. In some embodiments, the document string may adhere to a format (e.g., syntax) that is not fully understandable to humans. A document string may provide information about functions, methods, modules, or classes related to using or interacting with computer programming code. For example, a document string may provide information about what a function (or any code) does, what arguments a function accepts, the return value produced by a function (or any code), any potential exceptions raised by a function (or any code), how to use a function (or any code), or the expected behavior of a function (or any code). Natural language text as used herein may refer to any text written or spoken in human language to express thoughts, ideas, or information. Natural language text can be characterized by fluidity and variability, and it can include syntax, syntax, and / or semantics. For example, natural language text can include any sequence of words or sentences that convey meaning and can be understood by humans. As used herein, a result can refer to an outcome, effect, achievement, impact, or product. For example, a result can include digital programming results (e.g., program output, program behavior, computational functionality, a collection of generated data elements), generated automated computerized actions, generated images or other media, or any other outcome that can be caused by programming code.

[0049] In some embodiments, the method may also include generating one or more computer code samples based on a trained machine learning model and a docstring, the computer code samples being configured to produce corresponding candidate results. A trained machine learning model can refer to a mathematical or computational representation created and trained using machine learning algorithms to make predictions or decisions based on received inputs. The machine learning model can be trained based on examples or past data (e.g., training data) to generalize and / or make predictions on new, unseen data based on patterns, relationships, or trends identified in past data. In some embodiments, the machine learning model may be stored in an ML algorithm database 890.

[0050] Training data can include, for example, datasets collected from various public software repositories (e.g., datasets of hundreds, thousands, millions, or even billions). In some embodiments, training data can include text (e.g., natural language) and / or code (e.g., compiled or uncompiled code). Training data can be collected from one or more sources (such as web pages maintained on one or more websites). In some embodiments, data within such datasets can be initially filtered to exclude, for example, files that may be automatically generated, have a large average line length (e.g., greater than 100), a large maximum line length (e.g., greater than 1000), or contain a small percentage of alphanumeric characters. After filtering, the filtered data can be used as training data input to train a machine learning model. In some embodiments, training the machine learning model can include linear warm-up combined with other learning rate scheduling techniques (such as learning rate decay or cyclic learning rate) to further optimize the training process and enhance model performance (e.g., 175-step linear warm-up and cosine learning rate decay). During the warm-up phase, the learning rate can be initially set to a low value and then linearly increased over a number of training steps or epochs. Non-linear increases can also be used. This allows the model to start with a smaller learning rate, which helps it explore the parameter space more efficiently as the model's parameters are randomly initialized and avoids large and potentially harmful updates at the start of training. By increasing the learning rate linearly (or with any increasing trend), the model can quickly tune its parameters and adapt to the training data while avoiding sudden and potentially unstable changes. Once the warm-up phase is complete, the learning rate can be reduced according to a predefined schedule, such as using learning rate decay or employing adaptive optimization methods like Adam or Adagrad. For example, a machine learning model can be trained on a total of 100 billion tokens using an Adam optimizer with I = 0.9, 2 = 0.95, E = 10⁻⁸, and a weight decay factor of 0.1. Of course, machine learning models can be trained on other total amounts of tokens, such as millions of tokens (e.g., one million, ten million, one hundred million).

[0051] As an example, trained machine learning models can include linear regression models, decision tree models, random forest models, support vector machine models, convolutional neural networks, recurrent neural networks, or other artificial intelligence models, such as those related to... Figure 8The models discussed herein. As used herein, a computer code sample can refer to any combination of phrases, functions, procedures, scripts, strings, or connections of computer programming code. As used herein, a candidate result can refer to a potential or possible outcome that may or may not be equivalent to the desired result indicated by the natural language input (e.g., a candidate result or candidate solution may not necessarily solve the problem at hand). In some embodiments, a higher number of computer code samples provided by a trained machine learning model can produce more accurate candidate results based on the input docstring. For example, a machine learning model that outputs one computer code sample in response to an input docstring provides one candidate result and does not provide other potential computer code samples that might provide other candidate results; however, a machine learning model that outputs 100 computer code samples in response to an input docstring provides 100 corresponding candidate results, any one of which could provide the desired result.

[0052] Consistent with some disclosed embodiments, trained machine learning models can be developed by applying training data, including annotated computer code, to a precursor model, which includes a machine learning model trained based on natural language prompts. In some embodiments, the training data may also include a set of training problems constructed from independent functions that are correctly implemented (e.g., verified or validated). Such independent functions can be collected from programming web sources (e.g., competitive programming websites or software development interview preparation websites) and / or from software repositories (e.g., open-source repositories or projects utilizing continuous integration). For example, such sources may provide problem statements, function signatures, and solutions, which can be collected and used as training data, using the problem description as a docstring. As another example, competitive programming websites may provide additional unit test data for determining functional correctness (automatically determining the functional correctness of submissions made by developers to the website based on hidden unit tests used by such websites). As yet another example, additional unit tests may be created based on examples found in the problem statements, and / or additional test cases may be extracted by submitting incorrect solutions; a combination of the above can be used to curate the training data set. As yet another example, continuous integration can be used to manage additional programming issues based on open-source projects (e.g., by tracking and collecting the inputs and outputs of functions called during integration tests to create unit tests for those functions). This can be done, for example, by following setup and test commands in a continuous integration configuration file, which are used to set up virtual environments, install dependencies, and run integration tests. Example continuous integration repositories include GitHub repositories using the Travis and / or Tox frameworks, and pip packages within the Python Package Index (PyPI).

[0053] In some embodiments, the method may further include causing each of one or more computer code samples to be executed. As used herein, execution (or being executed) may refer to running or executing a computer code sample such that instructions contained within the computer code sample can be processed by a processor or interpreted by an interpreter or virtual machine and executed to perform one or more tasks. As used herein, a task may refer to, for example, interacting with input and / or output devices, accessing data, modifying data, transferring digital information, performing calculations, modifying variables, making decisions, and / or performing other computerized operations as defined by the computer code sample. Candidate results may include, for example, code segments configured to perform one or more specific functions, the provision of printed messages, the provision of updated data values, changes in computerized state, interaction with external systems or resources, performance of reading from a file, performance of writing to a file, performance of accessing a database, performance of making network requests, and / or data generated by the performance of one or more of these operations.

[0054] In some embodiments, the method may also include, based on execution, identifying at least one of computer code samples configured to produce a specific candidate result associated with a numerical programming result (e.g., a candidate result corresponding to the numerical programming result, a candidate result most closely corresponding to the numerical programming result, or a candidate result corresponding to a performance metric). As used herein, identification can refer to recognizing, distinguishing, or determining. For example, based on the execution of two computer code samples generated by a machine learning model, it may be found that the first executed computer code sample causes a result that does not correspond to (e.g., does not meet a tolerance threshold) the result indicated by the natural language input, and it may be found that the second executed computer code sample causes a result that corresponds to (e.g., meets a tolerance threshold) the desired result based on the natural language input. In such a scenario, the second computer code sample can be identified, while the first computer code sample can be ignored or discarded.

[0055] In some embodiments, the method may include performing at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code sample (e.g., locally and / or remotely), and / or re-executing at least one identified computer code sample.

[0056] As used herein, output can refer to sending, transmitting, producing, or providing. As used herein, user interface can refer to any means by which a user interacts with a software application or computer system. For example, a user interface can include a graphical user interface (GUI), a command-line interface, a touch user interface, a voice user interface, or a virtual reality user interface.

[0057] As used in this article, compilation can refer to the process of converting human-readable source code written in a programming language into machine-readable code that can be executed by a computer or target platform (i.e., not easily understood by humans). For example, samples of computer code can be transformed into lower-level representations, such as machine code or bytecode, which can then be understood and executed by a processor (although not understandable to humans).

[0058] As used herein, transmission can refer to the process of sending or transferring data or information from one location or device to another. Transmission can involve the propagation or communication of data through a medium or network, allowing that data to be received and accessed by an intended recipient (e.g., a user) or a user's receiving device.

[0059] As used herein, storage can refer to any action that preserves and / or retains data, information, or content in a manner that allows for access, retrieval, and / or use at a later time. Storage may involve the process of saving or recording data in a persistent and organized manner for future reference or use.

[0060] The key takeaway is that, as described herein, the technical implementations of generating computer code based on natural language input achieve solutions rooted in computer technology, rather than simply following rules. Furthermore, it is important to understand that such technical implementations facilitate solving the complex problem of automating computer code generation by using massive amounts of data to train artificial intelligence models and leveraging those models to generate accurate predictions of computer code.

[0061] In some embodiments, the method may further include verifying each of one or more executed computer code samples, wherein verification may include calculating a functional correctness score for each of the one or more executed computer code samples, wherein identifying at least one computer code sample may be based on the functional correctness score. In some embodiments, the functional correctness score may be based on comparing the functional behavior and / or output of the computer code sample with validation data representing desired or known behavior and / or results. Consistent with some disclosed embodiments, verification may include evaluating each of one or more generated computer code samples based on at least one unit test, without necessarily providing a functional correctness score. Further consistent with some disclosed embodiments, identifying at least one computer code sample may also be based on verification based on at least one unit test.

[0062] As used herein, validation can refer to confirming, verifying, or establishing truth, authenticity, validity, or accuracy. As used herein, a functional correctness score can refer to a value indicating the quantity of unit tests passed by a particular sample of computer code (e.g., the number of units flowing from a complete set of unit tests). As used herein, unit testing can refer to software testing or inspection where individual components (e.g., a sample of computer code or a portion thereof) can be tested in isolation to ensure proper functionality. The goal of unit testing can be to confirm the behavior and / or functionality of a small piece of self-constrained computer code and ensure that the piece of computer code operates as expected and meets specific desired requirements (e.g., a sample of computer code configured to provide candidate results actually provides the desired result). Unit tests can be written by developers, generated by machine learning models, or a combination of both. Unit tests can be based on known combinations of input-output data (e.g., test cases), known behavioral data (e.g., attributes, invariants, edge data), or a combination of both. This can provide a set of unit tests to verify one or more computer code samples generated according to this disclosure, and a functional correctness score can be calculated based on the number of unit tests within the set of unit tests that a particular computer code sample passes.

[0063] Consistent with some disclosed embodiments, verification may also include evaluating each of one or more generated computer code samples based on a threshold associated with at least one unit test. As used herein, a threshold can refer to a value or boundary that can be static or variable (e.g., dependent on another value) that determines whether a certain condition is met or not, for example, as determined by the unit test. As an example, a threshold may indicate a timeout value (e.g., 3 seconds) or another time-related value (e.g., execution time). For example, a threshold can be used to make a binary decision based on a continuous value or a probability value, both of which can be determined by the unit test. A threshold can also be used to determine the outcome of a decision or to categorize data into different categories. A threshold can also be used to determine when a computer code sample fails a unit test, for example, when a computer code sample fails a unit test within a given time-related value.

[0064] Consistent with some published embodiments, trained machine learning models can be fine-tuned based on validated samples of computer code. As used herein, fine-tuning can refer to the process of further training a trained machine learning model on a specific task or dataset (e.g., initializing, partially training, training against a larger or more general dataset) to further improve its performance and adapt it to a specific domain or problem. Fine-tuning can allow machine learning models to leverage knowledge and learned representations from larger, pre-existing models, and, for example, refine them for the specific task at hand using smaller, task-specific datasets, thereby producing more accurate outputs (e.g., outputs that more closely correspond to the desired output). During fine-tuning, the parameters of the trained machine learning model can be tuned or updated using a dataset representative of the target task. The purpose of fine-tuning can be to adjust or update the learned features and weights of the machine learning model to better align with the patterns and characteristics of the target task, which can lead to further improved performance and / or generalization.

[0065] Consistent with some disclosed embodiments, each of one or more generated computer code samples may be associated with at least one text token. As used herein, a token may refer to a unit or element of text used as a basic building block in natural language processing (NLP) or other machine learning tasks. A text token may represent a single unit of meaning or linguistic component within a phrase or sentence. For example, a block of text, such as a word, phrase, sentence, paragraph, or document, may be broken down into smaller units (e.g., text tokens) for further analysis. Depending on the specific tokenization strategy employed, text tokens may include, for example, individual words, punctuation marks, or even smaller sub-word units (e.g., a portion of a word). Tokenization may refer to the process of breaking down text into tokens, which may aid in performing machine learning tasks, including text classification, sentiment analysis, machine translation, and named entity recognition. In some embodiments, tokens may be generated based on natural language input, such as by transforming the natural language input into a format that is understandable to computerized models (but which may be impractical or impossible for humans to understand). By breaking text down into tokens, statistical and machine learning techniques can be applied to analyze and process natural language data.

[0066] Consistent with some disclosed embodiments, each of one or more generated computer code samples may also be associated with at least one space token. As used herein, a space token may refer to a token representing open space between words, punctuation marks, sentences, paragraphs, and other text formats (as previously described and illustrated).

[0067] In some embodiments, the method may further include outputting an associated result of at least one identified computer code sample via a user interface. As used herein, the associated result may refer to the actual result that occurs based on executing the computer code sample. Therefore, the method may include both outputting the identified computer code sample and / or the result of executing the computer code sample. Furthermore, the computer code sample may be provided to the user in conjunction with an associated docstring, which may confirm to the user that the identified computer code sample will provide the desired result.

[0068] Consistent with some disclosed embodiments, a trained machine learning model can be fine-tuned based on at least one public web source or software repository (as previously described and illustrated). In some embodiments, a device (e.g., using specific modules, programs, or applications) can retrieve digital information from one or more public web sources or software repositories, such as by crawling one or more web pages, websites, and / or Hypertext Token Language (HTML) code. As used herein, a public web source can refer to any information or content accessible to the general public via the Internet and / or the World Wide Web. Public web sources can include websites, web pages, blogs, forums, news articles, social media posts, and other publicly available online resources that can be accessed by anyone using an Internet connection. As used herein, a software repository (also referred to as a package repository or software source) can refer to a centralized location comprising a collection of software components that is organized and managed to facilitate software development, distribution, and updates, and where software packages, libraries, and related files are stored and available for distribution and installation. Non-restricted examples of software repositories include Debian package repositories, the Ubuntu Software Center, PyPI (Python Package Index), npm (Node Package Manager), and GitHub repositories. Trained machine learning models can be fine-tuned based on at least one of the following: packages (e.g., software components such as applications, libraries, drivers, and plugins), metadata (e.g., information associated with each package, such as version number, description, dependencies, and other data related to specific software components), or text (e.g., blog entries, comments, descriptions). Any or all of these can be stored in a software repository.

[0069] Consistent with some disclosed embodiments, the execution results of each of one or more computer code samples being executed can be used to further train a trained machine learning model. For example, the execution results of each of one or more computer code samples being executed can be used as additional training data for the machine learning model. Therefore, this method allows the machine learning model to generate additional training data for itself, becoming more accurate due to the self-generated training data, without requiring external data to be used as the additional training data.

[0070] Consistent with some disclosed embodiments, a trained machine learning model may include multiple layers, at least one of which has a transformer-decoder architecture. As used herein, transformer-decoder architecture may refer to a machine learning architecture that includes an encoder and a decoder, both of which are built on a self-attention mechanism, wherein the encoder focuses on encoding the input sequence and the decoder generates an output sequence based on the encoded representation.

[0071] Consistent with some publicly available embodiments, a trained machine learning model can be further trained or fine-tuned based on a set of training questions built from examples within at least one public web source or software repository.

[0072] Consistent with some disclosed embodiments, identifying at least one of the computer code samples can also be based on the average log probability. As used herein, the average log probability can refer to a value calculated by determining the logarithm of each predicted probability associated with a particular computer code sample (or other output provided by a machine learning model), and then calculating the average result across all predictions. By taking the logarithm of the predicted probabilities, the average log probability metric can provide a way to measure the confidence of a machine learning model in its predictions. A higher average log probability can indicate that the machine learning model is assigning a higher probability to the correct output (e.g., the correct computer code sample), thus indicating better performance. The average log probability metric can be particularly useful where the magnitude of the predicted probabilities is important, rather than just the correctness of the predictions made by the machine learning model. Additionally or alternatively, other statistical values ​​or techniques can be used. For example, one or more outliers can be removed before performing the logarithmic calculation. As another example, identifying at least one of the computer code samples may include determining a probability distribution.

[0073] Consistent with some disclosed embodiments, it is permissible to execute at least a portion of one or more computer code samples in a sandboxed computing environment. As used herein, a sandboxed computing environment can refer to a restricted and isolated environment (e.g., a virtual computing environment) in which software programs and processes can run safely without affecting any underlying systems or applications and / or the computer code samples not directly associated with the sandboxed computing environment. The sandboxed computing environment thus provides a controlled space for testing, experimenting with, and executing potentially untrusted or unknown computer code samples generated by trained machine learning models.

[0074] In some embodiments, the method may further include outputting definitions of functions, methods, classes, or modules associated with at least one identified computer code sample via a user interface. Therefore, the method may include both outputting the identified computer code sample and additional information associated with that computer code sample. Furthermore, the identified computer code sample may be provided to the user in conjunction with the additional information associated with it, which may confirm to the user that the identified computer code sample will provide the desired result, or it may provide the user with helpful information related to one or more of the functions, methods, classes, or modules associated with the computer code sample.

[0075] According to other disclosed embodiments, an exemplary system may include a memory storing at least one instruction and at least one processor configured to execute the instruction to perform a set of operations for generating computer code based on natural language input. The set of operations may mirror one or more steps of the method 100 described herein. Thus, the system may be configured to receive a document string representing natural language text specifying a numerical programming result. The system may also be configured to generate one or more computer code samples configured to produce corresponding candidate results using a trained machine learning model and based on the document string. The system may also be configured to cause one or more generated computer code samples to be executed and / or verified. In some embodiments, the system may also be configured to identify computer code samples configured to produce numerical programming results based on execution and / or verification. Furthermore, the system may be configured to perform at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code sample, and / or re-executing at least one identified computer code sample.

[0076] According to another embodiment of this disclosure, a computer-readable medium includes instructions for performing steps to generate computer code based on natural language input. The steps embodied in the instructions of the computer-readable medium may mirror one or more steps of the method 100 described herein. Thus, these steps may be configured to receive a document string representing natural language text specifying a numerical programming result. These steps may also be configured to generate one or more computer code samples using a trained machine learning model and based on the document string, the one or more computer code samples being configured to produce corresponding candidate results. These steps may also be configured to cause one or more generated computer code samples to be executed and / or verified. In some embodiments, these steps may also be configured to identify computer code samples configured to produce numerical programming results based on execution and / or verification. In some embodiments, these steps may also be configured to perform at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code sample, and / or re-executing at least one identified computer code sample.

[0077] This disclosure can be used to perform a range of natural language processing-based coding tasks. Example tasks include code completion (e.g., suggesting code completion to developers as they write code; for example, if a developer starts typing a line of code and then pauses, a code snippet can be suggested to complete the task, saving time and improving the accuracy of the code being written), automated testing (e.g., generating test cases and test code, which helps developers ensure their code works correctly and catches defects from the outset), code refactoring (e.g., suggesting changes to existing code to improve its efficiency, readability, and maintainability, which helps developers optimize their code and reduce technical debt), natural language processing (e.g., processing natural language queries and generating code based on those queries, which can be useful for developers unfamiliar with a particular programming language or who need to write code quickly), and intelligent coding assistants (e.g., building intelligent coding assistants that can help developers perform complex coding tasks; for example, coding assistants can help developers build machine learning models or...). Optimizing database queries), code generation for low-code platforms (e.g., integrating with platforms designed to make it easier for non-technical users to build applications to help generate the code needed to build applications, which can help reduce the technical knowledge required to build applications and accelerate the development process), code synthesis for code review (e.g., automatically generating code changes based on code review comments, which can save time and improve the efficiency of the code review process), rapid prototyping (e.g., quickly prototyping new ideas and testing different ways of coding a problem, which can help developers iterate more quickly and efficiently based on their ideas), code analysis and optimization (e.g., analyzing existing code and proposing ways to improve it; for example, proposing ways to reduce code complexity, improve its performance, or reduce its memory footprint), and game development (e.g., generating code for game development, including game engines, physics simulations, and artificial intelligence algorithms, which can enable game developers to create more complex and realistic games more efficiently).

[0078] Now refer to Figure 1 , 3 Figures 5 and 6 illustrate exemplary embodiments of this disclosure.

[0079] Figure 1 The figure illustrates an exemplary method 100 for generating computer code based on natural language input, consistent with disclosed embodiments such as those discussed herein. Figure 1 Any component step of the process shown or its constituent steps may be implemented using operating environment 700, system 800 (e.g., using at least one processor and at least one memory component) or any of the above components. Figure 1The steps illustrated are exemplary, and in some embodiments, steps may be added, merged, split, copied, repeated (e.g., as part of a machine learning process), modified, executed sequentially, executed in parallel, and / or deleted.

[0080] like Figure 1 As illustrated, exemplary method 100 may include step 110 of receiving a document string representing natural language text of a specified numerical programming result. Figure 1 As further illustrated, exemplary method 100 may also include step 120, which uses a trained machine learning model and, based on a docstring, generates one or more computer code samples configured to produce corresponding candidate results. Also as... Figure 1 As illustrated, the exemplary method 100 may further include step 130, which causes each of one or more computer code samples to be executed. Figure 1 The exemplary method 100 shown may further include step 140 of identifying at least one of computer code samples configured to produce specific candidate results associated with digital programming results based on execution. Figure 1 As further illustrated, the exemplary method 100 may also include step 150, which performs at least one of the following: outputting at least one identified computer code sample via a user interface, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device, storing at least one identified computer code, and / or re-executing at least one identified computer code sample. Figure 1 As illustrated in the figure, the exemplary method 100 may also include step 160 of further training the machine learning model based on the output or other data.

[0081] Figure 3 The description of some embodiments of this disclosure is used for implementation. Figure 1A functional block diagram of an exemplary operating environment 300 for the method is provided. In some embodiments, the operating environment 300 may include a system 304, which includes at least one memory (not shown) storing instructions and at least one processor (not shown) configured to execute the instructions to perform a set of operations for generating computer code based on natural language input. System 304 may be an instance of system 500 and / or include features of system 500. The set of operations may map to one or more steps of the method 100 described herein. Thus, system 304 may be configured to receive a document string 302 representing natural language text of a specified numerical programming result. System 304 may also be configured to generate one or more computer code samples 306 using a trained machine learning model 305 and based on the document string 302, the one or more computer code samples 306 being configured to produce corresponding candidate results. In some embodiments, the machine learning model 305 may be trained using data from public web sources and / or repositories 314. System 304 may also be configured to cause one or more generated computer code samples 306 to be executed via execution module 308 and / or verified via verification module 310. In some embodiments, one or more generated computer code samples 306 may be executed or verified in a sandbox computing environment 316. In some embodiments, system 304 may also be configured to identify computer code samples configured to produce digital programming results based on execution and / or verification. Furthermore, system 304 may be configured to perform at least one of the following: outputting at least one identified computer code sample via a user interface (not shown) 312, compiling at least one identified computer code sample, transmitting at least one identified computer code sample to a receiving device 318, storing at least one identified computer code sample, and / or re-executing at least one identified computer code sample.

[0082] Figure 5 The illustration depicts an exemplary method 500 for training a machine learning model, consistent with the disclosed embodiments. In some embodiments, the machine learning model can be trained such that it can be configured to generate computer code based on natural language input, or to generate natural language based on computer code input, as discussed herein. Figure 5 Any component step of the process shown or its constituent steps may be implemented using operating environment 700, system 800 (e.g., using at least one processor and at least one memory component) or any of the above components. Figure 5 The steps illustrated are exemplary, and in some embodiments, steps may be added, merged, split, copied, repeated (e.g., as part of a machine learning process), modified, executed sequentially, executed in parallel, and / or deleted.

[0083] like Figure 5 The exemplary method 500 for training a machine learning model, as illustrated in the figure, may include a step 510 of collecting and preparing data. Collecting and preparing data may include, for example, acquiring, accessing, or generating training data, cleaning and preprocessing data (e.g., handling missing values, normalizing data, or encoding categorical variables), and splitting the data into training and validation datasets. Figure 5 As further illustrated, exemplary method 500 may also include step 520 of selecting a machine learning model and architecture. Step 520 may include, for example, selecting an appropriate machine learning algorithm or model type, defining the model architecture (e.g., number of layers, neurons, activation functions), and setting hyperparameters (e.g., learning rate, regularization). Figure 5 As illustrated in the figure, the exemplary method 500 may further include a step 530 of training a machine learning model using the collected and prepared data. For example, step 530 may include initializing the machine learning model with random weights and biases, iterating through the training data (e.g., performing forward pass, calculating the loss / error between the predicted output and the true output, and performing backward pass), and / or repeating the iterations until convergence or a predefined stopping criterion is met. Furthermore, as... Figure 5 As shown, the exemplary method 500 may include a step 540 of evaluating a trained machine learning model. Step 540 may include, for example, evaluating the performance of the machine learning model based on a validation dataset (e.g., calculating evaluation metrics such as accuracy, precision, recall, and / or F1 score, and / or analyzing the results and adjusting the model architecture). Also as Figure 5 As shown, the exemplary method 500 may include a step 550 of fine-tuning the trained machine learning model (as previously described and illustrated). Figure 5 The exemplary method 500 shown may also include a step 560 of deploying a trained machine learning model. For example, step 560 may include deploying the machine learning model to a production environment, monitoring the performance of the machine learning model, updating the machine learning model as needed, and / or using the machine learning model to generate output based on a given input.

[0084] In yet another disclosed embodiment, an exemplary method for generating natural language text based on computer code input may include accessing a document string generation model configured to generate document strings based on computer code. As used herein, a document string generation model may refer to any natural language generation (NLG) machine learning model. For example, a document string generation model may include a template-based NLG model, a rule-based NLG model, a statistical NLG model, a neural NLG model, a hybrid NLG model, a data-to-text NLG model, a reinforcement learning NLG model, a controlled NLG model, an extractive NLG model, an abstract NLG model, a neural machine translation model, an image captioning model, a chatbot NLG model, or other artificial intelligence models that provide natural language output. As used herein, access may refer to any process of obtaining, receiving, or retrieving data, information, or resources from a given source or location, which may involve the ability to connect, enter, or log in to obtain specific content, files, functionality, or output.

[0085] In some embodiments, the method may also include receiving one or more computer code samples. As used herein, receiving can refer to requesting, accessing, obtaining, acquiring, accepting, identifying, selecting, highlighting, and / or collecting. For example, one or more computer code samples may be received when a user highlights (or otherwise selects) at least a portion of computer code, such as by evaluating, performing, and / or modifying the code by providing input to a user interface. As another example, one or more computer code samples may be received when a user enters at least a portion of computer code into a prompt field. In some embodiments, a user may enter at least a portion of computer code and natural language text into the prompt field.

[0086] In some embodiments, the method may further include using the document string generation model and generating one or more candidate document strings representing natural language text based on one or more received computer code samples, each of the one or more candidate document strings being associated with at least a portion of the one or more computer code samples. As used herein, a candidate document string may refer to a potential or possible document string that may or may not be an exact description of the corresponding computer code input. In some embodiments, a higher number of candidate document strings provided by the document string generation model may produce a more accurate natural language output based on the input computer code. For example, a document string generation model that outputs one candidate document string in response to input computer code provides one possible output and does not provide other outputs that may provide other potential candidate document strings; however, a document string generation model that outputs 100 candidate document strings in response to input computer code provides 100 possible outputs, any one of which (or a combination thereof) may provide an exact description of the input computer code. The term “associated with” as used herein may indicate a connection, relationship, correspondence, relevance, or involvement between two or more entities, concepts, or elements, and may imply that one thing is linked or connected to another thing in some way. This correlation can be based on various factors, such as similarity, causality, correlation, dependence, or participation.

[0087] In some embodiments, the method may also include at least one of one or more candidate docstrings identifying the intent of providing at least a portion of one or more computer code samples. As used herein, intent may refer to a goal, purpose, or description associated with computer code or a portion thereof. Intent may also refer to the intended functionality or purpose of a particular piece of code or a specific block of code in order to provide clarity and understanding for a user (e.g., a developer, administrator, student, or any other individual wishing to read or maintain computer code). For example, the intent of a computer code sample Figure 1 Generally, intent can explain the basic function, purpose, or goal that a sample of computer code (or a portion thereof) performs when executed or interacts with within a system, application, environment, or other software or hardware component. As another example, intent can include one or more of the function-method intents (e.g., an intent reflecting the purpose or action performed by a coded function or method), annotation intents (e.g., text that provides information or code explanation to convey the intent of the code), class-module intents (e.g., the name of a class or module that reflects the purpose or intent of the code it requires), API endpoint intents (e.g., the name of an endpoint that reflects the purpose or intent of that endpoint), or conditional intents (e.g., conditions or associated code / text that reflect the expected behavior or threshold).

[0088] As used herein, an identifier can refer to identification, differentiation, or determination. For example, based on the analysis of two candidate document strings generated by a document string generation model, it can be found that the first candidate document string provides an intent that is not equivalent to or less accurate (e.g., does not meet the tolerance threshold) compared to the actual intent indicated by the computer code sample, while the second candidate document string provides an intent that is equivalent to or more accurate (e.g., meets the tolerance threshold) compared to the actual intent indicated by the computer code. In such a scenario, the second candidate document string can be identified, while the first candidate document string can be ignored or discarded.

[0089] In some embodiments, the method may further include outputting at least one identified docstring and at least a portion of one or more computer code samples via a user interface. For example, the method may include both outputting computer code samples and at least one identified docstring that accurately indicates the intent of the computer code samples. Thus, the computer code samples can be provided to the user in conjunction with the associated docstring, which may confirm to the user that the computer code samples will provide the desired result, or this may provide the user with an explanation or description of the computer code samples in generated natural language. It should be understood that in many embodiments, given the confusing and number-based nature of computer code samples, the human mind is not equipped to perform the operation of determining the intent of computer code samples beyond simple evaluation.

[0090] Consistent with some disclosed embodiments, the document string generation model can also generate a similarity between the intent and the accompanying natural language text. The similarity between the intent and the accompanying natural language text can be generated, for example, by determining one or more of cosine similarity, edit distance (e.g., Levenshtein distance), Jaro-Winkler distance, Jaccard index, longest common subsequence (LCS), n-gram syntactic similarity, or Hamming distance between the intent and the accompanying natural language text. As used herein, the accompanying natural language text can refer to any natural language text that is not the output of the computer code sample. For example, a similarity can be generated between an intent (e.g., as indicated by the natural language description of the generated computer code sample) and a known value associated with the computer code sample (e.g., as indicated by training data including a known description of the computer code sample). The purpose of such generated similarities can be, for example, to verify, confirm, or validate a correctness score. The key takeaway is that implementing solutions rooted in computer technology, rather than simply following rules, helps solve the complex problem of obtaining the output provided by a document string generation model instantly (or almost instantly) using one or more sophisticated statistical methods.

[0091] Consistent with some disclosed embodiments, the docstring generation model can be further trained using at least one identified docstring associated with at least a portion of one or more computer code samples. For example, the output of the docstring generation model associated with the corresponding computer code sample can be used as additional training data for the docstring generation model or an additional machine learning model. Therefore, this method allows the docstring generation model to generate additional training data for itself or another machine learning model, becoming more accurate due to the self-generated training data (or allowing another machine learning model to become more accurate), without requiring external data to be used as additional training data. It is important to understand that combining the output of the docstring generation model (or another natural language-based learning model) with the training (or fine-tuning) process of that model forms an unconventional and ungeneral arrangement that helps address the technical problem of using self-generated data to improve the accuracy and effectiveness of a model without requiring the collection or preparation of any additional external training data.

[0092] Consistent with some disclosed embodiments, a docstring generation model can be trained using concatenated strings, each concatenated string including at least two of a function signature, a reference solution, or a docstring. As used herein, concatenated strings can refer to a sequence of strings that are combined, merged, joined, or associated together to create a unified or cohesive data entity. For example, concatenated strings can include two or more strings (e.g., function signature and reference solution, reference solution and docstring, function signature and docstring, or function signature, reference solution, and docstring) placed consecutively and separated by punctuation (e.g., commas) or operators (e.g., "+"). As used herein, a function signature can refer to the declaration or definition of a function in a programming language. A function signature provides information about a function, including its name, parameters (e.g., input arguments), return type, and any additional modifiers or qualifiers. A function signature can serve as a contract or specification defining a function interface, indicating how it should be invoked and what it should return. Function signatures can also help other parts of a program understand how to interact with the function and can allow the program or associated devices to implement appropriate function overloading or polymorphism. As used herein, a reference solution can refer to a predetermined and / or authoritative value that serves as a benchmark or standard for evaluating the correctness or quality of the generated output. For example, a reference solution can be established as a credible or ideal representation of the problem being addressed (e.g., determining an accurate docstring to indicate the intent of a corresponding sample of computer code). Reference solutions can be created by experts and / or machine learning models, such as by obtaining reference solutions through extensive testing and / or derived from theoretical analysis. The purpose of a reference solution may be to provide a point of comparison for other solutions, allowing for evaluation, validation, or verification. Thus, a reference solution can serve as a baseline or standard against which alternative solutions are measured and evaluated.

[0093] Consistent with some published embodiments, the docstring generation model can be further trained by minimizing the negative log-likelihood (NLL) associated with the docstring in each linked string. As used herein, negative log-likelihood can refer to the value calculated by taking the negative logarithm of the likelihood function. In some embodiments, the formula for negative log-likelihood may be: NLL = -log(likelihood) Where NLL represents negative log-likelihood.

[0094] Minimizing the negative log-likelihood is equivalent to maximizing the likelihood, and therefore, it yields model parameters that best fit a given dataset. Thus, the negative log-likelihood can be used as training data, serving as a criterion for estimating model parameters and evaluating the performance of probabilistic models.

[0095] Consistent with some disclosed embodiments, identifying at least one of one or more candidate document strings can be based on a correctness score calculated for one or more candidate document strings (e.g., each candidate document string). As used herein, a correctness score can refer to a metric used to evaluate the performance of a document string generation model. A correctness score can indicate the proportion of instances or predictions that are correctly classified compared to the total number of candidate document strings generated by the document string generation model. A correctness score can be expressed as a percentage, rating, or value to provide an indication of how accurately the document string generation model can predict or classify data. As an example, to calculate a correctness score, each prediction from the document string generation model (i.e., each generated document string) can be compared to one or more ground truth labels or otherwise known correct answers. Furthermore, each portion of a prediction that matches a ground truth label can be considered correct, while each incorrect portion of a prediction can be counted as an error. The correctness score can then be calculated as: Correctness score = (Number of correct parts / Total number of parts) * 100.

[0096] As an example, if the model correctly classifies 8 out of 10 parts of the generated document string, the accuracy score can be 80%.

[0097] In some embodiments, the method may further include validating each of one or more candidate document strings (as previously described and illustrated). In some embodiments, validation may include determining a correctness score for each of one or more candidate document strings (as previously described and illustrated). In some embodiments, identifying at least one of one or more candidate document strings (as previously described and illustrated) may be based on the determined correctness score.

[0098] Consistent with some publicly available embodiments, the document string generation model can be fine-tuned (as previously described and illustrated) based on validated candidate document strings.

[0099] In some embodiments, the method may further include ranking one or more candidate document strings based on the determined correctness score. In some embodiments, identifying one of the one or more candidate document strings may be based on selecting the top k candidate document strings. As used herein, the top k candidate document strings may refer to one of the top "k" or highest-ranking "k" candidate document strings from the complete set of generated candidate document strings, wherein the selection of the top k candidate document strings may be based on a specific criterion or scoring mechanism (e.g., correctness score), and wherein "k" represents a positive integer indicating the number of candidate document strings to be selected.

[0100] Consistent with some disclosed embodiments, the document string generation model may be a trained machine learning model (as previously described and illustrated).

[0101] Consistent with some publicly disclosed embodiments, trained machine learning models can have between one thousand and 14 billion parameters (e.g., between 10 billion and 14 billion parameters). Of course, other quantities of parameters can be used, such as tens or hundreds of thousands, tens of millions or hundreds of millions, or even hundreds of billions. As used herein, a parameter can refer to a variable or set of variables that influences and / or defines the behavior or configuration of a machine learning model by, for example, determining the model's ability to capture patterns and make predictions. Parameters can be tuned during training phases (e.g., fine-tuning phases) to minimize a predefined objective function, where the goal of the tuning is to find the optimal set of parameter values ​​that best fits the given training data and generalizes well to unseen data. Depending on the algorithm and problem domain, parameters can take different forms. For example, parameters can be weights assigned to individual features, coefficients in a mathematical equation, thresholds for decision boundaries, or architecture choices. The key takeaway is that this aspect improves the training, output, and accuracy of natural language-based learning models by including a high number of parameters that would otherwise be impossible to combine in other ways (e.g., in the human mind).

[0102] Consistent with some disclosed embodiments, a trained machine learning model may include multiple layers, each having a transformer-decoder architecture (as previously described and illustrated).

[0103] Consistent with some disclosed embodiments, the transformer decoder architecture may include at least one masked self-attention head and / or at least one feedforward network. As used herein, a self-attention head may refer to a subsequence (e.g., a head) representing different viewpoints or attention distributions. The self-attention head may perform a weighted summation on the elements of the input sequence, assigning different weights or importance to these positions based on their relevance to each other. For example, within a self-attention head, the input sequence may be transformed into three vectors: a query, a key, and a value. These vectors can be derived from the input sequence using a learned linear transformation. The query vector may represent the position being noticed, the key vector may represent all positions in the input sequence, and the value vector may store information or features associated with each position. The self-attention mechanism can then compute a weighted sum of the value vectors, where the weights are determined by the compatibility (or similarity) between the query vector and the key vector. Thus, the resulting weighted sum may represent the output of the self-attention head for a specific position. As used in this paper, a masked self-attention head can refer to a variant of self-attention where the machine learning model should not have access to future information during training or inference, and / or where the machine learning model needs to generate outputs sequentially. A masked self-attention head may achieve this by introducing a mask to prevent attention from noticing future locations. The mask could be, for example, an upper triangular matrix where each element above the main diagonal is set to a very large negative value (or negative infinity). This, in turn, causes the softmax operation in the attention mechanism to assign near-zero weights to future locations, effectively preventing their influence.

[0104] As used herein, a feedforward network (e.g., a fully connected network or a multilayer perceptron (MLP)) can refer to an artificial neural network comprising multiple interconnected layers of nodes, wherein multiple nodes (e.g., each node) in a given layer are connected to multiple nodes (e.g., each node) in a subsequent layer. In a feedforward network, information can flow unidirectionally from the input layer through one or more hidden layers to the output layer. Each node (e.g., a neuron or unit) in one or more hidden layers can apply an activation function to a weighted sum of the inputs received at that node from the previous layer. Furthermore, the weights associated with each connection can include learnable parameters of the network that can be adjusted during the training process.

[0105] Consistent with some publicly available embodiments, the docstring generation model can be fine-tuned based on at least one public web source or software repository (as previously described and illustrated).

[0106] Consistent with some publicly available embodiments, the docstring generation model (as previously described and illustrated) can be fine-tuned based on a training dataset built from examples within at least one public web source or software repository.

[0107] Consistent with some disclosed embodiments, identifying at least one of one or more candidate document strings can also be based on average log probability (as previously described and illustrated).

[0108] Consistent with some disclosed embodiments, document string generation models can be developed by applying training data, including annotated computer code, to a precursor model, which includes a machine learning model trained on natural language prompts (as previously described and illustrated).

[0109] In some embodiments, the method may further include training a machine learning model for generating computer code based on natural language input using training data, said training data including at least one identified docstring of output associated with at least a portion of one or more computer code samples. For example, the output of a docstring generation model associated with a corresponding computer code sample can be used as additional training data for a trained machine learning model for generating computer code based on natural language input. Therefore, this method can allow the docstring generation model to generate additional training data for another machine learning model, making the machine learning model more accurate due to the automatically generated training data, without requiring external data as additional training data. It is to be understood that combining the output of a docstring generation model (or another natural language-based learning model) with the training process of another machine learning model forms an unconventional and ungeneral arrangement that helps address the technical problem of improving the accuracy and effectiveness of another machine learning model without requiring additional external training data.

[0110] According to the disclosed embodiments, such as by including aspects of the embodiments described above, an exemplary system may include a memory storing at least one instruction and at least one processor configured to execute the instruction to perform a set of operations for generating natural language text based on computer code input. This set of operations may map to one or more steps of the method 200 described herein. Therefore, the system may be configured to access a document string generation model configured to generate document strings based on computer code input. The system may also be configured to receive one or more computer code samples. The system may further be configured to use the document string generation model and, based on the received computer code samples(s), generate candidate document strings representing natural language text associated with the computer code samples. In some embodiments, the system may also be configured to identify one or more candidate document strings representing the intent to provide one or more computer code samples(s). Furthermore, the system may be configured to output one or more identified document strings and one or more associated computer code samples(s) via a user interface. The system may also be configured to further train the document string generation model or another model based on the output.

[0111] According to another embodiment of this disclosure, a computer-readable medium includes instructions for performing the step of generating natural language text based on computer code input. The steps embodied in the instructions of the computer-readable medium may mirror one or more steps of the method 200 described herein. Therefore, these steps may be configured to access a document string generation model configured to generate document strings based on computer code input. These steps may also be configured to receive one or more computer code samples. The steps may also be configured to use the document string generation model and, based on the received computer code samples(s), generate candidate document strings representing natural language text, said candidate document strings being associated with the computer code samples. In some embodiments, these steps may also be configured to identify one or more candidate document strings representing the intent to provide one or more computer code samples(s). In some embodiments, the steps may also be configured to output one or more identified document strings and one or more associated computer code samples via a user interface. The system may also be configured to further train the document string generation model or another model based on the output.

[0112] This disclosure can be used to perform a range of code-related natural language processing tasks. Example tasks include summarizing (e.g., generating summaries of code to provide a high-level overview of its functionality), transformation (e.g., translating code comments and documentation into multiple languages, which can be useful for developers working with international teams or developing applications for users who speak different languages), code documentation (e.g., generating documentation of code, including descriptions of functions, variables, and classes, which can be used to help other developers understand the code and how it works), and question answering (e.g., answering questions about the code, such as "What does this function do?"). "or how to use this variable" This could be useful for developers trying to understand code written by others or for developers using legacy code, as well as for code completion (e.g., code completion based on natural language descriptions).

[0113] Using this disclosure, generating natural language from computer code input can have several applications in fields including software documentation, chatbots, and natural language processing. Example applications may include automated code documentation (e.g., automatically generating documentation for software code, which can help developers more easily understand how the code works and how to use it correctly), chatbots and virtual assistants (e.g., generating natural language responses for chatbots and virtual assistants, which can help make them more natural and engaging), code comments and annotations (e.g., automatically generating comments and annotations for code, thereby helping other developers more easily understand the code), technical writing (e.g., writing content based on code generation techniques, such as blog posts, articles, and white papers, which can help explain complex technical concepts in a more accessible and understandable way), code-to-speech (e.g., generating natural language audio descriptions of code, which may be useful for visually impaired developers who rely on audio descriptions to understand code), code summaries (e.g., generating natural language summaries of code, which would be useful for quickly understanding the purpose and functionality of a codebase), and automated error messages (e.g., generating natural language error messages that are easier for users to understand, which can help reduce frustration and improve user experience). Now refer to Figure 2 , Figure 4 and Figure 6-8 These figures illustrate exemplary embodiments of the present disclosure.

[0114] Consistent with the disclosed embodiments, Figure 2 The figure illustrates an exemplary method 200 for generating natural language text based on computer code input. Figure 2Any component step of the process shown or its constituent steps may be implemented using operating environment 700, system 800 (e.g., using at least one processor and at least one memory component) or any of the above components. Figure 2 The steps illustrated are exemplary, and in some embodiments, steps may be added, merged, split, copied, repeated (e.g., as part of a machine learning process), modified, executed sequentially, executed in parallel, and / or deleted.

[0115] like Figure 2 As illustrated, exemplary method 200 may include step 210 of accessing a docstring generation model configured to generate docstrings based on computer code input. Figure 2 As further illustrated, exemplary method 200 may also include step 220 of receiving one or more computer code samples. Also as Figure 2 As illustrated in the figure, the exemplary method 200 may further include step 230, where a document string generation model is used, and candidate document strings representing natural language text are generated based on one or more received computer code samples, the candidate document strings being associated with the computer code samples. Furthermore, as... Figure 2 As shown, the exemplary method 200 may include step 240, which identifies one or more candidate document strings representing the intent to provide one or more samples of computer code. Also as Figure 2 As shown, the exemplary method 200 may include step 250, whereby outputting (one or more) identified docstrings and one or more associated computer code samples via a user interface. Figure 2 The exemplary method 200 shown may also include step 260 of further training a document string generation model or another model based on the output.

[0116] Figure 4 The description of some embodiments of this disclosure is for implementation Figure 2A functional block diagram of an exemplary operating environment 400 for the method is provided. In some embodiments, the operating environment 400 may include a system 404, which includes at least one memory (not shown) storing instructions and at least one processor (not shown) configured to execute instructions to perform a set of operations for generating natural language text based on computer code input. System 404 may be an instance of system 500 and / or include features of system 500. The set of operations may map to one or more steps of the method 200 described herein. Thus, system 404 may be configured to access a document string generation model 406, which is configured to generate document strings based on computer code input. In some embodiments, document string generation model 406 may be trained using data from public web sources and / or repositories 414, appended natural language text 416, and / or strings 418 linked to one or more links. System 404 may also be configured to receive one or more computer code samples 402 as computer code input. System 404 can also be configured to use document string generation model 406 and generate one or more candidate document strings 407 representing natural language text, associated with computer code samples 402, based on one or more received computer code samples 402. System 404 can also be configured to verify each candidate document string 407 via verification module 410. In some embodiments, system 404 can also be configured to identify one or more candidate document strings that provide the intent of one or more received computer code samples 402 via identification module 408. Furthermore, system 404 can be configured to output output 412 via a user interface, which includes one or more identified document strings and one or more associated computer code samples. System 404 can also be configured to further train document string generation model 406 or another model 420 based on output 412.

[0117] Figure 6 The illustration depicts an exemplary method 600 for training a document string generation model, consistent with the disclosed embodiments. In some embodiments, the document string generation model can be trained such that it can be configured to generate natural language based on computer code input as discussed herein. Figure 6 Any component step of the process shown or its constituent steps may be implemented using operating environment 700, system 800 (e.g., using at least one processor and at least one memory component) or any of the above components. Figure 6The steps illustrated are exemplary, and in some embodiments, steps may be added, merged, split, copied, repeated (e.g., as part of a machine learning process), modified, executed sequentially, executed in parallel, and / or deleted.

[0118] like Figure 6 The exemplary method 600 for training a docstring generation model, illustrated in the figure, may include a step 610 of collecting and preparing data. Collecting and preparing data may include, for example, acquiring, accessing, or generating training data including computer code samples and corresponding docstrings; cleaning and preprocessing the training data (e.g., handling missing values, normalizing data, encoding category variables, removing special characters); and splitting the data into a training dataset and a validation dataset. Figure 6 As further illustrated, exemplary method 600 may also include step 620 of selecting a model type and architecture. Step 620 may include, for example, selecting an appropriate sequence-to-sequence model architecture (e.g., LSTM, transformer), defining the model architecture (e.g., encoder and decoder components), and setting hyperparameters (e.g., learning rate, regularization, batch size, maximum sequence length). Also as Figure 6 As illustrated, the exemplary method 600 may further include a step 630 of performing embedding and / or tokenization. Step 630 may include, for example, converting training data into a numerical representation, applying word embedding techniques (e.g., Word2Vec, GloVe) to capture semantic relationships, tokenizing text into sub-word units, and / or using a vocabulary mapping to tokenize the text. Figure 6 As further illustrated, exemplary method 600 may also include step 640 of training a document string generation model using the collected and prepared data. For example, step 640 may include initializing the document string generation model with random weights and / or biases, iterating through training data (e.g., performing forward pass, feeding the embeddings to the decoder to generate predicted document strings, calculating the loss / cross-entropy between the predicted and target document strings, and performing backward pass), and / or repeating the iterations until convergence or a predefined stopping criterion is met. Furthermore, as... Figure 6 As shown, the exemplary method 600 may include a step 650 of evaluating a trained docstring generation model. Step 650 may include, for example, evaluating the performance of the docstring generation model based on a validation dataset (e.g., calculating evaluation metrics such as accuracy, precision, BLEU score, ROUGE score, and / or analyzing the results and adjusting the model architecture). Also as Figure 6The exemplary method 600 shown may include a step 660 of fine-tuning a trained document string generation model (fine-tuning may be defined as previously described and illustrated). In some embodiments, the trained document string generation model is configured to perform operations related to... Figure 1 The operations discussed, or other model operations discussed in this article. For example... Figure 6 The exemplary method 600 shown may also include step 670 of deploying a trained docstring generation model. For example, step 670 may include deploying the docstring generation model to a production environment, monitoring the performance of the docstring generation model, updating the docstring generation model as needed, and / or using the docstring generation model to generate output based on a given input.

[0119] exist Figure 7 The diagram illustrates an exemplary operating environment for implementing various aspects of this disclosure. Figure 7 As illustrated, the exemplary operating environment 700 may include a computing device 702 in the form of a computer (e.g., a general-purpose computing device). In some embodiments, the computing device 702 may be associated with a user. Components of the computing device 702 may include, but are not limited to, various hardware components such as one or more processors 706, data storage devices 708, system memory 704, other hardware 710, and a system bus (not shown) that couples (e.g., communicatively coupled, physically coupled, and / or electrically coupled) various system components, enabling these components to travel to and from each other to transfer data. The system bus may be any of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus, also known as the Mezzanine bus.

[0120] Further reference Figure 7An exemplary embodiment of the operating environment 700 includes at least one computing device 702. The computing device 702 may be a single-processor or multi-processor computing device. The operating environment 700 may include one or more computing devices (e.g., multiple computing devices 702) in a given computer system, which may be clustered, part of a local area network (LAN), part of a wide area network (WAN), client-server networked, peer-to-peer networked within a cloud, or otherwise communicably linked. The computer system may include individual machines or a group of cooperating machines. A given computing device 702 may be configured for end-user use, for example, having applications, for administrators, as a server, as a distributed processing node, as a dedicated processing device, or otherwise configured to train and / or use machine learning models. In some embodiments, multiple computing devices 702 (e.g., a network of GPUs) may be configured to train machine learning models.

[0121] One or more users can interact with a computer system comprising one or more computing devices 702 by using a display, keyboard, mouse, microphone, touchpad, camera, sensors (e.g., touch sensors) and other input / output devices 718 via typed text, touch, voice, motion, computer vision, gestures, and / or other forms of input / output. Input / output devices 718 may be detachable (e.g., a connectable mouse or keyboard) or essential to computing device 702 (e.g., a touchscreen, a built-in microphone). User interface 712 may support interaction between the embodiment and one or more users. User interface 712 may include one or more of a command-line interface, graphical user interface (GUI), natural user interface (NUI), voice command interface, and / or other user interface (UI) presentations, which may be presented as different options or may be integrated. Users can enter commands and information through the user interface or other input devices such as tablet devices, electronic digitizers, microphones, keyboards, and / or pointing devices, commonly referred to as mice, trackballs, or touchpads. Other input devices may include joysticks, game controllers, disc-shaped satellite antennas, scanners, etc. Additionally, voice input, gesture input using hands or fingers, or other NUIs can also be used with appropriate input devices such as microphones, cameras, tablet devices, touchpads, gloves, or other sensors. These and other input devices are typically connected to the processing unit via a user input interface coupled to the system bus, but can also be connected via other interfaces and bus architectures such as parallel ports, game ports, or Universal Serial Bus (USB) . Monitors or other types of display devices are also connected to the system bus via interfaces such as video interfaces. Monitors can also be integrated with touchscreen panels or the like. Note that monitors and / or touchscreen panels can be physically coupled to a housing in which a computing device is incorporated, such as in a tablet PC. Additionally, computers such as computing devices can also include other peripheral output devices such as speakers and printers, which can be connected via output peripheral interfaces or the like.

[0122] Based on input received from user interface 712 and / or from one or more networks 716, one or more application programming interface (API) calls can be made between input / output device 718 and computing device 702. As used throughout this document, "based on" can mean established or set up when it is used, modified, affected, caused, dependent on, or otherwise derived from it. In some embodiments, API calls can be configured for a specific API and can be interpreted and / or translated into API calls configured for different APIs. As used herein, API can refer to a defined (e.g., according to an API specification) interface or connection between computers or computer programs.

[0123] System administrators, network administrators, software developers, engineers, and end users are each specific types of users. Automated agents, scripts, playback software, etc., acting on behalf of one or more people can also constitute users. Storage devices and / or networking devices may be considered peripheral equipment in some embodiments, and in other embodiments may be considered part of a system including one or more computing devices 702, depending on their separability from processor(s) 706. Figure 7 Other computerized devices and / or systems, not shown, may technically interact with computing device 702 or another system via one or more connections through network interface 714 to network 716. Network interface 714 may include network interface equipment, such as physical network interface controller (NIC) or virtual network interface (VIF).

[0124] Computing device 702 includes at least one logic processor 706. The at least one logic processor 706 may include circuitry and transistors configured to execute instructions from memory (e.g., memory 704). For example, the at least one logic processor 706 may include one or more central processing units (CPUs), arithmetic logic units (ALUs), floating-point units (FPUs), and / or graphics processing units (GPUs). Like other suitable devices, computing device 702 also includes one or more computer-readable storage media, which may include, but are not limited to, memory 704 and data storage device 708. In some embodiments, memory 704 and data storage device 708 may be part of a single memory component. The one or more computer-readable storage media may be of different physical types. The media may be volatile memory, non-volatile memory, fixed-in-place media, removable media, magnetic media, optical media, solid-state media, and / or other types of physically persistent storage media (as opposed to merely propagated signals). Specifically, when inserted into or otherwise installed relative to one or more computing devices 702, a configured medium 720, such as a portable (i.e., external) hard disk drive, optical disc (CD), digital versatile optical disc (DVD), memory stick, or other removable non-volatile storage medium, can functionally become a technical part of the computer system, making it easily accessible for interaction with and use by one or more processors 706. The removable configured medium 720 is an example of a computer-readable storage medium. Some other examples of computer-readable storage media include built-in random access memory (RAM), read-only memory (ROM), hard disks, and other memory storage devices (e.g., memory 704) that are not easily removable by the user.

[0125] The configured medium 720 may be configured with instructions (e.g., binary instructions) executable by processor 706; "executable" is used in a broad sense herein and includes machine code, interpreted code, bytecode, compiled code, and / or any other code configured to run on a machine (including physical machines or virtualized computing instances (e.g., virtual machines or containers)). The configured medium 720 may also be configured with data created, modified, referenced, and / or otherwise used for technical effects by the execution of the instructions. Instructions and data may configure the memory or other storage medium in which they reside such that the instructions and data may also configure the computing device when the memory or other computer-readable storage medium is a functional part of a given computing device.

[0126] While embodiments may be described as software instructions executed by one or more processors in a computing device (e.g., a general-purpose computer, server, or cluster), such a description is not intended to exhaustively describe all possible embodiments. Those skilled in the art will understand that the same or similar functionality can often be implemented, in whole or in part, directly in hardware logic to provide the same or similar technical effects. Alternatively, or in addition to software implementations, the technical functionality described herein may be performed at least in part by one or more hardware logic components. For example, and without excluding other implementations, embodiments may include other hardware logic components 710, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and similar components. For example, components of embodiments may be grouped into interactive functional modules based on their inputs, outputs, and / or their technical effects.

[0127] In addition to processor(s) 706, memory 704, data storage device 708, and screen / display, the operating environment 700 may also include other hardware 710, such as batteries, buses, power supplies, wired and wireless network interface cards. The terms "screen" and "display" are used interchangeably herein. A display may include one or more touchscreens, a screen that responds to input from a pen or tablet device, or a screen operated solely for output. In some embodiments, other input / output devices 718, such as human user input / output devices (screen, keyboard, mouse, tablet device, microphone, speaker, motion sensor, etc.), will be present in operative communication with one or more processors 706 and memory.

[0128] In some embodiments, the system includes a plurality of computing devices 702 connected via one or more networks 716. The network interface equipment can use components (which may be part of network interface 714) to provide access to one or more networks 716, such as packet-switched network interface cards, wireless transceivers, or telephone network interfaces that may be present in a given computer system. However, embodiments may also transmit technical data and / or technical instructions via direct memory access, removable non-volatile media, or other information storage-retrieval and / or transmission methods.

[0129] Computing device 702 can operate in a networked or cloud computing environment using a logical connection to one or more remote devices, such as a remote computer (e.g., another computing device 702) (e.g., using one or more networks 716). The remote computer may include one or more of a personal computer, server, router, network PC, or peer-to-peer device or other common network node, and may include any or all of the elements described above relative to a computer. The logical connection may include one or more LANs, WANs, and / or the Internet.

[0130] When used in a networked or cloud computing environment, computing device 702 can be connected to a public or private network via a network interface or adapter. In some embodiments, a modem or other communication connection device can be used to establish communication over the network. The modem can be internal or external, and it can be connected to a system bus via a network interface or other suitable mechanism. Wireless networking components, such as a wireless networking component including an interface and an antenna, can be coupled to the network via a suitable device such as an access point or peer computer. In a networked environment, program modules or portions thereof depicted relative to a computer can be stored in a remote memory storage device. It will be understood that the network connections shown are exemplary, and other means of establishing communication links between computers can be used.

[0131] Computing device 702 may generally include any of a variety of computer-readable media. Computer-readable media can be any available medium accessible to a computer, and includes both volatile and non-volatile media, as well as removable and non-removable media, excluding transmitted signals. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, DVD or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information (e.g., program modules, data for machine learning models, and / or the machine learning model itself) and is accessible to a computer. Communication media may embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transport mechanisms, and includes any information delivery medium. The term "modulated data signal" means a signal whose characteristics are set or altered in such a way as to the information encoded in the signal. By way of example and not limitation, communication media include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media. Any combination of the above may also be included within the scope of computer-readable media. Computer-readable media can be embodied in computer program products, such as software (e.g., including program modules) stored on computer-readable storage media (such as non-transitory computer-readable storage media).

[0132] Data storage device 708 or system memory includes computer storage media in the form of volatile and / or non-volatile memory such as ROM and RAM. The Basic Input / Output System (BIOS) may be stored in ROM, containing basic routines that help transfer information between components within the computer, such as during startup. RAM may contain data and / or program modules that are readily accessible to and / or currently being operated by the processing unit. By way of example and not limitation, the data storage device holds the operating system, applications, and other program modules and program data.

[0133] Data storage device 708 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, the data storage device may be a hard disk drive that reads from or writes to a non-removable, non-volatile magnetic medium, a disk drive that reads from or writes to a removable, non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable, non-volatile optical disk such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital multifunction disks, digital videotapes, solid-state RAM, solid-state ROM, etc.

[0134] Exemplary disclosed embodiments include systems, methods, and computer-readable media for generating computer code or natural language using language models trained on computer code. For example, in some embodiments, and as such... Figure 7 As illustrated, the operating environment 700 may include at least one computing device 702, which includes at least one processor 706, at least one memory 704, at least one data storage device 708, and / or the components described above. Figure 7 Any other components discussed.

[0135] Figure 8 This is a block diagram illustrating an exemplary machine learning platform for implementing various aspects of this disclosure, according to some embodiments thereof.

[0136] System 800 may include a data input engine 810, which may further include a data retrieval engine 804 and a data transformation engine 806. The data retrieval engine 804 may be configured to access, retrieve, interpret, request, or receive data that may be adjusted, reformatted, or altered (e.g., interpreted by other engines such as the data input engine 810). For example, the data retrieval engine 804 may use an API to request data from a remote source. The data input engine 810 may be configured to access, interpret, request, format, reformatted, or receive input data from one or more data sources 802. For example, the data input engine 810 may be configured to use the data transformation engine 806 to perform data reconfiguration or other changes, such as data dimensionality reduction. One or more data sources 802 may be present at one or more memories 704 and / or data storage devices 708. In some embodiments, one or more data sources 802 may be associated with a single entity (e.g., an organization) or multiple entities. One or more data sources 802 may include one or more of the following: training data 802a (e.g., input data for feeding a machine learning model as part of one or more training processes), validation data 802b (e.g., data that at least one processor can compare against the model output, such as data for determining the quality of the model output), and / or reference data 802c. In some embodiments, the data input engine 810 may be implemented using at least one computing device (e.g., computing device 702). For example, data from data source 802 may be obtained through one or more I / O devices and / or network interfaces. Furthermore, data may be stored (e.g., during the execution of one or more operations) in a suitable storage device or system memory. The data input engine 810 may also be configured to interact with a data storage device 708, which may be implemented on a computing device that stores data in a storage device or system memory. System 800 may include a characterization engine 820. The featureization engine 820 may include a feature annotation and labeling engine 812 (e.g., configured to annotate or label features from a model or data that can be extracted by a feature extraction engine 814), a feature extraction engine 814 (e.g., configured to extract one or more features from a model or data), and / or a feature scaling and selection engine 816. The feature scaling and selection engine 816 may be configured to determine, select, restrict, constrain, link, or define features (e.g., AI features) for use with the AI ​​model. System 800 may also include a machine learning (ML) modeling engine 830, which may be configured to perform one or more operations on the machine learning model (e.g., model training, model reconfiguration, model validation, model testing), such as those described in the processes described herein.For example, the ML modeling engine 830 can perform operations to train a machine learning model, such as adding, removing, or modifying model parameters. Training of the machine learning model can be supervised, semi-supervised, or unsupervised. In some embodiments, training of the machine learning model can include multiple rounds of a machine learning model process (e.g., a training process) or the passing of data (e.g., training data 802a). In some embodiments, different rounds can have different degrees of supervision (e.g., supervised, semi-supervised, or unsupervised). Data input to the model to train the model can include input data (e.g., as described above) and / or data previously output from the model (e.g., forming a cyclical learning feedback). Model parameters can include one or more of the following: seed values, model nodes, model layers, algorithms, functions, model connections (e.g., between other model parameters or between models), model constraints, or any other digital components that affect the model output. Model connections can include or represent relationships between model parameters and / or models, which can be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters and the relationships between model parameters discussed herein are cognitively infeasible for human minds to maintain or use. Without limiting the disclosed embodiments in any way, the machine learning model may include millions, trillions, or even billions of model parameters. The ML modeling engine 830 may include a model selector engine 832 (e.g., configured to select a model from a plurality of models, such as based on input data), a parameter selector engine 834 (e.g., configured to add, remove, and / or change one or more parameters of a model), and / or a model generation engine 836 (e.g., configured to generate one or more machine learning models, such as based on model input data, model output data, comparison data, and / or validation data). Similar to the data input engine 810, the featureization engine 820 may be implemented on a computing device. In some embodiments, the model selector engine 832 may be configured to receive input and / or transmit output to an ML algorithm database 890 (e.g., a data storage device 708). Similarly, the featureization engine 820 may utilize a storage device or system memory to store data and may utilize one or more I / O devices or network interfaces to transmit or receive data. The ML algorithm database 890 (or other data storage device 708) can store one or more machine learning models, any of which can be fully trained, partially trained or untrained.Machine learning models can be, or include, (e.g., in the case of meta-models) one or more of the following: statistical models, algorithms, neural networks (NNs), convolutional neural networks (CNNs), generative neural networks (GNNs), Word2Vec models, bag-of-words models, term frequency-inverse document frequency (TF-IDF) models, GPT (Generative Pretrained Transformer) models (or other autoregressive models), policy nearest neighbor (PPO) models, nearest neighbor models (e.g., k-nearest neighbor models), linear regression models, k-means clustering models, Q-learning models, temporal difference (TD) models, deep adversarial network models, or any other type of model further described herein.

[0137] System 800 may further include a predictive output generation engine 840, an output validation engine 850 (e.g., configured to apply validation data to machine learning model outputs), a feedback engine 870 (e.g., configured to apply feedback from users and / or machines to the model), and a model refinement engine 860 (e.g., configured to update or reconfigure the model). In some embodiments, feedback engine 870 may receive inputs and / or transmit outputs (e.g., outputs from trained, partially trained, or untrained models) to outcome metric database 880. Outcome metric database 880 may be configured to store outputs from one or more models and may also be configured to associate outputs with one or more models. In some embodiments, outcome metric database 880 or other devices (e.g., model refinement engine 860 or feedback engine 870) may be configured to correlate outputs, detect trends in output data, and / or infer changes in inputs or model parameters to elicit specific model outputs or specific types of model outputs. In some embodiments, model refinement engine 860 may receive outputs from predictive output generation engine 840 or output validation engine 850. In some embodiments, the model refinement engine 860 may transmit the received output to the featureization engine 820 or the ML modeling engine 830 in one or more iteration cycles.

[0138] Any or each engine in system 800 may be a module (e.g., a program module), which may be an encapsulated functional hardware unit designed to be used in conjunction with other components or part of a program that perform a specific function (e.g., its associated functions). Any or each of these modules may be implemented using computing devices. In some embodiments, the functionality of system 800 may be split across multiple computing devices to allow distributed processing of data, which can increase output speed and reduce computational load on individual devices. In some embodiments, system 800 may use load balancing to maintain a stable resource load (e.g., processing load, memory load, or bandwidth load) across multiple computing devices and reduce the risk of computing devices or connections becoming overloaded. In these or other embodiments, different components may communicate through one or more I / O devices and / or network interfaces.

[0139] System 800 can be associated with different use domains or fields. The description of embodiments involving specific domains (such as natural language processing or language modeling) is not intended to limit the disclosed embodiments to those specific domains, and embodiments consistent with this disclosure can be applied to any domain utilizing predictive modeling based on available data.

[0140] The embodiments may be further described using the following terms: 1. A computer-implemented method, comprising: Receive a document string representing natural language text, which specifies the result of numeric programming; Using a trained machine learning model and based on the document string, generate one or more computer code samples configured to produce corresponding candidate results; This causes each of one or more computer code samples to be executed; Based on execution, the identifier is at least one of a sample of computer code configured to produce a specific candidate result associated with the result of digital programming; and Perform at least one of the following: Output at least one identified computer code sample via the user interface; Compile at least one of the identified computer code samples; Transmit at least one identified sample of computer code to the recipient device; Store at least one identified computer code; or Re-execute at least one of the identified computer code samples.

[0141] 2. The method according to Clause 1 further includes verifying each of one or more executed computer code samples, wherein verification includes calculating a functional correctness score for each of the one or more executed computer code samples, wherein at least one computer code sample is identified based on the functional correctness score.

[0142] 3. The method according to Clause 2, wherein the trained machine learning model is fine-tuned based on verified samples of computer code.

[0143] 4. The method according to any one of Clauses 1 to 3 further includes verifying each of the one or more generated computer code samples, wherein verification includes evaluating each of the one or more generated computer code samples based on at least one unit test.

[0144] 5. The method according to Clause 4, wherein identifying at least one of the computer code samples is also based on verification.

[0145] 6. The method according to Clause 4 or 5, wherein the trained machine learning model is fine-tuned based on verified samples of computer code.

[0146] 7. The method according to any one of Clauses 4 to 6, wherein the verification further includes evaluating each of one or more generated computer code samples based on a threshold associated with at least one unit test.

[0147] 8. The method according to any one of Clauses 1 to 7, wherein each of the one or more generated computer code samples is associated with at least one text token.

[0148] 9. The method according to Clause 8, wherein each of the one or more generated computer code samples is also associated with at least one space token.

[0149] 10. The method according to any one of clauses 1 to 9 further includes outputting a specific candidate result of at least one identified computer code sample via a user interface.

[0150] 11. The method according to any one of clauses 1 to 10, wherein the trained machine learning model is fine-tuned based on at least one of public web sources or software repositories.

[0151] 12. The method according to Clause 11, wherein the trained machine learning model is fine-tuned based on a set of training questions constructed from examples within at least one public web source or software repository.

[0152] 13. The method according to any one of clauses 1 to 12, wherein identifying at least one of the computer code samples is further based on the average log probability.

[0153] 14. The method according to any one of Clauses 1 to 13, wherein at least a portion of one or more computer code samples is executed in a sandbox computing environment.

[0154] 15. The method according to any one of clauses 1 to 14 further includes outputting, via a user interface, the definition of a function, method, class, or module associated with at least one identified sample of computer code output.

[0155] 16. The method according to any one of Clauses 1 to 15, wherein the trained machine learning model is developed by applying training data including annotated computer code to a precursor model, the precursor model comprising a machine learning model trained based on natural language prompts.

[0156] 17. The method according to any one of Clauses 1 to 16, wherein the trained machine learning model is further trained using the execution results of each of the one or more computer code samples being executed.

[0157] 18. The method according to any one of clauses 1 to 17, wherein the trained machine learning model comprises a plurality of layers, at least one of which has a transformer decoder architecture.

[0158] 19. A system comprising: At least one memory for storing instructions; and At least one processor is configured to execute instructions to cause the system to perform the method according to any one of clauses 1 to 18.

[0159] 20. A computer-readable medium comprising instructions executable by one or more processors to perform the method according to any one of clauses 1 to 18.

[0160] 21. A computer-implemented method, comprising: Access is provided to a docstring generation model configured to generate docstrings from computer code. Receive one or more computer code samples; The document string generation model is used to generate one or more candidate document strings representing natural language text based on one or more received computer code samples, each of the one or more candidate document strings being associated with at least a portion of the one or more computer code samples; At least one of one or more candidate document strings that identify the intent to provide at least a portion of one or more computer code samples; and Output at least one identified docstring and at least a portion of one or more computer code samples via a user interface.

[0161] 22. The method according to Clause 21, wherein the document string generation model further generates a similarity between the intent and the attached natural language text.

[0162] 23. The method according to Clause 22, wherein at least one identified docstring associated with at least a portion of one or more computer code samples is used to further train the docstring generation model.

[0163] 24. The method according to any one of clauses 21 to 23, wherein the document string generation model is trained using concatenated strings, each concatenated string comprising at least two of a function signature, a reference solution, or a document string.

[0164] 25. The method according to Item 24, wherein the document string generation model is further trained by minimizing the negative log-likelihood associated with the document string in at least one linked string.

[0165] 26. The method according to any one of clauses 21 to 25, wherein identifying at least one of one or more candidate document strings is based on a correctness score calculated for at least one candidate document string.

[0166] 27. The method according to any one of clauses 21 to 26 further includes verifying each of one or more candidate document strings, wherein verification includes determining a correctness score for each of the one or more candidate document strings, wherein identifying at least one of the one or more candidate document strings is based on the determined correctness score.

[0167] 28. The method according to any one of clauses 21 to 27, wherein the document string generation model is fine-tuned based on validated candidate document strings.

[0168] 29. The method according to Clause 27 or 28 further includes ranking one or more candidate document strings based on the determined correctness score, wherein identifying one of the one or more candidate document strings is based on selecting the top k candidate document strings.

[0169] 30. The method according to any one of clauses 21 to 29, wherein the document string generation model is a trained machine learning model.

[0170] 31. The method according to Clause 30, wherein the trained machine learning model has between 10 billion and 14 billion parameters.

[0171] 32. The method according to clause 30 or 31, wherein the trained machine learning model comprises multiple layers, at least one of which has a transformer decoder architecture.

[0172] 33. The method according to Clause 32, wherein the transformer decoder architecture includes at least one of a masked self-attention head or a feedforward network.

[0173] 34. The method according to any one of clauses 21 to 33, wherein the document string generation model is fine-tuned based on at least one public web source or software repository.

[0174] 35. The method according to Clause 34, wherein the docstring generation model is fine-tuned based on a training dataset constructed from examples within the at least one public web source or software repository.

[0175] 36. The method according to any one of clauses 21 to 35, wherein identifying at least one of one or more candidate document strings is also based on the average log probability.

[0176] 37. The method according to any one of clauses 21 to 36, wherein the DocString Generation Model is developed by applying training data including annotated computer code to a precursor model, the precursor model comprising a machine learning model trained based on natural language prompts.

[0177] 38. The method according to any one of clauses 21 to 37 further includes using training data to train a machine learning model for generating computer code based on natural language input, said training data including at least one identified document string of output associated with at least a portion of one or more computer code samples.

[0178] 39. A system comprising: At least one memory for storing instructions; and At least one processor is configured to execute instructions to cause the system to perform the method according to any one of clauses 21 to 38.

[0179] 40. A computer-readable medium comprising instructions executable by one or more processors to perform the method according to any one of clauses 21 to 38.

[0180] As used herein, unless otherwise specifically stated, the term "or" covers all possible combinations except for those that are not feasible. For example, if it is stated that a component may include A or B, then the component may include A, or B, or A and B, unless otherwise specifically stated or not feasible. As a second example, if it is stated that a component may include A, B, or C, then the component may include A, or B, or C, or A and B, or A and C, or B and C, or A and B and C, unless otherwise specifically stated or not feasible.

[0181] The above-described exemplary embodiments are illustrated by flowcharts or block diagrams of methods, apparatus (systems), and computer program products. It will be understood that each block in a flowchart or block diagram, and combinations of blocks in a flowchart or block diagram, can be implemented by a computer program product or instructions on a computer program product. These computer program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, executable via the processor of the computer or other programmable data processing apparatus, create components for implementing the functions / actions specified in one or more blocks of the flowchart or block diagram.

[0182] These computer program instructions may also be stored in a computer-readable medium that directs one or more hardware processors, other programmable data processing devices, or other equipment of a computer to function in a particular manner, such that the instructions stored in the computer-readable medium form an article of manufacture, including instructions that implement functions / actions specified in one or more blocks of a flowchart or block diagram.

[0183] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed (e.g., executed) on the computer, other programmable apparatus or other device, thereby producing a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide a process for implementing the functions / actions specified in one or more boxes of a flowchart or block diagram.

[0184] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a non-transitory computer-readable storage medium. In the context of this document, a computer-readable storage medium can be any tangible medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

[0185] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, IR, or any suitable combination thereof.

[0186] The computer program code used to perform the operation can, for example, be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network (including LAN or WAN), or it can be connected to an external computer (e.g., via the internet through an internet service provider).

[0187] The flowcharts and block diagrams in the figures illustrate examples of the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each box in a flowchart or block diagram may represent a code module, code segment, or portion of code, which includes one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions indicated in the boxes may occur in a different order than those shown in the figures. For example, two boxes shown consecutively may actually execute substantially concurrently, or these boxes may sometimes execute in reverse order, depending on the functionality involved. It will also be noted that each box in a block diagram or flowchart, and combinations of boxes in a block diagram or flowchart, can be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified functions or actions.

[0188] It should be understood that the described embodiments are not mutually exclusive, and elements, components, materials, or steps combined with one example embodiment may be combined with other embodiments in a suitable manner, or eliminated from other embodiments, to achieve a desired design goal.

[0189] In the foregoing description, numerous specific details have been described with reference to which may vary depending on the implementation. Certain adaptations and modifications can be made to the described embodiments. Other embodiments will be apparent to those skilled in the art in light of the specification and practice of the invention disclosed herein. It is intended that this specification and examples be considered exemplary only. It is also intended that the sequence of steps shown in the figures be for illustrative purposes only and is not intended to be limited to any particular sequence of steps. Therefore, those skilled in the art will appreciate that these steps can be performed in different orders while implementing the same method.

Claims

1. A computer-implemented method, comprising: Receive a document string representing natural language text, which specifies the result of numeric programming; Using a trained machine learning model and based on the document string, generate one or more computer code samples configured to produce corresponding candidate results; This enables each of the one or more computer code samples to be executed in a test environment associated with the trained machine learning model; Based on the results of the test environment, identify at least one of the computer code samples that produce a specific candidate result associated with the digital programming result; The trained machine learning model is used to generate natural language text associated with at least one identified computer code sample; as well as The system outputs at least one identified computer code sample and the natural language text associated with the at least one identified computer code sample via a user interface.

2. The method of claim 1, further comprising verifying each of the one or more executed computer code samples, wherein the verification includes calculating a functional correctness score for each of the one or more executed computer code samples, wherein identifying at least one of the computer code samples is based on the functional correctness score.

3. The method of claim 2, wherein the trained machine learning model is fine-tuned based on validated computer code samples.

4. The method of any one of claims 1 to 3, further comprising verifying each of the one or more generated computer code samples, wherein the verification includes evaluating each of the one or more generated computer code samples based on at least one unit test generated by the trained machine learning model.

5. The method of claim 4, wherein identifying at least one of the computer code samples is further based on verification.

6. The method of claim 4 or 5, wherein the trained machine learning model is fine-tuned based on the verified computer code sample.

7. The method of any one of claims 4 to 6, wherein the verification further comprises evaluating each of the one or more generated computer code samples based on a threshold associated with the at least one unit test.

8. The method according to any one of claims 1 to 7, wherein each of the one or more generated computer code samples is associated with at least one text token.

9. The method of claim 8, wherein each of the one or more generated computer code samples is further associated with at least one space token.

10. The method according to any one of claims 1 to 9, further comprising outputting specific candidate results of the at least one identified computer code sample via the user interface.

11. The method according to any one of claims 1 to 10, wherein the trained machine learning model is fine-tuned based on at least one of public web sources or software repositories.

12. The method of claim 11, wherein the trained machine learning model is fine-tuned based on a set of training questions constructed from examples within at least one public web source or software repository.

13. The method according to any one of claims 1 to 12, wherein identifying at least one of the computer code samples is further based on the average logarithmic probability.

14. The method according to any one of claims 1 to 13, further comprising: Compile at least one of the identified computer code samples; Transmit the at least one identified computer code sample to the receiving device; Store at least one of the identified computer code samples; as well as Re-execute at least one of the identified computer code samples.

15. The method according to any one of claims 1 to 14, wherein the natural language text associated with the at least one identified computer code sample includes definitions of functions, methods, classes, or modules associated with the output of the at least one identified computer code sample.

16. The method of any one of claims 1 to 15, wherein the trained machine learning model is developed by applying training data comprising annotated computer code to a precursor model, the precursor model comprising a machine learning model trained based on natural language prompts.

17. The method of any one of claims 1 to 16, wherein the trained machine learning model generates training data based on the results of the execution, wherein the generated training data is used to further train the trained machine learning model.

18. The method according to any one of claims 1 to 17, wherein the trained machine learning model comprises a plurality of layers, at least one of the layers having a transformer-decoder architecture.

19. A system comprising: At least one memory for storing instructions; At least one processor is configured to execute instructions to perform the method according to any one of claims 1 to 18.

20. A computer-readable medium comprising instructions executable by one or more processors to perform the method according to any one of claims 1 to 18.

Images