Supervised training data generation

Abstract

Implementations relate to supervised training data. Synthetic intermediate representations of source code may be synthesized and translated using a rules-based translator to generate synthetic source code snippets in a high-level programming language. A translation machine learning (ML) model may be trained using pairs of (i) a respective synthetic intermediate representation, and (ii) a corresponding synthetic source code snippet in the high-level programming language. The now partially trained translation ML model may be applied to reference source code snippets in the high-level programming language to generate next stage intermediate representations. After filtering deficient next stage intermediate representations, remaining next stage intermediate representations may be translated using the rules-based translator to generate next stage source code snippets in the high-level programming language. Additional supervised training may be performed, e.g., of the translation ML model or another ML model using training data that includes at least the next stage source code snippets.

Description

SUPERVISED TRAINING DATA GENERATIONBackground

[0001] Computer software programming often requires developers to read and / or write source code (i.e., to program) in a specific high-level programming language, e.g. Java, C++, C, Python, etc. Each high-level programming language has its own strengths, weaknesses, nuances, idiosyncrasies, etc. Most programmers obtain at least a superficial understanding of multiple high-level programming languages but only master a few. Consequently, each high-level programming language tends to have its own talent pool. Language models such as transformer networks have become increasingly popular for translating between programming languages. Training a machine learning model to translate between different programming languages historically has required and / or benefited from supervised training data. This supervised training data may include pairs of semantically equivalent source code examples (referred to herein as “snippets”) in different languages. However, curating a supervised dataset can be expensive both in terms of time and cost.Summary

[0002] Implementations are described herein for leveraging data synthesis to reduce the costs associated with curating supervised training data for training a programming language translation model. More particularly, but not exclusively, implementations are described herein for iteratively and / or recursively generating, with little to no human intervention (akin to reinforcement learning), training data at distributions that are increasingly reflective of real- world source code. Later stages of training data may be used to effectively train translation machine learning models such as transformers to translate between programming languages.

[0003] In some implementations, a method may be implemented by one or more processors and may include: synthesizing a plurality of synthetic intermediate representations of source code snippets; translating the plurality of synthetic intermediate representations using a rules-based translator to generate a plurality of synthetic source code snippets in a high-level programming language; performing supervised training of a translation machine learning model using pairs of training data, each pair of training data including: a respective synthetic intermediate representation of the plurality of synthetic intermediate representations of source code snippets, and a corresponding synthetic source code snippet of the plurality of synthetic source codesnippets in the high-level programming language; using the trained translation machine learning model to translate a plurality of reference source code snippets in the high-level programming language to generate a superset of next stage intermediate representations of source code snippets; based on one or more criteria, filtering a subset of next stage intermediate representations from the superset of next stage intermediate representations of source code snippets; translating the remaining next stage intermediate representations using the rules-based translator to generate a plurality of next stage source code snippets in the high-level programming language; and performing additional supervised training of the translation machine learning model or a different translation machine learning model using additional training data that includes at least the next stage source code snippets of the plurality of next stage source code snippets in the high-level programming language.

[0004] In various implementations, the translation machine learning model may be a transformer model or a large language model. In various implementations, the additional supervised training may include training the different translation machine learning model to translate source code directly between the high-level programming language and another high- level programming language.

[0005] In various implementations, the superset of next stage intermediate representations may include a superset of syntactically constrained pseudocode snippets. In various implementations, the one or more criteria may include detection of one or more syntactic and / or semantic errors.

[0006] In various implementations, the supervised training includes a denoising objective. In various implementations, the synthesizing may include randomly generating variable names for inclusion in the synthetic intermediate representations of source code snippets.

[0007] In addition, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

[0008] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.Brief Description of the Drawings

[0009] Fig. 1 schematically depicts an example environment in which selected aspects of the present disclosure may be implemented, in accordance with various implementations.

[0010] Fig. 2 and Fig. 3 schematically depict an example of how various components configured with selected aspects of the present disclosure may carry out techniques described herein, in accordance with various implementations.

[0011] Fig. 4 depicts a flowchart illustrating an example method for practicing selected aspects of the present disclosure.

[0012] Fig. 5 illustrates an example architecture of a computing device.Detailed Description

[0013] Implementations are described herein for leveraging data synthesis to reduce the costs associated with curating supervised training data for training a programming language translation model. More particularly, but not exclusively, implementations are described herein for iteratively and / or recursively generating, with little to no human intervention (akin to reinforcement learning), training data at distributions that are increasingly reflective of real- world source code. The later stages of training data may be used to effectively train translation machine learning models such as transformers to translate between high-level programming languages. In various implementations, the training data may include training pairs (or more generally, tuples) that each include a source code snippet in a high-level programming language and an intermediate representation of the source code snippet.

[0014] In some implementations, a plurality of synthetic intermediate representations of source code may be generated. These synthetic intermediate representations may take various forms, such as syntactically constrained pseudocode, graphs (e.g., unified modeling language), flow charts, and so forth. Synthetic intermediate representations may be generated manually or programmatically. In the latter case, the synthetic intermediate representations may be generated randomly and / or stochastically, e.g., based on various rules or heuristics that can be manually defined, learned over time, etc. In various implementations, this initial set of synthetic intermediate representations may be translated, e.g., using a rules-based translator, into synthetic source code snippets in a high-level programming language, such as C, C++, Python, JavaScript, Java, Go, Kotlin, PHP, C#, Swift, Ruby, Perl, etc.

[0015] The initial set of synthetic intermediate representations may be of low quality. Variables may be assigned random names (e.g., strings of numbers and / or characters) that do not resemble human-interpretable variable names typically used by programmers. Logic implemented by these synthetic intermediate representations, while syntactically correct, may not be useful or sensical. Consequently, the initial set of synthetic source code snippets also may be of low quality.

[0016] Initial training data may be formed by pairing the synthetic intermediate representations with corresponding synthetic source code snippets. These training pairs may be used to perform supervised training (e.g., pretraining or bootstrapping) of a translation machine learning model, e.g., using techniques such as gradient descent, back propagation, etc. This translation machine learning model may take various forms, such as various types of transformers that are sometimes used as large language models (LLMs). These may include, for instance, PaLM, BARD, BERT, LaMDA, Meena, GPT, and / or any other LLM, such as any other LLM that is encoder-only based, decoder-only based, sequence-to-sequence based and that optionally includes an attention mechanism or other memory. Other types of machine learning models may be trained as well, such as recurrent neural networks, long short-term memory (LSTM) networks, gated recurrent unit (GRU) networks, and so forth. In some instances, the model selected may have less parameters than is typically used for an LLM, as the task to be performed using the model — translating source code — may be more constrained than responding to natural language requests.

[0017] The partially trained (e.g., pretrained) translation machine learning model may then be applied to reference source code en masse to generate what will be referred to herein as “next stage” intermediate representations of source code. For instance, publicly available repositories of reference source code snippets may be processed using the partially trained translation machine learning model to generate a superset of next stage intermediate representations (e.g., syntactically constrained pseudocode) of the reference source code snippets.

[0018] As implied by their name, the next stage intermediate representations may be higher quality than the initial intermediate representations because they are generated from real world reference source code snippets, and therefore are more reflective of distributions found in the real-world reference source code. Nevertheless, because the translation machine learning model is only partially trained, these next stage intermediate representations may still include any number of syntactic and / or semantic errors. Accordingly, in various implementations, various criteria, such as detected syntactic and / or semantic errors, may be used to programmaticallyfilter a subset of the next stage intermediate representations from the superset. For instance, any intermediate representations that are incapable of compilation (e.g., using a compiler designed to parse and / or compile syntactically constrained pseudocode) may be filtered out.

[0019] In many cases only a small percentage of next stage intermediate representations may remain after filtering. However, these remaining next stage intermediate representations may be of considerably higher quality than the initial set of synthetic intermediate representations. Accordingly, in various implementations, these remaining next stage intermediate representations may be processed using a similar sequence of operations as was used to process the initial synthetic intermediate representations.

[0020] For instance, the remaining next stage intermediate representations may be translated using the same rules-based translator into a plurality of next stage synthetic source code snippets in the high-level programming language. Next stage snippets of this new plurality may be used, e.g., in training pairs with the remaining intermediate representations, to train the same translation machine learning model. Because the next stage intermediate representations and synthetic source code snippets used for this stage of training are of higher quality and / or more reflective of real-world source code distributions than the previous iteration, the translation machine learning model is consequently better trained to generate realistic and / or high quality source code translations.

[0021] This process may be repeated until a final set of training data includes source code snippets that are highly reflective of real-world source code. This final set of training data may then be usable, for instance, to train the translation machine learning model or another machine learning model to translate between the intermediate representation (e.g., syntactically constrained pseudocode) and a high-level programming language, or even to translate directly between high-level programming languages.

[0022] In some implementations, in addition to or instead of the intermediate representation, two or more high-level programming language source code snippets in different high-level programming languages may be included in each training tuple during each stage. This may facilitate training of a direct, programming-language-to-programming-language translator. For instance, the initial synthetic set of intermediate representations may be translated using one or more rules-based translators into two different sets of source code snippets, one set in a first high-level programming language (e.g., Python) and the other set in a second high-level programming language (e.g., Java). The process may proceed similarly as described previously,except at the end, the training data may include semantically equivalent source code snippets in both the first and second high-level programming languages.

[0023] Fig. 1 schematically depicts an example environment in which selected aspects of the present disclosure may be implemented, in accordance with various implementations. Any computing devices depicted in Fig. 1 or elsewhere in the figures may include logic such as one or more microprocessors (e.g., central processing units or “CPUs”, graphical processing units or “GPUs”, tensor processing units or "TPUs") that execute computer-readable instructions stored in memory, or other types of logic such as application-specific integrated circuits (“ASIC”), field-programmable gate arrays (“FPGA”), and so forth. Some of the systems depicted in Fig. 1, such as a code knowledge system 100, may be implemented using one or more server computing devices that form what is sometimes referred to as a “cloud infrastructure,” although this is not required.

[0024] A code knowledge system 100 may be provided for helping clients 110-1 to 110-P manage their respective code bases 112-1 to 112-P. Code knowledge system 100 may include, among other things, a machine learning-based (ML-based) code translator 101 that is configured to help one or more clients 110-1 to 110-P to translate source code stored in one or more corresponding code bases 112-1 to 112-P. Each client 110 may be, for example, an entity or organization such as a business (e.g., financial institute, bank, etc.), non-profit, club, university, government agency, or any other organization that operates one or more software systems. For example, a bank may operate one or more software systems to manage the money under its control, including tracking deposits and withdrawals, tracking loans, tracking investments, and so forth. An airline may operate one or more software systems for booking / canceling / rebooking flight reservations, managing delays or cancellations of flight, managing people associated with flights, such as passengers, air crews, and ground crews, managing airport gates, and so forth.

[0025] ML code translator 103 may be configured to leverage knowledge of multiple different programming languages to aid clients 110-1 to 110-P in translating between programming languages when editing, updating, re-platforming, migrating, or otherwise acting upon their code bases 112-1 to 112-P. For example, ML code translator 103 may be configured to use one or more machine learning models 106-1 to 106-N to translate code snippets from one high-level programming language to another, e.g., on the fly or in batches. This may, for instance, enable a developer fluent in a first high-level programming language to view and / or edit source code that was originally written in a second, less-familiar programming language in the first programminglanguage. It may also significantly decrease the time and / or costs associated with migrating code bases 112 between different programming languages.

[0026] In various implementations, code knowledge system 100 may include a machine learning (“ML” in Fig. 1) database 105 that includes data indicative of one or more machine learning models 106-1 to 106-N. These machine learning models 106-1 to 106-N may take various forms that will be described in more detail below, including but not limited to models often used as large language models (LLMs), such as PaLM, BARD, BERT, LaMDA, Meena, GPT, and / or any other LLM, such as any other LLM that is encoder-only based, decoder-only based, sequence-to-sequence based and that optionally includes an attention mechanism or other memory. Additionally or alternatively, these machine learning models 106-1 to 106-N may include a graph-based network such as a graph neural network (“GNN”), graph attention neural network (“GANN”), graph convolutional neural network (“GCN”), or graph attention network (“GAT”), various flavors of a recurrent neural network (“RNN”, e.g., long short-term memory, or “LSTM”, gate recurrent units, or “GRU”, etc.), and any other type of machine learning model that may be applied to facilitate selected aspects of the present disclosure.

[0027] In some implementations, code knowledge system 100 may also have access to one or more programming-language-specific repositories (or “corpuses”) 108-1 to 108-M. In some implementations, these programming-language-specific corpuses 108-1 to 108-M may be used, for instance, to train one or more of the machine learning models 106-1 to 106-N and / or to generate high-quality supervised training data. In some implementations, the programminglanguage-specific corpuses 108-1 to 108-M may include reference source code snippets (e.g., entire code bases, libraries, etc.), inline comments, textual metadata associated with source code (e.g., commits), documentation such as textbooks and programming manuals, programming language-specific discussion threads, presentations, academic papers, and so forth. Some of corpuses 108-1 to 108-M may be publicly available repositories of source code that are widely available for download and use.

[0028] In some implementations, a client 110 that wishes to enable manipulation of its code base 112 in programming language(s) other than that / those used originally to write the source code may establish a relationship with an entity (not depicted in Fig. 1) that hosts code knowledge system 100. When a developer wishes to view / edit a source code snippet of the entity’s code base 112 but is unfamiliar with the native programming language, ML code translator 103 may provide one or more versions of the source code snippet that is translated to atarget programming language preferred by the developer. In some such implementations, ML code translator 103 may generate the translated source code snippet on the fly, e.g., in real time. In other implementations, ML code translator 103 may operate, e.g., in a batch mode, to preemptively translate all or selection portions of an entity’s code base 112 into a targeted programming language. In some implementations in which the developer then edits the translated source code snippet, the edited version may be translated back into the native programming language or left in the new, target programming language, assuming other necessary infrastructure is in place.

[0029] Machine learning models 106-1 to 106-N may take various forms, including those mentioned previously, to translate between different programming languages, or in some cases, to translate between different versions of the same programming language. As noted above, obtaining supervised (e.g., paired) training data to train these neural translation models can be challenging. Accordingly, code knowledge system 100 includes various other components that can aid in the automatic and / or systematic generation of supervised training data that can be used to train machine learning models 106-1 to 106-N.

[0030] A pseudocode synthesizer 101 may be configured to generate synthetic intermediate representations of source code snippets, such as syntactically constrained pseudocode. In some implementations, pseudocode synthesizer 101 may generate these intermediate representations of source code randomly, e.g., with little-to-no effort in generating representations that are readily interpretable by humans. Consequently, the initial synthetic intermediate representations may be low-quality, unnatural, ill-formed, etc.

[0031] In some implementations, pseudocode synthesizer 101 may randomly generate variable names for inclusion in synthetic pseudocode. These randomly generated variable names may differ from variable names typically used by human programmers. The latter would typically have some semantic and / or interpretable meaning, whereas randomly generated variable names may include, for instance, random sequences of characters and numbers that lack any semantic meaning.

[0032] In other implementations, a machine learning model such as a transformer may be trained to predict somewhat more natural variable names, rather than completely random. This may reduce computational requirements later, such as during inference, translation, filtering, training iterations, etc. In some implementations, the transformer may be trained to generate more natural variable names as follows. Source code in a high-level programming language(e.g., Python) may be modified so that all variable names are replaced with selected mask tokens. Each example of source code may then be paired with the selected mapping of mask- token-to-variable-name. The transformer may be trained to predict the mask-token-to-variable- name from a masked high-level programming language snippet.

[0033] Then, for each synthetic intermediate representation (e.g., syntactically constrained pseudocode), pseudocode synthesizer 101 may (e.g., via cooperation with rules-based translator 102) compute a synthetic translation into the high-level programming language, mask all variable names appearing in the original high-level programming language in the translation, use the transformer to predict realistic variable names in the masked translation, and apply the resulting substitution of masked random names to more realistic predicted names in the original synthetic intermediate representation. This will result in a distribution of variable names that, while not perfectly reflective of real-world variable names, is somewhat closer than purely random variable names.

[0034] A rules-based translator 102 may be configured to programmatically translate intermediate representations of source code snippets, such as syntactically constrained pseudocode snippets generated by pseudocode synthesizer 101, to one or more target high-level programming languages, such as C, C++, Python, Java, etc. Based on this translation, rules- based translator 102 may generate, for instance, source code snippets in one or more target high- level programming languages.

[0035] It may be the case that some, if not a substantial portion, of syntactically constrained pseudocode snippets generated during early stages by pseudocode synthesizer 101 include semantic and / or syntactic errors. This may be due, for instance, to the random (and hence, imperfect) synthesis of syntactically constrained pseudocode snippets by pseudocode synthesizer 101. Consequently, the synthetic source code snippets generated by rules-based translator 102, at least in the early stages, may likewise be of low quality.

[0036] Nonetheless, and as will be described in more detail with regard to Figs. 2 and 3, ML code translator 103 and / or training module 105 may be configured to perform supervised training of translation machine learning model(s) (e.g., 106-1 to 106-N) using tuples (e.g., pairs) of training data. In various implementations, each tuple (e.g., pair) of training data may include (i) a respective synthetic intermediate representation of the plurality of synthetic intermediate representations of source code snippets, and (ii) a corresponding synthetic source code snippetof the plurality of synthetic source code snippets that was generated by rules-based translator 102 based on the respective syntactically constrained pseudocode snippet.

[0037] In various implementations, the supervised training performed by ML code translator 103 and / or training module 105 may include applying data indicative of (e.g., the raw data, an embedding generated therefrom) the synthetic source code snippet as input across the translation machine learning model (one of 106-1 to 106-N). This application results in generation of a predicted synthetic intermediate representation (e.g., pseudocode snippet). This predicted synthetic intermediate representation (e.g., pseudocode snippet) may be compared to the training tuple’s reference or label synthetic intermediate representation to determine error(s). Based on these errors, a loss function (e.g., cross entropy) of the translation machine learning model may be minimized, e.g., using techniques such as gradient descent, simulated annealing, back propagation, etc.

[0038] Once the translation machine learning model (one of 106-1 to 106-N) has been initially trained in this manner, ML code translator 103 may apply it to data indicative of (e.g., raw data, embeddings generated therefrom) reference source code snippets in the target programming language. These reference source code snippets may be available from, for instance, one or more corpuses of source code (e.g., 108-1 to 108-M). This application by ML code translator 103 may result in translation of the reference source code snippets in the high-level programming language to generate a superset of next stage intermediate representations (e.g., syntactically constrained pseudocode) of source code snippets. Because the reference source code snippets are real world examples, e.g., composed and / or curated by humans, they typically will have realistic distributions that are reflected in this superset of next stage intermediate representations of source code snippets. Consequently, the next stage intermediate representations may be higher quality than the initial synthetic intermediate representations synthesized by pseudocode synthesizer 101.

[0039] In some implementations, filtering module 104 may be configured to evaluate the next stage intermediate representations and filter out those that fail one or more criteria. For instance, filtering module 104 may be configured to perform lexical analysis, syntactic analysis (e.g., parsing), semantic analysis, intermediate code generation, code optimization, and / or any other operation commonly associated with a compiler in order to evaluate each next state intermediate representation (which as noted previously may be syntactically constrained pseudocode in many examples). In various implementations, filtering module 104 may filter out next stageintermediate representations that, when evaluated in this manner, raise lexical, syntax, and / or semantic errors.

[0040] Rules-based translator 102 may then be configured to translate the next stage intermediate representations that remain after the filtering performed by filtering module 104 to generate a plurality of next stage source code snippets in the target high-level programming language. Because the remaining next stage intermediate representations are presumably high- quality and / or error-free, the resulting next stage source code snippets are likewise higher quality than, for instance, the synthetic source code snippets generated by rules-based translator 102 previously.

[0041] In various implementations, the components of Fig. 1 may repeat these operations until the latest tuples of next stage intermediate representations and corresponding next stage source code snippets in the high-level programming language are of sufficiently high quality to be usable, e.g., by ML code translator 103 and / or training module 105, as high-quality supervised training data. This high-quality supervised training data can then be used to perform supervised training of one or more translation machine learning models 106-1 to 106-N, such as the high- level programming language to intermediate representation translation machine learning model used to this point, or of another translation machine learning model.

[0042] In some cases, the operations described above (and illustrated in Figs. 2-3) are performed multiple times in parallel for multiple different programming languages, such as Java and Python. Two high-quality sets of supervised training data may be generated, one that includes pairs of intermediate representations and Java snippets, another that includes pairs of intermediate representations and Python snippets. The intermediate representations (e.g., syntactically constrained pseudocode snippets) of each supervised training data set may be used to cross reference Java snippets with semantically equivalent Python snippets. The intermediate representations may then be dropped, leaving pairs of semantically equivalent, and high quality, Java and Python snippets. These pairs of Java and Python snippets may then be used, e.g., by training module 105, to train a translation machine learning model to translate directly between Java and Python, without necessarily translating to intermediate representations in between.

[0043] Fig. 2 schematically depicts an example of how components described herein may cooperate to perform selected aspects of the present disclosure. Starting at top, pseudocode synthesizer 101 may generate, e.g., randomly, stochastically, etc., synthetic pseudocode snippets 220 (or more generally, intermediate representations). These synthetic pseudocodesnippets 220 may then be translated by rules-based translator 102 into synthetic source code snippets 222.

[0044] Tuples (e.g., pairs) of synthetic source code snippets 222 and corresponding synthetic pseudocode snippets 220 may then be formed as first stage supervised training data 224. First stage supervised training data 224 may be used, e.g., by ML code translator 103 and / or training module 105, to train a translation machine learning model 206. For instance, ML code translator 103 may apply data indicative of synthetic source code snippets 222 across the translation machine learning model 206 to generate predicted synthetic pseudocode snippets (not depicted). These predicted synthetic pseudocode snippets may be compared to the corresponding synthetic pseudocode snippets 220 of first stage supervised training data 224 (e.g., as labels) to determine error(s). These error(s) may then be used, e.g., by training module 105 as indicated by the black arrow, to train the translation machine learning model 206, e.g., by minimizing a loss function using techniques such as gradient descent, back propagation, simulated annealing, etc. The result of this training may be an at least partially trained translation machine learning model 206’.

[0045] Referring now to Fig. 3, the process may continue where a corpus of reference source code snippets, such as PL-1 corpus 308-1, may be accessed to collect a plurality of reference source code snippets 330. These reference source code snippets 330 may be processed by ML code translator 103 based on the partially trained translation machine learning model 206’ to generate a superset 332 of next stage pseudocode snippets.

[0046] In some implementations, filtering module 104 may then evaluate superset 332 of next stage pseudocode snippets to filter out those pseudocode snippets that include errors, such as lexical errors, syntactic errors, semantic errors, logical errors, etc. The remaining set 332’ of next stage pseudocode snippets may potentially be much smaller than the superset 332 of next stage pseudocode snippets.

[0047] In various implementations, the remaining set 332’ of next stage pseudocode snippets may then be translated by rules-based translator 102 to generate next stage source code snippets 334 (which are not referred to as “synthetic” herein because their origins lie in “genuine” reference source code snippets 330). These next stage source code snippets 334 may be paired with the remaining set 332’ of next stage pseudocode snippets as next stage supervised training data 336. Next stage supervised training data 336 may be higher quality than first stage supervised training data 224 because next stage supervised training data 336 is generated fromreal-world reference source code snippets 330, and therefore is more reflective of real-world source code distributions than, for instance, synthetic source code snippets 222.

[0048] As shown at bottom of Fig. 3, in some implementations, next stage source code snippets 334 may also be used as final source code snippets 334 in the first programming language (“PL1”, e.g., Java). Final source code snippets 334 in the first programming language 334 may be paired with additional final source code snippets in a second programming language 338 (“PL2”, e.g., Python) that were generated before, after, or in parallel using the same process of Figs. 2-3. These pairs 334 / 338 may form additional supervised training data 340 that can be used, e.g., by ML code translator 103 and / or training module 105, to train another translation machine learning model 342 to translate directly between the first and second high-level programming languages.

[0049] Fig. 4 is a flowchart illustrating an example method 400 of practice selected aspects of the present disclosure, in accordance with various implementations. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of code knowledge system 100. Moreover, while operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0050] At block 402, the system, e.g., by way of pseudocode synthesizer 101, may synthesize a plurality of synthetic intermediate representations (e.g., 220) of source code snippets. At block 404, the system, e.g., by way of rules-based translator 102, may translate the plurality of synthetic intermediate representations (e.g., 220) to generate a plurality of synthetic source code snippets (e.g., 222) in a high-level programming language (e.g., C, C++, Python, JavaScript, Java, Go, Kotlin, PHP, C#, Swift, Ruby, Perl, etc.). In some cases, rules-based translator 102 may translate the synthetic intermediate representations into multiple different high-level programming languages at once, to facilitate, for instance direct translation between different high-level programming languages.

[0051] At block 406, the system, e.g., by way of ML code translator 103 and / or training module 105, may perform supervised training of a translation machine learning model (e.g., 206) using tuples (e.g., pairs) of training data. Each pair of training data may include a respective synthetic intermediate representation of the plurality of synthetic intermediate representations of source code snippets that was generated at block 402, and a corresponding synthetic source codesnippet of the plurality of synthetic source code snippets in the high-level programming language that was generated at block 404. In some implementations, the supervised training of block 406 may include a denoising objective, which may facilitate more effective learning of distribution(s) of real code, such as reference source code snippets. As noted previously, in some implementations in which multiple synthetic source code snippets are generated for each tuple, each synthetic source code snippet being in a different high-level programming language. In these implementations, the additional synthetic source code snippets can be used at block 406 to train additional translation machine learning models.

[0052] As noted elsewhere herein, at this early stage these synthetic source code snippets may be relatively low quality because the underlying synthetic intermediate representations of source code snippets were generated at block 402 randomly, such that variables may be assigned unmeaningful names, formatting may be missing, instructions may be formed and / or organized in manners that are not intuitive / are random, and so forth.

[0053] At block 408, the system, e.g., by way of ML code translator 103, may use the now- partially trained translation machine learning model (e.g., 206’) to translate a plurality of reference source code snippets (e.g., 330) in the high-level programming language to generate a superset of next stage intermediate representations of source code snippets (e.g., 332). These next stage intermediate representations may be, for instance, pseudocode snippets that are better formed and / or higher quality than the synthetic intermediate representations (e.g., 220) that were generated at block 402, e.g., because they may reflect real -world distributions found across the reference source code. If multiple translation machine learning models are being trained in tandem, then multiple sets of next stage intermediate representations may be generated, one set from each of the multiple translation machine learning models being trained.

[0054] Based on one or more criteria, at block 410, the system, e.g., by way of filtering module 104, may filter a subset (e.g., 332’) of next stage intermediate representations from the superset (e.g., 332) of next stage intermediate representations of source code snippets. These criteria may include, for instance, lexical correctness, parse-ability, syntactic correctness, semantic correctness, etc. Again, if multiple sets of next stage intermediate representations are generated using multiple different translation machine learning models, then the multiple different sets of intermediate representations may each be filtered similarly.

[0055] At block 412, the system, e.g., by way of rules-based translator 102, may translate the remaining next stage intermediate representations (or multiple sets of remaining next stageintermediate representations where applicable) to generate a plurality of next stage source code snippets (e.g., 334) in the high-level programming language (or multiple different programming languages where applicable). In some implementations, the system, e.g., by way of ML code translator 103 and / or training module 105, may, at block 414, provide additional supervised training data (340) that includes, for instance, pairs of next stage source code snippets of the plurality of next stage source code snippets in the high-level programming language and either the remaining next stage intermediate representations (e.g., 332’) or other source code snippets (e.g., 338).

[0056] At block 416, the system, e.g., by way of ML code translator 103 and / or training module 105, may perform additional supervised training , e.g., of the translation machine learning model 206 or of another translation machine learning model (e.g., 342) using the additional training data (e.g., 340) provided at block 414.

[0057] Fig. 5 is a block diagram of an example computing device 510 that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device 510 typically includes at least one processor 514 which communicates with a number of peripheral devices via bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, user interface output devices 520, user interface input devices 522, and a network interface subsystem 516. The input and output devices allow user interaction with computing device 510. Network interface subsystem 516 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

[0058] User interface input devices 522 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 510 or onto a communication network.

[0059] User interface output devices 520 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of theterm "output device" is intended to include all possible types of devices and ways to output information from computing device 510 to the user or to another machine or computing device.

[0060] Storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include the logic to perform selected aspects of the method of Fig. 4, as well as to implement various components depicted in Figs. 1-3.

[0061] These software modules are generally executed by processor 514 alone or in combination with other processors. Memory 525 used in the storage subsystem 524 can include a number of memories including a main random-access memory (RAM) 530 for storage of instructions and data during program execution and a read only memory (ROM) 532 in which fixed instructions are stored. A file storage subsystem 526 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem 526 in the storage subsystem 524, or in other machines accessible by the processor(s) 514.

[0062] Bus subsystem 512 provides a mechanism for letting the various components and subsystems of computing device 510 communicate with each other as intended. Although bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0063] Computing device 510 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 510 depicted in Fig. 5 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 510 are possible having more or fewer components than the computing device depicted in Fig. 5.

[0064] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations willdepend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

CLAIMSWhat is claimed is:

1. A method implemented using one or more processors and comprising: synthesizing a plurality of synthetic intermediate representations of source code snippets; translating the plurality of synthetic intermediate representations using a rules-based translator to generate a plurality of synthetic source code snippets in a high-level programming language; performing supervised training of a translation machine learning model using pairs of training data, each pair of training data including: a respective synthetic intermediate representation of the plurality of synthetic intermediate representations of source code snippets, and a corresponding synthetic source code snippet of the plurality of synthetic source code snippets in the high-level programming language; using the trained translation machine learning model to translate a plurality of reference source code snippets in the high-level programming language to generate a superset of next stage intermediate representations of source code snippets; based on one or more criteria, filtering a subset of next stage intermediate representations from the superset of next stage intermediate representations of source code snippets; translating the remaining next stage intermediate representations using the rules-based translator to generate a plurality of next stage source code snippets in the high-level programming language; and performing additional supervised training of the translation machine learning model or a different translation machine learning model using additional training data that includes at least the next stage source code snippets of the plurality of next stage source code snippets in the high-level programming language.

2. The method of claim 1, wherein the translation machine learning model comprises a transformer model or a large language model.

3. The method of claim 1, wherein the additional supervised training comprises training the different translation machine learning model to translate source code directly between the high-level programming language and another high-level programming language.

4. The method of claim 1, wherein the superset of next stage intermediate representations comprise a superset of syntactically constrained pseudocode snippets.

5. The method of claim 4, wherein the one or more criteria include detection of one or more syntactic errors.

6. The method of claim 4, wherein the one or more criteria include detection of one or more semantic errors.

7. The method of claim 1, wherein the supervised training includes a denoising objective.

8. The method of claim 1, wherein the synthesizing includes randomly generating variable names for inclusion in the synthetic intermediate representations of source code snippets.

9. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to: synthesize a plurality of synthetic intermediate representations of source code snippets; translate the plurality of synthetic intermediate representations using a rules-based translator to generate a plurality of synthetic source code snippets in a high-level programming language; perform supervised training of a translation machine learning model using pairs of training data, each pair of training data including: a respective synthetic intermediate representation of the plurality of synthetic intermediate representations of source code snippets, and a corresponding synthetic source code snippet of the plurality of synthetic source code snippets in the high-level programming language; use the trained translation machine learning model to translate a plurality of reference source code snippets in the high-level programming language to generate a superset of next stage intermediate representations of source code snippets; based on one or more criteria, filter a subset of next stage intermediate representations from the superset of next stage intermediate representations of source code snippets; translate the remaining next stage intermediate representations using the rules-based translator to generate a plurality of next stage source code snippets in the high-level programming language; and perform additional supervised training of the translation machine learning model or a different translation machine learning model using additional training data that includes the nextstage source code snippets of the plurality of next stage source code snippets in the high-level programming language.

10. The system of claim 9, wherein the translation machine learning model comprises a transformer model.

11. The system of claim 9, wherein the translation machine learning model comprises a large language model.

12. The system of claim 9, wherein the superset of next stage intermediate representations comprise a superset of syntactically constrained pseudocode snippets.

13. The system of claim 12, wherein the one or more criteria include detection of one or more syntactic errors.

14. The system of claim 12, wherein the one or more criteria include detection of one or more semantic errors.

15. The system of claim 9, wherein the supervised training includes a denoising objective.

16. The system of claim 9, wherein the instructions to synthesize include instructions to randomly generate variable names for inclusion in the synthetic intermediate representations of source code snippets.

17. At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to: synthesize a plurality of synthetic intermediate representations of source code snippets; translate the plurality of synthetic intermediate representations using a rules-based translator to generate a plurality of synthetic source code snippets in a high-level programming language; perform supervised training of a translation machine learning model using pairs of training data, each pair of training data including: a respective synthetic intermediate representation of the plurality of synthetic intermediate representations of source code snippets, and a corresponding synthetic source code snippet of the plurality of synthetic source code snippets in the high-level programming language; use the trained translation machine learning model to translate a plurality of reference source code snippets in the high-level programming language to generate a superset of next stage intermediate representations of source code snippets;based on one or more criteria, filter a subset of next stage intermediate representations from the superset of next stage intermediate representations of source code snippets; translate the remaining next stage intermediate representations using the rules-based translator to generate a plurality of next stage source code snippets in the high-level programming language; and perform additional supervised training of the translation machine learning model or a different translation machine learning model using additional training data that includes the next stage source code snippets of the plurality of next stage source code snippets in the high-level programming language.

18. The at least one non-transitory computer-readable medium of claim 17, wherein the translation machine learning model comprises a transformer model.

19. The at least one non-transitory computer-readable medium of claim 17, wherein the translation machine learning model comprises a large language model.

20. The at least one non-transitory computer-readable medium of claim 17, wherein the superset of next stage intermediate representations comprise a superset of syntactically constrained pseudocode snippets.

Images