Generating synthetic training data using syntax mapping

Abstract

Automatic generation of synthetic training data is disclosed, which may be used to train a language model to generate code examples that follow a code language based on natural language input. Accordingly, a new language model may be created, or an existing language model may be fine tuned, to be suitable for automatically generating code, without having to manually generate a large amount of training data. Instead, a many-to-many syntax mapping is navigated to generate training data. In particular, a many-to-many syntax mapping maps code syntax to natural syntax. Each training data is then generated by navigating the many-to-many syntax mapping definition to generate a mapping of the respective code expressions to the respective natural language expressions.

Description

Background Art

[0001] Conventional large-scale pre-trained language models are capable of performing a wide variety of tasks using natural language input. For example, such language models can be fine-tuned using additional training data to perform tasks such as generating code in response to natural language input. For example, language models are traditionally fine-tuned to generate code for widely used programming languages such as C, C++, and Python. Training data for fine-tuning language models to generate code for these widely used programming languages is available through open source channels.

[0002] Many software products provide their own libraries, SDKs, and programming languages. Such programming languages are called "syntax-specific programming languages," or "DSLs." DSLs can have niche syntax and complex or incomplete documentation. To fine-tune language models to generate code for DSLs, training data is primarily generated through crowdsourcing, and then improvements are made through model evolution, such as using specialized model architectures and novel training techniques.

[0003] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is provided merely to illustrate one exemplary technology area where some embodiments described herein may be practiced. Summary of the Invention

[0004] This summary is provided to introduce some concepts in a simplified form that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] According to the principles described herein, training data suitable for training a language model to generate code based on natural language input is automatically generated. Specifically, the synthesized training data is suitable for training a language model to generate code that follows a code grammar based on the natural language input, rather than a natural grammar. Thus, new language models can be created, or existing language models can be fine-tuned to be suitable for automatically generating code with different code grammars and / or natural grammars, without having to manually generate large amounts of training data.

[0006] In contrast, according to the principles described herein, a many-to-many grammar mapping is navigated to generate training data. Specifically, the many-to-many grammar mapping maps code grammar to natural grammar. A code grammar is associated with a code language and defines how to form code strings from the code language's alphabet so that the generated code strings are valid according to the code language's syntax. A natural grammar, on the other hand, is associated with a natural language and defines how to form natural language strings from the natural language's alphabet so that the generated natural language strings at least approximate the natural language's syntax.

[0007] Then, for each of at least some of the training data, a plurality of training data is generated by navigating a many-to-many grammar mapping definition to generate a mapping of each code expression to a corresponding natural language expression. The navigation is at least partially random, such that each of the plurality of training data is different from at least a majority of other training data in the plurality of training data. The expression mapping may also potentially use context of the name-value pairs to further populate values into the expression mapping.

[0008] Thus, training data can be synthesized so that complex language models can be more easily trained and / or fine-tuned to generate code. This is particularly advantageous as new programming languages (such as domain-specific programming languages) are constantly being introduced and developed. Thus, grammar mappings can be used instead to synthesize sufficient training data, rather than having to manually develop training data for each of such programming languages. Thus, language models can be more effectively used to generate code that conforms to new and continuously developing programming languages. Thus, the power of language models (particularly large language models) can be more fully utilized.

[0009] Other features and advantages will be described in the description that follows, and in part will be apparent from the description, or may be learned by practicing the description herein. The features and advantages of the present invention may be realized and obtained by the instruments and combinations particularly pointed out in the appended claims. The features of the present invention will become more apparent from the following description and the appended claims, or may be learned by practicing the invention as set forth below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To illustrate the manner in which the foregoing and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments thereof that are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments and are therefore not to be considered limiting of scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1An environment in which the principles described herein may be run is shown, including a training data generator that generates expression mapping training data based on a grammar mapping definition;

[0012] Figure 2 shows a mapping environment in which code syntax is mapped to natural syntax according to the principles described herein;

[0013] Figure 3 A flowchart illustrating a method for generating training data according to the principles described herein; and

[0014] Figure 4 An example computing system is shown that may employ the principles described herein. DETAILED DESCRIPTION

[0015] According to the principles described herein, training data suitable for training a language model to generate code based on natural language input is automatically generated. Specifically, the synthesized training data is suitable for training a language model to generate code that follows a code grammar based on the natural language input, rather than a natural grammar. Thus, new language models can be created, or existing language models can be fine-tuned to be suitable for automatically generating code with different code grammars and / or natural grammars, without having to manually generate large amounts of training data.

[0016] In contrast, according to the principles described herein, a many-to-many grammar mapping is navigated to generate training data. Specifically, the many-to-many grammar mapping maps code grammar to natural grammar. A code grammar is associated with a code language and defines how to form code strings from the code language's alphabet so that the generated code strings are valid according to the code language's syntax. A natural grammar, on the other hand, is associated with a natural language and defines how to form natural language strings from the natural language's alphabet so that the generated natural language strings at least approximate the natural language's syntax.

[0017] Then, for each of at least some of the training data, a plurality of training data is generated by navigating a many-to-many grammar mapping definition to generate a mapping of each code expression to a corresponding natural language expression. The navigation is at least partially random, such that each of the plurality of training data is different from at least a majority of other training data in the plurality of training data. The expression mapping may also potentially use context of the name-value pairs to further populate values into the expression mapping.

[0018] Thus, training data can be synthesized so that complex language models can be more easily trained and / or fine-tuned to generate code. This is particularly advantageous as new programming languages (such as domain-specific programming languages) are constantly being introduced and developed. Thus, grammar mappings can be used instead to synthesize sufficient training data, rather than having to manually develop training data for each of such programming languages. Thus, language models can be more effectively used to generate code that conforms to new and continuously developing programming languages. Thus, the power of language models (particularly large language models) can be more fully utilized.

[0019] Figure 1 An environment 100 is shown in which the principles described herein may be implemented. The environment 100 includes a syntax mapping definition 110 and a training data generator 120. The training data generator 120 accesses (as indicated by arrow 101) the syntax mapping definition 110 to generate (as indicated by arrow 102) synthetic training data 130. The training data generator 120 may be a computer-executable component. As an example, if the training data generator 120 is provided by a program such as the following reference Figure 4 While the computing system operation of the computing system 400 is described, the training data generator 120 may be constructed, for example, as follows for Figure 4 406 is described in detail with reference to the executable component 406 of FIG.

[0020] The syntax mapping definition 110 is a many-to-many syntax mapping definition for mapping code syntax and natural syntax. The syntax mapping is "many-to-many" because the mapping can map multiple different code syntax patterns to multiple different language syntax patterns. The synthetic training data 130 includes multiple expression mappings 131, 132, 133 and 134, each of which represents a mapping of code expressions that follow the code syntax to natural expressions that follow the natural syntax. The expression mappings 131, 132, 133 and 134 are different from each other, although each expression mapping conforms to the many-to-many syntax mapping definition 110 because the code expressions in the expression mappings follow the syntax of the code syntax pattern, and the language expressions follow the syntax of the natural syntax pattern.

[0021] As indicated by ellipsis 135, training data generator 120 can generate any number of expression mappings 130. However, each expression mapping represents a piece of training data. Thus, training data generator 120 can be used to synthesize a sufficient amount of training data to train or fine-tune a language model to generate code that conforms to the code grammar.

[0022] Figure 2A mapping environment 200 is shown in which a code grammar 210 is mapped (as represented by line 201) to a natural grammar 220. As an example, a grammar mapping definition 110 defines a mapping 201 between the code grammar 210 and the natural grammar 220.

[0023] The code grammar 210 is associated with a code language and defines how to form strings (also referred to herein as "code strings") from the code language so that the code strings generated according to the syntax of the code language are valid. The code language can be any programming language, whether currently existing or yet to be developed. The code language can be a general programming language (such as C, C++, Python, Etch), or it can be a domain-specific programming language. The principles described herein are not limited to the identification of a code language.

[0024] The natural grammar 220 is associated with a natural language and defines how to form strings (also referred to herein as "natural language strings") from the natural language so that the generated natural language strings at least approximate the syntax of the natural language. As an example, a natural language can be any spoken and / or written language that is used to communicate naturally with humans.

[0025] For example, a natural language may be English. However, a natural language may be any other language used by humans to communicate (e.g., by speaking, writing, and / or signing). Examples include Mandarin, Hindi, Spanish, French, Arabic, Russian, Portuguese, Indonesian, or any other natural language, as the principles described herein are not limited to the identification of natural languages.

[0026] A natural language string is "at least approximately" the syntax of a natural language. This means that a natural language string either follows the identified syntax of the natural language exactly, or follows a syntax with some deviations, but these deviations are still identifiable to speakers of the natural language. Therefore, a natural language string does not have to follow the exact grammatical rules of the natural language. For example, the natural language "create a program definition" follows the exact English grammar, but the language "create a program definition" does not follow the exact English grammar because the indefinite article is omitted. However, an English speaker can understand the natural language "create a program definition". Therefore, a provider of natural language can provide more comfortable natural language input to the trained language model and still obtain appropriate generated code. In addition, this flexibility allows some migration of natural language grammatical rules, as well as colloquial use of natural language, while still obtaining correct generated code.

[0027] Figure 3 A training data generation method 300 according to the principles described herein is shown. As an example, the method 300 may be performed by Figure 1The method 300 includes accessing (act 301) a many-to-many grammar mapping definition for mapping code grammar and natural grammar. As an example, the many-to-many grammar mapping definition may be Figure 1 The syntax mapping definition 110 of , and perhaps more specifically, is Figure 2 In addition, the access (action 301) defined by the grammar mapping can be performed by Figure 1 Indicated by arrow 101.

[0028] Then, the method 300 includes generating a plurality of training data (act 310). The generation is achieved by performing the plurality of actions shown in act 310 to generate the training data in the form of expression mappings. Specifically, navigating (act 311) the many-to-many grammar mapping definition to generate mappings of the code expressions to the corresponding natural language expressions. Figure 1 , navigation (act 311) may be performed four times (one navigation each time) to generate expression map 131, expression map 132, expression map 133, and expression map 134. However, the navigation is at least partially random such that each expression map is different from at least most of the other expression maps.

[0029] A specific example will now be provided. In this example, a many-to-many grammar mapping definition is represented by a tree structure. Furthermore, navigation is performed from the root node of the tree structure downward to the leaf nodes. In this case, navigation is performed by navigating downward the tree structure at least partially randomly to form at least an intermediate form of the expression mapping.

[0030] An example syntax mapping definition is represented below in YAML format, with line numbers added for easy reference.

[0031] 1. Group name: "Problem"

[0032] 2. Root judgment: True

[0033] 3. Template:

[0034] 4. -MR mode: [“filter(@data source@, @condition@)”]

[0035] 5. NL mode: [“Show @1@ of @0@”] 6.

[0037] 7. Group name: "Condition"

[0038] 8. Template:

[0039] 9. -MR mode: [“@conditional base@”]

[0040] 10[“@0@"]

[0041] 11.-MR mode: [“@conditional base@ and @conditional base@”]

[0042] 12.NL mode: [“@0@ and @1@”]

[0043] 13.-MR mode: [“@conditional base@ or @conditional base@”]

[0044] 14.NL mode: [“@0@ or @1@”] 15.

[0046] 16. Group name: "Conditional Base"

[0047] 17. Templates:

[0048] 18.-MR mode: [“@number of columns@@comparators@@number@”]

[0049] 19.NL mode: [“@0@@1@@2@”] 20.

[0051] 21. Group Name: "Comparator"

[0052] 22. Template:

[0053] 23.-MR mode: [">"]

[0054] 24.NL mode: [“is @ greater than @”]

[0055] 25.-MR mode: [“<”]

[0056] 26.NL mode: [“is @ less than @”]

[0057] In this example, the syntax mapping definition pattern is a hierarchical structure based on groups to templates to patterns. Each group consists of one or more templates. Each template consists of one or more pairs, each pair including an MR pattern and an NL pattern. MR patterns are patterns associated with code syntax. NL patterns are patterns associated with natural syntax. When a group has a root predicate name value of "true", this is the root node in the tree structure. In this example, the generation of the expression mapping occurs by randomly navigating the tree structure from the root node,<MR,NL> The expression maps are sampled and combined using each sampling.

[0058] In this example, the computing system can take an example random navigation through the tree structure, which will now be described. The system starts from the group named "question" (see line 1) because this group has a root judgment parameter value of "true" (see line 2). Here, there is only one template (see line 3) with a single<MR,NL> Yes. When the template has only a single<MR,NL> When selecting from this template<MR,NL> The pair is reduced to simply selecting the single<MR,NL> Yes. So, after sampling from the "question" group, the expression mapping combination starts with:

[0059] MR: [“filter(@datasource@,@condition@)”]

[0060] NL: “Showing @0@ of @1@”.

[0061] The system then recursively iterates through each of the groups in the sampled pair until a terminal group (e.g., a leaf node in a tree structure) is sampled (where the atomicity parameter value is "true"). In the example above, there is no group where the atomicity parameter value is "true." However, this description will focus on the first navigation to illustrate the general principles of navigation syntax definition.

[0062] Returning to the first navigation example, after sampling the group named “question”, the computing system obtains the following expression mapping combinations:

[0063] MR: [“Filter(@DataSource@1,@Condition@1)”]

[0064] NL: “Showing @0@ of @1@”.

[0065] The MR pattern contains references to two non-terminal groups including a data source and a condition. The number after the reference refers to the order in which the groups named the same appear in the MR pattern. Since the data source group and the condition group appear only once within the MR pattern, they are each suffixed with "1". The NL representation also contains two non-terminal groups. Specifically, @0@ refers to the first group mentioned in the corresponding MR representation (i.e., the data source), and @1@ refers to the second group mentioned in the corresponding MR representation (i.e., the condition). Generally, the NL pattern includes the reference @X@, where "X" is an integer that represents the order of the groups within the MR representation.

[0066] Here, there is a group named "Condition" in the syntax mapping definition (defined from line 7 to line 14). In addition, the template element includes three possible conditions that can be sampled.<MR,NL> Yes. This is where random navigation comes in. Because there are multiple possible<MR,NL> Right, so we randomly select one of them. For this example, let’s assume that we sample the second<MR,NL> Yes. In other words, the system samples the following from the template element of the conditional group:<MR,NL> right:

[0067] MR mode: [“@conditional base@ and @conditional base@”]

[0068] NL pattern: [“@0@ and @1@”]

[0069] Then, the expression mapping combination is achieved by combining the<MR,NL> for and sampled from the "condition" group<MR,NL> Then the expression mapping combination becomes the following:

[0070] MR: "Filter (@DataSource@1, @ConditionBase@1 and @ConditionBase@2)"

[0071] NL: "Show @1@ and @2@'s @0@"",

[0072] Here, there are two groups called "conditional bases" involved in the combined MR pattern. The "conditional base" group is defined in lines 16 to 19. Here, the template element only includes one<MR,NL> Yes, that is:

[0073] MR mode: [“@number of columns@@comparators@@number@”]

[0074] NL mode: [“@0@@1@@2@”]

[0075] Since this is the only pair in the conditional basis set,<MR,NL> The expression mapping combination can then be rewritten again with this additional sampling, but this time as follows:

[0076] MR: "Filter(@datasource@1, @numbercolumn@1@comparator@1@number@1 and @numbercolumn@2@comparator@2@number@2)"

[0077] NL: "Show @0@ of @1@@2@@3@ and @4@@5@@6@"

[0078] Here, there are two groups named "Comparators" involved in the combined MR pattern. The "Comparator" group is defined in lines 21 to 26. Here, the template element only includes two<MR,NL> The first pair is row 23 to row 24 and the second pair is row 25 to row 26. Therefore, for each comparator instance in the MR pattern combined so far, these two pairs can be randomly sampled. In this example, it is assumed that for the first instance of the comparator, the second instance of the comparator group is randomly sampled.<MR,NL> The first instance of the comparator group is sampled and the second instance of the comparator group is sampled.<MR,NL> In this case, the expression mapping combination will be rewritten as follows:

[0079] MR: "Filter (@data source@1, @number column@1 <@number@1 and @number column@2 >@number@2)"

[0080] NL: "Show @0@ where @1@@ is less than @@2@ and @3@ is greater than @4@"

[0081] The Greater Than and Less Than groups can be built-in dictionary groups in the example framework. The Data Source and Number Column groups can be built-in context groups. In other words, their non-terminal symbols can be further written by sampling from the provided context. Sampling from the context will be discussed below.

[0082] The combination of expression mappings up to this point can be viewed as an intermediate form of expression mapping. However, it should be noted that due to the <mr nl>By randomly sampling, another navigation may result in a different intermediate form of the expression mapping. This intermediate form of the expression mapping can also be used for context application in the form of name-value pairs. For example, suppose such a context is as follows:

[0083] sheet:

[0084] - Name: "Student"

[0085] List:

[0086] -Display Name: "Name"

[0087] Original name: "Student Name"

[0088] Data type: "text"

[0089] -Display Name: "Age"

[0090] Original name: "Student Age"

[0091] Data type: "age"

[0092] -Display name: "Height"

[0093] Original name: "Student Height"

[0094] Data type: "numeric"

[0095] - Display Name: "Advisor"

[0096] Original title: "Instructor"

[0097] Data type: "Search:Teacher"

[0098] -Name: "Teacher"

[0099] List:

[0100] -Display Name: "Name"

[0101] Original name: "Teacher Name"

[0102] Data type: "text"

[0103] - Display name: "Salary"

[0104] Original name: "Salary"

[0105] Data type: "numeric"

[0106] The intermediate expression mapping combination can then be applied to this context to generate the following expression mapping:

[0107] NL: "Show students whose age is less than @number@1 and whose height is greater than @number@2"

[0108] MR: "Filter('student', 'age'<@0@ and 'height'<@1@)"

[0109] The application of diverse contexts allows the generation of more diverse training data, ultimately resulting in a language model that is more capable of generating code in response to various natural languages. In addition, the YAML syntax definition example is only a small example of the syntax mapping. Having more groups and / or more per template<MR,NL> The definition of the grammar mapping for can also similarly result in various expression mappings. Further diversity can be achieved by perturbing the natural language expressions. This perturbation can be different from the natural grammar, but different in the way it is defined. For example, a comma can be added to the NL pattern "Show students whose age is less than @number@1 and whose height is greater than @number@2" to make it "Show students whose age is less than @number@1 and whose height is greater than @number@2". This perturbation allows for further diversity in the training data to ensure that the language model can respond well to different ways of speaking or writing the same natural language.

[0110] In one embodiment, the<MR,NL> Weighted sampling of . This feature helps control the data distribution within the training data. As an example, a weight (e.g., 1.0 by default) can be assigned to each group, each template, or each pattern. For example, in one example, in the synthetic training data, the number of simple conditions should be twice that of compound conditions, and the condition group in the syntax mapping definition can become the following (again, line numbers are added for easy reference):

[0111] 1. Group name: "Condition"

[0112] 2. Template:

[0113] 3. -MR mode: ["@conditional base@"]

[0114] 4. ["@0@"]

[0115] 5. Sample coefficient: 4.0

[0116] 6. -MR mode: ["@conditional base@ and @conditional base@"]

[0117] 7. NL mode: ["@0@ and @1@"]

[0118] 8. -MR mode: ["@conditional base@ or @conditional base@"]

[0119] 9. NL mode: ["@0@ or @1@"]

[0120] See in particular the Sample Coefficient parameter having a value of 4.0.

[0121] The concept of built-in groups has been introduced previously. Built-in groups allow users who generate syntax mapping definitions to start building the definition more quickly. Such built-in groups may include context groups, such as data source, number column and age column, to provide data placeholders that can be replaced from the context definition. Alternatively or additionally, such built-in groups may include dictionary groups, such as greater than, less than and sort by, to provide alternative ways of describing the same operation. For example, sort by may be described in natural language as "sort by" or "sort in the order of". Alternatively or additionally, such built-in groups may include repeating groups, such as conditions from 2 to 4, to indicate that the condition group is repeated 2 to 4 times. The delimiters in MR and NL may also be configurable.

[0122] In some embodiments, the syntax mapping definition can be used to constrain the data space to a smaller and more reasonable range. For example, each MR pattern can have zero or more controlled conditions based on the placeholders used in the pattern to describe the possible generated data space. For example, consider the following<MR,NL> right:

[0123] MR mode: ["@number of columns@>=@number@ and @number of columns@<=@number@"]

[0124] NL mode: ["@0@ is between @1@ and @3@"]

[0125] The MR pattern contains two number column placeholders, and these two number column placeholders should be instantiated in the same column. In other words, the MR pattern should be associated with the constraint @0@=@2@. An example syntax that allows for concise specification of such a constraint is as follows:

[0126] This syntactic sugar provided by data generators allows:

[0127] -MR mode: ["@number of columns@>=@number@ and @0@<=@number@"]

[0128] NL mode: ["@0@ is between @1@ and @2@"]

[0129] Alternatively or additionally, constraints can be specified based on binary relations. One example is the relationship between a table and a column. Consider the following MR pattern: filter(@datasource@, @condition@). In this example, all sampled columns within the condition placeholder belong to the sampling table of the datasource placeholder. These general constraints can be implemented as built-in constraints (e.g. activated by default). In this case, the columns will be sampled based on the most appropriate table (determined by a built-in automatic algorithm) rather than a random table. For complex cases, the system can also explicitly specify such constraints, such as:

[0130] -MR mode: ["Select from a.@datasource@count(*)@global unique identifier column {t1}@ as join to @datasource@, because b opens a.@0@=b.@global unique identifier column {t2 and fk(@0@,its)}@group

[0131] By a.@0@”]

[0132] NL pattern: ["For each @2@, how many @1@ exist?"]

[0133] In some embodiments, the extensible validator pattern can allow users to generate<MR,NL> The generated MR can be validated using syntactic validation, but users can also implement custom validation, such as argument validators for specific application development tools. Users can filter generated examples based on pairs that pass the validator, thereby assigning higher confidence to the generated data points.

[0134] The user can construct a grammar mapping definition. However, to make this process easier, built-in groups can be provided as already described. In addition, much of the process of writing a grammar mapping definition can be further automated. This can be called "grammar induction", which generates a grammar from a seed mapping as an AI-assisted parsing task, where the generation process is supervised by a human. For example, a user may only have the corresponding<MR,NL> Several (e.g. 3 to 7) mapping examples for the "mapped" seed mapping annotation<MR,NL> Sequence semantically equivalent subspans. In addition, the user can provide a list (e.g., ten or more) of seed mappings for each desired function. The "seed" mapping can be the corresponding<MR,NL> Sequence pairs.

[0135] Thus, the principles described herein allow for the generation of training data, each of which maps code expressions in a particular code language to natural expressions in a particular natural language. According to some embodiments described herein, this can be implemented for more than just one code language and / or more than just one natural language. In this case, depending on the identification of the code language, there may be different grammar mapping definitions. Alternatively, depending on the identification of the natural language, there may be different grammar mapping definitions. Alternatively, depending on a combination of the identification of the code language and the identification of the natural language, there may be different grammar mapping definitions. In this case, the method further comprises identifying the code language and / or natural language to which the training data conforms and then selecting an appropriate grammar mapping definition.

[0136] Since the principles described herein are implemented in the context of a computing system, reference will now be made to Figure 4 However, the principles described herein are not limited to Figure 4 To describe operations within a computing system.

[0137] Computing systems are now increasingly taking a variety of forms. A computing system can be, for example, a handheld device, an appliance, a laptop computer, a desktop computer, a mainframe, a distributed computing system, a data center, or even a device not traditionally considered a computing system, such as a wearable device (e.g., glasses). In this specification and claims, the term "computing system" is broadly defined to include any device or system (or combination thereof) that includes at least one physically tangible processor and physically tangible memory capable of storing computer-executable instructions that can be executed by the processor. The memory can take any form and can depend on the nature and form of the computing system. The computing system can be distributed in a network environment and can include multiple component computing systems.

[0138] like Figure 4 As shown in , in the most basic configuration, computing system 400 includes at least one hardware processing unit 402 and memory 404. Processing unit 402 includes a general-purpose processor. Although not required, processing unit 402 may also include a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other dedicated circuit. In one embodiment, memory 404 includes physical system memory. The physical system memory can be volatile memory, non-volatile memory, or some combination of the two. In a second embodiment, the memory is a non-volatile mass storage device, such as a physical storage medium. If the computing system is distributed, then the processing, memory, and / or storage capabilities may also be distributed.

[0139] Computing system 400 also has a number of structures, often referred to as "executable components." For example, memory 404 of computing system 400 is shown as including executable component 406. The term "executable component" is a name for a structure that can be software, hardware, or a combination thereof, as is well known to those skilled in the computing arts. For example, when implemented in software, those skilled in the art will understand that the structure of the executable component can include software objects, routines, methods (and the like) that can be executed on the computing system. Such executable components reside in the computing system's stack, on a computer-readable storage medium, or a combination thereof.

[0140] Those skilled in the art will recognize that the structure of an executable component exists on a computer-readable medium such that the structure of the executable component, when interpreted by one or more processors (e.g., processor threads) of a computing system, causes the computing system to perform a function. This structure can be a computer-readable structure that is directly readable by a processor (as in the case where the executable component is binary). Alternatively, the structure can be constructed to be interpretable and / or compiled (whether in a single stage or in multiple stages) to generate such a binary that can be directly interpreted by a processor. When the term "executable component" is used, this understanding of the example structure of an executable component is fully within the understanding of those skilled in the art of computing.

[0141] The term "executable component" is also well known by those skilled in the art to include structures such as hard-coded or hard-wired logic gates that are implemented exclusively or almost exclusively in hardware, such as in a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other special purpose circuit. Therefore, the term "executable component" is a term that refers to structures well known to those of ordinary skill in the computing arts, whether implemented in software, hardware, or a combination thereof. In this specification, the terms "component," "agent," "manager," "service," "engine," "module," "virtual machine," etc. may also be used. As used in this specification and in this context, these terms (whether expressed in modified terms or not) are also intended to be synonymous with the term "executable component" and therefore also have structures well known to those of ordinary skill in the computing arts.

[0142] In the following description, various embodiments are described with reference to actions performed by one or more computing systems. If these actions are implemented in software, then in response to having executed the computer-executable instructions constituting the executable components, one or more processors (of the relevant computing system performing the action) direct the operation of the computing system. For example, such computer-executable instructions may be included on one or more computer-readable media forming a computer program product. One example of such an operation involves the manipulation of data. If these actions are implemented exclusively or nearly exclusively in hardware, such as in an FPGA or ASIC, the computer-executable instructions may be hard-coded or hard-wired logic gates. The computer-executable instructions (and the manipulated data) may be stored in memory 404 of computing system 400. Computing system 400 may also include a communication channel 408 that allows computing system 400 to communicate with other computing systems via, for example, a network 410.

[0143] Although not all computing systems require a user interface, in some embodiments, computing system 400 includes a user interface system 412 for interacting with a user. User interface system 412 may include an output mechanism 412A and an input mechanism 412B. The principles described herein are not limited to precise output mechanism 412A or input mechanism 412B, as this will depend on the nature of the device. However, output mechanism 412A may include, for example, a speaker, a display, tactile output, virtual or augmented reality, a hologram, etc. Examples of input mechanism 412B may include, for example, a microphone, a touch screen, virtual or augmented reality, a hologram, a camera, a keyboard, a mouse or other indicator input, any type of sensor, etc.

[0144] The embodiments described herein may include or utilize a dedicated or general-purpose computing system, including computer hardware, such as, for example, one or more processors and system memory, as discussed in more detail below. The embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or dedicated computing system. A computer-readable medium that stores computer-executable instructions is a physical storage medium. A computer-readable medium that carries computer-executable instructions is a transmission medium. Thus, by way of example and not limitation, embodiments of the present invention may include at least two distinct computer-readable media: a storage medium and a transmission medium.

[0145] Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other physical, tangible storage media that can be used to store desired program code means in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computing system.

[0146] A "network" is defined as one or more data links that enable the transmission of electronic data between computing systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computing system over a network or other communications connection (hardwired, wireless, or a combination of hardwired and wireless), the computing system properly considers the connection to be a transmission medium. Transmission media can include networks and / or data links that can be used to carry desired program code means in the form of computer-executable instructions or data structures and that can be accessed by general-purpose or special-purpose computing systems. Combinations of the above should also be included within the scope of computer-readable media.

[0147] Furthermore, upon reaching various computing system components, program code means in the form of computer-executable instructions or data structures can be automatically transferred from a transmission medium to a storage medium (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then ultimately transferred to computing system RAM and / or a less volatile storage medium at the computing system. Thus, it should be understood that storage media can be included in computing system components that also (or even primarily) utilize transmission media.

[0148] Computer-executable instructions include, for example, instructions and data that, when executed at a processor, cause a general-purpose computing system, a special-purpose computing system, or a special-purpose processing device to perform a specific function or group of functions. Alternatively or additionally, the computer-executable instructions may configure a computing system to perform a specific function or group of functions. Computer-executable instructions may be, for example, binary code or even instructions, such as intermediate format instructions, such as assembly language or even source code, that undergo some kind of translation (such as compilation) before being directly executed by a processor.

[0149] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter described in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0150] It will be appreciated by those skilled in the art that the present invention can be implemented in a network computing environment with various types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, data centers, wearable devices (such as glasses), etc. The present invention can also be practiced in a distributed system environment, where local and remote computing systems linked by a network link (by a hardwired data link, a wireless data link, or a combination of a hardwired and wireless data link) all perform tasks. In a distributed system environment, program modules can be located in local and remote memory storage devices.

[0151] Those skilled in the art will also appreciate that the present invention can be practiced in a cloud computing environment. A cloud computing environment can be distributed, although this is not required. In the case of a distributed environment, the cloud computing environment can be organizationally distributed internationally and / or have components owned by multiple organizations. In this specification and the appended claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the other many advantages that can be obtained from such a model when properly deployed.

[0152] For the processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in different orders. In addition, the operations outlined are provided as examples only, and some operations may be optional, combined into fewer steps and operations, supplemented or expanded into additional operations with other operations without departing from the essence of the disclosed embodiments.

[0153] The present invention may be implemented in other specific forms without departing from the spirit or features of the present invention. The described embodiments should be considered in all respects to be merely illustrative and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All variations within the meaning and scope of the equivalents of the claims are intended to be included within their scope.< / mr>

Claims

1. A computing system comprising: one or more processors; as well as One or more computer-readable media having computer-executable instructions thereon, the computer-executable instructions being structured such that, when executed by the one or more processors, the computing system is configured to generate synthetic training data by: accessing a many-to-many grammar mapping definition for mapping a code grammar to a natural grammar, the code grammar being associated with a code language and defining how to form code strings from an alphabet of the code language in such a way that the generated code strings are valid according to the syntax of the code language, and the natural grammar being associated with a natural language and defining how to form natural language strings from the alphabet of the natural language in such a way that the generated natural language strings at least approximate the syntax of the natural language; as well as The plurality of training data is generated by performing, for each of at least some of the plurality of training data, navigation of the many-to-many grammar mapping definition to generate mappings of respective code expressions to corresponding natural language expressions, the navigation being at least partially random such that each of the plurality of training data is different from at least most of the other training data in the plurality of training data.

2. The computing system of claim 1 , wherein the computer-executable instructions are further configured such that when the computer-executable instructions are executed by the one or more processors, the computing system is configured to: Identifying the code language that the training data will conform to; and The many-to-many grammar mapping definition corresponding to the identified code language is selected, wherein there are different applicable many-to-many grammar mapping definitions depending on at least the identification of the code language.

3. The computing system of claim 2 , wherein the computer-executable instructions are further configured such that when the computer-executable instructions are executed by the one or more processors, the computing system is configured to: identifying a natural language to which the training data will conform; and selecting the many-to-many grammar mapping definition that also corresponds to the identified natural language, wherein there are different applicable many-to-many grammar mapping definitions depending on at least the identification of the code language and the identification of the natural language.

4. The computing system of claim 1 , wherein the computer-executable instructions are further configured such that when the computer-executable instructions are executed by the one or more processors, the computing system is configured to: identifying the natural language that the training data will be in; and The many-to-many grammar mapping definition corresponding to the identified natural language is selected, wherein there are different applicable many-to-many grammar mapping definitions depending on at least the identification of the natural language.

5. The computing system of claim 1 , wherein depending on at least one of the identification of the code language and the identification of the natural language, there are different applicable many-to-many grammar mapping definitions, the different many-to-many grammar mapping definitions following a common grammar definition pattern.

6. The computing system of claim 1 , wherein the many-to-many grammar mapping definition comprises a tree structure that can be navigated downward from a root node to leaf nodes, wherein the navigation of the many-to-many grammar mapping definition is performed at least in part by randomly navigating the tree structure downward from the root node to construct at least an intermediate form of the mapping of each code expression to a corresponding natural language expression.

7. The computing system of claim 6, wherein the computer-executable instructions are constructed such that when the computer-executable instructions are executed by the one or more processors, the intermediate form of the mapping expressed by each code is also used for application of a context in the form of name-value pairs.

8. The computing system of claim 1 , wherein the computer-executable instructions are constructed such that, when executed by the one or more processors, the computing system automatically generates the many-to-many grammar mapping based on a plurality of seed mappings between natural language expressions and code expressions.

9. The computing system of claim 1 , wherein the computer-executable instructions are constructed such that, when executed by the one or more processors, the computing system further generates the plurality of training data by perturbing a representation of each of at least some of the mappings of respective code expressions and corresponding natural language expressions.

10. The computing system of claim 1 , wherein the computer-executable instructions are constructed such that, when executed by the one or more processors, the computing system further generates the plurality of training data by filtering representations of each of at least some of the mappings of respective code expressions and corresponding natural language expressions using validation rules.

11. The computing system of claim 1 , wherein the computer-executable instructions are constructed such that, when executed by the one or more processors, the computing system further generates the plurality of training data by using validation rules to change a representation of each of at least some of the mappings of respective code expressions and corresponding natural language expressions.

12. A method for generating synthetic training data for training a language model to generate code examples conforming to a code language based on natural language input, the method comprising: accessing a many-to-many grammar mapping definition for mapping a code grammar to a natural grammar, the code grammar being associated with a code language and defining how to form code strings from an alphabet of the code language in such a way that the generated code strings are valid according to the syntax of the code language, and the natural grammar being associated with a natural language and defining how to form natural language strings from the alphabet of the natural language in such a way that the generated natural language strings at least approximate the syntax of the natural language; as well as The plurality of training data is generated by performing, for each of at least some of the plurality of training data, navigation of the many-to-many grammar mapping definition to generate mappings of respective code expressions to corresponding natural language expressions, the navigation being at least partially random such that each of the plurality of training data is different from at least most of the other training data in the plurality of training data.

13. The method according to claim 12, further comprising: Identifying the code language that the training data will have; as well as The many-to-many grammar mapping definition corresponding to the identified code language is selected, wherein there are different applicable many-to-many grammar mapping definitions depending on at least the identification of the code language.

14. The method according to claim 13, further comprising identifying the natural language that the training data will be in; and The many-to-many grammar mapping definition corresponding to the identified natural language is selected, wherein there are different applicable many-to-many grammar mapping definitions depending on at least the identification of the natural language.

15. According to the method according to claim 12, the many-to-many grammar mapping definition includes a tree structure that can be navigated from a root node downward to a leaf node, and the navigation of the many-to-many grammar mapping definition is at least partially performed in the following manner: randomly navigating the tree structure downward from the root node to at least construct an intermediate form of the mapping of each code expression to the corresponding natural language expression.

16. The method according to claim 15, further comprising: The intermediate form of the mapping expressed by each code is also used for application of the context in the form of the name-value pairs.

17. The method of claim 12, wherein the computer executable instructions are constructed such that when the computer executable instructions are executed by the one or more processors, the computing system automatically generates the many-to-many grammar mapping based on a plurality of seed mappings between natural language expressions and code expressions.

18. The method according to claim 12, wherein the generating of the plurality of training data further comprises: A representation of each of at least some of the mappings of respective code expressions and corresponding natural language expressions is scrambled.

19. The method according to claim 12, wherein the generating of the plurality of training data further comprises: Validation rules are used to filter representations of each of at least some of the mappings of respective code expressions and corresponding natural language expressions.

20. The method of claim 12, the generating of the plurality of training data being further performed by using validation rules to change a representation of each of at least some of the mappings of respective code expressions and corresponding natural language expressions.

Images