Machine learning structured result generation

Abstract

Aspects of the present application relate to machine learning (ML) structured result generation. In an example, an instruction to invoke programmatic code of an ML model is to indicate a result interface in which a model output is to be stored. The result interface is processed to generate a data format description for the result interface such that the inputs of the ML model further include the data format description. As a result of providing the data format description as input to the ML model, the ML model is directed to generate a structured model output corresponding to the result interface. The result model output is processed to generate an instance of the result interface, e.g., with one or more corresponding attributes from the structured model output. Thus, the programmatic code can reliably perform subsequent processing based on the generated instance of the result interface.

Description

BACKGROUND OF THE DISCLOSURE

[0001] Output generated by a machine learning (ML) model may have a degree of variability, making it challenging to incorporate ML processing into procedural code. For example, an ML model may generate output that includes different terms and / or has different structures. Accordingly, these and other variations in the ML model output may result in unexpected or unintended behavior of the procedural code, or may cause the procedural code to fail altogether, among other potential harms.

[0002] Embodiments have been described with respect to these and other general considerations. Moreover, although relatively specific problems have been discussed, it should be understood that embodiments are not limited to solving the specific problems identified in the background. SUMMARY OF THE DISCLOSURE

[0003] Aspects of the present application relate to machine learning (ML) structured result generation. In an example, instructions of procedural code that invoke an ML model are used to indicate a result interface in which the output of the model will be stored. The result interface is processed to generate a data format description for the result interface such that the input to the ML model also includes the data format description. As a result of providing the data format description as input to the ML model, the ML model is guided to generate a structured model output corresponding to the result interface. The result model output is processed to generate an instance of the result interface, e.g., having one or more corresponding attributes from the structured model output. Accordingly, the procedural code is able to reliably perform subsequent processing based on the generated instance of the result interface.

[0004] The present Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. The present Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting examples and non-exhaustive examples are described with reference to the following drawings.

[0006] Figure 1 An overview of an example system in which machine learning structured result generation may be used in accordance with aspects of the present disclosure is shown.

[0007] Figure 2 An overview of an example method for obtaining and processing structured machine learning model output in accordance with aspects described herein is shown.

[0008] Figure 3 An overview of an example method for generating structured machine learning model output in accordance with aspects described herein is shown.

[0009] Figure 4Aand Figure 4B shows an overview of an example generative machine learning model that can be used in accordance with aspects described herein.

[0010] Figure 5 is a block diagram showing example physical components of a computing device in which aspects of the present disclosure may be practiced.

[0011] Figure 6 is a simplified block diagram of a computing device in which aspects of the present disclosure may be practiced.

[0012] Figure 7 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0013] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments or examples. Aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Embodiments may be practiced as a method, system, or device. Thus, embodiments may take the form of a hardware implementation, a fully software implementation, or an implementation combining software and hardware aspects. Accordingly, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0014] In an example, a machine learning (ML) model produces a model output based on an input (e.g., that may be generated by procedural code or otherwise provided and / or obtained from a user). For example, a generative ML model is used to process natural language input to correspondingly produce a model output. However, in cases where such model interactions are included as part of procedural code, parsing or otherwise processing the model output can be challenging because the output generated by the ML model may not be deterministic or may otherwise have a degree of variability. Such challenges may be contrary to an application programming interface (API), where the API includes defined functions that produce output according to a predefined format.

[0015] As a result of such unpredictability, procedural code can incorporate various pattern matching techniques, data cleaning techniques, and / or any of various other processes to incorporate the model output from the ML model into the processes performed by the procedural code. Such aspects may increase code complexity and / or development time while potentially reducing the reliability of the procedural code and other impairments. For example, if the procedural code fails to handle edge cases where the ML model generates output different from what the procedural code expects, the procedural code may exhibit unexpected / unintended behavior or may fail to operate.

[0016] Accordingly, aspects of the present disclosure relate to machine learning structured result generation. In an example, instructions of a procedural code that invokes an ML model are identified. The identified procedural instructions may include an input to the ML model (or, as another example, a model output that references an earlier procedural instruction generated previously), and an output invoked by the ML model (e.g., generated based on the input and / or a previously generated model output) is subsequently processed according to one or more other procedural instructions of the code.

[0017] The procedural instructions that invoke the ML model (also referred to herein as "procedural ML calls") may also indicate a result interface in which the model output will be stored. As used herein, a result interface includes, but is not limited to, a basic data type (e.g., an integer, a string, or a floating point number) or a complex data type (e.g., a class, a struct, an array, a map, a dictionary, or a dynamic type), and any data structure among various other data structures. For example, the procedural code may define a class or other data structure as the result interface of the ML model output. Example procedural code includes, but is not limited to, source code (e.g., in an interpreted or compiled language), bytecode, and / or machine code, etc. As another example, a development environment or library defines one or more result interfaces for use by the procedural code. Accordingly, the result interface is processed to generate a data format description for the result interface such that the input to the ML model also includes the data format description according to the aspects described herein.

[0018] For example, the input to the ML model includes a background context, instructions related to the procedural instructions (e.g., at least a part of the procedural instructions or other representations of the procedural instructions), a request to generate a model output according to the provided data format description, the data format description, and / or an indication to complete the input using the subsequently generated model output, etc. As a result of providing the data format description as part of the input to the ML model, the ML model is guided to generate a structured model output corresponding to the result interface. The resulting model output is processed to generate an instance of the result interface, e.g., having one or more corresponding attributes from the structured model output. As a result, the procedural code is able to reliably perform subsequent processing based on the generated instance of the result interface.

[0019] As used herein, structured model outputs include, but are not limited to, one or more key / value pairs, Extensible Markup Language (XML) output, YAML output, and / or JavaScript Object Notation (JSON) output, and any one of a variety of other structured data formats. In an example, the data format description generated for the result interface similarly includes such structured data formats. The generated data format description is used to describe a set of properties of the result interface, where each property of the result interface can itself be a primitive type or a complex type. For example, the data format description includes the property name, an indication of whether the property is optional, the default value of the property, an indication as an expected data type / format (e.g., email address, phone number, Uniform Resource Locator (URL), date, time, or date-time), a description of the property, a minimum and / or maximum value for the property, a set of enumerated values for the property, and / or a maximum character and / or list length for the property, and any one of a variety of additional or alternative property characteristics. In an example, the minimum length property / maximum length property is converted to a maximum token length / minimum token length, which can improve the accuracy of the ML model when generating structured output corresponding to such a property.

[0020] A generative model (which is also generally referred to herein as a type of ML model) used in accordance with the aspects described herein can generate any output type among a variety of output types (and thus can be a multimodal generative model in some examples), and can be a generative transformer model and / or a large language model (LLM) in some examples, and a generative image model in some examples. Example ML models include, but are not limited to, Generative Pretrained Transformer 3 (GPT-3), BigScience BLOOM (Large Open Science Open Access Multilingual Language Model), DALL-E, DALL-E2, Stable Diffusion, or Jukebox. Additional examples of these aspects are discussed below with respect to Figure 4A - Figure 4B the generative ML models shown in

[0021] It should be understood that, according to aspects described herein, any one of a variety of techniques can be used to generate a data format description for a result interface. For example, the procedural code defining the result interface is processed to generate a schema corresponding to the result interface, which can thus form part of the data format description accordingly. The data format description can also include instructions for generating a model output that follows the original schema. Describing the result interface using such an original schema can improve the accuracy of model output generation because the ML model is provided with an accurate, complete, and / or detailed definition of the result interface, and the model output is generated accordingly using the result interface. However, using the original schema to describe the result interface may consume a larger amount of available tokens for a given ML model (compared to other example aspects described herein), thereby limiting other inputs that can be provided and / or inputs that can be generated by the ML model.

[0022] As another example, the data format description of the result interface includes an example structured output corresponding to the result interface, such that the model output generated by the ML model follows the example structured output. For example, the example structured output can be obtained from a previous ML process associated with a reference interface, can be specified by a developer of the procedural code, and / or can be a serialized representation of one or more instances of the result interface (e.g., as may have been defined by the developer), among other examples. In some examples, the example structured output also includes one or more comments describing aspects of the result interface instance. The data format description can also include instructions for generating a model output that follows the included example format (e.g., as well - formed XML / YAML / JSON output). Compared to the original schema, the example structured output can consume fewer available tokens for a given ML model. However, the example structured output may include less detail (e.g., optional attributes may not be included in the example structured output, or may not include an indication that a defined attribute is otherwise optional), resulting in a lower - quality structured output from the ML model.

[0023] As another example, the data format description of the result interface includes a summary representation of the original schema. For example, the summary representation is generated based on the original schema, where the summary representation is structured according to the format in which the structured output will be generated. In an example, one or more attributes of the result interface are included as strings, where each string includes the type, attribute description, whether the attribute can be null, whether the attribute has a default value, and / or associated constraints, and other attribute characteristics for the corresponding attribute. The data format description can also include instructions for generating a model output that follows the summary representation accordingly. Thus, the summary representation includes similar information about the result interface (and in some examples, more information will be included in instances using the example structured output), while consuming a reduced amount of tokens to be processed by the ML model.

[0024] Accordingly, it should be understood that, in accordance with aspects described herein, any one of a variety of techniques can be used to generate a data format description. As another example, a representation of a result interface can be provided as input to an ML model (e.g., with instructions that summarize or otherwise describe the result interface), where the ML model accordingly generates a data format description for the result interface. The resulting model output can be cached for future use. Even so, compared to other example data format description generation techniques, data format descriptions generated using ML may incur additional processing overhead (e.g., as a result of the associated ML processing).

[0025] Figure 1 An overview of an example system 100 in which machine learning structured result generation can be used, in accordance with aspects of the present disclosure, is shown. As shown, system 100 includes a structured result service 102, a computing device 104, and a network 106. In an example, the structured result service 102 and the computing device 104 communicate via the network 106, which can include a local area network, a wireless network, the Internet, or any combination thereof, among other examples.

[0026] As shown, the computing device 104 includes an application 116, a structured result manager 118, a schema generator 120, and an object generator 122. In an example, the application 116 includes procedural code that invokes a machine learning model (e.g., of the machine learning engine 114). Accordingly, the structured result manager 118 can identify such a procedural ML call of the application 116, which can indicate the result interface in which the model output will be stored. For example, a procedural ML call can be identified as a result of executing or otherwise processing procedural instructions.

[0027] Accordingly, the schema generator 120 processes the result interface to generate a schema. As described above, the result interface can be a basic type or a complex type, and the generated schema can thus indicate one or more attributes of the result interface and one or more attributes associated with each of the attributes. Although the computing device 104 is shown as including a schema generator 120, it should be understood that any one of a variety of other techniques can be used to generate a schema. Additionally, although system 100 is shown as an example of generating a data format description based on a schema for the result interface, it should be understood that in other examples, a schema need not be generated, and the data format description can be generated or otherwise obtained according to any one of a variety of other techniques. For example, the application 116 can alternatively include an example structured output that is used as a supplement or alternative to other aspects of the data format description.

[0028] In an example, at least some aspects described herein can be performed before executing application 116. For example, a schema and / or associated data format description can be pre-generated (e.g., at compile time or as a preprocessing step) such that the schema and / or associated data format description can be used for subsequent processing in accordance with aspects described herein.

[0029] The structured result manager 118 provides an indication of the generated schema to the structured result service 102. In an example, the indication also includes an input specified by the application 116 (e.g., an input that may have been generated by the application 116 and / or obtained from a user of the computing device 104, etc.). In other examples, the indication refers to a previously generated model output, such as may be the case when a previous procedural instruction of the application 116 generates a model output and a subsequent procedural instruction is processing that model output to generate an instance of a result interface in accordance with aspects described herein.

[0030] As shown, the structured result service 102 includes a request processor 108, a prompt generator 110, a structured result validator 112, and a machine learning engine 114. In an example, the request processor 108 receives an ML processing request from the computing device 104 (e.g., including an indication of a schema and / or input for processing by an ML model, as may have been generated by the schema generator 120 and the application 116, respectively).

[0031] Accordingly, an indication of the request is provided to the prompt generator 110, which generates a prompt for the ML model of the machine learning engine 114. In an example, the generated prompt includes an input received from the computing device 104 (or as another example, a reference to a previously generated model output) and a data format description in accordance with aspects described herein. For example, the prompt generator 110 processes the schema received from the computing device 104 to generate a summary representation of the schema, which is then used as part of the data format description accordingly. As another example, the schema is used as part of the data format description. As another example, the prompt generator 110 provides an indication of the schema to the ML model of the machine learning engine 114 (e.g., in conjunction with a prompt request for the data format description) such that the model output received as a response is used as part of the data format description accordingly.

[0032] Examples of prompts that can be generated by the prompt generator 110 are provided below for reference:

[0033] / / / Start prompt

[0034] <Input content>

[0035] The result is well-formed <json xml>Format and strictly adhere to the following format.

[0036] <Example structured output, raw mode, or summary mode, and other examples>

[0037] / / / End of prompt

[0038] Thus, the generated prompt includes the received input, an indication of the structured data format (e.g., JSON or XML in this example), and a description of the data format for the result interface for application 116. The generated prompt is processed by machine learning engine 114 to generate a structured model output accordingly. Although structured result service 102 is shown as including machine learning engine 114, it should be understood that in other examples, a third-party or remote machine learning service may be additionally or alternatively used.

[0039] In an example, an ML model is selected from a set of ML models, which may be the case where the prompt is associated with a given ML model. As another example, the request received by request processor 108 includes an indication of a specific ML model that processes the input using the request. In some cases, structured result service 102 determines to generate multiple ML interactions (e.g., each ML interaction having an associated prompt and result model output). For example, structured result service 102 may determine that an ML processing request (e.g., as may have been received from computing device 104) will exceed the token limit of the associated ML model, such that structured result service 102 generates multiple prompts, each prompt including at least a portion of the input. Structured result service 102 may then generate additional prompts to combine a set of resulting model outputs, which may include, for example, a data format description, such that the ML model generates a structured output based on the set of model outputs accordingly. In other examples, a first ML interaction may combine a set of model outputs, while a second ML interaction generates a structured output accordingly.

[0040] In an example, the structured result validator 112 evaluates a structured model output from an ML model (e.g., generated based on a prompt including a data format description) to determine, for example, whether the structured model output is syntactically correct and / or whether the structured model output conforms to a schema (e.g., as may have been generated by the schema generator 120), among other examples. In cases where the structured result validator 112 determines that the structured model output fails validation, the structured result validator 112 may attempt a remedial action and / or may provide an indication to the computing device 104 (e.g., for further processing by the application 116 and / or for viewing by a user of the computing device 104). Example remedial actions include, but are not limited to, attempting to correct a malformed structured output (e.g., by closing symbols / tags), requesting that the ML model correct the structured output (e.g., providing a prompt to generate a structured output that is syntactically corrected based on the malformed output and / or to complete the generation of the structured output), and / or selecting a different instance of the structured model output generated by the ML model.

[0041] If the structured model output is successfully validated by the structured result validator 112, the structured model output is provided to the computing device 104 (e.g., in response to a request for the model output) such that the object generator 122 processes the structured model output to generate an instance of the result interface accordingly. For example, the object generator 122 processes the structured model output to instantiate an instance of the result interface that includes one or more attributes defined by the structured model output. It should be understood that the format of the structured model output need not be the same as the structure of the result interface. For example, the object generator 122 may process a structured model output that is JSON to generate an instance of the result interface that need not be a JSON object.

[0042] While examples have been described in which the result interface is instantiated, it should be understood that similar techniques may be used in which the structured model output itself is provided accordingly for subsequent processing by procedural code. For example, procedural code may process XML / YAML / JSON output as an alternative or supplement to an instance of the generated result interface according to the aspects described herein.

[0043] Thus, as a result of combining the input for ML processing as providing a schema, the structured result service 102 effectively operates as an API for any of a variety of languages and / or data types / formats. That is, the structured result service 102 may generate any one of a variety of structured model outputs for a similar input, where the structured model output is structured according to a schema provided in conjunction with the request for the model output.

[0044] As another example, aspects of the present disclosure can be used to project an object of a first data type into an object of a second data type. For example, a serialized representation of an object in the first data type can be provided in conjunction with a schema for the second data type such that the structured result service 102 generates a structured model output including a representation of the object according to the second data type. The structured model output can then be processed (e.g., by the object generator 122) to instantiate an object of the second data type accordingly (e.g., which includes attributes indicated by the structured model output).

[0045] Figure 2 An overview of an example method 200 for obtaining and processing structured machine learning model output in accordance with aspects described herein is shown. In an example, aspects of method 200 are performed by a computing device (such as Figure 1 the computing device 104 in).

[0046] As shown, method 200 begins at operation 202, where programmatic instructions (e.g., of an application (such as Figure 1 the application 116 in)) including an ML model call are identified. Aspects of operation 202 can be performed by a structured result manager (such as Figure 1 the structured result manager 118 in). For example, the programmatic instructions are identified as the result of executing or otherwise processing programmatic code.

[0047] At operation 204, a result interface associated with the identified programmatic instructions is determined. Aspects of operation 204 can be performed by a structured result manager (such as Figure 1 the structured result manager 118 in). In an example, the result interface is determined based on the indication of the programmatic instructions, and the model output generated as a result of the ML model call will be stored using the result interface.

[0048] Thus, at operation 206, a schema is generated for the determined result interface. In an example, aspects of operation 206 are performed by a schema generator (such as Figure 1 the schema generator 120 in). For example, operation 206 includes evaluating the definition of the result interface to generate a schema such that the schema includes an indication of one or more attributes of the result interface and, in some examples, includes one or more corresponding attribute characteristics for each attribute. In other examples, operation 206 includes providing an indication of the definition of the resource interface to a schema generation service such that the schema is received as a response. Thus, it should be understood that any of a variety of techniques can be used to generate a schema in accordance with aspects described herein. Additionally, it should be understood that method 200 is described in an example where a data format description is generated by the structured result service, and in other examples, these aspects can additionally or alternatively be performed at operation 206.

[0049] The process proceeds to operation 208, where a request for the ML model output is provided, where the request includes the generated schema. In an example, the request also includes an input for processing by the ML model and / or an indication of a previously generated model output, etc. As an example, the request is provided to a structured results service, such as Figure 1 the structured results service 102 in

[0050] At operation 210, in response to the request provided at operation 208, a structured model output is received. In an example, the structured model output includes data formatted according to one or more structured data formats (including but not limited to XML, YAML, and / or JSON, etc.).

[0051] Method 200 proceeds to operation 212, where the structured model output is processed to correspondingly generate an instance of a result interface. In an example, aspects of operation 212 are performed by an object generator (such as Figure 1 the object generator 122 in Figure 1 Note that, in response to a request for ML processing, the structured results service may verify the structured model output before providing the structured output. In some examples, such verification and / or remedial actions may additionally or alternatively be performed as part of operation 212.

[0052] Move to operation 214, where, in accordance with aspects described herein, an instance of the result interface generated at operation 212 is returned for subsequent processing (e.g., by one or more other instructions of procedural code, such as Figure 1 the application 116 in

[0053] Figure 3 An overview of an example method 300 for generating a structured machine learning model output in accordance with aspects described herein is shown. In the example, aspects of method 300 are performed by a structured results service (such as Figure 1 the structured results service 102 in

[0054] As shown, method 300 begins at operation 302, where an ML processing request including an indication of input and a schema is received. In the example, the request is received by a request processor (e.g., Figure 1 the request processor 108 in Figure 2 from a structured results manager (e.g., structured results manager 118). The request may have been generated as the computing device performs aspects of operation 208 discussed above with respect to Figure 1 method 200. As described above, the request may include input for ML processing and / or may reference previously generated model outputs, etc. The schema received as part of the request may have been generated by a schema generator (such as Figure 1 the schema generator 120 in

[0055] Method 300 proceeds to operation 304, where the schema is processed to generate a data format description. In the example, aspects of operation 304 are performed by a prompt generator (such as Figure 1 the prompt generator 110 in

[0056] In the example, the generated data format description includes the schema received at operation 302 (e.g., as a raw schema), includes a summary representation of the generated schema, or includes processing the schema using an ML model to correspondingly generate the data format description. In other examples, the received request alternatively includes an example structured output such that operation 304 may be omitted.

[0057] However, in the case where the number of tokens is below a predetermined threshold, the process branches "yes" to operation 308, where an input segment is determined for chained ML processing such that the input is processed according to a set of ML evaluations linked together. In an example, determining the input segment includes selecting a subpart of the input such that one or more other subparts are processed in subsequent iterations of operation 308, operation 310, operation 312, and operation 314. In other examples, the input segment is determined as a result of processing the input to identify one or more skills for processing the input. Thus, it should be understood that any one of a variety of techniques can be used to generate multiple input segments, and the multiple input segments are used to perform chained ML model processing according to the aspects described herein.

[0058] At operation 310, a prompt is generated for processing by the ML model, the prompt including at least a portion of the input (e.g., in some examples, according to the determined input segment) and a data format description generated at operation 304. Aspects of operation 310 can be performed by a prompt generator, such as prompt generator 110 discussed above with respect to Figure 1 In the case where chained ML processing is determined to be performed, operation 310 may not include the data format description such that when combining multiple model outputs, the data format description may alternatively be included at operation 316, as discussed in more detail below.

[0059] The process proceeds to operation 312, where a model output corresponding to the generated prompt is obtained. For example, the prompt is provided for processing by an ML model (e.g., by a machine learning engine, such as the machine learning engine 114 in Figure 1 ). Thus, the ML model generates a model output in response. Additional examples of such aspects are discussed below with respect to Figure 4A and 4B In the case where the prompt includes a data format description, the ML model generates a structured model output according to the aspects described herein.

[0060] Method 300 proceeds to determination 314, where it is determined whether there are any remaining input segments to process. Thus, if it is determined to process one or more additional input segments, method 300 branches "yes" and returns to operation 308, where the additional input segments are processed according to the operations described above. Thus, method 300 can loop between operation 308, operation 310, operation 312, and operation 314 to perform chained ML processing on the received input in the case where the ML model processing would exceed one or more capabilities of the ML model.

[0061] Finally, the process can branch "no" to operation 316, where a set of model outputs (e.g., obtained as a result of multiple iterations of operation 312) are combined into a single model output. In an example, operation 316 includes concatenating, appending, or otherwise combining each model output in a set of model outputs to produce a single model output. For example, if each model output is a structured model output, the structure of the model output can correspondingly facilitate combining the model outputs (e.g., by including attributes, labels, or other sub-parts of the model output into the single model output). Additionally or alternatively, a set of model outputs are processed by an ML model, e.g., in conjunction with the data format description generated at operation 304, such that the ML model correspondingly generates a structured model output. Thus, it should be understood that according to aspects described herein, any one of a variety of techniques can be used to combine a set of model outputs. Operation 316 is shown with a dashed box to indicate that in some examples (e.g., where it is determined at determination 306 that the input does not exceed the capabilities of the ML model), operation 316 can be omitted. In such examples, the process instead proceeds from determination 314 to operation 318.

[0062] At operation 318, the structured model output is verified. Aspects of operation 318 can be performed by a structured result validator (such as Figure 1 structured result validator 112 in

[0063] . In an example, operation 318 includes processing the model output to determine whether it is syntactically correct and / or whether the structured model output conforms to a schema (e.g., as received at operation 302), among other examples. It should be appreciated that in other examples any one of a variety of additional or alternative evaluations can be performed.

[0064] Although method 300 is shown in an example of performing a remedial action for a requested updated model output, it should be understood that in other examples any one of a variety of alternative or additional remedial actions may be performed. For example, operation 322 may alternatively or additionally include attempting to correct a malformed structured output and / or requesting the ML model to correct the structured output, etc.

[0065] Returning to determination 320, if alternatively it is determined that the structured model output is successfully verified, the process branches "yes" to operation 320, where the structured model output is provided in response to the request received at operation 302. For example, the structured model output is provided to a computing device (e.g., performing Figure 2 aspects of method 200 therein), such that the computing device generates an instance of the result interface based on the structured output (e.g., according to aspects of operations 210 and 212). It should be understood that the structured output may be provided for any one of a variety of alternative or additional processes, e.g., in the case where procedural code processes the structured output itself rather than an object instantiated based on the structured output. Method 300 terminates at operation 324.

[0066] Figure 4A and Figure 4B shows an overview of an example generative machine learning model that may be used in accordance with aspects described herein. Referring first to Figure 4A , the conceptual diagram 400 depicts an overview of a pre-trained generative model package 404 in accordance with aspects described herein, which processes an input and data format description 402 to generate a structured model output 406 for a result interface. Examples of the pre-trained generative model package 404 include but are not limited to the Megatron-Turing Natural Language Generation Model (MT-NLG), Generative Pretrained Transformer 3 (GPT-3), Generative Pretrained Transformer 4 (GPT-4), BigScience BLOOM (Large Open Science Open Access Multilingual Language Model), DALL-E, DALL-E2, Stable Diffusion, or Jukebox.

[0067] In an example, the generative model package 404 is pre-trained based on various inputs (e.g., various human languages, various programming languages, and / or various content types), and thus does not need to be fine-tuned or trained for a specific scenario. Instead, the generative model package 404 can be pre-trained more generally such that the input 402 includes a prompt that is generated, selected, or otherwise designed to guide the generative model package 404 to produce a specific generative model output 406. For example, the prompt includes context and / or one or more completion prefixes to pre-load the generative model package 404 accordingly. As a result, the generative model package 404 is guided based on the prompt to generate an output that includes a predicted sequence of tokens related to the prompt (e.g., not exceeding the token limit of the generative model package 404). In an example, the predicted sequence of tokens is further processed (e.g., via output decoding 416) to produce the output 406. For example, each token is processed to identify the corresponding word, word fragment, or other content that forms at least a part of the output 406. It should be understood that the input 402 and the generative model output 406 can each include any one of various content types, including but not limited to text output, image output, audio output, video output, procedural output, and / or binary output, among other examples. In an example, the input 402 and the generative model output 406 can have different content types, as can be the case when the generative model package 404 includes a generative multimodal machine learning model.

[0068] Accordingly, the generative model package 404 can be used in any of a variety of scenarios, and further, different generative model packages can be used in place of the generative model package 404 with substantially no modification to other associated aspects (e.g., similar to those aspects described herein with respect to Figure 1 , Figure 2 and Figure 3 . Accordingly, the generative model package 404 operates as a tool for performing machine learning processing, where a specific input 402 to the generative model package 404 is programmatically generated or otherwise determined such that the generative model package 404 produces a model output 406 that can then be used for further processing.

[0069] The generative model package 404 can be provided or otherwise used according to any of a variety of paradigms. For example, the generative model package 404 can be used in a computing device (e.g., Figure 1 for local use by the computing device 104), or can be remotely accessed from a machine learning service (e.g., the structured results service 102). In other examples, aspects of the generative model package 404 are distributed across multiple computing devices. In some instances, the generative model package 404 can be accessed via an application programming interface (API), such as may be provided by an operating system of a computing device and / or by other examples such as a machine learning service.

[0070] Now referring to the illustrated aspects of the generative model package 404, the generative model package 404 includes input tokenization 408, input embedding 410, model layers 412, output layer 414, and output decoding 416. In an example, the input tokenization 408 processes the input 402 to generate the input embedding 410, which includes a sequence of symbolic representations corresponding to the input 402. Thus, the input embedding 410 is processed by the model layers 412, output layer 414, and output decoding 416 to produce the model output 406. Figure 4B An example architecture corresponding to the generative model package 404 is depicted in, and this example architecture is discussed in further detail below. Even so, it should be understood that the architectures shown and described herein should not be taken in a limiting sense, and in other examples, any of a variety of other architectures may be used.

[0071] Figure 4B is a conceptual diagram depicting an example architecture 450 of a pre-trained generative machine learning model that can be used in accordance with the aspects described herein. As noted above, various alternative architectures and corresponding ML models may be used in other examples without departing from the aspects described herein.

[0072] As shown, the architecture 450 processes the input 402 to produce the generative model output 406, aspects of which have been discussed above with respect to Figure 4A The architecture 450 is depicted as a transformer model including an encoder 452 and a decoder 454. The encoder 452 processes the input embedding 458 (aspects of which may be similar to Figure 4A the input embedding 410 in), and the input embedding includes a sequence of symbolic representations corresponding to the input 456. In an example, the input 456 includes the input and data format description 402 corresponding to the result interface in which the model output will be stored, as may have been generated by Figure 1 the pattern generator 120 and / or the prompt generator 110 in, for example, by performing aspects of Figure 2 and Figure 3 operations 202 to 206 and / or operations 302 to 304 in respectively.

[0073] Additionally, the positional encoding 460 can introduce information about the relative and / or absolute positions of the tokens of the input embedding 458. Similarly, the output embedding 474 includes a sequence of symbolic representations corresponding to the output 472, and the positional encoding 476 can similarly introduce information about the relative and / or absolute positions of the tokens of the output embedding 474.

[0074] As shown, the encoder 452 includes an example layer 470. It should be understood that any number of such layers can be used, and the depicted architecture is simplified for illustrative purposes. The example layer 470 includes two sub-layers: the multi-head attention layer 462 and the feed-forward layer 466. In the example, residual connections are included around each of the layers 462, 466, followed by a normalization layer 464 and a normalization layer 468, respectively.

[0075] The decoder 454 includes an example layer 490. Similar to the encoder 452, any number of such layers can be used in other examples, and the architecture of the depicted decoder 454 is simplified for illustrative purposes. As shown, the example layer 490 includes three sub-layers: the masked multi-head attention layer 478, the multi-head attention layer 482, and the feed-forward layer 486. Aspects of the multi-head attention layer 482 and the feed-forward layer 486 can be similar to those discussed above with respect to the multi-head attention layer 462 and the feed-forward layer 466, respectively. Additionally, the masked multi-head attention layer 478 performs multi-head attention on the output of the encoder 452 (e.g., the output 472). In the example, the masked multi-head attention layer 478 prevents positions from attending to subsequent positions. This mask in combination with the offset embedding (e.g., offset by one position, as shown for the multi-head attention layer 482) ensures that the prediction for a given position depends on the known outputs of one or more positions less than the given position. As shown, residual connections are also included around the layers 478, 482, and 486, followed by a normalization layer 480, 484, and 488, respectively.

[0076] The multi-head attention layers 462, 478, and 482 can each linearly project the queries, keys, and values to corresponding dimensions using a set of linear projections. Each linear projection can be processed using an attention function (e.g., dot product or additive attention) to produce an n-dimensional output value for each linear projection. The resulting values can be concatenated and projected again such that the values are then processed as Figure 4B shown (e.g., by the corresponding normalization layers 464, 480, or 484).

[0077] The feed-forward layers 466 and 486 can each be a fully-connected feed-forward network applied to each position. In an example, the feed-forward layers 466 and 486 each include a plurality of linear transformations with rectified linear unit activations between the linear transformations. In an example, each linear transformation is the same across different positions and can use different parameters compared to other linear transformations of the feed-forward network.

[0078] Additionally, aspects of the linear transformation 492 can be similar to the linear transformations discussed above with respect to the multi-head attention layers 462, 478, and 482 and the feed-forward layers 466 and 486. The Softmax (normalized exponential function) 494 can also convert the output of the linear transformation 492 into predicted next token probabilities, as shown by the output probabilities 496. It should be understood that the illustrated architecture is provided as an example, and in other examples, any of a variety of other model architectures can be used in accordance with the disclosed aspects. In some cases, multiple iterations of processing are performed in accordance with the above aspects (e.g., using the generative model package 404 in Figure 4A or the encoder 452 and decoder 454 in Figure 4B ) to generate a sequence of output tokens (e.g., words), which can then be combined, for example, to produce a complete sentence (and / or any of a variety of other things). It should be understood that other generative models can generate multiple output tokens in a single iteration and can thus use a reduced number of iterations or a single iteration.

[0079] Thus, the output probabilities 496 can thus form a structured model output 406 in accordance with the aspects described herein such that the output of the generative ML model (e.g., which can include a structured output) is used, for example, processed to generate an instance of a result interface (e.g., similar to the aspects of operation 212 of method 200 in Figure 3 ). In other examples, the structured model output 406 is processed by procedural code (e.g., as a supplement to or alternative to generating an instance of the result interface as described above).

[0080] Figure 5 - Figure 7 and the associated description provide a discussion of various operating environments in which aspects of the present disclosure can be practiced. However, the devices and systems shown and discussed with respect to Figure 5 - Figure 7 are for purposes of example and illustration and are not limiting of the large number of computing device configurations that can be used to practice the aspects of the present disclosure described herein.

[0081] Figure 5 is a block diagram showing the physical components (e.g., hardware) of a computing device 500 in which aspects of the present disclosure can be practiced. The computing device components described below can be applicable to the computing devices described above, including one or more devices associated with the structured result service 102, and with respect to Figure 1 The computing device 104 discussed. In a basic configuration, the computing device 500 can include at least one processing unit 502 and system memory 504. Depending on the configuration and type of the computing device, system memory 504 can include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of these memories.

[0082] System memory 504 can include an operating system 505 and one or more program modules 506 suitable for running software applications 520, such as one or more components supported by the systems described herein. As an example, system memory 504 can include a prompt generator 524 and an object generator 526. The operating system 505 can be suitable for controlling the operation of the computing device 500, for example.

[0083] Furthermore, embodiments of the present disclosure can be practiced in conjunction with graphics libraries, other operating systems, or any other application programs, and are not limited to any particular application or system. This basic configuration is Figure 5 illustrated by those components within the dashed line 508. The computing device 500 can have additional features or functionality. For example, the computing device 500 can also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage is Figure 5 illustrated by removable storage device 509 and non-removable storage device 510.

[0084] As described above, multiple program modules and data files can be stored in system memory 504. When executed on the processing unit 502, program modules 506 (e.g., applications 520) can perform processes including, but not limited to, aspects described herein. Other program modules that can be used in accordance with aspects of the present disclosure can include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, etc.

[0085] Furthermore, embodiments of the present disclosure can be practiced on the following circuits: including discrete electronic elements, a package or integrated electronic chip including logic gates, a circuit utilizing a microprocessor, or a single chip including electronic elements or a microprocessor. For example, embodiments of the present disclosure can be practiced via a system on a chip (SOC), where Figure 5 Each or many of the components shown can be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operating via the SOC, the functions described herein regarding the capabilities of the client switching protocol can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of computing device 500. Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluidic, and quantum technologies. Additionally, embodiments of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0086] Computing device 500 may also have one or more input devices 512, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. (Multiple) output devices 514, such as a display, speaker, printer, etc., may also be included. The above devices are examples, and other devices may be used. Computing device 500 may include one or more communication connections 516 that allow communication with other computing devices 550. Examples of suitable communication connections 516 include but are not limited to radio frequency (RF) transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0087] As used herein, the term computer-readable medium may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 504, removable storage device 509, and non-removable storage device 510 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic tape cartridges, tapes, disk storage, or other magnetic storage devices, or any other article that can be used to store information and can be accessed by computing device 500. Any such computer storage media may be part of computing device 500. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0088] A communication medium is embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery medium. The term "modulated data signal" can describe a signal having one or more characteristics set or changed in a manner that encodes information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0089] Figure 6 System 600 is shown, which can be, for example, a mobile computing device such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., and embodiments of the present disclosure can be practiced using system 600. In one embodiment, system 600 is implemented as a "smart phone" capable of running one or more applications (e.g., a browser, an email client, a calendar, a contact manager, a messaging client, a game, and a media client / player). In some aspects, system 600 is integrated as a computing device such as an integrated personal digital assistant (PDA) and a wireless phone.

[0090] In a basic configuration, such a mobile computing device is a handheld computer having both input and output elements. System 600 generally includes a display 605 and one or more input buttons that allow a user to input information into system 600. The display 605 can also be used as an input device (e.g., a touch screen display).

[0091] If included, an optional bypass input element allows for further user input. For example, the bypass input element can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, system 600 can include more or fewer input elements. For example, in some embodiments, the display 605 may not be a touch screen. In another example, an optional keypad 635 can also be included, and the optional keypad 635 can be a physical keypad or a "soft" keypad generated on a touch screen display.

[0092] In various embodiments, the output elements include a display 605 for presenting a graphical user interface (GUI), visual indicators (e.g., light-emitting diodes 620), and / or audio transducers 625 (e.g., speakers). In some aspects, a vibration transducer is included to provide haptic feedback to the user. In yet another aspect, input and / or output ports are included, such as an audio input (e.g., a microphone jack) for sending signals to or receiving signals from an external device, an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0093] One or more applications 666 may be loaded into the memory 662 and run on or in association with the operating system 664. Examples of applications include a telephone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. System 600 also includes a non-volatile storage area 668 within the memory 662. The non-volatile storage area 668 may be used to store persistent information that should not be lost when system 600 is powered down. Applications 666 may use and store information in the non-volatile storage area 668, such as emails or other messages used by an email application, etc. A synchronization application (not shown) also resides on system 600 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 668 synchronized with the corresponding information stored at the host computer. It should be understood that other applications may be loaded into the memory 662 and run on system 600 described herein.

[0094] System 600 has a power supply 670, which may be implemented as one or more batteries. The power supply 670 may also include an external power source, such as an AC adapter or a charging docking station that supplements or recharges the battery.

[0095] System 600 may also include a radio interface layer 672 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 672 facilitates a wireless connection between system 600 and the "outside world" via a communication carrier or service provider. Transmissions to and from the radio interface layer 672 are under the control of the operating system 664. In other words, communications received by the radio interface layer 672 may be propagated to the applications 666 via the operating system 664, and vice versa.

[0096] The visual indicator 620 can be used to provide visual notifications, and / or the audio interface 674 can be used to generate audible notifications via the audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light-emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to the power supply 670 such that when activated, these devices remain on for a duration specified by the notification mechanism even if the processor 660 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes an action to indicate the powered-on state of the device. The audio interface 674 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 674 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to facilitate control of notifications, as will be described below. The system 600 can also include a video interface 676, which enables operation of the on-board camera 630 to record still images, video streams, etc.

[0097] It should be understood that the system 600 can have additional features or functionality. For example, the system 600 can also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage is Figure 6 shown by the non-volatile storage area 668.

[0098] As described above, the data / information generated or collected and stored via the system 600 can be stored locally, or the data can be stored on any number of storage media that can be accessed by the device via the radio interface layer 672 or via a wired connection between the system 600 and a separate computing device associated with the system 600 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via the radio interface layer 672 or via a distributed computing network. Similarly, such data / information can be easily transferred between computing devices for storage and use according to any of a variety of data / information transfer and storage components, including email and collaborative data / information sharing systems.

[0099] Figure 7 An aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a personal computer 704, a tablet computing device 706, or a mobile computing device 708, is shown, as described above. The content displayed at the server device 702 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 724, a web portal 725, a mailbox service 726, an instant messaging store 728, or a social networking site 730.

[0100] A structured results manager 720 (e.g., similar to application 520) may be employed by a client communicating with server device 702. Additionally or alternatively, a structured results service 721 may be employed by server device 702. Server device 702 may provide data to and receive data from client computing devices via network 715, such as personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smart phone). As an example, the computer system described above may be embodied in personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smart phone). In addition to receiving graphical data that may be used for preprocessing at a graphical source system or postprocessing at a receiving computing system, any of these examples of computing devices may obtain content from storage 716.

[0101] It should be understood that the aspects and functions described herein may operate on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions may operate remotely from each other on a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information may be displayed via an on-board computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information may be displayed on and interacted with a wall surface on which various types of user interfaces and information are projected. Interaction with the numerous computing systems with which embodiments of the present disclosure may be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, where the associated computing device is equipped with detection (e.g., camera) functions for capturing and interpreting user gestures to control the functions of the computing device, etc.

[0102] As will be appreciated from the foregoing disclosure, one aspect of the present technology relates to a system that includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: generating a data format description for a result interface of program code, where the result interface is defined by the program code to receive an output of a machine learning model; generating a machine learning (ML) processing request that includes the generated data format description for the result interface; in response to the ML processing request, receiving a structured model output corresponding to the data format description for the result interface; based on the structured model output, generating an instance of the result interface; and providing the instance of the result interface for processing by the program code. In an example, generating an instance of the result interface includes: identifying attributes of the structured model output; and populating corresponding attributes of the result interface based on values of the attributes identified from the structured model output. In another example, the ML processing request is a first ML processing request; and generating a data format description for the result interface includes: generating a second ML processing request that includes a schema for the result interface; and in response to the second ML processing request, receiving a model output that includes the data format description. In another example, generating a data format description for the result interface includes: generating a schema for the result interface; and processing the generated schema to generate the data format description. In yet another example, processing the generated schema to generate the data format description includes: generating a summary representation for the result interface based on the schema. In yet another example, the data format description includes instructions for generating a model output in accordance with the summary representation for the result interface. In another example, the ML processing request further includes a representation of a first type of object; and the result interface is a second type of object different from the first type. In another example, the ML processing request includes at least one of: an input to be processed by an ML model associated with the ML processing request; or an indication of a previously generated model output.

[0103] In another aspect, the technology relates to a method. The method includes: receiving, from a computing device, a machine learning (ML) processing request that includes a description of a result interface; processing the ML processing request using an ML model to generate a structured model output according to the description of the result interface; validating the structured model output; and providing the structured model output in response to the ML processing request based on determining that the structured model output is validated. In an example, validating the structured model output includes at least one of the following: validating the syntax of the structured model output; or evaluating the structured model output compared to the description of the result interface. In another example, the description of the result interface is the original schema of the result interface; the method further includes processing the original schema to generate a summary representation of the original schema of the result interface; and the ML processing request is processed according to the generated summary representation for the result interface. In another example, the ML processing request is a first ML processing request; and the structured model output is a first instance of the structured model output; and the method further includes: receiving a second ML processing request; generating a second instance of the structured model output for the second ML processing request; validating the second instance of the structured model output; and performing a remedial action based on determining that the second instance of the structured model output is not validated. In yet another example, the remedial action is at least one of the following: processing the second instance of the structured output to correct a malformed syntax of the second instance of the structured output; evaluating a secondary output of the ML model generated based on the second ML processing request; or providing a request to the ML model to process the second instance of the structured output and generate a third instance of the structured output.

[0104] In another aspect, the technology relates to another method. The method includes: as a process, generating an ML processing request as a result of programming machine learning (ML) calls that define a result interface to receive the output of a machine learning model, the ML processing request including a description of the result interface and at least one of the following: an input to be processed or an indication of a previously generated model output; in response to the ML processing request, receiving a structured model output corresponding to the description of the result interface; based on the structured model output, generating an instance of the result interface; and providing the instance of the result interface for processing by program code. In an example, generating an instance of the result interface includes: identifying an attribute of the structured model output; and populating a corresponding attribute of the result interface based on a value of the attribute identified from the structured model output. In another example, the ML processing request is a first ML processing request; and the method further includes: generating a second ML processing request including a schema for the result interface; and in response to the second ML processing request, receiving a model output including a description of the result interface. In another example, the method further includes: generating a schema for the result interface; and processing the generated schema to generate a description of the result interface. In yet another example, processing the generated schema to generate a description includes: generating a summary representation for the result interface based on the schema. In yet another example, the ML processing request includes instructions for generating a model output in accordance with the description of the result interface. In another example, the ML processing request further includes a representation of an object of a first type; and the result interface is an object of a second type different from the first type.

[0105] For example, aspects of the present disclosure have been described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to aspects of the present disclosure. The functions / actions recited in the blocks may occur out of the order shown in any flowchart. For example, depending on the functionality / action involved, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order.

[0106] The description and illustration of one or more aspects provided in this application are not intended to limit or define the scope of the present disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the claimed aspects of the present disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Whether shown and described in combination or separately, various features (both structures and methods) are intended to be selectively included or omitted to produce embodiments having a particular set of features. Having provided the description and illustration of this application, those skilled in the art can envision variations, modifications, and alternative aspects that fall within the spirit of the broader aspects embodied in this application, which do not depart from the broader scope of the claimed disclosure.< / json>

Claims

1. A system, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations including: generating a data format description for a result interface, wherein the result interface is defined by the program code to receive an output of a machine learning model; generating a machine learning (ML) processing request that includes the generated data format description for the result interface; in response to the ML processing request, receiving a structured model output corresponding to the data format description for the result interface; generating an instance of the result interface based on the structured model output; and providing the instance of the result interface for processing by the program code.

2. The system according to claim 1, wherein generating the instance of the result interface includes: identifying attributes of the structured model output; and populating corresponding attributes of the result interface based on values of the attributes identified from the structured model output.

3. The system according to claim 1, wherein: the ML processing request is a first ML processing request; and generating the data format description for the result interface includes: generating a second ML processing request that includes a schema for the result interface; and in response to the second ML processing request, receiving a model output that includes the data format description.

4. The system according to claim 1, wherein generating the data format description for the result interface includes: generating a schema for the result interface; and processing the generated schema to generate the data format description.

5. The system according to claim 4, wherein processing the generated pattern to generate the data format description includes: Generating a summary representation for the result interface based on the schema.

6. The system according to claim 5, wherein the data format description includes instructions for generating a model output in accordance with the summary representation for the result interface.

7. The system according to claim 1, wherein: the ML processing request further includes a representation of a first type of object; and the result interface is a second type of object different from the first type.

8. The system according to claim 1, wherein the ML processing request includes at least one of the following: an input to be processed by an ML model associated with the ML processing request; or an indication of a previously generated model output.

9. A method, comprising: receiving, from a computing device, a machine learning (ML) processing request that includes a description of a result interface; processing the ML processing request using an ML model to generate a structured model output according to the description of the result interface; validating the structured model output; and in response to determining that the structured model output is validated, providing the structured model output in response to the ML processing request.

10. The method according to claim 9, wherein validating the structured model output includes at least one of the following: validating the syntax of the structured model output; or evaluating the structured model output against the description of the result interface.

11. The method according to claim 9, wherein: the description of the result interface is the original schema of the result interface; the method further comprises processing the original schema to generate a summary representation of the original schema of the result interface; and the ML processing request is processed according to the generated summary representation for the result interface.

12. The method according to claim 9, wherein: the ML processing request is a first ML processing request; and the structured model output is a first instance of the structured model output; and the method further comprises: receiving a second ML processing request; generating a second instance of the structured model output for the second ML processing request; validating the second instance of the structured model output; and based on determining that the second instance of the structured model output is not validated, performing a remedial action.

13. The method according to claim 12, wherein the remedial action is at least one of the following: processing the second instance of the structured output to correct the malformed syntax of the second instance of the structured output; evaluating a secondary output of the ML model generated based on the second ML processing request; or providing a request to the ML model to process the second instance of the structured output and generate a third instance of the structured output.

14. A method, comprising: generating an ML processing request as a result of processing a programming machine learning (ML) call that defines a result interface to receive an output of a machine learning model, the ML processing request including a description of the result interface and at least one of the following: an indication of an input to be processed or a previously generated model output; in response to the ML processing request, receiving a structured model output corresponding to the description of the result interface; based on the structured model output, generating an instance of the result interface; and providing the instance of the result interface for processing by the program code.

15. The method according to claim 14, wherein generating the instance of the result interface comprises: identifying attributes of the structured model output; and populating corresponding attributes of the result interface based on values of the attributes identified from the structured model output.

Images