Multi-stage machine learning model chaining

Abstract

A skill chain is generated that includes a set of ML model evaluations for processing the inputs and is ultimately used to generate a model output accordingly. Each ML model evaluation corresponds to a "model skill" in the skill chain. The intermediate output generated by a first ML evaluation for a first model skill in the skill chain may then be processed as input to a second ML evaluation for a second model skill in the skill chain, thereby ultimately generating a model output for the given input. Such a skill chain may include any number of skills according to any of a variety of structures, and the evaluations need not all use the same ML model.

Description

[Background technology]

[0001] background A single evaluation using a machine learning model may have limited utility, especially for more complex tasks and when the user is unfamiliar with the machine learning model and / or the task at hand. Thus, such evaluation may result in a poor user experience, increased user frustration, and / or wasted computational resources, among other disadvantages.

[0002] It is with respect to these and other general considerations that the embodiments are described, and while relatively specific problems are discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. Summary of the Invention [Means for solving the problem]

[0003] overview Aspects of the present application relate to multi-stage machine learning model chaining, in which a skill chain is created that includes a set of ML model evaluations for processing inputs, and ultimately used to generate model outputs accordingly. Each ML model evaluation corresponds to a "model skill" in the skill chain. For example, a model skill has an associated prompt template that is used to generate prompts (e.g., including inputs and / or context) that are processed using the corresponding ML model to generate model outputs accordingly. In other examples, the ML model associated with a model skill need not have an associated prompt template, such as when prompting is not used by the ML model in processing inputs and generating model outputs.

[0004] The intermediate output produced by a first ML evaluation for a first model skill in the skill chain may then be processed as input to a second ML evaluation for a second model skill in the skill chain, thereby ultimately producing a model output for the given input. Such a skill chain may include any number of skills according to any of a variety of structures and need not be evaluated using the same ML model.

[0005] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] BRIEF DESCRIPTION OF THE DRAWINGS Non-limiting and non-exhaustive examples are described with reference to the following figures: [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates a schematic of an example system in which multi-stage machine learning model chaining may be used in accordance with aspects of the present disclosure. [Figure 2] FIG. 1 illustrates an example conceptual diagram for processing user input to generate model output using chained machine learning models, according to aspects described herein. [Figure 3] 1 illustrates an overview of an example method for processing user input to generate model output according to aspects described herein. [Figure 4] 1 illustrates an overview of an example method for processing user input according to prompts using a generative ML model, according to aspects described herein. [Figure 5A] 1 illustrates a schematic of an exemplary generative machine learning model that may be used in accordance with aspects described herein. [Figure 5B]1 illustrates a schematic of an exemplary generative machine learning model that may be used in accordance with aspects described herein. [Figure 6] FIG. 1 is a block diagram illustrating exemplary physical components of a computing device in which aspects of the present disclosure may be implemented. [Figure 7] FIG. 1 is a simplified block diagram of a computing device in which aspects of the present disclosure may be implemented. [Figure 8] FIG. 1 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0008] Detailed Description In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which specific embodiments or examples are shown by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the disclosure. The embodiments may be embodied as methods, systems, or devices. Thus, the embodiments may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0009] In some examples, a machine learning (ML) model generates a model output based on an input (e.g., as may be received from a user). For example, natural language input from a user is processed using a generative ML model to appropriately generate a model output for the natural language input. However, the limited nature of such a single evaluation can result in reduced practicality, especially when a user is inexperienced / unfamiliar with the ML model (e.g., the user may provide input that results in limited use of the ML model and / or unexpected behavior of the ML model). Similarly, using an ML model with a single evaluation can limit the tasks for which the model can be used, among other disadvantages.

[0010] Accordingly, aspects of the present application relate to multi-stage machine learning model chaining, in which inputs are processed using a set of ML model evaluations (e.g., chained together in a "skill chain") to ultimately generate a model output for a given input. Each ML model evaluation corresponds to a "model skill" in the skill chain. For example, a model skill has an associated prompt template, which is used to generate prompts (e.g., including input and / or context) that are processed using the corresponding ML model and model output is generated accordingly. In other examples, the ML model associated with a model skill need not have an associated prompt template, such as when prompting is not used by the ML model in processing inputs and generating model output.

[0011] The intermediate output generated by a first skill in the skill chain may then be processed by a second skill to generate a subsequent output, as appropriate. It will be understood that such a skill chain can include any number of skills, and the model skills need not be associated with the same ML model. For example, a generative model may generate natural language output, while a recognition model (or any of various other types of ML models) may process the intermediate output from the generative model to generate a model output, as appropriate. The output of the recognition model may be provided as a final model output or may be an intermediate output that is processed using another skill in the skill chain.

[0012] A generative model (also generally referred to herein as a type of ML model) used in accordance with the embodiments described herein may generate any of a variety of output types (and thus, in some examples, may be a multimodal generative model), and in some examples, may be a generative Transformer model and / or a large-scale language model (LLM), a generative image model. Exemplary ML models include, but are not limited to, Megatron-Turing Natural Language Generation model (MT-NLG), Generative Pre-trained Transformer 3 (GPT-3), Generative Pre-trained Transformer 4 (GPT-4), BigScience BLOOM (Large Open-science Open-access Multilingual Language Model), DALL-E, DALL-E 2, Stable Diffusion, or Jukebox. Additional examples of such embodiments are described below with respect to the generative ML models shown in Figures 5A-5B.

[0013] The output of a skill that is processed by a subsequent skill may be referred to herein as an "intermediate output." Exemplary intermediate outputs include, but are not limited to, natural language output, image data output, video data output, program output, and / or binary output. Thus, it will be understood that the intermediate output may have any number of output "streams" (e.g., each having an associated content type).

[0014] In some examples, the intermediate output includes a structured output, which may include one or more tags, key / value pairs, and / or metadata, among other examples. For example, a stream may be presented according to an associated tag in such a structured output. In some examples, a skill's prompt template defines or otherwise includes instructions related to such a structured output, thereby causing a generative ML model associated with the skill to generate the structured output accordingly. As such, the use of structured output can increase the degree of determinism of the model output and, therefore, improve reliability when chaining multiple ML model evaluations together in accordance with aspects described herein. In other examples, the intermediate output may resemble a final model output, which may be provided to a user in other ways or for further processing by an application, among other examples.

[0015] According to aspects described herein, skills may be chained together according to any of a variety of techniques. For example, a skill chain may include, among other examples, one or more sequential skills, a hierarchical set of skills, a set of parallel skills, and / or a skill that depends on or otherwise processes output from two or more skills. Accordingly, it will be understood that a skill chain may be a graph or arranged according to any of a variety of other data structures. Furthermore, a skill chain may include any of a variety of other types of skills. For example, one or more model skills may be chained together with a program skill. For example, a program skill may read the contents of a file, obtain data from a data source and / or a user, send an electronic message including model output, create a file including model output, and / or execute program output generated by a model skill.

[0016] In some examples, a skill library stores model skills and / or program skills, from which a set of skills can be identified and used to generate a skill chain accordingly (e.g., to perform a set of related ML model evaluations). As an example, a chain orchestrator can extract intent from a given input (e.g., from a user and / or application), which may be mapped to one or more skills in the skill library, thereby generating a chain model and / or program skills for processing the input. In some examples, new skills are dynamically identified and added to the skill library (e.g., by an application or by a user or developer) so that they can be used as part of a skill chain for processing a given input according to aspects described herein.

[0017] As described above, a skill may include or otherwise be associated with a prompt template. One or more fields, regions, and / or other portions of the prompt template may be populated (e.g., with input and / or context) to generate a prompt that is processed by an ML model in accordance with aspects described herein. For example, the prompt may be used to pre-set up an ML model, thereby directing the model to generate output corresponding to the prompt template. It will be understood that a prompt template may include any of a variety of data, including, but not limited to, natural language, image data, audio data, video data, and / or binary data, among other examples.

[0018] Thus, in some examples, a skill, as used herein, invokes processing by an ML model (e.g., according to an associated prompt template) to process a given input (e.g., as may be received from a user or as intermediate output from another skill). In some examples, context is processed as part of the ML model evaluation. For example, among other examples, the input may include instructions regarding the context, the skill may define the context provided to the ML model, and / or a chain orchestrator (e.g., that defines and / or manages the processing of a skill chain) may determine the context used for the ML model evaluation, as appropriate.

[0019] In addition to chaining ML model evaluation prompts, associated contexts may be shared among or otherwise used by multiple model skills. For example, at least a portion of the context used for processing associated with a first model skill (or, in other examples, multiple model skills) may be used by a second model skill. In some examples, the context is modified by a first model evaluation (e.g., of a first model skill) prior to or concurrent with processing by a second ML model evaluation (e.g., for a second model skill) such that the second ML model evaluation uses the updated context accordingly.

[0020] As a result of the disclosed chaining techniques, it may be possible to accomplish tasks and / or generate model outputs that would not have been possible using a single ML model evaluation. For example, during the processing of a skill chain, information may be obtained from one or more data sources and / or user input may be requested, which is then used in subsequent processing (e.g., by one or more subsequent skills in the skill chain). As another example, the skill chain's evaluation may be dynamically adapted as a result of component evaluations, thereby affecting one or more future evaluations of the skill chain (e.g., by adding, removing, or modifying evaluations). Furthermore, the skill chain itself may be managed, orchestrated, and / or derived by an ML model (e.g., by a generative ML model based on natural language input received from a user and / or input generated by an application or otherwise received from an application). Furthermore, assuming that different ML models (e.g., each may produce a different type of model output) can be chained together, the resulting model output may be an output that would not otherwise be produced as a result of processing by a single ML model.

[0021] 1 illustrates an overview of an example system 100 in which multi-stage machine learning model chaining may be used in accordance with aspects of the present disclosure. As shown, system 100 includes a machine learning service 102, a computing device 104, and a network 106. In some examples, machine learning service 102 and computing device 104 communicate over network 106, which may include a local area network, a wireless network, or the Internet, or any combination thereof, among other examples.

[0022] As shown, the machine learning service 102 includes a chain orchestrator 108, a model repository 110, a skill library 112, and a semantic memory store 114. In some examples, the machine learning service 102 receives a request from a computing device 104 (e.g., from a multi-stage machine learning framework 118) to generate a model output that can be generated using a skill chain as described herein. As mentioned above, the request can include input (e.g., as can be user input received from a user at the computing device 104 and / or generated by an application 116).

[0023] The received request is processed by the chain orchestrator 108, which may identify one or more ML models from the model repository 110 and process the input accordingly. In an example, the chain orchestrator 108 processes the request to generate a skill chain for producing a model output (e.g., using one or more models from the model repository 110). For example, the chain orchestrator 108 uses a generative ML model to process at least a portion of the input (e.g., in conjunction with prompts generated from a prompt template using the input), thereby generating a skill chain that includes one or more model skills (and, in some examples, one or more program skills), as appropriate. The chain orchestrator 108 then processes the resulting skill chain according to aspects described herein, for example, using one or more models from the model repository 110, skills from the skill libraries 112 and / or 120, and / or context from the semantic memory stores 114 and / or 122.

[0024] In another example, the request includes a prompt used to pre-set up the ML model (e.g., generated using a prompt template for the skill from the skill library 120 of the computing device 104). As another example, the request includes an indication of a skill in the skill library 112, such that the chain orchestrator 108 generates a corresponding prompt based on the skill from the skill library 112, as appropriate. Additionally or alternatively, the request includes a context in which the request should be processed (e.g., from the semantic memory store 122 of the computing device 104). As a further example, the request includes an indication of a context in the semantic memory store 114, such that the chain orchestrator 108 retrieves the context from the semantic memory store 114, as appropriate. Additional examples of these and other aspects are described below with respect to the semantic memory store 218 and corresponding recall engine 210 of FIG. 2. Such aspects may be used when the multi-stage machine learning framework 118 performs aspects similar to the chain orchestrator 108, such that the multi-stage machine learning framework 118 generates skill chains and / or manages the processing of skill chains accordingly.

[0025] In some examples, the chain orchestrator 108 obtains additional information to be used in processing the request (e.g., as may be obtained from a remote data source or as may be requested from a user of the computing device 104). For example, the chain orchestrator 108 may determine to obtain additional information for a given evaluation of a skill chain, among other examples. As one example, the additional information may be obtained via a programmatic skill (e.g., as may have been included in the skill chain by the chain orchestrator 108). Examples of such aspects are described in more detail below with respect to methods 300 and 400 of FIGS. 3 and 4, respectively.

[0026] The model repository 110 may include any number of different ML models. For example, the model repository 110 may include a base model, a language model, a speech model, a video model, and / or an audio model. As used herein, a base model is a model pre-trained on a wide range of data applicable to a wide range of tasks (e.g., a model capable of processing a variety of different tasks or modalities). In some examples, the multimodal machine learning models in the model repository 110 may have been trained using training data having multiple content types. Thus, given a first type of content, the ML models in the model repository 110 may generate content having any of a variety of related types. It will be understood that the model repository 110 may include, among other examples, base models and fine-tuned models (e.g., for a particular context and / or a particular user or set of users).

[0027] Referring now to computing device 104, computing device 104 includes application 116, multi-stage machine learning framework 118, skill library 120, and semantic memory store 122. In some examples, application 116 uses multi-stage machine learning framework 118 to process user input and generate model output accordingly, which may be presented to a user of computing device 104 and / or used for subsequent processing by application 116, among other examples.

[0028] In some examples, aspects of the multi-stage machine learning framework 118 are similar to the chain orchestrator 108 and therefore will not necessarily be described again in detail. For example, in addition to or as an alternative to generating a skill chain by the chain orchestrator 108, the multi-stage machine learning framework 118 may generate and / or manage the evaluation of the skill chain in accordance with aspects described herein. For example, the multi-stage machine learning framework 118 provides instructions of user input to the machine learning service 102 such that a skill chain is generated by the machine learning service 102 and received by the computing device 104 in response. Thus, the multi-stage machine learning framework 118 manages the evaluation of the skill chain (e.g., generating subsequent requests to the machine learning service 102 for component model skills) according to one or more associated prompt templates (e.g., as may be stored by the skill library 112 / 120) and / or based on associated context (e.g., from the semantic memory store 114 / 122). In some examples, the multi-stage machine learning framework 118 requests model outputs from the machine learning service 102 for model skills in a skill chain, and the program skills in the skill chain can be processed locally on the computing device 104 (or in other examples, remotely from the computing device 104).

[0029] Accordingly, it will be appreciated that aspects of the disclosure may be implemented according to any of a variety of paradigms. For example, skill chain generation / orchestration and / or prompt generation (e.g., based on prompt templates of model skills) may be performed client-side (e.g., by the multi-stage machine learning framework 118), server-side (e.g., by the chain orchestrator 108), or any combination thereof, among other examples. For example, the multi-stage machine learning framework 118 may perform a first ML evaluation associated with a first model skill stored by the skill library 120 of the computing device 104, and a second ML evaluation may be performed by the machine learning service 102 based on a second model skill stored by the skill library 112. The multi-stage machine learning framework 118 may be provided as part of the operating system of the computing device 104 (e.g., as a service, application programming interface (API), and / or framework), made available as a library included by (or more directly embedded by) the application 116, or provided as a standalone application, among other examples.

[0030] As another example, a user interface is provided through which a user may interact with the multi-stage machine learning framework and / or chain orchestrator. For example, the machine learning service 102 may additionally or alternatively implement aspects similar to the multi-stage machine learning framework 118 such that the machine learning service 102 additionally or alternatively provides a website through which a user may optionally interact with a console or terminal interface of the multi-stage machine learning framework. The console may include a text-based user interface through which a user inputs skills (e.g., model skills and / or program skills) that may be chained together. For example, a pipe ("|") operator may be used to chain skills together, thereby passing the output of one skill (e.g., a first model skill) as input to another skill (e.g., a second model skill). It will be appreciated that the input and / or output of one or more skills may additionally or alternatively be redirected according to any of a variety of other techniques (e.g., using the "<<" operator, the ">" operator, and / or the ">>" operator).

[0031] As a further example, the computing device 104 may include a user interface that is part of an application (e.g., application 116) or multiple applications (e.g., as a shared framework or as functionality provided by the operating system of the computing device 104). In such examples, natural language input may be provided via the user interface (e.g., as text input and / or as voice input), which may be processed according to aspects described herein and used to generate a skill chain accordingly. In some examples, skills in a skill chain may interact with one or more command interfaces, each of which may be associated with an application (e.g., application 116) and / or the operating system of the computing device 104, among other examples. For example, the operating system may provide a command interface through which interactions may be performed, for example, using an accessibility API and / or an extensibility API.

[0032] In some examples, a model skill may generate programmatic output based on received natural language input that is executed, parsed, or otherwise processed (e.g., as a programmatic skill) to interact with various functionality and / or other aspects of the computing device 104 (e.g., applications 116, system preferences, etc.). As such, a user of the computing device 104 may use such an interface to interact with application / device functionality using multi-stage machine learning model chaining in accordance with aspects described herein. While an example is described in which user input is received as natural language input and then results in a chain of ML evaluations, it will be understood that similar techniques may be used in examples in which such multi-stage machine learning model chaining is tied to user interface elements (e.g., in response to user actuation of a button, scrollbar, window, or menu) or other software processes, among other examples.

[0033] 2 illustrates an example conceptual diagram 200 for processing user input to generate model output using chained machine learning models according to aspects described herein. As shown, diagram 200 processes user input 202 according to a set of models (e.g., ML models 204 and 206, as orchestrated by chain orchestrator 203) to generate model output 208. For example, user input 202 may be received from a computing device, such as computing device 104 of FIG. 1. Aspects of chain orchestrator 203 may be similar to those described above with respect to chain orchestrator 108 and, therefore, will not necessarily be described again in detail below.

[0034] User input 202 may include any of a variety of inputs, including, but not limited to, natural language input, command line input, input received via a framework or application, and / or input received via a central service (e.g., of an operating system) or a uniform resource identifier (URI) handler, among other examples. While examples are described herein with respect to natural language input, it will be understood that any of a variety of additional input types may be received, including, but not limited to, image input and / or video input. Furthermore, natural input may include any of a variety of inputs, such as text input or voice input.

[0035] As shown, user input 202 is processed by chain orchestrator 203. In some examples, chain orchestrator 203 processes user input 202 to generate a skill chain including multiple model skills (e.g., including evaluation by ML model 204 and ML model 206), and ultimately generates model output 208 according to aspects described herein. Such aspects may be similar to those described above with respect to chain orchestrator 108, such that user input 202 is processed to extract intent that is mapped to one or more skills (e.g., from skill library 212).

[0036] To generate the skill chain, an orchestration prompt may be generated by the chain orchestrator 203, which includes an indication of one or more skills (also referred to herein as a "skill list") from the skill library 212 and at least a portion of the user input 202, such that the generative ML model generates a skill chain in which the user input 202 is processed. Thus, the chain orchestrator 203 appropriately maps one or more intents of the user input 202 to one or more model skills and / or program skills in the skill library 212. Additional examples of these and other aspects of the chain orchestrator 203 are described below with respect to operation 304 of the method 300 of FIG. 3 .

[0037] In some examples, the skill list is dynamically generated. As one example, the skill library 212 may include one or more files that each define one or more skills by which input (e.g., user input and / or intermediate output) can be processed. As another example, the skill library 212 includes a database that stores a list of skills. In some examples, a new skill may be registered (e.g., in a database or index), thereby indicating that the skill is available as part of a skill chain. For example, a plug-in application 214 (e.g., which may be similar in aspect to application 116) may include one or more skills that are registered in the skill library 212 such that processing using the skills of the plug-in application 214 can be performed in accordance with aspects described herein. It will therefore be understood that the set of skills in the skill library 212 may be stored using any of a variety of techniques.

[0038] In one example, when generating a skill list, the chain orchestrator 203 lists the contents of the skill library 212. A skill may include or otherwise have an associated description of its functionality (e.g., a manual page or usage information, such as syntax and / or one or more input / output instructions), at least a portion of which may be included in the skill list generated by the chain orchestrator 203 and used to generate the skill chain accordingly.

[0039] Thus, chain orchestrator 203 manages the processing of user input 202 according to the generated skill chain. As described above, such a skill chain may include, among other examples, one or more sequential skills, a hierarchical set of skills, a set of parallel skills, and / or a skill that depends on or otherwise processes the output from two or more previous skills. In some examples, the order of evaluation of the skill chain is determined based on available skills in skill library 212. Additionally or alternatively, although this example uses two model skills corresponding to ML models 204 and 206, the skill chain may include one or more programmatic skills.

[0040] As shown, once a skill chain is generated by chain orchestrator 203, user input 202 is processed by ML model 204 to generate intermediate output. For example, a prompt template corresponding to a first model skill may be populated or otherwise processed to generate a prompt (e.g., including at least a portion of user input 202 and / or context from semantic memory store 218) that is processed by ML model 204 accordingly. The intermediate output from ML model 204 is then processed by ML model 206. Each of ML models 204 and 206 may be the same or similar models (e.g., generating one or more of the same content types and / or trained using similar training data), or as another example, each of ML models 204 and 206 may be different models (e.g., generating different sets of content types). When processing inputs (e.g., user inputs 202 or intermediate outputs from a previous evaluation), each of ML models 204 and 206 may use skills from skills library 212 (e.g., as may have been determined by chain orchestrator 203 in accordance with aspects described herein or as may have been identified in other ways).

[0041] In some examples, ML model 204 and ML model 206 each use context obtained from recall engine 210, as may be stored by semantic memory store 218. For example, it may be determined (e.g., by chain orchestrator 203 and / or by ML model 204 or 206) that processing associated with a skill should be performed according to context from recall engine 210. In other examples, a skill from skill library 212 may indicate (e.g., as part of an associated prompt template) that context should be obtained from semantic memory store 218, such that recall engine 210 is used to obtain such context accordingly. Thus, it will be understood that context may be obtained for a skill as a result of any of a variety of decisions and / or instructions, among other examples.

[0042] As one example, semantic memory store 218 stores semantic embeddings (also referred to herein as "semantic addresses") associated with ML model 204 and / or ML model 206, each of which may correspond to one or more content objects. In some examples, entries in semantic memory store 218 include one or more semantic embeddings corresponding to context objects, and / or the context objects themselves or references to the context objects, among other examples.

[0043] In some examples, the semantic memory store 218 stores embeddings associated with one or more models (e.g., ML models 204 and / or 206) and their particular versions, which may thus represent the same or similar content, but in different semantic embedding spaces (e.g., as associated with each model / version). Furthermore, as new models are added or existing models are updated, one or more entries in the semantic memory store 218 may be re-encoded (e.g., by generating new semantic embeddings according to the new embedding spaces). In this way, a single content object entry in the semantic memory store 218 may have locatable semantic addresses across models / versions, thereby enabling search for content objects based on a similarity determination (e.g., as a result of an algorithmic comparison) between the corresponding semantic addresses and semantic context indications.

[0044] As a result, an input embedding may be generated (e.g., as may be associated with user input 202 and / or processing by ML model 204 or 206). For example, the input embedding may be generated by a machine learning model that appropriately encodes the intent corresponding to user input 202. Additionally or alternatively, the input embedding may be generated based on any of a variety of other inputs (e.g., audio input and / or visual input) received by the computer. Additional and / or alternative methods for generating input embeddings may be recognized by those skilled in the art.

[0045] Thus, based on the input embeddings, recall engine 210 may identify one or more content objects to serve as context for processing associated with a skill. For example, a set of semantic embeddings that match the input embeddings (e.g., using cosine distance, another geometric n-dimensional distance function, or other algorithmic similarity metric) may be identified and used to identify one or more corresponding content objects accordingly. As described above, processing by ML model 204 and / or ML model 206 may add, remove, or otherwise modify one or more entries in semantic memory store 218 such that the context from recall engine 210 used by a subsequent skill may be influenced by one or more previous skills.

[0046] As a result of the multi-stage ML model chaining performed by ML models 204 and 206, a model output 208 is generated. Thus, each of the one or more model skills in the skill chain may generate an intermediate output (e.g., a structured output), and the final skill in the skill chain (e.g., ML model evaluation by ML model 206, as shown) may generate model output 208 based on such intermediate outputs. By way of example, the final model output generated by ML model 206 may include, but is not limited to, a natural language output, a voice and / or audio output, an image output, a video output, and / or a program output.

[0047] Diagram 200 is shown in an example where the skill chain includes two model skills (e.g., corresponding to ML model 204 and ML model 206). Arrow 216 is provided to indicate that in other examples, additional model skills (e.g., associated with any of ML model 204, ML model 206, and / or various other ML models not shown) may be included. Furthermore, while diagram 200 shows an example where the model skills are sequential, it will be understood that parallel skills, hierarchical ML skills, and / or skills that depend on the output from multiple previous skills, among other examples, may be used in other examples. Furthermore, as noted above, the skill chain may in other examples further include one or more programmatic skills.

[0048] 3 illustrates an example method 300 for processing user input to generate model output according to aspects described herein. In some examples, aspects of method 300 are performed by a chain orchestrator (e.g., chain orchestrator 108 of FIG. 1 or chain orchestrator 203 of FIG. 2), among other examples, and / or by a multi-stage machine learning framework (e.g., multi-stage machine learning framework 118).

[0049] As shown, method 300 begins at operation 302, where user input is received. In some examples, the user input may be received from a computing device (e.g., computing device 104 of FIG. 1 ), such as when aspects of method 300 are performed by a machine learning service (e.g., machine learning service 102 of FIG. 1 ). As another example, the user input may be received from an application (e.g., application 116), from a service, or from other software on the computing device (e.g., computing device 104), such as when aspects of method 300 are performed by a multi-stage machine learning framework (e.g., multi-stage machine learning framework 118). In some examples, the received input may be similar to user input 202 described above with respect to FIG. 2 . The received user input may include natural language input (e.g., text and / or voice input), image input, and / or video input, among any of a variety of other inputs.

[0050] Method 300 proceeds to operation 304, where the received input is processed to generate a skill chain (e.g., corresponding to a set of ML ratings, in accordance with aspects described herein, and in some examples, further corresponding to a set of program ratings). Prompts are generated that are processed, for example, by a generative ML model to generate the skill chain accordingly. The prompts may be generated based on a prompt template populated to include at least a portion of the input received at operation 302. In some examples, the prompt template is further populated with a list of skills (e.g., from a skill library, such as skill libraries 112, 120, and / or 212 of FIGS. 1 and 2).

[0051] In some examples, a chain orchestrator (e.g., chain orchestrator 108) of a machine learning service generates the skill chain. As another example, a request to process input is provided to a machine learning service, such that skill chain instructions are received in response, such as when aspects of method 300 are performed by a multi-stage machine learning framework of a client computing device (e.g., computing device 104 of FIG. 1). As a further example, at least a portion of such skill chain generation is performed locally on the computing device, such as when a generative ML model for performing such aspects is locally available. As described above, the generated skill chain may include, among other examples, one or more sequential skills, a hierarchical set of skills, a set of parallel skills, and / or a skill that depends on or otherwise processes output from two or more previous skills.

[0052] While examples of generating skill chains using generative ML models are described (e.g., with respect to chain orchestrator 108, chain orchestrator 203, and operation 304 in FIGS. 1, 2, and 3, respectively), it will be understood that any of a variety of additional or alternative techniques may be used. As one example, a semantic store similar to semantic memory store 218 may additionally or alternatively be used to store one or more embeddings associated with skills in a skill library (e.g., as may be generated based at least in part on the associated skill's description, manual page, and / or associated prompt template). For example, an input embedding may be generated for the input received in operation 302 (e.g., thereby indicating one or more associated intents) and may be used to identify one or more skills having associated embeddings that match the input embedding (similar to the manner described above with respect to recall engine 210). The identified skills may then form a skill chain accordingly.

[0053] As another example, context may be provided to the generative ML model when generating a skill chain, as may be determined by the recall engine from a semantic memory engine (e.g., it may be included as part of the generated prompt), similar to the recall engine 210 and semantic memory store 218 described above with respect to FIG. 2.

[0054] Flow continues to operation 306, where a skill is selected from the skill chain generated in operation 304. In some examples, the skill chain generated in operation 304 indicates an order, hierarchy, and / or one or more interdependencies of the component skills such that the skill is selected appropriately in operation 306.

[0055] At decision 307, it is determined whether the selected skill is a program skill. As described above, a skill chain, in some examples, may include a program skill in which any of a variety of operations are performed by a computing device. Thus, if it is determined that the selected skill is a program skill, flow branches “YES” to operation 309 where the program skill is processed. In several examples, operation 309 includes, among other examples, executing a command, obtaining additional information (e.g., from a user or a data source), and / or affecting the operation of an operating system or application (e.g., via an API or command interface). In subsequent iterations of method 300, executing the program skill at operation 309 may include executing a program output generated by a machine learning model in accordance with aspects described herein. Thus, it will be understood that any of a variety of program operations may be performed when evaluating a skill chain. Flow then proceeds to decision 314, described below.

[0056] However, if it is instead determined that the selected skill is not a program skill (e.g., such that it is instead a model skill), the flow branches “NO” to decision 308, where it is determined whether to invoke context from a semantic memory store (e.g., semantic memory stores 114, 122, and / or 210 of FIGS. 1 and 2). As described above, the determination may be based on a prompt template corresponding to the selected model skill. For example, the prompt template may indicate that context should be retrieved from the semantic memory store and / or may include instructions regarding what context, if available, should be retrieved. As another example, the determination to invoke context from the semantic memory store may be made automatically, as may be determined based on a previous model skill that used the same or a similar prompt. Thus, it will be understood that context may be retrieved from the semantic memory store of a model skill as a result of any of a variety of decisions and / or instructions, among other examples.

[0057] If it is determined to recall context from the semantic memory, flow branches “YES” to operation 310, where a context is generated based on the semantic memory store. Such aspects may be similar to those described above with respect to recall engine 210 of FIG. 2 and, therefore, will not necessarily be described again in detail below. For example, an input semantic embedding is generated based on a user input and / or a prompt template on which ML evaluation is performed, such that one or more matching semantic embeddings can be identified from the semantic memory store. Content corresponding to the one or more identified semantic embeddings is retrieved and used as context for the ML evaluation of model skill, as appropriate. As described above, the retrieved content can be included in a prompt generated according to the prompt template. It will be appreciated that the context can be obtained from any of a variety of sources, including, but not limited to, a user's computing device (e.g., computing device 104 of FIG. 1 ) and / or a machine learning service (e.g., machine learning service 102), among other examples. In contrast, if it is instead decided not to retrieve the context from the semantic memory store, flow instead branches "NO" from decision 308 to operation 312, described below.

[0058] Flow ultimately proceeds to operation 312, where an output is generated for the selected machine learning skill. As described above, a prompt may be generated based on a prompt template, such that the prompt includes at least a portion of the input and, in some examples, the generated context. It will be understood that in other examples, the ML model associated with the model skill may not use prompting. Similar to operation 304, when the skill chain generation aspect of method 300 is executed locally on a client computing device, a request for ML processing may be provided to a machine learning service, such that a generated output is received from the machine learning service in response. In some examples, the generated output (e.g., as may be received in response from the machine learning service) may be an intermediate output, including, for example, a structured output generated as a result of a prompt that includes instructions regarding the structured output, as described above. Additional example aspects of operation 312 are described below with respect to method 400 of FIG. 4.

[0059] At decision 314, it is determined whether there are any skills remaining in the skill chain created at operation 304. In some examples, the skill chain is updated as a result of operation 309 and / or operation 312, described above. Decision 314 may include evaluating the skill chain (e.g., as created at operation 304 and / or as may have been updated as a result of operation 309 and / or operation 312) to determine whether there are any skills that have not yet been processed. If it is determined that there are no skills remaining, the flow branches "NO" to operation 316, described below.

[0060] In contrast, if it is instead determined that there are skills remaining, the flow branches "YES" back to operation 306, where a subsequent skill is selected. Thus, if one or more skills remain, the flow loops through operations 306-314. A subsequent iteration of operation 312 may use the generated output of an earlier iteration of operation 309 and / or operation 312 as input to the model skill in generating the subsequent model output. Similarly, a subsequent iteration of operation 309 may, in some examples, use the generated output of an earlier iteration of operation 309 and / or operation 312. Additionally or alternatively, at least a portion of the received user input is used as input for the subsequent iteration of operation 309 and / or operation 312.

[0061] In addition to chaining together ML evaluations by subsequent iterations of operation 312, in some examples, one or more contexts may be chained together as a result of subsequent iterations of operation 310. For example, a context corresponding to a previous ML evaluation (e.g., as may have been generated by a previous iteration of operation 310 and / or updated by a previous iteration of operation 312) may be used as the context for the subsequent ML evaluation by operation 312.

[0062] Finally, method 300 reaches operation 316, where an indication of the generated output is provided. For example, the indication may be provided to a client computing device, as in an example where aspects of method 300 are performed by a chain orchestrator of a machine learning platform. Additionally or alternatively, the indication may be provided by a multi-stage machine learning framework of the client computing device. For example, the indication may be provided to an application (e.g., application 116 of FIG. 1 ) for subsequent processing. In some examples, an indication of at least some of the generated output is provided to a user of the computing device. As described above, the resulting output may include any of a variety of content, including, but not limited to, natural language output, voice and / or audio output, image output, video output, and / or program output. Method 300 ends at operation 316.

[0063] 4 illustrates an example method 400 for processing user input according to prompts using a generative ML model (also referred to herein as ML model evaluation) in accordance with aspects described herein. In some examples, aspects of method 400 are performed as part of operation 312 described above with respect to method 300 of FIG.

[0064] As shown, method 400 begins at operation 402 where input is obtained. Aspects of the obtained input may be similar to user input 202 described above with respect to FIG. 2 or that received at operation 302 of method 300 of FIG. 3, and thus are not necessarily described again in detail below. For example, the input may be obtained from a user of a computing device (e.g., computing device 104 of FIG. 1). In some examples, the input is received as part of a request to generate model output according to aspects described herein (e.g., as a result of performing aspects of operation 312 described above with respect to method 300 of FIG. 3).

[0065] At operation 404, a context may be obtained. Operation 404 is shown with a dashed box to indicate that in other examples, operation 404 may be omitted. Similar to operation 402, in some examples, the context may be obtained as part of a request to generate a model output. In other examples, the context may be obtained from a semantic memory store, such as may be generated by a recall engine similar to recall engine 210 from semantic memory store 218, as described above with respect to FIG. 2.

[0066] Flow continues to operation 406, where a prompt is generated. In some examples, an indication of a model skill (e.g., corresponding to a prompt template) is received as part of the received request (as described above with respect to operation 402). In some examples, the prompt template may be obtained based on an association with a model skill in a skill library (e.g., skill libraries 112 and / or 120 of FIG. 1 and / or skill library 212 of FIG. 2). The prompt template is processed to incorporate at least a portion of the obtained input, and in some examples, the obtained context. For example, one or more fields, regions, or other portions of the prompt template may be replaced or otherwise populated in such a manner to generate a prompt from which a model output may be generated for a given model skill.

[0067] Moving to act 408, a model is determined from the set of models. In some examples, the skill for which the prompt was generated at act 406 may include an indication regarding the model with which the generated prompt will be processed. As another example, the received request may include such an indication. The model may be identified from a model repository, such as model repository 110 of FIG. 1. In other examples, such a determination need not be made, such as when the machine learning service and / or associated API from which the request was received performs processing on a single ML model.

[0068] Flow proceeds to operation 410 where a model output is generated. In some examples, operation 410 includes processing the prompt generated in operation 406 according to the ML model determined in operation 408. Aspects of an example ML model that may be used to perform such processing are described below with respect to FIGS.

[0069] At operation 412, an indication of the generated outputs is provided. For example, a response to the request received as part of operation 402, operation 404, and / or operation 406 may be generated that includes at least a portion of the model outputs. As described above, the model outputs may include intermediate outputs and / or structured outputs, such as when the request corresponds to an intermediate ML evaluation of a skill chain. Thus, the indication of the generated outputs may be received by the computing device, and subsequent processing may be performed accordingly (e.g., by a multi-stage machine learning framework and / or application, such as multi-stage machine learning framework 118 and / or application 116, respectively). Method 400 ends at operation 412.

[0070] 5A and 5B illustrate schematics of exemplary generative machine learning models that may be used in accordance with aspects described herein. Referring first to FIG. 5A, a conceptual diagram 500 illustrates a schematic of a pre-trained generative model package 504 that processes skill inputs and prompts 502 of a skill chain to generate model outputs 506 for multi-stage ML model chaining in accordance with aspects described herein. Examples of pre-trained generative model packages 504 include, but are not limited to, Megatron-Turing Natural Language Generation model (MT-NLG), Generative Pre-trained Transformer 3 (GPT-3), Generative Pre-trained Transformer 4 (GPT-4), BigScience Large Open-science Open-access Multilingual Language Model (BLOOM), DALL-E, DALL-E 2, Stable Diffusion, or Jukebox.

[0071] In some examples, the generative model package 504 is pre-trained according to a variety of inputs (e.g., various human languages, various programming languages, and / or various content types) and therefore does not need to be fine-tuned or trained for a particular scenario. Rather, the generative model package 504 may be more generally pre-trained such that the input 502 includes prompts generated, selected, or otherwise designed to guide the generative model package 504 to generate a particular generative model output 506. For example, the prompt includes one or more completion prefixes that pre-populate the generative model package 504 appropriately as a context and / or result. As a result, the generative model package 504 is guided to generate a prompt-based output that includes a predicted sequence of tokens related to the prompt (e.g., up to the token limit of the generative model package 504). In some examples, the predicted sequence of tokens is further processed (e.g., by output decoding 516) to result in the output 506. For example, each token is processed to identify a corresponding word, word fragment, or other content that forms at least a portion of the output 506. It will be appreciated that the input 502 and the generative model output 506 may each include any of a variety of content types, including, but not limited to, text output, image output, audio output, video output, programmatic output, and / or binary output, among other examples. In some examples, the input 502 and the generative model output 506 may have different content types, such as when the generative model package 504 includes a generative multimodal machine learning model.

[0072] As such, the generative model package 504 may be used in any of a variety of scenarios, and even a different generative model package may be used in place of the generative model package 504 without substantially modifying other related aspects (e.g., similar to those described herein with respect to Figures 1, 2, 3, and 4). Thus, the generative model package 504 operates as a tool upon which machine learning processes are performed, where specific inputs 502 to the generative model package 504 are programmatically generated or otherwise determined, thereby causing the generative model package 504 to generate model outputs 506 that may then be used for further processing.

[0073] The generative model package 504 may be provided or otherwise used according to any of a variety of paradigms. For example, the generative model package 504 may be used locally on a computing device (e.g., computing device 104 of FIG. 1 ) or may be accessed remotely from a machine learning service (e.g., machine learning service 102). In other examples, aspects of the generative model package 504 are distributed across multiple computing devices. In some examples, the generative model package 504 is accessible via an application programming interface (API), such as may be provided by the operating system of a computing device and / or by a machine learning service, among other examples.

[0074] Referring now to the illustrated embodiment of the generative model package 504, the generative model package 504 includes an input tokenizer 508, input embeddings 510, a model layer 512, an output layer 514, and output decoding 516. In some examples, the input tokenizer 508 processes the input 502 to generate an input embedding 510, which includes a sequence of symbolic representations corresponding to the input 502. The input embedding 510 is then processed by the model layer 512, the output layer 514, and output decoding 516 to generate the model output 506. An exemplary architecture corresponding to the generative model package 504 is shown in FIG. 5B and is described in further detail below. Nevertheless, it will be understood that the architecture shown and described herein should not be taken in a limiting sense, and that in other examples, any of a variety of other architectures may be used.

[0075] 5B is a conceptual diagram illustrating an example architecture 550 of a pre-trained generative machine learning model that may be used in accordance with aspects described herein. As noted above, any of a variety of alternative architectures and corresponding ML models may be used in other examples without departing from aspects described herein.

[0076] As shown, architecture 550 processes input 502 to generate generative model output 506 (aspects of which are described above with respect to FIG. 5A ). Architecture 550 is shown as a Transformer model including an encoder 552 and a decoder 554. Encoder 552 processes input embeddings 558 (which may be similar in aspect to input embeddings 510 of FIG. 5A ) that include a sequence of symbolic representations corresponding to input 556. In some examples, input 556 includes inputs and prompts 502 corresponding to skills in a skill chain, which may be similar in aspect to prompts generated based on user input 202, context from semantic memory store 218, and / or prompt templates for skills from skill libraries 112, 120, and / or 212 according to aspects described herein.

[0077] Additionally, positional encoding 560 may introduce information regarding the relative and / or absolute position of input embedding 558 relative to the token. Similarly, output embedding 574 includes a sequence of symbolic representations corresponding to output 572, and positional encoding 576 may similarly introduce information regarding the relative and / or absolute position of output embedding 574 relative to the token.

[0078] As shown, the encoder 552 includes an exemplary layer 570. It will be understood that any number of such layers may be used, and that the illustrated architecture is simplified for illustrative purposes. The exemplary layer 570 includes two sublayers: a multi-head attention layer 562 and a feedforward layer 566. In some examples, residual connections are included around each layer 562, 566, followed by normalization layers 564 and 568, respectively.

[0079] Decoder 554 includes exemplary layer 590. Similar to encoder 552, any number of such layers may be used in other examples, and the illustrated architecture of decoder 554 is simplified for illustrative purposes. As shown, exemplary layer 590 includes three sublayers: a masked multi-head attention layer 578, a multi-head attention layer 582, and a feedforward layer 586. Aspects of multi-head attention layer 582 and feedforward layer 586 may be similar to those described above with respect to multi-head attention layer 562 and feedforward layer 566, respectively. Additionally, masked multi-head attention layer 578 performs multi-head attention on the output of encoder 552 (e.g., output 572). In some examples, masked multi-head attention layer 578 prevents a position from processing a subsequent position. Such masking may be combined with offsetting the embedding (e.g., by one position, as illustrated by multi-head attention layer 582) to ensure that predictions for a given position depend on known outputs for one or more positions below the given position. As shown, residual connections are also included around layers 578, 582, and 586, followed by normalization layers 580, 584, and 588, respectively.

[0080] Each of the multi-head attention layers 562, 578, and 582 may linearly project the query, key, and value using a set of linear projections onto the corresponding dimensions. Each linear projection may be processed using an attention function (e.g., dot product or additive attention), resulting in an n-dimensional output value for each linear projection. The resulting values ​​may be concatenated and projected again so that the values ​​can be subsequently processed (e.g., by the corresponding normalization layer 564, 580, or 584) as shown in FIG. 5B.

[0081] Each of the feedforward layers 566 and 586 may be a fully connected feedforward network applied to each location. In some examples, each of the feedforward layers 566 and 586 includes multiple linear transformations with rectified linear unit activations between them. In some examples, each linear transformation may be the same across different locations, but may use different parameters compared to other linear transformations in the feedforward network.

[0082] Further, aspects of the linear transform 592 may be similar to the linear transforms described above with respect to the multi-head attention layers 562, 578, and 582 and the feedforward layers 566 and 586. A softmax 594 may further convert the output of the linear transform 592 into a probability of the predicted next token, as indicated by output probability 596. It will be understood that the illustrated architecture is provided by way of example, and that in other examples, any of a variety of other model architectures may be used in accordance with aspects of the disclosure. In some examples, multiple iterations of processing are performed in accordance with the above aspects (e.g., using the generative model package 504 of FIG. 5A or the encoder 552 and decoder 554 of FIG. 5B) to generate a sequence of output tokens (e.g., words), which are then combined, for example, to produce a complete sentence (and / or any of a variety of other content). It will be understood that other generative models may generate multiple output tokens in a single iteration and thus may use a reduced number of iterations or a single iteration.

[0083] Thus, in this manner, output probabilities 596 may form chained ML evaluation output 506 according to aspects described herein, such that the output of the generative ML model (which may include, e.g., a structured output) is used as an input for a subsequent skill in the skill chain according to aspects described herein (e.g., similar to a "YES" decision in decision 314 of method 300 of FIG. 3). In other examples, chained ML evaluation output 506 is provided as a generated output after processing of the skill chain (e.g., similar to aspects of operation 316 of method 300), which may be further processed according to aspects of the disclosure.

[0084] 6-8 and the associated description provide an illustration of various operating environments in which aspects of the present disclosure may be implemented. However, the devices and systems shown and described with respect to Figures 6-8 are for purposes of example and explanation and are not intended to limit the vast number of computing device configurations that may be utilized to implement aspects of the present disclosure described herein.

[0085] 6 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600 in which aspects of the present disclosure may be implemented. The computing device components described below may be suitable for the computing devices described above, including one or more devices associated with the machine learning service 102, and the computing device 104 described above with respect to FIG. 1. In a basic configuration, the computing device 600 may include at least one processing unit 602 and system memory 604. Depending on the configuration and type of computing device, the system memory 604 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memory.

[0086] The system memory 604 may include an operating system 605 and one or more program modules 606 suitable for executing software applications 620, such as one or more components supported by the system described herein. By way of example, the system memory 604 may store a chain orchestrator 624 and a recall engine 626. For example, the operating system 605 may be suitable for controlling the operation of the computing device 600.

[0087] Additionally, embodiments of the present disclosure may be implemented in conjunction with graphics libraries, other operating systems, or any other application programs and are not limited to any particular application or system. This basic configuration is illustrated in FIG. 6 by those components within dashed line 608. Computing device 600 may have additional features or functionality. For example, computing device 600 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 6 by removable storage device 609 and non-removable storage device 610.

[0088] As mentioned above, a number of program modules and data files may be stored in the system memory 604. While executing on the processing unit 602, the program modules 606 (e.g., applications 620) may perform processes including, but not limited to, aspects such as those described herein. Other program modules that may be used in accordance with aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc.

[0089] Furthermore, embodiments of the present disclosure may be implemented in electrical circuits including discrete electronic elements, in packaged or integrated electronic chips including logic gates, in microprocessor-based circuits, or on a single chip including electronic elements or a microprocessor. For example, embodiments of the present disclosure may be implemented via a system-on-chip (SOC), where each or many of the components illustrated in FIG. 6 may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units, and various application functionalities, all integrated (or "baked") onto the chip substrate as a single integrated circuit. When operating via a SOC, the functionality described herein with respect to the client's ability to switch protocols may operate via application-specific logic integrated with other components of the computing device 600 on a single integrated circuit (chip). Embodiments of the present disclosure may also be implemented using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including, but not limited to, mechanical, optical, fluidic, and quantum technologies. Furthermore, embodiments of the present disclosure may be implemented in a general purpose computer or in any other circuits or systems.

[0090] The computing device 600 may also have one or more input devices 612, such as a keyboard, mouse, pen, sound or voice input device, touch or swipe input device, etc. One or more output devices 614, such as a display, speakers, printer, etc., may also be included. The foregoing devices are examples, and others may be used. The computing device 600 may include one or more communication connections 616 that enable communication with other computing devices 650. Examples of suitable communication connections 616 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0091] The term computer-readable medium as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and that can be accessed by computing device 600. Any such computer storage media may be part of computing device 600. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0092] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may refer to a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media.

[0093] 7 illustrates a system 700 (with which embodiments of the present disclosure may be implemented), which may be, for example, a mobile computing device such as a mobile phone, a smartphone, a wearable computer (such as a smartwatch), a tablet computer, a laptop computer, etc. In one embodiment, system 700 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, a calendar, a contact manager, a messaging client, games, and a media client / player). In some aspects, system 700 is integrated as a computing device such as an integrated personal digital assistant (PDA) and wireless telephone.

[0094] In its basic configuration, such a mobile computing device is a handheld computer having both input and output elements. System 700 typically includes a display 705 and one or more input buttons that allow a user to input information into system 700. Display 705 may also function as an input device (e.g., a touchscreen display).

[0095] If included, optional side input elements allow for further user input. For example, the side input elements may be rotary switches, buttons, or any other type of manual input element. In alternative aspects, system 700 may incorporate more or fewer input elements. For example, display 705 may not be a touchscreen in some embodiments. In another example, optional keypad 735 may also be included, which may be a physical keypad or a "soft" keypad generated on the touchscreen display.

[0096] In various embodiments, the output elements include a display 705 for showing a graphical user interface (GUI), visual indicators (e.g., light emitting diodes 720), and / or audio transducers 725 (e.g., speakers). In some aspects, vibration transducers are included to provide tactile feedback to the user. In yet other aspects, input and / or output ports such as audio input (e.g., microphone jack), audio output (e.g., headphone jack), and video output (e.g., HDMI port) for sending signals to or receiving signals from external devices are included.

[0097] One or more application programs 766 may be loaded into memory 762 and run on or in conjunction with operating system 764. Examples of application programs include a phone dialer program, an email program, a personal information manager (PIM) program, a word processing program, a spreadsheet program, an internet browser program, a messaging program, etc. System 700 also includes a non-volatile storage area 768 within memory 762. Non-volatile storage area 768 may be used to store persistent information that should not be lost even if system 700 is powered off. Application programs 766 may use and save information in non-volatile storage area 768, such as emails or other messages used by an email application. A synchronization application (not shown) also resides on system 700 and is programmed to interact with a corresponding synchronization application resident on the host computer to keep information stored in non-volatile storage area 768 synchronized with corresponding information stored on the host computer. As should be understood, other applications may be loaded into memory 762 and run on system 700 as described herein.

[0098] The system 700 includes a power supply 770, which may be implemented as one or more batteries, and may further include an external power source, such as an AC adapter or a powered docking cradle, that replenishes or charges the batteries.

[0099] System 700 may also include a radio interface layer 772 that performs the function of transmitting and receiving radio frequency communications. Radio interface layer 772 facilitates wireless connectivity between system 700 and the "outside world" via a carrier or service provider. Transmissions to and from radio interface layer 772 are under the control of operating system 764. That is, communications received by radio interface layer 772 can be disseminated to application programs 766 via operating system 764, and vice versa.

[0100] The visual indicator 720 may be used to provide visual notifications, and / or the audio interface 774 may be used to generate audible notifications via the audio transducer 725. In the illustrated embodiment, the visual indicator 720 is a light-emitting diode (LED), and the audio transducer 725 is a speaker. These devices may be directly coupled to the power source 770 so that, when activated, they remain on for a duration dictated by the notification mechanism, even though the processor 760 and other components may shut down to conserve battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 774 is used to provide audible signals to the user and to receive audible signals from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 774 may also be coupled to a microphone for receiving audible input, such as to facilitate a telephone conversation. In accordance with embodiments of the present disclosure, the microphone may also function as an audio sensor to facilitate control of notifications, as described below. The system 700 may further include a video interface 776 that enables operation of the onboard camera 730 for recording still images, video streams, and the like.

[0101] It will be understood that system 700 may have additional features or functionality. For example, system 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tape. Such additional storage is represented in FIG. 7 by non-volatile storage area 768.

[0102] Data / information generated or captured and stored via system 700 may be stored locally, as described above, or the data may be stored in any number of storage media that can be accessed by the device via wireless interface layer 772 or via a wired connection between system 700 and a separate computing device associated with system 700 (e.g., a server computer in a distributed computing network such as the Internet). As should be understood, such data / information may be accessed via wireless interface layer 772 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to any of a variety of data / information transfer and storage means, including email and collaborative data / information sharing systems.

[0103] 8 illustrates one aspect of an architecture for a system for processing data received at a computing system from a remote source, such as a personal computer 804, a tablet computing device 806, or a mobile computing device 808, as described above. Content displayed at the server device 802 may be stored on different communication channels or other storage types. For example, various documents may be stored using a directory service 824, a web portal 825, a mailbox service 826, an instant messaging store 828, or a social networking site 830.

[0104] A multi-stage machine learning framework 820 (e.g., similar to application 620) may be employed by clients communicating with server device 802. Additionally or alternatively, a chain orchestrator 821 may be employed by server device 802. Server device 802 may provide data to and from client computing devices, such as personal computer 804, tablet computing device 806, and / or mobile computing device 808 (e.g., smartphone) via network 815. By way of example, the computer systems described above may be embodied in personal computer 804, tablet computing device 806, and / or mobile computing device 808 (e.g., smartphone). Any of these examples of computing devices may retrieve content from store 816 in addition to receiving graphics data available for pre-processing at a graphics originating system or for post-processing at a receiving computing system.

[0105] It will be understood that aspects and functionality described herein may operate on a distributed system (e.g., a cloud-based computing system), where application functionality, memory, data storage and retrieval, and various processing functions may operate remotely from one another over a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information may be displayed via on-board computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information may be displayed and interacted with on a wall surface onto which the various types of user interfaces and information are projected. Interactions with numerous computing systems in which embodiments of the present invention may be implemented include keystroke input, touchscreen input, voice or other audio input, gesture input with detection (e.g., camera) functionality for an associated computing device to capture and interpret user gestures to control functionality of the computing device, and the like.

[0106] As can be appreciated from the foregoing disclosure, one aspect of the present technology relates to a system comprising at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, including: obtaining user input from a user; generating, based on the user input, a skill chain including a set of skills for processing the user input; generating, for a first model skill in the skill chain, a first prompt including at least a portion of the obtained user input based on a first prompt template associated with the first model skill, and processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; generating, for a second model skill in the skill chain, a second prompt including at least a portion of the intermediate output as an input for a second model skill based on a second prompt template associated with the second model skill, and processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and providing an indication of the model output for display to a user. In one example, generating a skill chain includes generating a skill list corresponding to a set of skills from a skill library, the skill list including a description of each skill in the set of skills; providing a user input and an indication of the skill list to a machine learning service; and receiving from the machine learning service the skill chain corresponding to the user input. In another example, generating a skill chain includes generating, for a user input, an input embedding that encodes the intent of the user input; determining, from a skill library, a set of skills each having an associated semantic embedding that matches the generated input embedding; and generating a skill chain based on the determined set of skills. In a further example, a semantic embedding is determined to match the input embedding based on an algorithmic similarity metric between the semantic embedding and the input embedding.In yet another example, processing the first prompt to obtain the intermediate output includes providing a request to a machine learning service to process the first prompt using the first machine learning model and receiving a response from the machine learning service including the intermediate output. In yet a further example, the intermediate output of the first model skill includes a structured output. In another example, at least a portion of the first prompt corresponds to the structured output.

[0107] In another aspect, the technology relates to a method including, in a computing device, obtaining a skill chain corresponding to an input; generating, for a first model skill in the skill chain, a first prompt including at least a portion of the user input based on a first prompt template associated with the first model skill; processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; generating, for a second model skill in the skill chain, a second prompt including at least a portion of the intermediate output as an input for the second model skill based on a second prompt template associated with the second model skill; processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and processing, by the computing device, at least a portion of the model output to affect operation of the computing device. In one example, the first machine learning model is a second machine learning model. In another example, the skill chain further includes a program skill executed by the computing device, and an output of the program skill is processed as an input for the second model skill. In a further example, the intermediate output of the first model skill includes a structured output processed by the program skill. In yet another example, processing the portion of the model output includes displaying the portion of the model output to a user of the computing device, and in yet a further example, processing the portion of the model output includes parsing the portion of the model output by an application of the computing device to affect the operation of the application.

[0108] In a further aspect, the technology relates to another method including: acquiring user input from a user; generating, based on the user input, a skill chain including a set of skills for processing the user input; generating, for a first model skill in the skill chain, a first prompt including at least a portion of the acquired user input based on a first prompt template associated with the first model skill; processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; generating, for a second model skill in the skill chain, a second prompt including at least a portion of the intermediate output as an input for the second model skill based on a second prompt template associated with the second model skill; processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and providing an indication of the model output for display to a user. In one example, generating the skill chain includes generating a skill list corresponding to the set of skills in a skill library, the skill list including a description of each skill in the set of skills; providing the user input and the indication of the skill list to a machine learning service; and receiving the skill chain corresponding to the user input from the machine learning service. In another example, generating a skill chain includes generating, for a user input, an input embedding that encodes an intent of the user input, determining a set of skills from a skill library, each skill having an associated semantic embedding that matches the generated input embedding, and generating a skill chain based on the determined set of skills. In a further example, the semantic embedding is determined to match the input embedding based on an algorithmic similarity metric between the semantic embedding and the input embedding. In yet another example, processing the first prompt to obtain an intermediate output includes providing a request to a machine learning service to process the first prompt using a first machine learning model, and receiving a response from the machine learning service including the intermediate output.In yet a further example, the intermediate output of the first model skill includes a structured output. In another example, at least a portion of the first prompt corresponds to the structured output.

[0109] Aspects of the present disclosure are described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to aspects of the present disclosure. The functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality / acts involved.

[0110] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict in any way the scope of the claimed disclosure. The aspects, examples, and details provided in this application are believed to be sufficient to convey proprietary rights and enable others to make and use the claimed aspects of the disclosure. The claimed disclosure is not to be construed as limited to any aspects, examples, or details provided in this application. Various features (both structural and methodological), whether shown and described in combination or separately, are intended to be selectively included or omitted to produce embodiments having a particular set of features. Given the description and illustrations provided in this application, those skilled in the art will be able to envision variations, modifications, and alternative embodiments that are within the spirit of the broader aspects of the general inventive concept embodied in this application without departing from the broader scope of the claimed disclosure.

Claims

1. 1. A system comprising: at least one processor; a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising: obtaining user input from a user; generating a skill chain based on the user input, the skill chain including a set of skills for processing the user input; For a first model skill in the skill chain: generating a first prompt that includes at least a portion of the captured user input based on a first prompt template associated with the first model skill; and processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; For a second model skill in the skill chain: generating a second prompt based on a second prompt template associated with the second model skill, the second prompt including at least a portion of the intermediate output as an input to the second model skill; and processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and providing an indication of the model output for display to the user; a memory including: A system comprising:

2. generating the skill chain, generating a skill list corresponding to a set of skills in a skill library, the skill list including a description of each skill in the set of skills; providing the user input and the skill list indication to a machine learning service; receiving the skill chain from the machine learning service in response to the user input; The system of claim 1 , comprising:

3. generating the skill chain, generating, for the user input, an input embedding that encodes the intent of the user input; determining a set of skills from a skills library, each of which has an associated semantic embedding that matches the generated input embedding; generating the skill chain based on the determined set of skills; The system of claim 1 , comprising:

4. The system of claim 3 , wherein a semantic embedding is determined to match the input embedding based on an algorithmic similarity metric between the semantic embedding and the input embedding.

5. processing the first prompt to obtain the intermediate output; providing a request to a machine learning service to process the first prompt using the first machine learning model; receiving a response from the machine learning service that includes the intermediate output; The system of claim 1 , comprising:

6. The system of claim 1 , wherein the intermediate output of the first model skill comprises a structured output.

7. The system of claim 6 , wherein at least a portion of the first prompt corresponds to the structured output.

8. obtaining, at a computing device, a skill chain corresponding to the input; For a first model skill in the skill chain: generating a first prompt including at least a portion of the input based on a first prompt template associated with the first model skill; and processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; For a second model skill in the skill chain: generating a second prompt including at least a portion of the intermediate output as an input to the second model skill based on a second prompt template associated with the second model skill; and processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and processing, by the computing device, at least a portion of the model output to affect operation of the computing device; A method comprising:

9. The method of claim 8 , wherein the first machine learning model is the second machine learning model.

10. the skill chain further includes a program skill executed by the computing device; and The method of claim 8 , wherein an output of the program skill is processed as an input of the second model skill.

11. The method of claim 10 , wherein the intermediate output of the first model skill comprises a structured output that is processed by the program skill.

12. The method of claim 8 , wherein processing the portion of the model output comprises displaying the portion of the model output to a user of the computing device.

13. The method of claim 8 , wherein processing the portion of the model output comprises parsing the portion of the model output by an application on the computing device to affect the behavior of the application.

14. obtaining user input from a user; generating a skill chain based on the user input, the skill chain including a set of skills for processing the user input; For a first model skill in the skill chain: generating a first prompt that includes at least a portion of the captured user input based on a first prompt template associated with the first model skill; and processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; For a second model skill in the skill chain: generating a second prompt based on a second prompt template associated with the second model skill, the second prompt including at least a portion of the intermediate output as an input to the second model skill; and processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and providing an indication of the model output for display to the user; A method comprising:

15. generating the skill chain, generating a skill list corresponding to a set of skills in a skill library, the skill list including a description of each skill in the set of skills; providing the user input and the skill list indication to a machine learning service; receiving the skill chain from the machine learning service in response to the user input; 15. The method of claim 14, comprising:

Images