Multi-stage machine learning model chain

Abstract

A skill chain including a set of ML model evaluations is generated, the input is processed with the set of ML model evaluations, and the skill chain is used to eventually generate a model output accordingly. Each ML model evaluates a "model skill" corresponding to the skill chain. The intermediate output generated by the first ML evaluation for the first model skills of the skill chain may then be processed as an input for the second ML evaluation for the second model skills of the skill chain, thereby ultimately generating a model output for the given input. Such a skill chain may include any number of skills according to any of the various structures and do not need to be evaluated using the same ML model.

Description

Background Art

[0001] A single evaluation using a machine learning model may have limited utility, especially for more complex tasks and in instances where the user is not familiar with the machine learning model and / or the task at hand. Thus, such an evaluation may result in a degraded user experience, increased user frustration, and / or wasted computing resources, among other things.

[0002] Embodiments have been described with respect to these and other general considerations. Moreover, although relatively specific problems have been discussed, it should be understood that embodiments are not limited to solving the specific problems identified in the background art. Summary of the Invention

[0003] Aspects of the present application relate to a multi-stage machine learning model chain, in which a set of ML model evaluations are generated, the input is processed using the set of ML model evaluations, and a skill chain is used to ultimately generate a model output accordingly. Each ML model evaluation corresponds to a "model skill" of the skill chain. For example, a model skill has an associated prompt template that is used to generate a prompt (e.g., including an input and / or context) that is processed using the corresponding ML model to generate a model output accordingly. In other examples, the ML model associated with a model skill does not need to have an associated prompt template, such as when a prompt is not used by the ML model when processing the input to generate the model output.

[0004] An intermediate output generated by a first ML evaluation for a first model skill of a skill chain can subsequently be processed as an input for a second ML evaluation for a second model skill of the skill chain, thereby ultimately generating a model output for a given input. Such a skill chain can include any number of skills according to any of various structures and does not require the same ML model to be evaluated.

[0005] The present summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. The present summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Brief Description of the Drawings

[0006] Non-limiting examples and non-exhaustive examples are described with reference to the following drawings.

[0007] Figure 1 An overview of an example system in which a multi-stage machine learning model chain can be used is shown in accordance with aspects of the present disclosure.

[0008] Figure 2 An overview of an example conceptual diagram for using a chained machine learning model to process user input to generate a model output is shown in accordance with aspects described herein.

[0009] Figure 3 shows an overview of an example method for processing user input to generate a model output according to aspects described herein.

[0010] Figure 4 shows an overview of an example method for processing user input according to a prompt using a generative ML model according to aspects described herein.

[0011] Figure 5A and Figure 5B shows an overview of an example generative machine learning model that can be used according to aspects described herein.

[0012] Figure 6 is a block diagram showing example physical components of a computing device in which aspects of the present disclosure may be practiced.

[0013] Figure 7 is a simplified block diagram of a computing device in which aspects of the present disclosure may be practiced.

[0014] Figure 8 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. Detailed Description

[0015] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments or examples. Aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Embodiments may be practiced as a method, system, or device. Thus, embodiments may take the form of a hardware implementation, a fully software implementation, or an implementation combining software and hardware aspects. Accordingly, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0016] In an example, a machine learning (ML) model generates a model output based on an input (e.g., as may be received from a user). For example, natural language input from a user is processed using a generative ML model to correspondingly generate a model output for the natural language input. However, the limited nature of such a single evaluation may result in reduced utility, particularly in instances where the user is inexperienced / unfamiliar with the ML model (e.g., such that the user may provide input that results in limited utilization of the ML model and / or causes the ML model to behave unexpectedly). Similarly, the use of a single evaluation of the ML model may limit the tasks for which the model can be used, among other detriments.

[0017] Accordingly, aspects of the present application relate to multi-stage machine learning model chains, where an input is processed by being evaluated using a set of ML models (e.g., linked together in a "skill chain") to ultimately produce a model output for a given input. Each ML model evaluation corresponds to a "model skill" of the skill chain. For example, a model skill has an associated prompt template that is used to generate a prompt (e.g., including the input and / or context), and the prompt is processed using the corresponding ML model to generate a model output accordingly. In other examples, the ML model associated with a model skill does not need to have an associated prompt template, as is the case when a prompt is not used by the ML model when processing the input to generate a model output.

[0018] An intermediate output generated by a first skill of the skill chain can subsequently be processed by a second skill to generate a subsequent output accordingly. Such a skill chain can include any number of skills, and it should be understood that model skills do not need to be associated with the same ML model. For example, a generative model can generate a natural language output, and a recognition model (or any of various other types of ML models) can process the intermediate output from the generative model to produce a model output accordingly. The output of the recognition model can be provided as the final model output or can be an intermediate output that is processed using another skill of the skill chain.

[0019] The generative models used according to the aspects described herein (which are also generally referred to herein as a type of ML model) can generate any output type among various output types (and thus can be a multimodal generative model in some examples), and can be a generative transformer model and / or a large language model (LLM), and in some examples, a generative image model. Example ML models include but are not limited to the Megatron-Turing Natural Language Generation Model (MT-NLG), Generative Pretrained Transformer 3 (GPT-3), Generative Pretrained Transformer 4 (GPT-4), BigScience BLOOM (Large Open Science Open Access Multilingual Language Model), DALL-E, DALL-E2, StableDiffusion, or Jukebox. Additional examples of these aspects are discussed below with respect to Figures 5A - 5B the generative ML models shown in

[0020] The output of a skill that is processed by a subsequent skill can be referred to herein as an "intermediate output". Example intermediate outputs include but are not limited to natural language output, image data output, video data output, procedural output, and / or binary output. Accordingly, it should be understood that an intermediate output can have any number of output "streams" (e.g., each stream having an associated content type).

[0021] In an example, the intermediate output includes structured output, which may include one or more tags, key / value pairs, and / or metadata, among other things. For example, a stream can be represented based on associated tags within such structured output. In an example, a prompt template of a skill defines or otherwise includes an indication related to such structured output such that a generative ML model associated with the skill produces structured output accordingly. In this way, the use of structured output can increase the degree of certainty of the model output and thus can improve reliability when chaining together multiple ML model evaluations according to aspects described herein. In other examples, the intermediate output can be similar to the final model output, which can otherwise be provided to a user or further processed by an application, among other examples.

[0022] According to aspects described herein, skills can be chained together according to any of a variety of techniques. For example, a skill chain can include one or more sequential skills, a set of hierarchical skills, a set of parallel skills, and / or a skill that depends on or otherwise processes the output from two or more skills, among other examples. Thus, it should be understood that a skill chain is a graph or can be arranged according to any of a variety of other data structures. Additionally, a skill chain can include any of a variety of other types of skills. For example, one or more model skills can be chained together with procedural skills. For example, a procedural skill can read the contents of a file, obtain data from a data source and / or from a user, send an electronic message including a model output, create a file including a model output, and / or execute a procedural output generated by a model skill.

[0023] In an example, a skill library stores model and / or procedural skills from which a set of skills can be identified and used to generate a skill chain accordingly (e.g., to perform a set of associated ML model evaluations). As an example, a chain coordinator can extract an intent from a given input (e.g., from a user and / or an application), which can be mapped to one or more skills of the skill library to generate a chain of model and / or procedural skills that are utilized to process the input. In an example, new skills are added to the skill library (e.g., by an application or by a user or developer) such that the new skills can be dynamically identified and used as part of a skill chain to process a given input according to aspects described herein.

[0024] As described above, a skill can include a prompt template or otherwise be associated with a prompt template. One or more fields, regions, and / or other portions of the prompt template can be populated (e.g., utilizing input and / or context) to generate a prompt to be processed by an ML model according to aspects described herein. For example, the prompt is used to initiate the ML model, thereby inducing the model to generate an output corresponding to the prompt template. It should be understood that the prompt template can include any of a variety of data, including but not limited to natural language, image data, audio data, video data, and / or binary data, among other examples.

[0025] Thus, in an example, a skill as used herein invokes processing of an ML model (e.g., according to an associated prompt template) to process a given input (e.g., as can be received from a user or as can be an intermediate output from another skill). In an example, context is processed as part of the ML model evaluation. For example, the input can include an indication of context, the skill can define the context to be provided to the ML model, and / or a chain coordinator (e.g., defining and / or managing processing of a skill chain) can accordingly determine the context to be used for the ML model evaluation, among other examples.

[0026] In addition to the chain prompt for ML model evaluation, the associated context can be shared among or otherwise used by multiple model skills. For example, at least a portion of the context used for processing associated with a first model skill (or in other examples, multiple model skills) can be used by a second model skill. In an example, the context is changed by a first model evaluation (e.g., of the first model skill) that occurs before or concurrently with processing by a second ML model evaluation (e.g., for the second model skill), such that the second ML model evaluation uses the updated context accordingly.

[0027] As a result of the disclosed chaining techniques, it is possible to complete tasks and / or generate model outputs that would otherwise not be possible via a single ML model evaluation. For example, information can be obtained from one or more data sources and / or user request inputs can be received when processing a skill chain, which are then used for subsequent processing (e.g., by one or more subsequent skills of the skill chain). As another example, the evaluation of a skill chain can be dynamically adapted as a result of a constituent evaluation, thereby affecting one or more future evaluations of the skill chain (e.g., by adding an evaluation, removing an evaluation, or changing an evaluation). Additionally, the skill chain itself can be managed, coordinated, and / or derived by an ML model (e.g., a generative ML model based on natural language input received from a user and / or input generated by or otherwise received from an application). Additionally, different given ML models can be chained together (e.g., each of which can generate different types of model outputs), and the resulting model output can be an output that would not be produced by the processing of a single ML model.

[0028] Figure 1 An overview of an example system 100 in which a multi-stage machine learning model chain can be used in accordance with aspects of the present disclosure is shown. As shown, system 100 includes a machine learning service 102, a computing device 104, and a network 106. In an example, the machine learning service 102 and the computing device 104 communicate via the network 106, which can include a local area network, a wireless network, the Internet, or any combination thereof, among other examples.

[0029] As shown, the machine learning service 102 includes a chain coordinator 108, a model repository 110, a skill library 112, and a semantic memory store 114. In an example, the machine learning service 102 receives a request from the computing device 104 (e.g., from a multi-stage machine learning framework 118) to generate a model output, which can be generated using a skill chain as described herein. As described above, the request can include an input (e.g., user input that can be received from a user at the computing device 104 and / or generated by an application 116).

[0030] The received request is processed by the chain coordinator 108, which can identify one or more ML models from the model repository 110 and process the input accordingly. In an example, the chain coordinator 108 processes the request to generate a skill chain and uses the skill chain to generate a model output (e.g., using one or more models of the model repository 110). For example, the chain coordinator 108 uses a generative ML model to process at least a portion of the input (e.g., a prompt generated from a prompt template using the input), thereby generating a skill chain that includes one or more model skills (and in some examples, one or more procedural skills). The chain coordinator 108 then processes the resulting skill chain according to the aspects described herein, such as using one or more models of the model repository 110, skills of the skill repository 112 and / or the skill repository 120, and / or context from the semantic memory store 114 and / or the semantic memory store 122.

[0031] In other examples, the request includes a prompt that is used to initiate an ML model (e.g., generated using a prompt template for a skill of the skill repository 120 from the computing device 104). As another example, the request includes an indication of a skill in the skill repository 112 such that the chain coordinator 108 generates a corresponding prompt based on the skill from the skill repository 112 accordingly. Additionally or alternatively, the request includes context that the request is to be processed with (e.g., from the semantic memory store 122 of the computing device 104). As another example, the request includes an indication of context in the semantic memory store 114 such that the chain coordinator 108 retrieves the context from the semantic memory store 114 accordingly. Additional examples of these and other aspects are discussed below with respect to Figure 2 the semantic memory store 218 and the corresponding recall engine 210 in

[0032] In some instances, the chain coordinator 108 obtains additional information used when processing the request (e.g., as may be obtained from a remote data source or from a user request of the computing device 104). For example, the chain coordinator 108 can determine to obtain additional information for a given evaluation of the skill chain, among other examples. As an example, the additional information can be obtained via a procedural skill (e.g., as may have been included in the skill chain by the chain coordinator 108). Examples of these aspects are discussed in more detail below with respect to Figure 3 and Figure 4 the methods 300 and 400 of

[0033] The model repository 110 may include any number of different ML models. For example, the model repository 110 may include a base model, a language model, a speech model, a video model, and / or an audio model. As used herein, a base model is a model that is pre-trained on a broad set of data that can be adapted to a wide range of tasks (e.g., a model capable of handling various different tasks or modalities). In an example, the multi-modal machine learning models of the model repository 110 may be trained using training data having multiple content types. Thus, given a first type of content, the ML models of the model repository 110 may generate content of any of the various associated types. It should be understood that the model repository 110 may include base models as well as models that have been fine-tuned (e.g., for a particular context and / or a particular user or group of users), among other examples.

[0034] Turning now to computing device 104, computing device 104 includes an application 116, a multi-stage machine learning framework 118, a skill library 120, and a semantic memory store 122. In an example, the application 116 uses the multi-stage machine learning framework 118 to process user input and generate a model output accordingly, which may be presented to a user of the computing device 104 and / or used for subsequent processing by the application 116, among other examples.

[0035] In an example, aspects of the multi-stage machine learning framework 118 are similar to the chain coordinator 108 and thus need not be described in detail again. For example, in addition to or as an alternative to being generated by the skill chains of the chain coordinator 108, the multi-stage machine learning framework 118 may generate and / or manage the evaluation of skill chains according to aspects described herein. For example, the multi-stage machine learning framework 118 provides an indication of user input to the machine learning service 102 such that the skill chains are generated by the machine learning service 102 and received by the computing device 104 as a response. Thus, the multi-stage machine learning framework 118 manages the evaluation of skill chains (e.g., generating subsequent requests to the machine learning service 102 for constituent model skills) according to one or more associated prompt templates (e.g., as may be stored by the skill library 112 / skill library 120) and / or based on an associated context (e.g., from the semantic memory store 114 / semantic memory store 122). In an example, the multi-stage machine learning framework 118 requests a model output from the machine learning service 102 for the model skills of a skill chain, while the procedural skills of the skill chain may be processed locally on the computing device 104 (or remotely in other examples).

[0036] Accordingly, it should be understood that the disclosed aspects may be implemented according to any of the various examples. For example, skill chain generation / coordination and / or prompt generation (e.g., prompt templates based on model skills) may be performed on the client side (e.g., via the multi-stage machine learning framework 118), on the server side (e.g., via the chain coordinator 108), or any combination thereof, among other examples. For example, the multi-stage machine learning framework 118 may perform a first ML evaluation associated with a first model skill stored in the skill library 120 of the computing device 104, while a second ML evaluation is performed by the machine learning service 102 based on a second model skill stored in the skill library 112. The multi-stage machine learning framework 118 may be provided as part of the operating system of the computing device 104 (e.g., as a service, application programming interface (API), and / or framework), may be available as a library included by the application 116 (or may be more directly incorporated by the application), or may be provided as a stand-alone application, among other examples.

[0037] As another example, a user interface is provided through which a user may interact with the multi-stage machine learning framework and / or the chain coordinator. For example, the machine learning service 102 may additionally or alternatively implement aspects similar to the multi-stage machine learning framework 118 such that the machine learning service 102 provides a website through which a user may interact with the console or terminal interface of the multi-stage machine learning framework accordingly. The console may include a text-based user interface through which a user inputs skills (e.g., model skills and / or procedural skills) that may be linked together. For example, skills may be linked together using the pipe (“|”) operator such that the output of one skill (e.g., a first model skill) is transmitted as input to another skill (e.g., a second model skill). It should be understood that the input and / or output of one or more skills may additionally or alternatively be redirected according to any of the various other techniques (e.g., using the “<<”, “>”, and / or “>>” operators).

[0038] As another example, computing device 104 can include a user interface that is part of an application (e.g., application 116) or multiple applications (e.g., such as a shared framework or functionality provided by the operating system of computing device 104). In such an example, natural language input can be provided via the user interface (e.g., such as text input and / or such as voice input), and the natural language input can be processed according to aspects described herein and used accordingly to generate a skill chain. In an example, the skills of the skill chain can interact with one or more command interfaces, each of which can be associated with an application (e.g., application 116) and / or the operating system of computing device 104, among other examples. For example, the operating system can provide a command interface via which interactions can be performed, e.g., through an accessibility API and / or an extensibility API.

[0039] In an example, a model skill can generate a procedural output that is executed, parsed, or otherwise processed (e.g., as a procedural skill) to interact with various functions and / or other aspects of computing device 104 (e.g., application 116, system preferences, etc.) based on the received natural language input. Thus, according to aspects described herein, a user of computing device 104 can use such an interface to interact with application / device functionality via a multi-stage machine learning model chain. Although examples have been described in which user input is received as natural language input and subsequently causes an ML evaluation chain, it should be understood that in instances where such a multi-stage machine learning model chain is tied to a user interface element (e.g., user actuation of a button, scroll bar, window, or menu) or other software processing, etc., similar techniques can be used.

[0040] Figure 2 An overview of an example conceptual diagram 200 for processing user input using a chained machine learning model to generate a model output according to aspects described herein is shown. As shown, diagram 200 processes user input 202 according to a set of models (e.g., ML models 204 and 206 coordinated by chain coordinator 203) to generate a model output 208. For example, user input 202 can be received from a computing device (such as Figure 1 computing device 104 therein). Aspects of chain coordinator 203 can be similar to those discussed above with respect to chain coordinator 108 and are therefore not necessarily described in detail below.

[0041] The user input 202 can include any one of a variety of inputs, including but not limited to natural language input, command line input, input received via a framework or application, and / or input received via a central service (e.g., of an operating system) or a Uniform Resource Identifier (URI) processor, etc. Although examples are described herein with reference to natural language input, it should be understood that any one of a variety of additional or input types can be received, including but not limited to image input and / or video input. Additionally, natural input can include any one of a variety of inputs, such as text input or voice input.

[0042] As shown, the user input 202 is processed by the chain coordinator 203. In an example, according to aspects described herein, the chain coordinator 203 processes the user input 202 to generate a skill chain that includes a plurality of model skills (e.g., an evaluation including ML model 204 and ML model 206), to ultimately generate a model output 208. Such aspects can be similar to those discussed above with respect to the chain coordinator 108, such that the user input 202 is processed to extract an intent that maps to one or more skills (e.g., of the skill library 212).

[0043] To generate the skill chain, a coordination prompt can be generated by the chain coordinator 203, which includes an indication of one or more skills from the skill library 212 (which is also referred to herein as the "skill list") and at least a portion of the user input 202, such that a generative ML model generates the skill chain by which the user input 202 is processed. Thus, the chain coordinator 203 correspondingly maps one or more intents of the user input 202 to one or more models and / or procedural skills of the skill library 212. Additional examples of these and other aspects of the chain coordinator 203 are discussed below with respect to Figure 3 operation 304 of method 300 in

[0044] In an example, the skill list is generated dynamically. As an example, the skill library 212 can include one or more files, each defining one or more skills by which an input (e.g., user input and / or intermediate output) can be processed. As another example, the skill library 212 includes a database storing a list of skills. In some cases, new skills can be registered (e.g., in the database or in an index), thereby indicating that the skill can be used as part of a skill chain. For example, a plug-in application 214 (e.g., aspects of which can be similar to application 116) can include one or more skills registered within the skill library 212, such that processing of skills using the plug-in application 214 can be performed according to aspects described herein. Thus, it should be understood that a set of skills in the skill library 212 can be stored using any one of a variety of techniques.

[0045] In an example, the chain coordinator 203 lists the contents of the skill library 212 when generating a skill list. A skill may include or otherwise have an associated description of its functionality (e.g., a man page or usage information such as the syntax and / or indication of one or more inputs / outputs), at least a portion of which may be included in the skill list generated by the chain coordinator 203 and used to generate a skill chain accordingly.

[0046] The chain coordinator 203 thus manages the processing of the user input 202 based on the generated skill chain. As described above, such a skill chain may include one or more sequential skills, a set of hierarchical skills, a set of parallel skills, and / or a skill that depends on or otherwise processes the output from two or more previous skills, among other examples. In an example, the evaluation order of the skill chain is determined based on the available skills in the skill library 212. Additionally or alternatively, the skill chain includes one or more procedural skills, although this example is one in which two model skills are used, corresponding to the ML model 204 and the ML model 206.

[0047] As shown, once the skill chain is generated by the chain coordinator 203, the user input 202 is processed by the ML model 204 to generate an intermediate output. For example, a prompt template corresponding to the first model skill may be populated or otherwise processed to generate a prompt that is accordingly processed by the ML model 204 (e.g., including at least a portion of the user input 202 and / or context from the semantic memory store 218). Thus, the intermediate output from the ML model 204 is then processed by the ML model 206. The ML model 204 and the ML model 206 may each be the same or similar models (e.g., generating the same type(s) of content and / or trained using similar training data), or as another example, the ML model 204 and the ML model 206 may each be different models (e.g., generating different sets of content types). When processing an input (e.g., the user input 202 or an intermediate output from a previous evaluation), the ML model 204 and the ML model 206 may each use skills from the skill library 212 (e.g., as may have been determined or otherwise identified by the chain coordinator 203 in accordance with the aspects described herein).

[0048] In an example, each of the ML models 204 and 206 uses context obtained from the recall engine 210, as may be stored by the semantic memory store 218. For example, it may be determined (e.g., by the chain coordinator 203 and / or by the ML model 204 or the ML model 206) that processing associated with a skill should be performed based on the context from the recall engine 210. In other examples, a skill from the skill library 212 may indicate (e.g., as part of an associated prompt template) that context should be obtained from the semantic memory store 218, such that the recall engine 210 is accordingly used to obtain such context. Thus, it should be understood that context may be obtained for a skill as a result of any one of the various determinations and / or indications and other examples.

[0049] As an example, the semantic memory store 218 stores semantic embeddings (also referred to herein as "semantic addresses") associated with the ML model 204 and / or the ML model 206, each semantic embedding corresponding to one or more content objects. In an example, entries in the semantic memory store 218 include one or more semantic embeddings corresponding to context objects and / or the context objects themselves or references to the context objects and other examples.

[0050] In an example, the semantic memory store 218 stores embeddings associated with one or more models (e.g., the ML model 204 and / or the ML model 206) and their specific versions, which can thus represent the same or similar content (e.g., associated with each model / version) in a varying semantic embedding space. Additionally, when a new model is added or an existing model is updated, one or more entries within the semantic memory store 218 may be re-encoded (e.g., by generating new semantic embeddings according to the new embedding space). In this way, a single content object entry within the semantic memory store 218 can have a locatable semantic address across models / versions, enabling the retrieval of content objects based on a determination (e.g., as a result of an algorithmic comparison) of similarity between the corresponding semantic address and a semantic context indication.

[0051] As a result, an input embedding may be generated (e.g., may be associated with the user input 202 and / or processed by the ML model 204 or the ML model 206). For example, the input embedding may be generated by a machine learning model that encodes the intent corresponding to the user input 202 accordingly. Additionally or alternatively, the input embedding may be generated based on any one of various other inputs (e.g., audio and / or visual inputs) received by the computer. Those skilled in the art will recognize additional and / or alternative methods for generating input embeddings.

[0052] The recall engine 210 can thus identify one or more content objects provided as context for processing associated with a skill based on the input embedding. For example, a set of semantic embeddings that match the input embedding (e.g., using cosine distance, another geometric n-dimensional distance function, or other algorithmic similarity metric) can be identified and used accordingly to identify one or more corresponding content objects. As described above, the processing of ML model 204 and / or ML model 206 can add, remove, or otherwise modify one or more entries in the semantic memory store 218 such that the context used by subsequent skills from the recall engine 210 may be affected by one or more previous skills.

[0053] As a result of the multi-stage ML model chain performed by ML model 204 and ML model 206, a model output 208 is generated. Thus, one or more model skills of the skill chain can each generate an intermediate output (e.g., a structured output), and the final skill of the skill chain (e.g., the ML model evaluation performed by ML model 206, as shown) can generate the model output 208 based on such intermediate outputs. As an example, the final model output produced by ML model 206 includes, but is not limited to, natural language output, speech and / or audio output, image output, video output, and / or procedural output.

[0054] FIG. 200 shows an example where the skill chain includes two model skills (e.g., corresponding to ML model 204 and ML model 206). Arrows 216 are provided to illustrate that in other examples, additional model skills can be included (e.g., associated with any of ML model 204, ML model 206, and / or various other ML models, not shown). Additionally, while FIG. 200 depicts an example where the model skills are sequential, it should be understood that parallel skills, hierarchical ML skills, and / or skills that depend on outputs from multiple previous skills, etc., can also be used in other instances. Additionally, as described above, in other examples, the skill chain can also include one or more procedural skills.

[0055] Figure 3 An overview of an example method 300 for processing user input to generate a model output in accordance with aspects described herein is shown. In an example, aspects of method 300 are performed by a chain coordinator (e.g., Figure 1 chain coordinator 108 in Figure 2 or chain coordinator 203 in

[0056] and / or by a multi-stage machine learning framework (e.g., multi-stage machine learning framework 118) and other examples. Figure 1The computing device 104) receives user input, which can be the case when aspects of method 300 are performed by a machine learning service (e.g., Figure 1 the machine learning service 102) in. As another example, user input is received from an application (e.g., application 116), from a service, or from other software of a computing device (e.g., computing device 104), which can be the case when aspects of method 300 are performed by a multi-stage machine learning framework (e.g., multi-stage machine learning framework 118). In an example, the received input can be similar to the user input 202 discussed above with respect to Figure 2 The received user input can include natural language input (e.g., text and / or voice input), image input, and / or video input, as well as any one of a variety of other inputs.

[0057] Method 300 proceeds to operation 304, where the received input is processed to generate a skill chain (e.g., corresponding to a set of ML evaluations and, in some examples, further corresponding to a set of procedural evaluations according to aspects described herein). For example, prompts processed by a generative ML model are generated to generate a skill chain accordingly. The prompts can be generated based on a prompt template that is populated to include at least a portion of the input received at operation 302. In an example, the prompt template is further populated with a list of skills (e.g., from a skill library such as Figure 1 and Figure 2 skill libraries 112, 120, and / or 212) in.

[0058] In an example, a chain coordinator of a machine learning service (e.g., chain coordinator 108) generates a skill chain. As another example, a request is provided to a machine learning service to process the input such that an indication of the skill chain is received as a response, which can be the case when aspects of method 300 are performed by a multi-stage machine learning framework of a client computing device (e.g., Figure 1 the computing device 104) in. As another example, at least a portion of such skill chain generation is performed locally on the computing device, which can be the case when the generative ML model used to perform such aspects is locally available. As described above, the generated skill chain can include one or more sequential skills, a set of hierarchical skills, a set of parallel skills, and / or skills that depend on or otherwise process the output from two or more previous skills, among other examples.

[0059] Although using a generative ML model to generate a skill chain is described (e.g., with respect to Figure 1 , Figure 2 and Figure 3Examples of the chain coordinator 108, chain coordinator 203, and operation 304) in [description], but it should be understood that any of a variety of additional or alternative techniques may be used. As an example, a semantic store similar to the semantic memory store 218 may additionally or alternatively be used to store one or more embeddings associated with the skills of the skill library (e.g., as may be generated based on at least a portion of the associated skill description, manual page, and / or associated prompt template). For example, input embeddings may be generated for the input received at operation 302 (e.g., to indicate one or more associated intents) and used to identify one or more skills having associated embeddings that match the input embeddings (similar to the aspects discussed above with respect to the recall engine 210). Thus, the identified skills may accordingly form a skill chain.

[0060] As another example, when generating a skill chain (e.g., which may be included as part of the generated prompt), context may be provided to the generative ML model, as may be determined by the recall engine from the semantic memory engine, similar to the recall engine 210 and semantic memory store 218 discussed above with respect to Figure 2 the discussion of the recall engine 210 and semantic memory store 218.

[0061] The process proceeds to operation 306, where a skill is selected from the skill chain generated at operation 304. In the example, the skill chain generated at operation 304 indicates the order, hierarchy, and / or one or more interdependencies that make up the skills, such that the skills are selected accordingly at operation 306.

[0062] At determination 307, it is determined whether the selected skill is a procedural skill. As described above, in some examples, the skill chain may include procedural skills for any of the various processes performed by the computing device. Thus, if it is determined that the selected skill is a procedural skill, the process branches "yes" to operation 309, where the procedural skill is processed. In the example, operation 309 includes executing commands, obtaining additional information (e.g., from the user or from a data source), and / or affecting the operation of the operating system or application (e.g., via an API or command interface), among other examples. In subsequent iterations of method 300, executing the procedural skill at operation 309 may include executing a procedural output generated by a machine learning model according to the aspects described herein. Thus, it should be understood that any of a variety of procedural operations may be performed when evaluating the skill chain. The process then proceeds to determination 314, which will be discussed below.

[0063] However, if it is not determined that the selected skill is not a procedural skill (e.g., such that it is alternatively a model skill), the process branches "no" to determination 308, where it is determined whether from the semantic memory store (e.g., Figure 1 andFigure 2 The semantic memory stores 114, semantic memory store 122, and / or 210) recall context. As described above, this determination can be based on a prompt template corresponding to the selected model skill. For example, the prompt template can indicate that context should be retrieved from the semantic memory store and / or can include an indication of what context should be retrieved (if available). As another example, it can be automatically determined to recall context from the semantic memory store, as can be determined based on previous model skills using the same or similar prompts. Thus, it should be understood that as a result of any of the various determinations and / or indications and other examples, context can be retrieved from the semantic memory store for the model skill.

[0064] If it is determined to recall context from the semantic memory, the process branches "yes" to operation 310, where the context is generated based on the semantic memory store. Such aspects can be similar to those discussed above with respect to Figure 2 the recall engine 210 in, and thus need not be described in detail below. For example, an input semantic embedding is generated based on the user input and / or prompt template on which the ML evaluation is to be performed, such that one or more matching semantic embeddings can be identified from the semantic memory store. The content corresponding to the identified (multiple) semantic embeddings is retrieved and used accordingly as the context for the ML evaluation of the model skill. As described above, the retrieved content can be included in the prompt generated according to the prompt template. It should be understood that context can be retrieved from any of a variety of sources, including but not limited to the user's computing device (e.g., Figure 1 the computing device 104 in) and / or a machine learning service (e.g., machine learning service 102), etc. In contrast, if alternatively it is determined not to recall context from the semantic memory store, the process alternatively branches "no" from determination 308 to operation 312, which will be discussed below.

[0065] The process finally proceeds to operation 312, where an output is generated for the selected machine learning skill. As described above, the prompt can be generated based on the prompt template such that the prompt includes at least a portion of the input and, in some examples, includes the generated context. It should be understood that in other examples, the ML model associated with the model skill may not use a prompt. Similar to operation 304, in instances where the skill chain generation aspect of method 300 is performed locally on the client computing device, a request for ML processing can be provided to the machine learning service such that the generated output is received as a response from the machine learning service. In some examples, the generated output (e.g., can be received as a response from the machine learning service) can be an intermediate output, which includes, for example, a structured output generated as a result of the prompt, the prompt including an indication of such a structured output, as described above. Below with respect toFigure 4 Method 400 discusses additional example aspects of operation 312.

[0066] At determination 314, it is determined whether there are remaining skills in the skill chain generated at operation 304. In an example, as a result of operation 309 and / or operation 312 above, the skill chain is updated. Determination 314 may include evaluating the skill chain (e.g., as generated at operation 304 and / or as may have been updated due to operation 309 and / or operation 312) to determine whether there are skills that have not yet been processed. If it is determined that there are no remaining skills, the process branches "no" to operation 316, which will be discussed below.

[0067] In contrast, if it is determined that there are remaining skills, the process branches "yes" and returns to operation 306, where a subsequent skill is selected. Thus, in instances where one or more skills are retained, the process loops between operations 306 - 314. When generating subsequent model outputs, subsequent iterations of operation 312 may use the outputs generated by previous iterations of operation 309 and / or operation 312 as inputs to the model skills. Similarly, in some examples, subsequent iterations of operation 309 may use the outputs generated by previous iterations of operation 309 and / or operation 312. Additionally or alternatively, at least a portion of the received user input is used as an input to subsequent iterations of operation 309 and / or operation 312.

[0068] In addition to chaining ML evaluations via subsequent iterations of operation 312, in some examples, one or more contexts may be chained together as a result of subsequent iterations of operation 310. For example, the context corresponding to a previous ML evaluation (e.g., as may have been generated by a previous iteration of operation 310 and / or updated by a previous iteration of operation 312) may be used as the context for a subsequent ML evaluation of operation 312.

[0069] Finally, method 300 reaches operation 316, where an indication of the generated output is provided. For example, the indication may be provided to a client computing device, as is the case in an instance where method 300 is executed by a chain coordinator of a machine learning platform in various aspects. Additionally or alternatively, the indication may be provided by a multi - stage machine learning framework of the client computing device. For example, the indication is provided to an application (e.g., Figure 1 application 116 in

[0070] Figure 4 shows an overview of an example method 400 for processing user input according to a prompt using a generative ML model (also referred to herein as ML model evaluation) according to aspects described herein. In the example, aspects of method 400 are performed as part of operation 312 discussed above with respect to Figure 3 method 300.

[0071] As shown, method 400 begins at operation 402, where an input is obtained. Aspects of the obtained input may be similar to the user input 202 discussed above with respect to Figure 2 or the user input received at operation 302 of method 300 in Figure 3 and thus need not be described in detail below. For example, the input may be obtained from a user of a computing device (e.g., Figure 1 computing device 104 in Figure 3 ). In some examples, according to aspects described herein, the input is received as part of a request for a generative model output (e.g., as a result of performing aspects of operation 312 discussed above with respect to Figure 3 method 300).

[0072] At operation 404, context may be obtained. Operation 404 is shown with a dashed box to indicate that in other examples, operation 404 may be omitted. Similar to operation 402, in some examples, the context may be obtained as part of a request for a generative model output. In other examples, the context may be obtained from a semantic memory store, such as may be generated by a recall engine similar to recall engine 210 from semantic memory store 218, as discussed above with respect to Figure 2 above.

[0073] The flow proceeds to operation 406, where a prompt is generated. In the example, an indication of a model skill (e.g., corresponding to a prompt template) is received as part of the received request (as described above with respect to operation 402). In some cases, the prompt template may be obtained based on an association with a model skill in a skill library (e.g., Figure 1 skill library 112 and / or skill library 120 in Figure 2 and / or skill library 212 in

[0074] Moving to operation 408, a model is determined from a set of models. In an example, the skill used to generate the prompt at operation 406 can include an indication of the model to process the generated prompt. As another example, the received request can include such an indication. The model can be identified from a model repository (such as the model repository 110 in Figure 1 ). In other examples, such a determination is not required, as can be the case when the machine learning service and / or the associated API through which the request is received utilize a single ML model to perform the processing.

[0075] The process proceeds to operation 410, where a model output is generated. In an example, operation 410 includes processing the prompt generated at operation 406 according to the ML model determined at operation 408. Aspects of example ML models that can be used to perform such processing are described below with reference to Figures 5A - 5B .

[0076] At operation 412, an indication of the generated output is provided. For example, a response to the request received as part of operation 402, operation 404, and / or operation 406 can be generated, the response including at least a portion of the model output. As described above, the model output can include intermediate output and / or structured output, which can be the case when the request corresponds to an intermediate ML evaluation of a skill chain. Thus, the indication of the generated output can be received by a computing device, where subsequent processing can be performed accordingly (e.g., by a multi-stage machine learning framework and / or application, such as multi-stage machine learning framework 118 and / or application 116). Method 400 terminates at operation 412.

[0077] Figure 5A and Figure 5B show an overview of an example generative machine learning model that can be used in accordance with the aspects described herein. First, with reference to Figure 5A , the conceptual diagram 500 depicts an overview of a pre-trained generative model package 504 according to the aspects described herein, which processes inputs and prompts 502 for skills of a skill chain to generate a model output 506 for a multi-stage ML model chain. Examples of the pre-trained generative model package 504 include, but are not limited to, the Megatron-Turing Natural Language Generation model (MT-NLG), Generative Pretrained Transformer 3 (GPT-3), Generative Pretrained Transformer 4 (GPT-4), BigScience BLOOM (Large Open Science Open Access Multilingual Language Model), DALL-E, DALL-E2, Stable Diffusion, or Jukebox.

[0078] In an example, the generative model package 504 is pre-trained based on various inputs (e.g., various human languages, various programming languages, and / or various content types), and thus does not need to be fine-tuned or trained for a specific scenario. Instead, the generative model package 504 can be pre-trained more generally such that the input 502 includes a prompt that is generated, selected, or otherwise designed to induce the generative model package 504 to produce a specific generative model output 506. For example, the prompt includes context and / or one or more completion prefixes to pre-load the generative model package 504 accordingly. As a result, the generative model package 504 is induced to generate an output based on the prompt, which includes a predicted sequence of tokens related to the prompt (e.g., not exceeding the token limit of the generative model package 504). In an example, the predicted sequence of tokens is further processed (e.g., via output decoding 516) to produce the output 506. For example, each token is processed to identify the corresponding word, word fragment, or other content that forms at least a part of the output 506. It should be understood that the input 502 and the generative model output 506 can each include any one of various content types, including but not limited to text output, image output, audio output, video output, procedural output, and / or binary output, among other examples. In an example, the input 502 and the generative model output 506 can have different content types, as can be the case when the generative model package 504 includes a generative multimodal machine learning model.

[0079] Accordingly, the generative model package 504 can be used in any of a variety of scenarios, and further, different generative model packages can be used in place of the generative model package 504 with substantially no modification to other associated aspects (e.g., similar to those aspects described herein with respect to Figure 1 , Figure 2 , Figure 3 and Figure 4 ). Accordingly, the generative model package 504 operates as a tool for performing machine learning processing, where a specific input 502 to the generative model package 504 is procedurally generated or otherwise determined such that the generative model package 504 produces a model output 506, which can then be used for further processing.

[0080] The generative model package 504 can be provided or otherwise used according to any of a variety of paradigms. For example, the generative model package 504 can be used in a computing device (e.g., Figure 1used locally by the computing device 104), or can be remotely accessed from a machine learning service (e.g., machine learning service 102). In other examples, aspects of the generative model package 504 are distributed across multiple computing devices. In some instances, the generative model package 504 can be accessed via an application programming interface (API), such as may be provided by the operating system of a computing device and / or by other examples such as a machine learning service.

[0081] Now referring to the illustrated aspects of the generative model package 504, the generative model package 504 includes input tokenization 508, input embedding 510, model layer 512, output layer 514, and output decoding 516. In an example, the input tokenization 508 processes the input 502 to generate the input embedding 510, which includes a sequence of symbolic representations corresponding to the input 502. Thus, the input embedding 510 is processed by the model layer 512, output layer 514, and output decoding 516 to produce the model output 506. Figure 5B An example architecture corresponding to the generative model package 504 is depicted in, which is discussed in further detail below. Even so, it should be understood that the architectures shown and described herein should not be taken in a limiting sense, and in other examples, any of a variety of other architectures may be used.

[0082] Figure 5B is a conceptual diagram depicting an example architecture 550 of a pre-trained generative machine learning model that can be used in accordance with the aspects described herein. As noted above, without departing from the aspects described herein, any of a variety of alternative architectures and corresponding ML models may be used in other examples.

[0083] As shown, the architecture 550 processes the input 502 to produce a generative model output 506, aspects of which were discussed above with respect to Figure 5A The architecture 550 is depicted as a transformer model including an encoder 552 and a decoder 554. The encoder 552 processes the input embedding 558 (aspects of which may be similar to Figure 5A the input embedding 510 in), the input embedding including a sequence of symbolic representations corresponding to the input 556. In an example, the input 556 includes an input corresponding to a skill of a skill chain and a prompt 502, aspects of which may be similar to the user input 202, the context from the semantic memory store 218, and / or a prompt template generated based on skills from the skill library 112, skill library 120, and / or skill library 212 in accordance with the aspects described herein.

[0084] Additionally, the positional encoding 560 can introduce information about the relative and / or absolute positions of the tokens of the input embedding 558. Similarly, the output embedding 574 includes a sequence of symbolic representations corresponding to the output 572, and the positional encoding 576 can similarly introduce information about the relative and / or absolute positions of the tokens of the output embedding 574.

[0085] As shown, the encoder 552 includes an example layer 570. It should be understood that any number of such layers can be used, and the depicted architecture is simplified for illustrative purposes. The example layer 570 includes two sub-layers: the multi-head attention layer 562 and the feed-forward layer 566. In the example, residual connections are included around each of the layers 562, 566, followed by a normalization layer 564 and a normalization layer 568, respectively.

[0086] The decoder 554 includes an example layer 590. Similar to the encoder 552, any number of such layers can be used in other examples, and the architecture of the depicted decoder 554 is simplified for illustrative purposes. As shown, the example layer 590 includes three sub-layers: the masked multi-head attention layer 578, the multi-head attention layer 582, and the feed-forward layer 586. Aspects of the multi-head attention layer 582 and the feed-forward layer 586 can be similar to those discussed above with respect to the multi-head attention layer 562 and the feed-forward layer 566, respectively. Additionally, the masked multi-head attention layer 578 performs multi-head attention on the output of the encoder 552 (e.g., the output 572). In the example, the masked multi-head attention layer 578 prevents positions from attending to subsequent positions. This mask in combination with the offset embedding (e.g., offset by one position, as shown for the multi-head attention layer 582) can ensure that the prediction for a given position depends on the known outputs of one or more positions less than the given position. As shown, residual connections are also included around the layers 578, 582, and 586, followed by a normalization layer 580, a normalization layer 584, and a normalization layer 588, respectively.

[0087] The multi-head attention layer 562, the multi-head attention layer 578, and the multi-head attention layer 582 can each linearly project the query, key, and value to corresponding dimensions using a set of linear projections. Each linear projection can be processed using an attention function (e.g., dot product or additive attention) to produce an n-dimensional output value for each linear projection. The resulting values can be concatenated and projected again such that the values are then processed as Figure 5B shown (e.g., by the corresponding normalization layer 564, normalization layer 580, or normalization layer 584).

[0088] The feed-forward layer 566 and the feed-forward layer 586 can each be a fully-connected feed-forward network applied to each position. In an example, the feed-forward layer 566 and the feed-forward layer 586 each include a plurality of linear transformations with rectified linear unit activations between the linear transformations. In an example, each linear transformation is the same across different positions and can use different parameters compared to other linear transformations of the feed-forward network.

[0089] Additionally, aspects of the linear transformation 592 can be similar to the linear transformations discussed above with respect to the multi-head attention layer 562, the multi-head attention layer 578, and the multi-head attention layer 582, as well as the feed-forward layer 566 and the feed-forward layer 586. The softmax function 594 can also convert the output of the linear transformation 592 into predicted next-token probabilities, as shown by the output probabilities 596. It should be understood that the architectures shown are provided as examples, and in other examples, any of a variety of other model architectures can be used in accordance with the disclosed aspects. In some cases, multiple iterations of processing are performed in accordance with the above aspects (e.g., using Figure 5A the generative model package 504 in Figure 5B or the encoder 552 and the decoder 554 in

[0090] Figure 3

[0091] Figures 6 - 8 Figures 6 - 8

[0092] Figure 6 ​​​​is a block diagram showing the physical components (e.g., hardware) of a computing device 600 that can practice aspects of the present disclosure. The computing device components described below can be applicable to the computing devices described above, including one or more devices associated with the machine learning service 102, and the computing device 104 discussed above with respect to Figure 1 In a basic configuration, computing device 600 can include at least one processing unit 602 and system memory 604. Depending on the configuration and type of the computing device, system memory 604 can include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of these memories.

[0093] System memory 604 can include an operating system 605 and one or more program modules 606 suitable for running software applications 620, such as one or more components supported by the systems described herein. As an example, system memory 604 can store a chain coordinator 624 and a recall engine 626. Operating system 605 can be suitable for controlling the operation of computing device 600, for example.

[0094] In addition, embodiments of the present disclosure can be practiced in conjunction with a graphics library, other operating systems, or any other application programs, and are not limited to any particular application or system. This basic configuration is shown by those components within dashed line 608 in Figure 6 Computing device 600 can have additional features or functionality. For example, computing device 600 can also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage is shown by removable storage device 609 and non-removable storage device 610 in Figure 6 As described above, multiple program modules and data files can be stored in system memory 604. When executed on processing unit 602, program modules 606 (e.g., applications 620) can perform processes including, but not limited to, aspects described herein. Other program modules that can be used in accordance with aspects of the present disclosure can include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided applications, etc.

[0095] In addition, embodiments of the present disclosure can be practiced on the following circuits: including discrete electronic elements, packages or integrated electronic chips including logic gates, circuits utilizing microprocessors, or single chips including electronic elements or microprocessors. For example, embodiments of the present disclosure can be practiced via a system-on-chip (SOC), where

[0096] In addition, embodiments of the present disclosure can be practiced on the following circuits: including discrete electronic elements, packages or integrated electronic chips including logic gates, circuits utilizing microprocessors, or single chips including electronic elements or microprocessors. For example, embodiments of the present disclosure can be practiced via a system-on-chip (SOC), where Figure 6Each or many of the components shown may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operating via the SOC, the functions regarding the capabilities of the client switching protocol described herein may be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 600. Embodiments of the present disclosure may also be practiced using other technologies capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluidic, and quantum technologies. Additionally, embodiments of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0097] The computing device 600 may also have one or more input devices 612, such as a keyboard, mouse, pen, voice or speech input device, touch or swipe input device, etc. (Multiple) output devices 614, such as a display, speaker, printer, etc., may also be included. The above devices are examples, and other devices may be used. The computing device 600 may include one or more communication connections 616 that allow communication with other computing devices 650. Examples of suitable communication connections 616 include but are not limited to radio frequency (RF) transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0098] As used herein, the term computer-readable medium may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other article that can be used to store information and can be accessed by the computing device 600. Any such computer storage media may be part of the computing device 600. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0099] A communication medium can be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and includes any information delivery medium. The term "modulated data signal" can describe a signal having one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0100] Figure 7 System 700 is shown, which can be, for example, a mobile computing device such as a mobile phone, a smartphone, a wearable computer (such as a smartwatch), a tablet computer, a laptop computer, etc., and embodiments of the present disclosure can be practiced using system 700. In one embodiment, system 700 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, system 700 is integrated as a computing device such as an integrated personal digital assistant (PDA) and a wireless phone.

[0101] In a basic configuration, such a mobile computing device is a handheld computer having both input elements and output elements. System 700 generally includes a display 705 and one or more input buttons that allow a user to input information into system 700. The display 705 can also be used as an input device (e.g., a touchscreen display).

[0102] If included, an optional bypass input element allows for further user input. For example, the bypass input element can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, system 700 can include more or fewer input elements. For example, in some embodiments, the display 705 can not be a touchscreen. In another example, an optional keypad 735 can also be included, and the optional keypad 735 can be a physical keypad or a "soft" keypad generated on a touchscreen display.

[0103] In various embodiments, the output elements include the display 705 for presenting a graphical user interface (GUI), visual indicators (e.g., light-emitting diode 720), and / or audio transducers 725 (e.g., speakers). In some aspects, a vibration transducer is included to provide haptic feedback to the user. In yet another aspect, input and / or output ports are included, such as an audio input (e.g., microphone jack) for sending signals to or receiving signals from an external device, an audio output (e.g., headphone jack), and a video output (e.g., HDMI port).

[0104] One or more applications 766 may be loaded into the memory 762 and run on or in association with the operating system 764. Examples of applications include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, etc. The system 700 also includes a non-volatile storage area 768 within the memory 762. The non-volatile storage area 768 may be used to store persistent information that should not be lost when the system 700 is powered down. The applications 766 may use and store information in the non-volatile storage area 768, such as emails or other messages used by an email application, etc. A synchronization application (not shown) also resides on the system 700 and is programmed to interact with a corresponding synchronization application residing on a host computer to keep the information stored in the non-volatile storage area 768 synchronized with the corresponding information stored at the host computer. It should be understood that other applications may be loaded into the memory 762 and run on the system 700 described herein.

[0105] The system 700 has a power supply 770, which may be implemented as one or more batteries. The power supply 770 may also include an external power supply, such as an AC adapter or a charging docking station that supplements or recharges the battery.

[0106] The system 700 may also include a radio interface layer 772 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 772 facilitates a wireless connection between the system 700 and the "outside world" via a communication carrier or service provider. Transmissions to and from the radio interface layer 772 are under the control of the operating system 764. In other words, communications received by the radio interface layer 772 may be propagated to the applications 766 via the operating system 764, and vice versa.

[0107] The visual indicator 720 can be used to provide visual notifications, and / or the audio interface 774 can be used to generate audible notifications via the audio transducer 725. In the illustrated embodiment, the visual indicator 720 is a light-emitting diode (LED), and the audio transducer 725 is a speaker. These devices can be directly coupled to the power supply 770 such that when activated, these devices remain on for a duration specified by the notification mechanism even if the processor 760 and other components may be turned off to conserve battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the powered-on state of the device. The audio interface 774 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 774 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to embodiments of the present disclosure, the microphone can also be used as an audio sensor to facilitate control of the notifications, as will be described below. The system 700 can also include a video interface 776 that enables operation of the on-board camera 730 to record still images, video streams, etc.

[0108] It should be understood that the system 700 can have additional features or functions. For example, the system 700 can also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage is Figure 7 shown by the non-volatile storage area 768.

[0109] As described above, the data / information generated or collected and stored via the system 700 can be stored locally, or the data can be stored on any number of storage media that can be accessed by the device via the radio interface layer 772 or via a wired connection between the system 700 and a separate computing device associated with the system 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via the radio interface layer 772 or via a distributed computing network. Similarly, such data / information can be easily transmitted between computing devices for storage and use according to any of a variety of data / information transfer and storage components, including email and collaborative data / information sharing systems.

[0110] Figure 8 An aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a personal computer 804, a tablet computing device 806, or a mobile computing device 808, is shown, as described above. The content displayed at the server device 802 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 824, a web portal 825, a mailbox service 826, an instant messaging store 828, or a social networking site 830.

[0111] A multi-stage machine learning framework 820 (e.g., similar to application 620) can be adopted by a client communicating with server device 802. Additionally or alternatively, a chain coordinator 821 can be adopted by server device 802. Server device 802 can provide data to and receive data from client computing devices via network 815, such as personal computer 804, tablet computing device 806, and / or mobile computing device 808 (e.g., smartphone). As an example, the computer system described above can be embodied in personal computer 804, tablet computing device 806, and / or mobile computing device 808 (e.g., smartphone). In addition to receiving graphical data that can be used for preprocessing at a graphical source system or postprocessing at a receiving computing system, any of these examples of computing devices can obtain content from storage 816.

[0112] It should be understood that the aspects and functions described herein can operate on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions can operate remotely from each other on a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information can be displayed via an on-board computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed on and interacted with a wall surface on which various types of user interfaces and information are projected. Interaction with the numerous computing systems in which the embodiments of the present disclosure can be practiced includes keystroke input, touchscreen input, voice or other audio input, gesture input, where the associated computing device is equipped with detection (e.g., camera) functions for collecting and interpreting user gestures to control the functions of the computing device, etc.

[0113] As will be understood from the foregoing disclosure, one aspect of the present technology relates to a system that includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: obtaining user input from a user; generating a skill chain based on the user input, the skill chain including a set of skills for processing the user input; for a first model skill of the skill chain: generating a first prompt including at least a portion of the obtained user input based on a first prompt template associated with the first model skill; and using a first machine learning model associated with the first model skill to process the first prompt to obtain an intermediate output; for a second model skill of the skill chain: generating a second prompt including at least a portion of the intermediate output as an input for the second model skill based on a second prompt template associated with the second model skill; and using a second machine learning model associated with the second model skill to process the second prompt to obtain a model output; and providing an indication of the model output for display to the user. In an example, generating the skill chain includes: generating a skill list corresponding to a set of skills in a skill library, where the skill list includes a description for each skill in the set of skills; providing an indication of the user input and the skill list to a machine learning service; and receiving a skill chain corresponding to the user input from the machine learning service. In another example, generating the skill chain includes: generating an input embedding encoding the intent of the user input for the user input; determining a set of skills from a skill library, each skill in the set of skills having an associated semantic embedding that matches the generated input embedding; and generating a skill chain based on the determined set of skills. In another example, it is determined that the semantic embedding matches the input embedding based on an algorithmic similarity measure between the semantic embedding and the input embedding. In yet another example, processing the first prompt to obtain the intermediate output includes: providing a request to a machine learning service to process the first prompt using the first machine learning model; and receiving a response from the machine learning service including the intermediate output. In yet another example, the intermediate output of the first model skill includes a structured output. In another example, at least a portion of the first prompt corresponds to the structured output.

[0114] In another aspect, the technology relates to a method. The method includes: obtaining, at a computing device, a skill chain corresponding to an input; for a first model skill of the skill chain: generating a first prompt including at least a portion of the user input based on a first prompt template associated with the first model skill; and processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; for a second model skill of the skill chain: generating a second prompt including at least a portion of the intermediate output as an input for the second model skill based on a second prompt template associated with the second model skill; and processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and processing at least a portion of the model output by the computing device to affect the operation of the computing device. In an example, the first machine learning model is the second machine learning model. In another example, the skill chain further includes a procedural skill executed by the computing device; and the output of the procedural skill is processed as an input for the second model skill. In another example, the intermediate output of the first model skill includes a structured output processed by the procedural skill. In yet another example, processing a portion of the model output includes displaying a portion of the model output to a user of the computing device. In yet another example, processing a portion of the model output includes parsing a portion of the model output by an application of the computing device to affect the operation of the application.

[0115] In another aspect, the technology relates to another method. The method includes: obtaining user input from a user; generating a skill chain based on the user input, the skill chain including a set of skills for processing the user input; for a first model skill of the skill chain: generating a first prompt including at least a portion of the obtained user input based on a first prompt template associated with the first model skill; and using a first machine learning model associated with the first model skill to process the first prompt to obtain an intermediate output; for a second model skill of the skill chain: generating a second prompt including at least a portion of the intermediate output as an input for the second model skill based on a second prompt template associated with the second model skill; and using a second machine learning model associated with the second model skill to process the second prompt to obtain a model output; and providing an indication of the model output for display to the user. In an example, generating the skill chain includes: generating a skill list corresponding to a set of skills in a skill library, the skill list including a description for each skill in the set of skills; providing an indication of the user input and the skill list to a machine learning service; and receiving from the machine learning service a skill chain corresponding to the user input. In another example, generating the skill chain includes: generating an input embedding encoding the intent of the user input; determining a set of skills from a skill library, each skill in the set of skills having an associated semantic embedding that matches the generated input embedding; and generating a skill chain based on the determined set of skills. In another example, it is determined that the semantic embedding matches the input embedding based on an algorithmic similarity measure between the semantic embedding and the input embedding. In yet another example, processing the first prompt to obtain the intermediate output includes: providing a request to a machine learning service to process the first prompt using the first machine learning model; and receiving from the machine learning service a response including the intermediate output. In yet another example, the intermediate output of the first model skill includes a structured output. In another example, at least a portion of the first prompt corresponds to the structured output.

[0116] For example, aspects of the present disclosure have been described above with reference to block diagrams and / or operational illustrations of methods, systems, and computer program products according to aspects of the present disclosure. The functions / actions recited in the blocks may not occur in any order shown in any flowchart. For example, depending on the functions / actions involved, two consecutive blocks shown may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0117] The description and illustration of one or more aspects provided in this application are not intended to limit or define the scope of the present disclosure in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the claimed aspects of the present disclosure. The claimed disclosure should not be construed as limited to any aspect, example, or detail provided in this application. Various features (both structures and methods), whether shown and described in combination or separately, are intended to be selectively included or omitted to produce embodiments having a particular set of features. Having provided the description and illustration of this application, those skilled in the art can envision variations, modifications, and alternative aspects that fall within the spirit of the broader aspects of the general inventive concept embodied in this application, and these variations, modifications, and alternative aspects do not depart from the broader scope of the claimed disclosure.

Claims

1. A system, comprising: At least one processor; And A memory storing instructions which, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations including: Obtaining user input from a user; Generating a skill chain based on the user input, the skill chain including a set of skills for processing the user input; For a first model skill of the skill chain: Generating a first prompt including at least a portion of the obtained user input based on a first prompt template associated with the first model skill; and Processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; For a second model skill of the skill chain: Generating a second prompt including at least a portion of the intermediate output as an input for the second model skill based on a second prompt template associated with the second model skill; and Processing the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and Providing an indication of the model output for display to the user.

2. The system according to claim 1, wherein generating the skill chain includes: Generating a skill list corresponding to a set of skills in a skill library, wherein the skill list includes a description for each skill in the set of skills; Providing an indication of the user input and the skill list to a machine learning service; And Receiving the skill chain corresponding to the user input from the machine learning service.

3. The system according to claim 1, wherein generating the skill chain includes: Generating an input embedding encoding the intent of the user input for the user input; Determining a set of skills from a skill library, each skill in the set of skills having an associated semantic embedding that matches the generated input embedding; And Generating the skill chain based on the determined set of skills.

4. The system according to claim 3, wherein it is determined that the semantic embedding matches the input embedding based on an algorithmic similarity metric between the semantic embedding and the input embedding.

5. The system according to claim 1, wherein processing the first prompt to obtain the intermediate output includes: Providing a request to a machine learning service to process the first prompt using the first machine learning model; And Receiving a response from the machine learning service including the intermediate output.

6. The system according to claim 1, wherein the intermediate output of the first model skill includes a structured output.

7. The system according to claim 6, wherein at least a portion of the first prompt corresponds to the structured output.

8. A method, comprising: Obtaining a skill chain corresponding to an input at a computing device; For a first model skill of the skill chain: Generating a first prompt including at least a portion of the user input based on a first prompt template associated with the first model skill; And Processing the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; For a second model skill of the skill chain: Generate a second prompt including at least a portion of the intermediate output as input for the second model skill, based on a second prompt template associated with the second model skill; and Process the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and Process at least a portion of the model output by the computing device to affect the operation of the computing device.

9. The method according to claim 8, wherein the first machine learning model is the second machine learning model.

10. The method according to claim 8, wherein: The skill chain further includes a procedural skill executed by the computing device; and The output of the procedural skill is processed as input for the second model skill.

11. The method according to claim 10, wherein the intermediate output of the first model skill includes a structured output processed by the procedural skill.

12. The method according to claim 8, wherein processing the portion of the model output includes displaying the portion of the model output to a user of the computing device.

13. The method according to claim 8, wherein processing the portion of the model output includes parsing the portion of the model output by an application of the computing device to affect the operation of the application.

14. A method, comprising: Obtain user input from a user; Generate a skill chain based on the user input, the skill chain including a set of skills for processing the user input; For a first model skill of the skill chain: Generate a first prompt including at least a portion of the obtained user input, based on a first prompt template associated with the first model skill; and Process the first prompt using a first machine learning model associated with the first model skill to obtain an intermediate output; For a second model skill of the skill chain: Generate a second prompt including at least a portion of the intermediate output as input for the second model skill, based on a second prompt template associated with the second model skill; and Process the second prompt using a second machine learning model associated with the second model skill to obtain a model output; and Provide an indication of the model output for display to the user.

15. The method according to claim 14, wherein generating the skill chain includes: Generate a skill list corresponding to a set of skills in a skill library, wherein the skill list includes a description for each skill in the set of skills; Provide an indication of the user input and the skill list to a machine learning service; and Receive the skill chain corresponding to the user input from the machine learning service.

Images