Embedding and / or modification of the action model used in the automatic generation of action sets in response to natural language requests.

Abstract

In contrast to inflexible approaches, the embodiments disclosed herein utilize a flexible approach when automatically determining an action set to utilize when attempting to perform a task requested by a user's natural language input. **Solution**: This approach is flexible in that at least the embedding techniques and / or the action model utilized when generating an action set for which an action set to utilize is determined are at least selectively changed. Stated another way, the embodiments utilize a framework through which different embedding techniques and / or different action models can be at least selectively utilized when generating different candidate action sets for a given NL input of the user. Further, one of those action sets can be selected for actual use when attempting to perform the real-world execution of a given task reflected by a given NL input. The selection can be based on a suitability metric and / or other considerations for the selected action set.

Description

Technical Field

[0001] Various model-based (e.g., machine learning model-based) techniques have been proposed to automatically generate actions that can be implemented when attempting to execute a task. However, such techniques may be inflexible in that the same model is always used when generating actions, and / or in that inputs processed using the model when generating actions are always generated in the same manner. As a result, such techniques may lack robustness in many situations, resulting in actions that fail to successfully execute the task in those situations. Additionally and / or alternatively, such techniques may result in performing the generated actions without first considering whether the action is suitable for executing the task and / or whether it is more suitable than alternative actions. This can lead to performance of actions that fail to successfully execute the task in many situations.

Summary of the Invention

[0002] Embodiments described herein relate to methods and apparatuses for robust automatic generation of an action set for use in executing a task in response to free-form natural language (NL) input (e.g., spoken utterance) provided by a user and requesting execution of the task. The generated action set may be provided for use in executing the task. For example, providing the generated action set may cause the action set to be automatically performed, thereby triggering automatic execution of the task.

[0003] Action sets can be generated for various tasks across various domains by utilizing the techniques disclosed herein. Some non-exclusive examples of tasks include automatically controlling a computer application, automatically monitoring a video feed for the occurrence of specific conditions and automatically performing actions in response, automatically translating source code in a first programming language to source code in a second programming language, automatically generating an application programming interface (API), and / or automatically monitoring changes to an inventory database and automatically performing actions in response, and / or automatically integrating with a new exchange subsystem (for example, automatically integrating with a new web application framework (e.g., a TypeScript-based framework) when switching from an older web application framework (e.g., a JavaScript-based framework)).

[0004] In contrast to inflexible methods, the embodiments disclosed herein employ a flexible method for automatically determining the set of actions to use when attempting to perform a task requested by a user's NL input. This method is flexible in that the embedding technique and / or action model used to generate the action set from which the action set to be used is determined is at least selectively modified. In other words, the embodiments leverage a framework through which different embedding techniques and / or different action models can be at least selectively used to generate different candidate action sets for a given user NL input. Furthermore, one of these action sets may be selected for actual use when attempting to perform a given task reflected by a given NL input in the real world. The selection may be based on suitability metrics and / or other considerations for the selected action set. For example, the selection may be based on an absolute threshold or a suitability metric that satisfies a threshold for the suitability metrics for the alternative action sets. Such a flexible method is more robust and enables the determination of a corresponding action set that is normal for a wide variety of corresponding NL inputs and / or corresponding states of a wide variety of domains.

[0005] In particular, for a given NL input and a given state of a domain, which embedding technique and / or action model is used to generate the selected action will vary depending on the different NL input and / or state of the domain. As a specific example, for a first NL request and a first state of the domain, the action set determined for use may be generated based on processing an embedding generated using the first technique with a specific action model. On the other hand, for a second NL request and / or a second state of the domain, the action set determined for use may be generated based on processing an alternative embedding generated using the second technique with a specific action model. As another specific example, for a first NL request for a first state of the domain, the action set determined for use may be generated based on processing an embedding generated using the first technique with the first action model. On the other hand, for a second NL request and / or a second state of the domain, the action set determined for use may be generated based on processing an embedding generated using the first technique with an alternative action model.

[0006] The embodiments disclosed herein additionally or alternatively simulate at least selectively embodiments of the generated action sets when determining which action sets to use when attempting to perform a task. As described herein, such simulations help ensure that the determined action sets are suitable for performing a task before real-world implementation of the determined action sets, thereby improving the accuracy of the determined action sets. Such simulations can additionally or alternatively be used to determine whether additional action sets should be generated using alternative embedding techniques and / or alternative action models, and / or to help guide the generation of additional action sets. In these and other forms, simulations can be used to determine whether and / or how to generate additional action sets for consideration, and to balance the requirements for robustness and accuracy with the requirements for efficient use of computational resources when generating and / or evaluating additional action sets.

[0007] As a non-limiting example of the embodiments disclosed herein, suppose a computer-aided design (CAD) application is running on a client device and is displaying (among other things) a “widget,” and the user provides an NL input which is a spoken utterance, “Make the widget 10% larger.” In this example, the domain for the task may be a specific CAD application, a class of applications (e.g., any CAD application, or more generally, any visual manipulation application), or another domain. Furthermore, the task in the example relates to the control of a computer application. However, as referenced above and elsewhere in this specification, the techniques disclosed herein can be used to generate and / or evaluate sets of actions for various tasks across various domains that utilize the techniques disclosed herein. In some embodiments, or for some domains, the same embedding technique and / or action model may be considered for each of multiple domains. In other embodiments, or for some other domains, the set of embedding techniques and / or action models considered for those domains may each be domain-specific.

[0008] Continuing with the example, a first request embedding can be generated using a first embedding technique that generates the request embedding based on NL input data that directly reflects the NL input. For example, the NL input data may include the text of the NL input (e.g., text generated by automatic speech recognition of spoken utterances). Using the first technique, the text can be processed using a large language model (LLM) or other machine learning (ML) model to generate an NL embedding, which is a lower-dimensional semantic representation of the NL input data. The first request embedding can be adapted to the NL embedding. The LLM model, or other machine learning model used to generate the NL embedding, may optionally be domain-specific.

[0009] A second request embedding can also be generated using a second embedding technique that generates request embeddings based on alternative NL input data generated by modifying and / or supplementing the NL input data based on domain-specific knowledge (DSK). For example, the NL input may be modified and / or supplemented based on the DSK before the LLM model processes the NL input (i.e., the modified and / or supplemented NL input will be processed). For example, terms in the NL input may be replaced or supplemented by domain-specific definitions for those terms.

[0010] To generate DSK embeddings, a third request embedding can also be generated using a third embedding technique that processes the DSKs related to the terms of the NL input separately, using an LLM model or a separate machine learning model. This DSK embedding can then be combined with, or otherwise combined with, the NL input embedding generated based on processing the NL input data, and the combined embedding can be used as a request embedding. For example, the DSK embedding and the NL input embedding can be processed together on an additional neural network to generate a combined low-dimensional embedding. In one specific case, domain-specific definitions for terms in the NL input can be processed separately to generate domain-specific knowledge embeddings (e.g., definitions for domain-specific "widgets"). In another specific example, domain-specific images for terms in the NL input can be processed separately to generate domain-specific knowledge embeddings (e.g., pictures of domain-specific "wickets").

[0011] A fourth request embedding can also be generated using a fourth embedding technique that generates request embeddings based on further alternative NL input data generated by modifying and / or supplementing the NL input data based on external knowledge that is not domain-specific. For example, a general web search can be performed based on some or all of the NL input data, and text from the response results (e.g., top results) can be used to supplement or replace terms in the NL input data. For example, a general web search for "widgets" can be performed, and text snippets from the top results can be used to modify and / or supplement the NL input data. As another example, nearest neighbor search can be performed additionally or alternatively, and the results from the nearest neighbor search can be used additionally or alternatively to modify and / or supplement the NL input data. The nearest neighbor search can be performed based on the NL input data and across at least one large text corpus to identify text (e.g., text from nearest neighbors to all or part of the NL input data), and the identified text is used to modify and / or supplement the NL input data.

[0012] A first request embedding can be processed using an action model to generate a first candidate action set. A second request embedding can be processed using an action model to generate a second candidate action set. A third request embedding can be processed using an action model to generate a third candidate action set. A fourth request embedding can be processed using an action model to generate a fourth candidate action set. Each of the first, second, third, and fourth action sets contains actions that can potentially be used alone or in combination with other actions to control the application according to the request of the NL input. The first, second, third, and fourth action sets may contain different actions and / or different sequences of actions, for example, based on the fact that each is generated using a different request embedding.

[0013] Each action in the first, second, third, and fourth action sets can be performed through interaction with the application, and when performed, can result in a generated output (e.g., usable by another action) and / or give some control over the application. Control over the application can be done through emulated input (e.g., emulated touch input) and / or through the application's API. Each action may include corresponding program code, such as code in JavaScript, Python, C++, or another programming language, and / or may include API calls.

[0014] Some actions can be atomic or granular. An example of such an atomic action is "class <x>"Select an object of ", where "X" is a variable that can be populated based on the terminology of the NL input. Another example of an atomic action is: <y>"Identify the target location using properties / multiple properties," where Y is a variable that can be populated based on the terminology of the NL input. Yet another example of an atomic action is "Drag the selected object to <target location>," where <target location> is "the target location <y>A variable that can be populated based on the output of an atomic action called "Identify by Property / Multiple Properties". Further examples of atomic actions include "Click a button to rotate 90 degrees to the right", "Select All", and "Delete", each of which does not involve any variables. Such actions that do not involve any variables are also referred to herein as state-independent actions. That is, state-independent actions will be executed the same way regardless of the corresponding state of the domain. For example, "Click a button to rotate 90 degrees to the right" will result in the button being "clicked", regardless of what else is being rendered in the CAD application. In contrast, some actions that involve state-dependent variables are also referred to herein as state-dependent actions. In other words, they can be state-dependent in that they are performed depending on the current state and result in different actions for different states. For example, "Class <x>In the example "Select an object of the class", the object that is selected is determined by the class <x>This will depend on the state of the CAD application, for example, "class <x>If the state is "red" and there is a red circle at the top of the active screen of the CAD application, then performing a state-dependent action will select the red circle. In contrast, if there is a red square at the bottom of the active screen instead, then performing that state-dependent action will select the red square. As another example, "class <x>In the case of a state-dependent action such as "count the objects of the class", the result of performing the state-dependent action is that the state is "class <x>This will depend on how many of the "objects" exist. As described herein, atomic actions can be configured together in an action set, and the application can be controlled based on the action set. Some actions may optionally include a coarser preconfigured set of multiple atomic actions, such as "select all and click the button to rotate 90 degrees to the right." An action set may also optionally consist of one or more "coarser" actions along with atomic actions.

[0015] In some embodiments, additional and / or alternative action sets may be generated additionally or alternatively using alternative action models. For example, a first request embedding may be processed using an alternative action model to generate a fifth action set. As another example, a second request embedding may be processed using an alternative action model to generate a sixth action set. The alternative action models may be of different types and / or may be trained in different ways. For example, an action model may be a model used to generate probability distributions over coarse and / or fine-grained actions based on processing request embeddings and embeddings of the current state of the domain. Candidate action sets can then be generated by configuring the highest-probability action shown by the generated probability distribution. For example, an action model may generate a probability distribution over 100 candidate actions A1 to A100. The first configured action set can be an ordered sequence of four highest-probability actions such as {A1, A25, A77, A42}, the second candidate set can be an alternative ordered sequence of four highest-probability actions such as {A1, A77, A25, A42}, and the third candidate set can be an ordered sequence of five actions sampled from ten highest-probability actions such as {A1, A25, A82, A29, A42}. As another example, the alternative action model can be an RL policy model used by the RL agent to iteratively process a sequence of domain states and generate the next action based on that processing. The RL agent can then perform the next action, resulting in the next corresponding state in the sequence. Thus, in this other example, the action set can be a sequence of actions generated by the RL agent in iterative processing (i.e., the corresponding action generated in each iteration).

[0016] Each of the generated action sets can be evaluated to determine its suitability, and the most suitable (and optionally satisfying suitability threshold) action set is selected for actual real-world implementation. In some embodiments, the evaluation of action sets may include deleting an action set if it violates one or more action rules defined for a domain. For example, with respect to a domain, a given action rule may define that a given action is not permitted at all, or is not permitted if it occurs before or after another action. If an action set contains a given action, or contains a given action before or after another action, that action set violates the given action rule and may be deleted.

[0017] In some embodiments, the evaluation of an action set may additionally or alternatively include determining the suitability of the action set by performing a simulation based on that action set. The simulation of an action set may be performed in a simulated environment that reflects the corresponding state of the domain. For example, the initial simulated environment may reflect the current state of the CAD application. Some action sets may be generated prior to the simulation and implemented during the corresponding simulation. Other action sets may be generated during the corresponding simulation. For example, an action set generated using an RL agent and an RL policy model may be generated during the corresponding simulation by processing simulated state data from the simulation. As described herein, some or all actions in an action set may be state-dependent, meaning that their implementation depends on the initial simulated environment and modifications to that simulated environment through any implementation of the action set or preceding actions.

[0018] In some implementations, determining the suitability of an action set based on its simulation may include determining whether the simulation violated one or more state rules defined for the domain. For example, a state rule for a domain might define that a certain state should never be encountered within that domain. If the simulation data from the simulation indicates that a certain state was encountered, the action set may be deemed unsuitable, or this may negatively impact the suitability metric for the action set.

[0019] In some embodiments, determining the suitability of an action set based on a simulation of that action set may additionally or alternatively include rendering simulation data from the simulation to the user who provided the NL request, and determining suitability based on user feedback in response to the rendering. For example, a screenshot of the simulated environment in its final state from the simulation, or other data reflecting the final state from the simulation, may be presented to the user, and the user may provide a user interface input reflecting whether the screenshot reflects the successful execution of the task. Cases of negative feedback may be used to remove the corresponding action set or to negatively impact the suitability metric for the corresponding action set. In contrast, cases of positive feedback may be used to select the corresponding action set as the most suitable or to positively impact the suitability metric for the corresponding action set.

[0020] In some embodiments, determining the suitability of an action set based on a simulation of that action set may, additionally or alternatively, be based on how closely the simulation fits the requirements of the NL input. For example, the final state of a simulation can be processed to generate an NL description of that final state, and this NL description (e.g., its embeddings) is compared to the NL input when generating a suitability metric. For example, a "closer" embedding may correspond to a better suitability metric, which indicates higher suitability. For example, a screenshot of the simulated final state can be processed using an image captioning model to generate an NL description. Another example is processing a video from a simulation (e.g., a series of screenshots) to generate an NL description of the simulation, and this NL description is compared to the NL input when generating a suitability metric. For example, each screenshot may be processed individually using a first machine learning model to generate a corresponding embedding, and the embeddings may be processed in a time sequence using a second machine learning model (e.g., a transformer) to generate an NL description of the simulation and / or an overall embedding that semantically describes the simulation (e.g., the embedding corresponding to the earliest screenshot processed first). As another example, non-visual (i.e., not image or video) simulation data from a simulation can be processed to generate a non-linear description of the simulation, such as audio data from the simulation (e.g., generated by the simulated application during the simulation), text data from the simulation (e.g., generated by the simulated application during the simulation), and / or other data from the simulation. More generally, one or more instances of simulation data, each reflecting a corresponding state of the simulated application in a corresponding time instance, can be processed to generate one or more corresponding embeddings and / or other representations.Furthermore, the generated embeddings and / or other representations can be used to generate suitability metrics (for example, by comparison with NL input embeddings and / or request embeddings).

[0021] In some embodiments of the work examples, the first, second, third, fourth, and / or additional action sets are generated in parallel. This can reduce waiting times when resolving the action sets to be used for real-world implementation. In other embodiments, only a subset of the action sets (e.g., only the first action set) is generated initially, and additional action sets are generated and evaluated only if the subset of action sets is determined to be unsuitable. For example, only the first action set may be generated and evaluated initially, and the second action set is generated and / or evaluated only if the evaluation indicates unsuitability. Furthermore, the third action set is generated and / or evaluated only if the evaluation of the second action set indicates unsuitability. This can reduce the amount of processing being performed in a given time and conserve computational resources by at least selectively eliminating the need to generate and / or evaluate subsequent action sets (e.g., when a previously generated action set is determined to be suitable).

[0022] In various embodiments, determining whether (or to what extent) heterogeneous action sets are generated in parallel may depend on one or more factors. For example, the domain of the request or the request itself may indicate whether the request is urgent (and therefore some degree of parallelism is utilized) (and in that case, whether parallelism is utilized or not, or less). As another example, determining whether heterogeneous action sets are generated in parallel may, additionally or alternatively, be based on the current load on the server performing the action set generation and / or simulation.

[0023] The above is provided as a non-limiting overview of some embodiments disclosed herein. These and other embodiments are described in further detail herein.

[0024] Furthermore, some embodiments comprise one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and the instructions are configured to cause any of the aforementioned methods to be performed. Some embodiments comprise at least one transitory or non-transitory computer-readable storage medium storing computer instructions executable by one or more processors to perform any of the aforementioned methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] [Figure 1] It is a diagram of an exemplary environment in which the embodiments disclosed in the present specification may be implemented. [Figure 2] The diagram schematically illustrates an example of how the components in FIG. 1 may interact when automatically determining an action set to be used when attempting to execute a task requested by a user's NL input. [Figure 3] It is a flow diagram illustrating an exemplary method for implementing selected aspects of the present disclosure in accordance with the embodiments disclosed in the present specification. [Figure 4] It is a flow diagram illustrating another exemplary method for implementing selected aspects of the present disclosure in accordance with the embodiments disclosed in the present specification. [Figure 5] It illustrates an exemplary architecture of a computing device. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] Figure 1 schematically illustrates an exemplary environment in which selected aspects of the present disclosure may be implemented in accordance with various embodiments. Any computing device illustrated in FIG. 1 or elsewhere in the figures may include logic such as one or more microprocessors that execute computer-readable instructions stored in memory (e.g., a central processing unit or "CPU", a graphical processing unit or "GPU", a tensor processing unit or "TPU"), or other types of logic such as an application-specific integrated circuit ("ASIC"), a field-programmable gate array ("FPGA"), etc. Some of the systems illustrated in FIG. 1, such as the neural arena system 120, may be implemented in whole or in part using one or more server computing devices that form what may be referred to as "cloud infrastructure", but this is not required.

[0027] The neural arena system 120 may be operatively coupled via one or more computer networks 114 to one or more client computing devices such as client computing device 110 (also referred to herein as clients). The neural arena system 120 may automatically determine, based on natural language input provided by a user of client 110, the set of actions to use when attempting to perform the requested task in the natural language input. For example, a user may provide free-form spoken natural language input reflecting a task to be performed in a domain via a microphone of client 110, and the neural arena system 120 may automatically determine, based on the natural language input, the set of actions to use when attempting to execute the task in the real world. Furthermore, the neural arena system 120 may be operatively coupled to an execution system 140 via network 114.

[0028] The implementation system 140 can execute a set of actions determined by the neural arena system 120 in the corresponding domain when attempting to perform a corresponding task in the real world. For example, if the task automatically monitors a video feed for the occurrence of a certain condition, the implementation system 140 can process the video feed using the set of actions when monitoring for the occurrence of a certain condition, and optionally use the determined set of actions to generate an alert in response to the detection of any of the conditions. In such an example, the implementation system 140 can be implemented, for example, on a server that receives the video feed as a stream over the internet, and / or on a computing device that is on a local network having a camera that provides the video feed and receives the video feed as a stream on the local network. In another example, if the task automatically converts source code in a first programming language to source code in a second programming language, the implementation system 140 can be implemented on a server and / or client device that can receive the source code in the first programming language and automatically convert it to source code in the second programming language using the determined set of actions.

[0029] The client device 110 may include one or more applications, such as application 112, that interact with the neural arena system 120. For example, application 112 may be able to provide user NL input (e.g., through spoken and / or typed inputs), and application 112 may be able to send NL input to the neural arena system 120. Also, application 112 may be able to render outputs generated by the neural arena system 120 to the user (e.g., for confirmation of the action set before automated real-world implementation), such as outputs requesting user feedback (e.g., outputs reflecting the final state of the simulation of candidate action sets) and / or outputs reflecting the action set determined by the neural arena system 120. In some embodiments, where the task involves controlling a computer application running on the client 110, application 112 (or another application) may be able to be controlled using the action set determined by the neural arena system 120. In these embodiments, the execution system 140 may interface with the application when executing the action set and / or be integrated (whole or partially) as part of the application.

[0030] In Figure 1, the neural arena system 120 and the implementation system 140 are shown as separate entities connected to the client 110 via the network 114. However, in various embodiments, one or more modes of the neural arena system 120 and the implementation system 140 can be combined and / or implemented locally on the client 110. For example, one or more engines of the neural arena system 120 can be implemented on the client 110, and / or one or more modes of the implementation system 140 can be implemented on the client 110.

[0031] In various embodiments, the neural arena system 120 includes an implantation engine 122, an action engine 124, a selection engine 126, a simulation (SIM) engine 128, and / or an evaluation engine 130.

[0032] The embedding engine 122 can interface with one or more embedding ML models 152 when generating the embeddings described herein. Which embedding ML models 152 the embedding engine 122 interfaces with, and / or the data that the embedding engine 122 processes when interfaceing with one or more of the embedding ML models 152, may depend on the embedding technique used by the embedding engine 122. For example, for a given NL input, the embedding engine 122 may, in one instance, generate a first embedding based on a first embedding technique that processes NL input data reflecting the NL input using a domain-specific LLM model of the embedding model 152. Furthermore, for a given NL input, the embedding engine 122 may, in another instance, generate a second embedding based on a second embedding technique that includes processing alternative NL input data using a domain-specific LLM model. The embedding engine 122 may generate alternative NL input data by modifying and / or supplementing the NL input data using a DSK. Furthermore, for a given NL input, the embedding engine 122 can generate a third embedding based on a third embedding technique, which in another example includes processing the image using the image embedding model of the embedding ML model 152. The image can be, for example, a DSK image associated with the NL input data, or an external image associated with the NL input data.

[0033] The embedding technique used by the embedding engine 122 in a given case may depend on various factors and, in some embodiments, may be indicated by the selection engine 126. For example, the embedding technique used may depend on the domain of the task, the NL input from which the embedding is being generated, and / or the action model used by the action engine 124 when generating the candidate action set. Also, for example, the embedding technique used in a given case of NL input may, additionally or alternatively, be based on the embedding techniques used in previous cases when generating the candidate action set for the NL input and / or the evaluation of those candidate action sets. Various embedding ML models 152 may be provided. For example, embedding ML models 152 may include those specific to a particular domain, those specific to a particular set of domains, and / or those that are domain-independent. As another example, embedding ML models 152 may, additionally or alternatively, include those specific to a first type of data (e.g., natural language data), those specific to a second type of data (e.g., images), and / or those specific to a third type of data (e.g., audio data).

[0034] The action engine 124 can interface with multiple action models 154 when generating candidate action sets as described herein. In some embodiments, the action engine 124 also interfaces with one or more action rules 164 that can be used to remove some of the generated candidate action sets from further consideration (e.g., from further consideration by the evaluation engine 130). The action rules 164 may be domain-specific and / or specific to the corresponding request entity, such as the user providing the corresponding NL request or the organization associated with the user providing the corresponding NL request. For example, for a particular domain and a particular organization, a given action rule may define that a given action is not permitted at all, or is not permitted if it occurs before or after some other action.

[0035] Which action model 154 the action engine 124 interfaces with in a given case may depend on various factors and, in some embodiments, may be indicated by the selection engine 126. For example, the action model 154 used may depend on the domain of the task, the NL input from which the embeddings are being generated, and / or the embeddings generated by the embedding engine 122. Also, for example, the action model 154 used in a given case of NL input may, additionally or alternatively, be based on the evaluation of action models and / or those candidate action sets used in previous situations when generating candidate action sets for the NL input. A variety of action models 154 may be provided. For example, action model 152 may include machine learning models and / or heuristic models. As another example, action model 152 may include those specific to a particular domain, those specific to a particular set of domains, and / or those that are domain-independent. As another example, action model 152 may include those used to generate action sets by iteratively generating the corresponding next action in the action set based on an RL policy and applying updated state data in each iteration, those used to generate one or more candidate action sets in a single iteration, those used to generate a value function and a measure that reflects the values ​​of the action set and the current state pair, and / or other action models. For example, an action model may include one or more of the following: an RL policy ML model, an action sequence ML model, a constraint satisfaction model, a SAT solver, and / or other models.

[0036] In some embodiments, the selection engine 126 can interact with the embedding engine 122 to instruct which embedding technique is used by the embedding engine 122 in a given case, and / or can interact with the action engine 124 to instruct which action model is used by the action engine 124 in a given case. For example, the selection engine 126 can instruct the embedding engine 122 to use a first embedding technique first. Then, only if the evaluation engine 130 indicates that the corresponding candidate action set generated based on the first embedding technique is inappropriate, the selection engine 126 can instruct the embedding engine 122 to use a second embedding technique when generating an additional candidate action set. As another example, the selection engine 126 can instruct the embedding engine 122 to use both the first and second embedding techniques first. Next, only if the evaluation engine 130 indicates that the corresponding candidate action sets generated based on the first and second embedding techniques are inappropriate, the selection engine 126 may instruct the embedding engine 122 to utilize a third and / or fourth embedding technique when generating additional candidate action sets.

[0037] In some embodiments, the selection engine 126 may selectively use one or more selection models 156 when determining which embedding technique and / or action should be used in a given case. For example, the selection model 156 may include a selection ML model that can be used to process the domain of the task and / or NL input data requesting the task (e.g., embeddings of the NL input data) and generate an output showing the corresponding probability for each of a plurality of embedding techniques and / or action models. The selection engine 126 may use the generated output when selecting which embedding technique and / or action model to use. For example, the selection engine 126 may use the output to select the highest-probability embedding technique and / or highest-probability action model to use first. Such a selection ML model may be trained on supervised training examples based on past NL inputs and action sets that were determined to be suitable (and optionally, confirmed to be suitable after their real-world implementation). For example, suppose a given action set was generated for an NL input using a first embedding technique and a first action model and was determined to be suitable. In response, a training example may be generated that includes, as training example input, the task domain and / or NL input data of the NL input, and as training example output, positive values ​​for the first embedding technique and the first action model, and negative values ​​for all other embedding techniques and action models.

[0038] Using the SIM engine 128, for each action set generated by the action engine 124, the execution of the action set can be simulated in a simulated environment, such as a simulated environment that reflects the current state of the domain. Furthermore, the SIM engine 128 generates simulation data for each simulation. In some situations, action sets can be generated independently of their simulation, and the SIM engine 128 is used to simulate the action set after it has been generated. In some other situations, action sets can be generated during the simulation via the SIM engine 128. For example, some RL policy models can be used in a simulation to generate candidate action sets, which are executed during the simulation, and their generation depends on the simulated state encountered during the simulation.

[0039] As one specific example, if the task is to control a computer application, the SIM engine 128 simulates the performance of controlling the application 112 by executing a set of candidate actions. In such an example, the SIM engine 128 can use an emulator when running the simulation and can optionally start each simulation from the current state of the application. As another specific example, if the task is to translate code from a first programming language to a second programming language, the SIM engine 128 simulates the execution of the translation by executing a set of candidate actions and optionally further simulates the execution of the resulting translated second programming language code. As yet another specific example, if the task is to automatically monitor changes to an inventory database and automatically take action in response, the SIM engine 128 simulates monitoring the simulated database by simulating a dynamically changing inventory database and executing a set of candidate actions. As a further specific example, if a task automatically monitors a video feed for the occurrence of certain conditions and automatically takes action in response, the SIM engine 128 simulates the video feed and / or plays back past real-world video feeds by executing a set of candidate actions, thereby simulating the monitoring of the video feed.

[0040] The evaluation engine 130 can determine whether a candidate action set is suitable for performing the corresponding task, and / or determine which of several candidate action sets is the most suitable. In various embodiments, when evaluating an action set, the evaluation engine 130 utilizes simulation data from the simulation of the action set by the SIM engine 128. In some of these embodiments, the evaluation engine 130 can compare the simulation data with one or more state rules 160, which can be used to determine that a candidate action set is unsuitable, and / or to negatively affect the suitability score for the candidate action set, which is used when determining the suitability of the candidate action set. The state rules 160 may be domain-specific and / or specific to the corresponding request entity, such as a user providing the corresponding NL request or an organization associated with a user providing the corresponding NL request. For example, for a particular domain and a particular organization, a given state rule may define that a given state should never be encountered, or that a particular sequence of states should never be encountered. If simulation data from a simulation of a candidate action set indicates that a given state and / or a specific sequence of states was encountered, the evaluation engine 130 may determine that the candidate action set is unsuitable. As another example, for a particular domain and a particular organization, a given state rule may define that a given state or a specific sequence of states is undesirable but not prohibited. If simulation data from a simulation of a candidate action set indicates that a given state and / or a specific sequence of states was encountered, the evaluation engine 130 may negatively impact the suitability metric for the candidate action set.

[0041] In some additional or alternative embodiments where simulation data is used when evaluating action sets, the evaluation engine 130 may additionally or alternatively solicit and utilize user feedback based on the simulation data and / or analyze how closely the simulation data fits the requirements of the corresponding NL input. For example, the evaluation engine 130 may have the user who provided the NL request render the simulation data from the simulation and, in response to the rendering, determine suitability based on the user's feedback. For example, the evaluation engine 130 may have the client 110 render a screenshot of the simulated environment in its final state from the simulation, and in response, the user may provide a user interface input that reflects whether the screenshot reflects the successful execution of the task. The evaluation engine 130 may use instances of negative feedback to remove the corresponding action set or negatively impact the suitability metric of the corresponding action set. Conversely, the evaluation engine 130 may use instances of positive feedback to select the corresponding action set as the most suitable or positively impact the suitability metric of the corresponding action set.

[0042] In some additional or alternative embodiments that utilize simulation data when evaluating action sets, the evaluation engine 130 may additionally or alternatively determine the suitability of an action set based on how closely the simulation data from the simulation of that action set fits the requirements of the NL input. For example, the evaluation engine 130 may process the final state of the simulation to generate an NL description of the final state and compare that NL description with the NL input data when generating a suitability metric. For example, an embedding of the NL description of the final state may be compared with an embedding based on NL description data (e.g., based solely on NL description data, or based on NL description data supplemented or modified as described herein). A comparison showing a greater degree of similarity (e.g., a “closer” embedding) may correspond to a better suitability metric, which indicates better suitability (i.e., a better suitability score). For example, a screenshot of the simulated final state may be processed using an image captioning model to generate an NL description.

[0043] The machine learning models described herein may be of various architectures and may be trained in various ways. For example, one or more of the models may be graph-based neural networks (e.g., graph neural networks (GNNs), graph attention neural networks (GANNs), or graph convolutional neural networks (GCNs)), transformers, encoder-decoders, or recurrent neural networks ("RNNs," e.g., long short-term memory, or "LSTM," or gate recurrent unit, or "GRU"), or BERT (Bidirectional Encoder Representations from Transformers). Furthermore, reinforcement learning, supervised learning, and / or imitation learning may be used, for example, when training one or more of the machine learning models. Additional descriptions of several embodiments of various machine learning models are provided herein.

[0044] Referring to Figure 2, examples of engines 122, 124, 126, 128, and 130 of the neural arena system 120 are provided. These include possible interactions between these engines and models 152, 154, and 156, as well as rules 160 and 164, which may be utilized by the neural arena system 120.

[0045] In Figure 2, the embedding engine 122 processes at least the NL input 101 when generating the request embedding 123. The NL input 101 is provided by the user through interaction with the user interface input device of the client device (e.g., client 110 in Figure 1), and the NL input 101 includes a request that generates an action for a task. For example, the NL input 101 could be spoken input from the user detected via the microphone of the client 110, and the embedding engine 122 could process the recognized text generated based on the spoken input (e.g., using automatic speech recognition, ASR) when generating the request embedding 123. Alternatively, the NL input 101 could be typed input provided via the virtual keyboard or hardware keyboard of the client 110, and the typed text could be processed by the embedding engine 122 when generating the request embedding 123. For example, the recognized text or typed text could be processed using the NL ML model 152A of the request ML model 152 to generate the NL embedding. The NL ML model 152A could be, for example, an LLM. Request embedding 123 can be an NL embedding, or it can be a function of an NL embedding and other NL embeddings.

[0046] In some embodiments, the request engine 122 additionally utilizes domain-specific knowledge (DSK) 102, context data 103, and / or external knowledge 104 when generating the request embedding 123. In those embodiments, whether the DSK 102, context data 103, and / or external knowledge 104 are utilized depends on the embedding technique used by the embedding engine 122. Furthermore, in some of those embodiments, the embedding technique used by the embedding engine 122 in a given case can be indicated by the selection engine 126. For example, for the first request embedding 123 for a given NL input 101, the selection engine 126 can be instructed to generate the request embedding 123 using a first embedding technique in which only the NL input 101 is utilized when generating the request embedding 123. Furthermore, for the next request embedding 123 for the same given NL input 101, the selection engine 126 can be instructed to generate the request embedding 123 using a second embedding technique that utilizes both the NL input 101 and the DSK 102. For example, the request engine 122 can use the DSK 102 to modify the NL input 101 and use the NL ML model 152 to process the modification of the NL input 101 and generate the request embedding 123. In a specific case, the NL input 101 might be "make the wicket 15% smaller," and the request engine 122 could use the DSK 102 to find a domain-specific definition of "wicket" as "a small door next to a larger door," and modify the NL input 101 to "make the small door 15% smaller" or "make the wicket, which is a small door next to a larger door, 15% smaller." Continuing with the specific case, the request embedding could be an NL embedding generated based on the modification of the NL input 101. The selection engine 126 can use the selection ML model 156 and / or consider which embedding techniques have already been used for a given NL input 101 when instructing which embedding technique to use in a given case.

[0047] Another example of an embedding technique that can be used by the embedding engine 122 is that the embedding engine 122 can identify specific domain-specific knowledge related to the terminology of the NL input 101 from the DSK 102. Furthermore, the embedding engine 122 can process the specific domain-specific knowledge using one or more of the request ML models 152 to generate a DSK embedding. The embedding engine 122 can then concatenate or otherwise combine the DSK embedding with an NL embedding and / or a context embedding (e.g., processed with or via an additional neural network to generate a lower-dimensional embedding), and the combined embedding is used as the request embedding 123. For example, the NL input 101 can be "make wicket 15% smaller", and the embedding engine 122 can generate an NL embedding based on the NL input 101 (unmodified). Furthermore, the embedding engine 122 can identify the domain-specific NL definition of "wicket" and / or the domain-specific image of "wicket" from the DSK 102. The embedding engine 122 can process the domain-specific NL definition of "Wicket" using the NL ML model 152A to generate a domain-specific NL definition embedding, and / or process the domain-specific "Wicket" image using the image ML model 152B to generate a domain-specific image embedding. The embedding engine 122 can then generate a requested embedding 123 in accordance with the NL embedding, the domain-specific NL definition embedding, and / or the domain-specific image embedding. The image ML model 152B can be, for example, a convolutional neural network (CNN) or other neural network trained to generate semantically rich embeddings of images based on image processing.

[0048] As referenced above, the embedding engine 122 may utilize context data 103 and / or external knowledge 104 when generating a request embedding 123, either additionally or alternatively, for one or more embedding techniques. For example, context data 103 may include current state data about the task's domain, and the embedding engine 122 may process the current state data to generate a context embedding, and generate a request embedding 123 depending on the context embedding. For example, the current state data may include an NL description of the domain's current state. Additional or alternative context data 103 may be used when generating a context embedding, such as instructions for an application currently running on the client device that provided the NL input 101, recent NL inputs provided via the client device, the current time or other current time data, the current location, and / or other context data. As another example, external knowledge 104 may include data obtained through a general-purpose search engine or other general-purpose knowledge base, and data obtained using the NL input 101. For example, external knowledge 104 may include search results returned from a general-purpose search engine in response to a query formulated using the NL input 101. The embedding engine 122 can process external knowledge 104 to generate knowledge embeddings and generate request embeddings 123 in accordance with the knowledge embeddings. In some embodiments, first context data can be used in a first embedding technique, and separate second context data can be used in a second embedding technique. For example, the first context data may include instructions for the application currently running on the client device but exclude recent NL inputs, and the second context data may include recent NL inputs but exclude application instructions.As another example, the first context data may include a pixel-level abstraction of the image of the current state (e.g., the current state of the application) (e.g., the pixels themselves), but exclude instructions for higher-level abstractions of the image (e.g., shapes or other objects derived from the pixels), while the second context data may include higher-level abstractions, but exclude pixel-level abstractions.

[0049] The action engine 124 processes the request embedding 123 using one or more action ML models to generate one or more candidate action sets 125. For example, in a given case, the action engine 124 may process the request embedding using one of the first RL policy model 154A, the second RL policy model 154B, the constraint satisfaction model 154C, the action sequence model 154N, or other models of action model 154 (e.g., other models indicated by the vertical ellipsis in Figure 2). In some of these embodiments, the selection engine 126 can indicate which of the action models 154 is used by the action engine 124 in a given case. For example, for an initial instance of a given NL input 101, the selection engine 126 may be instructed to generate a candidate action set 125 using a first RL policy model 154A; for a second instance of the same given NL input 101, the selection engine 126 may be instructed to generate a candidate action set 125 using a second RL policy model 154B; and for a third instance of the same given NL input 101, the selection engine 126 may be instructed to generate a candidate action set 125 using an action sequence model 154N. Optionally, when generating a candidate action set 125, the action engine 124 may delete one or more generated action sets based on action sets that violate action rule 164, as described herein.

[0050] Using the SIM engine 128, the execution of each of the candidate action sets 125 can be simulated in a simulated environment, such as a simulated environment that reflects the current state of the domain. Furthermore, the SIM engine 128 generates simulation data 129 for each simulation. In some situations, an action set from the action sets 125 can be generated independently of its simulation, and the SIM engine 128 is used to simulate the action set after it has been generated. In some other situations, the action set can be generated by the action engine 124 during the simulation via the SIM engine 128. This is reflected by the double-arrowed dashed line between the action engine 124 and the SIM engine 128.

[0051] Simulation data 129 is provided to the evaluation engine 130. The evaluation engine 130 can use the simulation data 129 to determine whether a corresponding candidate action set among the candidate action sets 125 is suitable for performing the corresponding task, and / or to determine which of the multiple candidate action sets is the most suitable. In some of these embodiments, the evaluation engine 130 can compare the simulation data with one or more state rules 160, which can be used to determine that a candidate action set is unsuitable, and / or to negatively affect the suitability score for the candidate action set, which is used when determining the suitability of the candidate action set. In some additional or alternative embodiments that utilize the simulation data 129 when evaluating action sets, the evaluation engine 130 can additionally or alternatively use the simulation data 129 to solicit and / or analyze how closely the simulation data 129 fits the requirements of the NL input 101. In some additional or alternative embodiments that utilize simulation data 129 when evaluating action sets, the evaluation engine 130 may additionally or alternatively determine the suitability of an action set based on how closely the simulation data 129 from the simulation of that action set fits the requirements of the NL input 101.

[0052] If the evaluation engine 130 determines that none of the candidate action sets 125 are suitable, the evaluation engine can output a "unsuitable" instruction 131 to the selection engine 126. In response, the selection engine 126 can adapt the embedding technique used by the embedding engine 122 and / or the action model 154 used by the action engine 124. It can then generate further candidate action sets 125 based on different request embeddings 123 (e.g., generated using alternative embedding techniques) and / or based on different action models of the action model 154. For example, the selection engine 126 can adapt the embedding technique being used, but not the action model 154 being used. In response, the embedding engine 122 can generate different request embeddings 123 using different adapted embedding techniques, and the action engine 124 will process the different request embeddings 123 using the same action model as before. This may result in the generation of different candidate action sets 125 due to the different request embeddings 123. Different action sets 125 can be simulated by the SIM engine 128, and the resulting simulation data 129 is used by the evaluation engine 130 when evaluating different candidate action sets 125. For example, this can be repeated multiple times until the evaluation engine 130 determines that the evaluated candidate action set is suitable.

[0053] If the evaluation engine 130 determines that one of the candidate action sets 125 is suitable in a given iteration, it can provide the action set 132 to the implementation system 140. The implementation system 140 can then execute the action set 132 in a real-world environment. For example, the implementation system 140 can execute the action set 132 automatically and without first prompting the user for verification. Alternatively, the implementation system 140 may first prompt the user for verification before execution, and only execute the action set 132 if a positive user input is received in response to the prompt.

[0054] Figure 3 is a flowchart illustrating an exemplary method 300 for carrying out a selected aspect of the disclosure according to embodiments disclosed herein. For convenience, the operations in the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of the neural arena system 120. Furthermore, although the operations of method 300 are shown in a particular order, this is not limiting. One or more operations can be rearranged, omitted, or added.

[0055] In block 302, the system receives NL input data that reflects a request, which automatically generates an action for the task. For example, the request may be a spoken request, and the NL input data may be its transcription, which is generated using automatic speech recognition.

[0056] In block 304, the system selects the request embedding technique and / or action model to use. In some embodiments, the system selects the request embedding technique and / or action model to use based on the domain of the task and / or based on the NL input data in block 302. For example, the system can use a defined heuristic indicating that a first embedding technique and a first action model should be used first for NL input data containing a particular domain and / or particular term. As another example, the system can use a trained selection model (e.g., one of the selection models 156 in Figure 1) to process the NL input data (e.g., its embedding) and / or domain indication to generate an output indicating which embedding technique and / or action model should be used. Based on the output, the system can select the embedding technique and / or action model to use first.

[0057] In block 306, the system generates a request embedding based on processing the NL input data using an embedding ML model. In some embodiments, the embedding ML model is selected from several candidate embedding ML models according to the currently selected embedding technique. In some implementations or iterations, block 306 includes subblocks 306A, 306B, and / or 306C. In some of these embodiments, which of the subblocks, if any, are executed in a given iteration of block 306 may depend on the currently selected embedding technique. For example, in the first embedding technique, none of the subblocks may be executed; in the second embedding technique, only subblock 306A may be executed; in the third embedding technique, only subblock 306B may be executed; in the fourth embedding technique, only subblock 306C may be executed; in the fifth embedding technique, only subblocks 306A and 306B may be executed; and / or in the sixth embedding technique, all of subblocks 306A, 306B, and 306C may be executed.

[0058] In subblock 306A, the system generates request embeddings based on a DSK, such as a pre-stored DSK. For example, the system can generate request embeddings based on modifying an NL input using a DSK and processing the modified NL input. Alternatively, the system can process DSKs associated with an NL input separately to generate domain-specific embeddings, and then generate request embeddings in accordance with the NL embeddings from processing the NL input and in accordance with the domain-specific embeddings.

[0059] Subblock 306A optionally includes a further subblock 306A1 in which the system interacts with a user to obtain at least several DSKs. For example, the system may interact with a user to obtain at least several DSKs in response to determining that there are no pre-stored DSKs corresponding to one or more terms of an received NL input. In some embodiments, the execution of subblock 306A without the execution of subblock 306A1 can be considered a first embedding technique, and the execution of subblock 306A with the execution of subblock 306A1 can be considered a separate second embedding technique.

[0060] In subblock 306B, the system generates request embeddings based on context data, such as context data that describes the context of an NL input but is not reflected by the content of the NL input itself. For example, context data may include temporal context data (e.g., time of day, day of week, and / or day of year), previous NL input context data (e.g., NL inputs provided by the same user within the last N seconds and / or the most recent N NL inputs), and client context data (e.g., images rendered on the client at or near the time the NL input was provided on the client, and / or applications running on the client at or near the time the NL input was provided). In some embodiments, in block 306B, the system may process the context data to generate context embeddings and generate request embeddings in response to the NL embeddings from processing the NL inputs and in response to the context embeddings.

[0061] In subblock 306C, the system generates request embeddings based on external data from external knowledge sources. For example, a general web search can be performed based on some or all of the NL input data, and text from the response results (e.g., top results) can be used to supplement or replace terms in the NL input data, and the system can generate request embeddings based on processing the modified NL input. As another example, the system can process the text from the response results separately to generate external data embeddings, and generate request embeddings in accordance with the NL embeddings from processing the NL input and in accordance with the external embeddings. As yet another example, an image search can be performed based on some or all of the NL input data, and images from the response results (e.g., top image results) can be processed using an image embedding ML model to generate image embeddings, and request embeddings can be generated in accordance with the image embeddings and in accordance with the NL embeddings from processing the NL input (e.g., NL embeddings generated using a domain-specific LLM).

[0062] In block 308, the system processes the generated request embeddings using an action model from the most recent iteration of block 306 to generate one or more predicted action sets. For example, the system may process the generated request embeddings using an action model to generate a single candidate action set or multiple candidate action sets. In some embodiments, the action model used is selected from multiple candidate action models according to the currently selected action model.

[0063] In some embodiments, block 308 includes subblock 308A and / or subblock 308B.

[0064] In subblock 308A, if the system determines that a candidate action set violates one or more defined action rules, it may delete one or more of the candidate action sets.

[0065] In subblock 308B, the system may remove one or more candidate action sets based on previous iterations of subblocks 314A and / or 314B (described below) for the NL input. In general, subblocks 314A and / or 314B can also be used to remove candidate action sets, but only on candidate action sets that produce an undesirable simulated state when performed. In some situations, the actions of the action set that produced that simulated state can be identified in subblocks 314A and / or 314B. In some of those embodiments, subblock 308B may include removing candidate action sets containing those actions. For example, a candidate action set may be removed if it contains a particular action identified in subblocks 314A and / or 314B, or a particular sequence of actions identified in subblocks 314A and / or 314B.

[0066] Additional and / or alternative techniques may be used to remove action sets in block 308. For example, when an ML model is used to generate action sets and probabilities or other measures indicating the likelihood that an action set is suitable for a request embedding, the embodiment may remove action sets for which the measure does not meet a threshold.

[0067] In block 310, the system determines whether there are any candidate action sets remaining from the most recent iteration of block 308. If not, the system proceeds to block 318 (described below). If there are, the system proceeds to block 312.

[0068] In block 312, the system executes the action set in the simulation for each of the action sets generated in the most recent iteration of block 308 (that were not deleted in block 308). As described herein, for some action models, the system can execute blocks 308 and 310 simultaneously. That is, for those action models, the system sequentially generates the actions of the action set through interaction with the simulated environment in the simulation. For example, the RL agent can use the RL policy model among the action models when selecting and executing actions of the action set in the simulation. For other action models, the system can generate a completed action set in block 308 before executing that action set in the simulation in block 310.

[0069] In block 314, the system determines, based on the simulation, whether each simulated action set is appropriate. The system determines whether an action set is appropriate based on simulation data from the simulation. For example, the system can generate a suitability metric for an action set based on the simulation data and determine whether an action set is appropriate based on the suitability metric. For example, if the suitability metric for an action set meets a threshold and shows higher suitability than any other action sets considered in the iteration of block 314, the action set can be determined to be suitable. Otherwise, the action set can be determined to be unsuitable.

[0070] In some embodiments, when determining whether an action set is suitable, block 314 includes subblock 314A, subblock 314B, and / or subblock 314C.

[0071] In subblock 314A, the system compares the action set with one or more state rules. For example, a state rule for a domain may define that a certain state should never be encountered in that domain. If the simulation data from the simulation indicates that a particular state was encountered, the system may determine that the action set is unsuitable. As another example, an additional or alternative state rule may define that an additional state is undesirable for the domain. If the simulation data from the simulation indicates that a certain state was encountered, the system may negatively impact the suitability metric for the action set. The simulation data being compared may include, for example, simulated state data reflecting the final simulated state and / or one or more intermediate simulated states.

[0072] In subblock 314B, the system requests user feedback regarding the simulation data and determines whether an action set is suitable based at least in part on the user feedback received in response to the request. For example, the system may allow the user to render (e.g., visually and / or audibly) a representation of the final simulated state and / or intermediate simulated states. The representation may be from the simulation data or generated from the simulation data. Furthermore, the system may process feedback received through one or more user interface inputs in response to the rendering and determine whether an action set is suitable based on the feedback. For example, instances of negative feedback may be used to eliminate the corresponding action set or to negatively affect the suitability metric for the corresponding action set. In contrast, instances of positive feedback may be used to select the corresponding action set as the most suitable or to positively affect the suitability metric for the corresponding action set.

[0073] In subblock 314C, the system compares the simulation data with the NL input data in block 302 and determines, at least partially, whether the action set is suitable. For example, a comparison showing at least a positive threshold similarity can have a positive effect on the suitability metric, while a comparison showing less than a negative threshold similarity can have a negative effect on the suitability metric. As a specific example, the system may process the final state of the simulation to generate an NL description of the final state and compare that NL description with the NL input data when generating the suitability metric. For example, an embedding of the NL description of the final state can be compared with an embedding based on NL description data (e.g., based solely on NL description data, or based on NL description data supplemented or modified as described herein).

[0074] In block 316, the system determines whether any of the action sets in the most recent iteration of block 314 are suitable. If suitable, the system proceeds to block 320 to trigger the real-world implementation of the most suitable action set (e.g., the one with the best suitability metric). In some embodiments or situations, block 320 may include automatically triggering the real-world implementation. In some other embodiments or situations, block 320 may include first prompting the user for affirmation of the action set and only causing the real-world implementation if affirmation is received in response. In yet another embodiment or situation, block 320 may include sending data reflecting the action set to one or more computing devices for presentation to the user for embodiments and / or future embodiments by computing devices.

[0075] If, in block 316, the system determines that none of the actions in the most recent iteration of block 314 are suitable, the system proceeds to block 318. In block 318, the system selects an alternative request embedding technique and / or an alternative action model. The system then returns to block 306 and uses the alternative request embedding technique and / or alternative action model to execute another iteration of blocks 306, 308, 310, and optionally 312, 314, and 316. This can continue until a suitable set of actions is determined, or until other conditions are met (for example, a threshold number of iterations are performed and / or all variations of the embedding technique and / or action model are exhausted).

[0076] The alternative request embedding technique and / or alternative action model selected in each iteration of block 318 is unique for any of those currently used in the current execution of method 300 for the current NL input. In some embodiments, the system can utilize heuristics and / or trained selection models when selecting possible suboptimal embedding techniques and / or action models. For example, a defined heuristic may indicate which embedding techniques and action models to use first, then which, and so on. As another example, the system can use a trained selection model (e.g., one of the selection models 156 in Figure 1) to process the NL input data (e.g., its embeddings) and / or domain instructions to generate an output indicating the probabilities of embedding techniques and / or action models. Based on these probabilities, the system can select the next embedding technique and / or action model to use.

[0077] In some embodiments, the system may additionally and / or alternatively use data from block 308 to determine whether to adjust the most recent embedding technique or, instead, the most recent action model being used. For example, if the action model used in the most recent iteration shows a confidence threshold in the generated candidate action set but is nevertheless found to be inappropriate, this may indicate that the most recent request embedding is inaccurate, but the most recent action model is likely the correct one to use. In response, the system may select an alternative embedding technique. On the other hand, if the action model used in the most recent iteration shows a threshold confidence level below the generated candidate action set, this may indicate that the most recent action model is likely not the correct one to use. In response, the system may select an alternative action model.

[0078] Figure 4 is a flowchart illustrating another exemplary method 400 for carrying out a selected aspect of the disclosure according to embodiments disclosed herein. For convenience, the operations in the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of the neural arena system 120. Furthermore, although the operations of method 400 are shown in a particular order, this is not limiting. One or more operations can be rearranged, omitted, or added.

[0079] First, it should be noted that Method 400 in Figure 4 includes many aspects common to Method 300 in Figure 3. However, in Method 400, several variations of the requirements embedding technique utilize combinations of action models in parallel when generating candidate action sets.

[0080] In block 402, the system receives NL input data that reflects a request to automatically generate an action for a task.

[0081] In block 404, the system selects N variations, where N is an integer greater than 1. Each variation includes a unique request embedding technique and / or a unique action model to utilize. The two combinations of request embedding techniques and / or action models are unique in that they do not exist in any other variation for the current iteration of block 404, or for the previous iteration of block 404 for the same request in block 402. As an example, the system can generate a first variation including a first embedding technique and a first action model, a second variation including a second embedding technique and a second action model, and a third variation including a third embedding technique and a first action model.

[0082] In block 406A, the system generates a request embedding using the first variant of the variants selected in the most recent iteration of block 404.

[0083] In block 408A, the system uses the action model of the first variant to process the request embedding in block 406A and generate a predicted set of actions.

[0084] In block 410A, the system executes each action set in the simulation.

[0085] In block 412A, the system determines a suitability metric for each action set based on simulation data from the simulation.

[0086] Similarly, in block 406N, the system generates a request embedding using the Nth variant from the variants selected in the most recent iteration of block 404. Furthermore, in block 408N, the system processes the request embedding of block 406N using the action model of the Nth variant to generate a predicted set of actions. Furthermore, in block 410N, the system performs each action set in a simulation. Furthermore, in block 412N, the system determines a suitability metric for each action set based on simulation data from the simulation.

[0087] As indicated by the horizontal ellipsis in Figure 4, in embodiments where there are more than two variations generated in an iteration of block 404, for each of those variations, the system can similarly use that variation to generate a request embedding and a predicted action set, perform each of the predicted action sets in a simulation, and use the corresponding simulated data to determine the corresponding suitability metric for the predicted action set.

[0088] In block 414, the system determines whether any of the candidate action sets are suitable based on the suitability metrics generated in blocks 412A-N. If none are suitable, the system returns to block 404. If any are suitable, the system proceeds to block 416 and triggers the real-world execution of the suitable candidate action set.

[0089] Figure 5 is a block diagram of an exemplary computing device 510 that can be optionally used to perform one or more embodiments of the techniques described herein. In some embodiments, a client device 110, a neural arena system 120, and / or other components may comprise one or more components of the exemplary computing device 510.

[0090] The computing device 510 typically includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to a corresponding interface device in another computing device.

[0091] The user interface input device 522 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphic tablets, scanners, touchscreens integrated into displays, voice input devices such as voice recognition systems and microphones, and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 510 or a communication network.

[0092] The user interface output device 520 may include non-visual displays such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include flat panel devices such as a cathode ray tube (CRT), liquid crystal display (LCD), projection devices, or any other mechanism for creating visible images. The display subsystem may also provide non-visual displays via an audio output device, etc. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0093] The storage subsystem 524 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 524 may include logic that performs selected embodiments of Method 300 in Figure 3, Method 400 in Figure 4, and / or other methods described herein.

[0094] These software modules are generally executed by processor 514 alone or in combination with other processors. The memory 525 used within the storage subsystem 524 may include several memories, including main random access memory (RAM) 530 for storing instructions and data during program execution, and read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules performing the functions of a particular embodiment may be stored by the file storage subsystem 526 within the storage subsystem 524 or on other machines accessible by processor 514.

[0095] The bus subsystem 512 provides a mechanism for various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is schematically shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[0096] The computing device 510 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly changing nature of computers and networks, the description of the computing device 510 shown in Figure 5 is intended only as a specific example to illustrate several embodiments. Many other configurations of the computing device 510 are possible, having more or fewer components than the computing device shown in Figure 5.

[0097] While several embodiments have been described and illustrated herein, various other means and / or structures can be utilized to perform the function and / or to obtain the results and / or one or more advantages described herein, and each of such variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials and configurations described herein are intended to be illustrative, and actual parameters, dimensions, materials and / or configurations are intended to depend on one or more specific applications in which this teaching is used. Those skilled in the art will recognize, or be able to investigate, many equivalents to the specific embodiments described herein using only routine experimentation. Therefore, it should be understood that the embodiments described above are presented only as examples, and that within the scope of the appended claims and their equivalents, embodiments may be practiced in ways other than those specifically described and claimed. Embodiments of this disclosure cover each individual feature, system, article, material, kit and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not inconsistent with each other.

[0098] In some embodiments, a method is provided which is implemented by one or more processors, and this method includes receiving natural language input data that reflects a user request for automatically generating actions to perform a task depending on the corresponding state of a domain. The method further includes generating request embeddings based on processing the natural language input data. The method further includes performing a simulation of the task by executing a predicted set of actions generated based on processing the request embeddings using one or more trained action models in a simulated environment that reflects the corresponding state of a domain. The method further includes determining, based on the simulation, that the predicted set of actions is not suitable for performing the task. In response to determining that the predicted set of actions is not suitable for performing the task, the method further includes generating an alternative predicted set of actions for performing the task. In some embodiments, generating an alternative predicted set of actions includes utilizing an alternative request embedding when generating the alternative predicted set of actions, and / or utilizing at least one alternative trained action model when generating the alternative predicted set of actions. The method further includes determining whether an alternative predicted set of actions is suitable for performing the task, and, in response to determining that an alternative predicted set of actions is suitable for performing the task, transmitting data to cause the alternative predicted set of actions to be performed in a real-world environment in order to perform the task.

[0099] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0100] In some embodiments, the method further includes generating a predicted set of actions based on processing request embeddings using one or more trained action models. In some of those embodiments, generating the predicted set of actions is performed before running the simulation.

[0101] In some embodiments, the method further includes generating a set of predicted actions based on processing request embeddings using one or more trained action models. In some of those embodiments, generating the set of predicted actions is performed during the run of the simulation and is further based on processing simulated state data generated during the run of the simulation using one or more trained action models. The simulated state data processed may depend on the trained action models used. For example, some trained action models may be configured to process simulated state data that includes pixels from simulated images from the simulation, while other trained action models may be trained to process shapes and / or other features detected from simulated images without processing the simulated images themselves.

[0102] In some embodiments, determining whether an alternative predicted action is suitable for performing the task includes performing an additional simulation of the task by executing the alternative predicted action in a simulated environment, and determining, based on the additional simulation, whether the alternative predicted action is suitable for performing the task.

[0103] In some embodiments, determining, based on a simulation, that a predicted action is unsuitable for performing a task includes processing simulation data from a simulation using the predicted action to generate a natural language output describing the processed simulation data, generating a metric based on comparing the natural language output with natural language input data, and determining that the predicted action is unsuitable for performing the task based on whether the metric does not meet a threshold. In some versions of those embodiments, the natural language output is natural language text or natural language embeddings. In some additional and / or alternative modifications of those embodiments, the simulation data includes the final state of the simulated environment from a simulation using the predicted action.

[0104] In some embodiments, determining, based on simulation, that a predicted action is unsuitable for performing a task includes processing simulation data from a simulation using the predicted action to determine whether one or more domain or task-specific rules are violated, and determining that the predicted action is unsuitable for performing the task in response to determining that at least one of the one or more domain or task-specific rules is violated. In some versions of those embodiments, the method further includes determining a particular action among the predicted actions such that the performance of that action in the simulation resulted in a violation of a given one of the one or more domain or task-specific rules. In some versions of those embodiments, determining that an alternative predicted action is suitable for performing a task includes determining that the alternative predicted action lacks a particular action.

[0105] In some embodiments, determining, based on a simulation, that a predicted action is unsuitable for performing a task includes rendering simulation data from a simulation using the predicted action on a client device that received a user request, receiving user interface inputs provided on the client device in response to rendering the simulation data on the client device, and determining, based on the user interface inputs, that a predicted action is unsuitable for performing a task. In some versions of those embodiments, the simulation data includes the final state of the simulated environment from a simulation using the predicted action. In some of those versions, the method further includes determining a particular action among the predicted actions, based on the user interface input being directed to a particular feature of the final state, such that the execution of the particular action in the simulation resulted in a particular feature. In some of those versions, determining that an alternative predicted action is suitable for performing a task includes determining that the alternative predicted action lacks a particular action.

[0106] In some embodiments, generating an alternative set of predicted actions involves utilizing at least one alternative trained action model and not using one or more trained action models that were used to generate the predicted action set.

[0107] In some alternative request embedding embodiments, generating an alternative predicted action set involves using an alternative request embedding when generating the alternative predicted action set.

[0108] In some alternative request embedding embodiments, generating a request embedding includes generating a natural language embedding based on processing natural language input data using a language model, and generating a request embedding based on the natural language embedding; generating an alternative request embedding includes generating an alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplemental terms from a domain-specific knowledge base for the task, generating an alternative natural language embedding based on processing the alternative natural language input data using a language model, and generating an alternative request embedding based on the alternative natural language embedding. In those embodiments, one or more supplemental terms are not used when generating the request embedding.

[0109] In some alternative request embedding embodiments, generating a request embedding includes generating a natural language embedding based on processing natural language input data using a language model, and generating a request embedding based on the natural language embedding; generating an alternative request embedding includes generating an alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplemental terms from an external knowledge source that is not specific to the domain or task, generating an alternative natural language embedding based on processing the alternative natural language input data using a language model, and generating an alternative request embedding based on the alternative natural language embedding. In those embodiments, one or more supplemental terms are not used when generating the request embedding.

[0110] In some alternative request embedding embodiments, generating an alternative request embedding involves generating an alternative request embedding based on the client device receiving the user request rendering a clarification prompt, receiving user feedback provided in response to the clarification prompt and via one or more user interface inputs on the client device, and processing the user feedback. In those embodiments, the user feedback is not utilized when generating the request embedding.

[0111] In some embodiments of alternative request embedding, generating an alternative request embedding includes generating a context embedding based on processing context data, and generating an alternative request embedding based on the context embedding. In those embodiments, context data is not used when generating the request embedding.

[0112] In some alternative manifestations of request embedding, generating a request embedding involves generating a request embedding based on processing natural language input data without processing second context data and processing first context data, and generating an alternative request embedding involves generating a request embedding based on processing natural language input data without processing first context data and processing second context data. In some versions of those manifestations, the first context data represents the current state of the domain at a first level of abstraction, and the second context data represents the current state of the domain at a second level of abstraction. In some of those versions, the current state of the domain includes a rendered image, the first level of abstraction is a pixel-level abstraction, and the second level of abstraction is a shape-level abstraction.

[0113] In some embodiments, generating an alternative predicted action set involves using at least one alternative trained action model to handle the alternative request embedding when generating the alternative predicted action set.

[0114] In some embodiments, the method further includes determining, based on one or more probabilities, whether to utilize alternative request embeddings or, instead, at least one alternative trained action model, when generating an alternative predicted set of actions. In some of those embodiments, the one or more probabilities are for the predicted set of actions and are generated based on processing the request embeddings using one or more trained action models.

[0115] In some embodiments, when generating an alternative predicted action set, determining whether to use an alternative request embedding or, instead, at least one alternative trained action model, based on one or more probabilities, includes determining to use an alternative request embedding in response to one or more probabilities satisfying one or more thresholds indicating at least threshold confidence in the predicted action set. In some of those embodiments, when generating an alternative predicted action set, determining whether to use an alternative request embedding or, instead, at least one alternative trained action model, based on one or more probabilities, includes determining to use an alternative trained action model in response to one or more probabilities failing to satisfy one or more thresholds indicating at least threshold confidence in the predicted action set.

[0116] In some embodiments, the predicted action set includes a first group of actions in a first sequence, and the alternative action set includes a first group of actions in a second sequence, or a second group of actions in a third sequence.

[0117] In some embodiments, sending data to perform an alternative predicted set of actions in a real-world environment to execute a task includes sending data to perform an alternative predicted set of actions automatically in response to a user request, without requiring any further user input after the user request has been provided.

[0118] In some embodiments, a method is provided which is performed by one or more processors, and the method includes receiving a natural language input containing a request that automatically generates an action for performing a task depending on a corresponding state of a domain. The method further includes generating a first request embedding based on performing a first process, the first process being based on at least the natural language input. The method further includes processing the first request embedding using at least one trained action model to generate a first predicted action for performing a task. The method further includes performing a first simulation of the task, in a simulated environment that reflects a corresponding state of a domain, in which the first predicted action is performed. The method further includes determining a first suitability metric for the first predicted action based on the first simulation. The method further includes generating a second request embedding different from the first request embedding based on performing a second process, the second process being based on at least the natural language input. The method further includes processing the second request embedding using at least one trained action model or at least one alternative trained action model to generate a second predicted action for performing a task. The method further includes performing a second simulation of the task, in which a second predicted action is performed in a simulated environment that reflects the corresponding state of the domain. The method further includes determining a second suitability metric for the second action based on the second simulation. The method further includes determining whether to perform the second action instead of the first action, based on a comparison of the first and second suitability metrics. In response to the determination to perform the second action, the method further includes transmitting data to cause the second action to be performed in a real-world environment in order to execute the task.

[0119] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features:

[0120] In some embodiments, the method further includes determining that a first suitability metric does not meet a threshold. In some of those embodiments, following and in response to determining that a first suitability metric does not meet a threshold, one or more of the following are performed: generating a second request embedding, processing the second request embedding to generate a second predicted action, performing a second simulation, or determining a second suitability metric.

[0121] In some embodiments, before or in parallel with determining the first suitability metric, one or more of the following are performed: generating a second request embedding, processing the second request embedding to generate a second predicted action, running a second simulation, and / or determining the second suitability metric.< / x> < / x> < / x> < / x> < / x> < / y> < / y> < / x>

Claims

1. A method carried out by one or more processors, wherein the method is Receiving natural language input data that reflects user requests to automatically generate actions to perform tasks according to the corresponding state of the domain, Based on the processing of the aforementioned natural language input data, the process involves generating request embeddings. The simulation of the task is performed by executing a predicted set of actions generated based on processing the request embedding using one or more trained action models in a simulated environment that reflects the corresponding state of the domain, Based on the simulation, it is determined that the predicted set of actions is not suitable for performing the task. In response to determining that the predicted set of actions is not suitable for performing the task, To generate an alternative predicted set of actions for performing the aforementioned task, the generation of the alternative predicted set of actions is When generating the aforementioned alternative predicted action set, alternative request embeddings may be used, and / or Generating the alternative predicted action set includes utilizing at least one alternative trained action model when generating the alternative predicted action set, Determining that the aforementioned alternative predicted set of actions is suitable for performing the task, In response to determining that the aforementioned alternative predicted set of actions is suitable for performing the task, A method comprising transmitting data that causes the alternative predicted set of actions to be performed in a real-world environment in order to perform the aforementioned task.

2. The process further includes generating the predicted set of actions based on processing the request embedding using one or more trained action models, The method according to claim 1, wherein generating the predicted set of actions is performed before running the simulation.

3. The process further includes generating the predicted set of actions based on processing the request embedding using one or more trained action models, The method according to claim 1, wherein generating the predicted set of actions is performed during the execution of the simulation, and further comprising processing the simulated state data generated during the execution of the simulation using the one or more trained action models.

4. Determining that the aforementioned alternative predicted action is suitable for performing the task means Performing additional simulations of the task by carrying out the alternative predicted actions in the simulated environment, The method according to claim 1, comprising determining, based on the additional simulation, that the alternative predicted action is suitable for performing the task.

5. Based on the simulation, determining that the predicted action is not suitable for performing the task means that Processing simulation data from the simulation using the predicted actions to generate natural language output describing the processed simulation data, Based on comparing the natural language output with the natural language input data, a metric is generated. The method according to claim 1, comprising determining that the predicted action is unsuitable for performing the task based on the metric not meeting a threshold.

6. The method according to claim 5, wherein the natural language output is natural language text or natural language embedding.

7. The method according to claim 5, wherein the simulation data includes the final state of the simulated environment from the simulation using the predicted actions.

8. Based on the simulation, determining that the predicted action is not suitable for performing the task means that The simulation data from the simulation using the predicted actions is processed to determine whether one or more domain or task-specific rules are violated. In response to determining that at least one of the aforementioned domain or task-specific rules has been violated, The method according to claim 1, comprising determining that the predicted action is not suitable for performing the task.

9. The process further includes determining a specific action among the predicted actions such that the execution of the specific action in the simulation resulted in a violation of one or more domain or task-specific rules. The method according to claim 6, wherein determining that the alternative predicted action is suitable for performing the task includes determining that the alternative predicted action lacks the particular action.

10. Based on the simulation, determining that the predicted action is not suitable for performing the task means that The simulation data from the simulation using the predicted action is rendered on the client device that received the user request. In response to rendering the aforementioned simulation data on the client device, the client device receives user interface input provided by the client device, The method according to claim 1, comprising determining, based on the user interface input, that the predicted action is not suitable for performing the task.

11. The method according to claim 10, wherein the simulation data includes the final state of the simulated environment from the simulation using the predicted actions.

12. Based on the fact that the user interface input is directed toward a specific characteristic of the final state, the further includes determining a specific action among the predicted actions, the action performed in the simulation resulting in the specific characteristic, The method according to claim 11, wherein determining that the alternative predicted action is suitable for performing the task includes determining that the alternative predicted action lacks the particular action.

13. The method according to claim 1, wherein generating the alternative predicted action set includes utilizing the at least one alternative trained action model and not using the one or more trained action models used in generating the predicted action set.

14. The method according to claim 1, wherein generating the alternative predicted action set includes using the alternative request embedding when generating the alternative predicted action set.

15. Generating the aforementioned request embedding is, This involves generating natural language embeddings based on processing the natural language input data using a language model, This includes generating the request embedding based on the natural language embedding, Generating the aforementioned alternative request embedding is Generating alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplementary terms from a domain-specific knowledge base for the task, Based on processing the alternative natural language input data using a language model, the process generates alternative natural language embeddings. This includes generating the alternative request embedding based on the alternative natural language embedding, The method of claim 14, wherein the one or more supplementary terms are not used when generating the request embedding.

16. Generating the aforementioned request embedding means This involves generating natural language embeddings based on processing the natural language input data using a language model, This includes generating the request embedding based on the natural language embedding, Generating the aforementioned alternative request embedding is Generating alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplementary terms from external knowledge sources that are not specific to the domain or the task, Based on processing the alternative natural language input data using a language model, the process generates alternative natural language embeddings. This includes generating the alternative request embedding based on the alternative natural language embedding, The method of claim 14, wherein the one or more supplementary terms are not used when generating the request embedding.

17. Generating the aforementioned alternative request embedding is The client device that received the aforementioned user request renders a clarification prompt, Receiving user feedback provided in response to the clarification prompt and via one or more user interface inputs on the client device, This includes, based on processing the user feedback, generating the alternative request embedding, The method according to claim 14, wherein the user feedback is not used when generating the request embedding.

18. Generating the aforementioned alternative request embedding is Based on processing context data, generating context embeddings, This includes generating the alternative request embedding based on the aforementioned context embedding, The method according to claim 14, wherein the context data is not used when generating the request embedding.

19. Generating the aforementioned request embedding means This includes processing the natural language input data without processing the second context data, and generating the request embedding based on processing the first context data, Generating the aforementioned alternative request embedding is The method according to claim 14, comprising processing the natural language input data without processing the first context data, and generating the request embedding based on processing the second context data.

20. The method according to claim 19, wherein the first context data represents the current state of the domain at a first level of abstraction, and the second context data represents the current state of the domain at a second level of abstraction.

21. The method according to claim 20, wherein the current state of the domain includes a rendered image, the first level of abstraction is a pixel-level abstraction, and the second level of abstraction is a shape-level abstraction.

22. Generating the aforementioned alternative predicted set of actions is The method according to claim 1, comprising using the at least one alternative trained action model to process the alternative request embedding when generating the alternative predicted action set.

23. In generating the aforementioned alternative predicted action set, the method further includes determining, based on one or more probabilities, whether to utilize the alternative request embedding or, instead, to utilize the at least one alternative trained action model. The method according to claim 1, wherein the one or more probabilities are for the predicted set of actions and are generated based on processing the request embedding using the one or more trained action models.

24. When generating the aforementioned alternative predicted action set, determining whether to utilize the alternative request embedding or, instead, utilize the at least one alternative trained action model, based on the one or more probabilities, is: The method according to claim 23, comprising determining to utilize the alternative request embedding in response that one or more probabilities satisfy one or more thresholds indicating at least threshold confidence in the predicted set of actions.

25. When generating the aforementioned alternative predicted action set, determining whether to utilize the alternative request embedding or, instead, utilize the at least one alternative trained action model, based on the one or more probabilities, is: The method according to claim 24, comprising determining to use the alternative trained action model in response to the fact that one or more probabilities do not satisfy one or more thresholds indicating at least threshold confidence in the predicted set of actions.

26. The method according to claim 1, wherein the predicted set of actions includes a first group of actions in a first sequence, and the alternative predicted set of actions includes the first group of actions in a second sequence, or a second group of actions in a third sequence.

27. To perform the aforementioned task, transmitting the data that causes the alternative predicted set of actions to be performed in the real-world environment is: The method according to claim 1, comprising transmitting data that causes the alternative predicted action to be automatically performed in response to the user request, and that causes the alternative predicted action to be automatically performed without requiring any further user input after the user request has been provided.

28. A method carried out by one or more processors, wherein the method is Receiving natural language input containing requests to automatically generate actions to perform tasks according to the corresponding state of the domain, At least generating a first request embedding based on performing a first processing based on the natural language input, Using at least one trained action model, process the first request embedding to generate a first predicted action for performing the task, Performing a first simulation of the task, in a simulated environment that reflects the corresponding state of the domain, to perform the first predicted action, Based on the first simulation, a first suitability metric for the first predicted action is determined, To generate a second request embedding that is different from the first request embedding, based at least on performing a second processing based on the natural language input, Using at least one trained action model or at least one alternative trained action model, the second request embedding is processed to generate a second predicted action for performing the task. Performing a second simulation of the task, which involves performing the second predicted action in the simulated environment that reflects the corresponding state of the domain, Based on the second simulation, a second suitability metric for the second predicted action is determined, Based on a comparison of the first suitability metric and the second suitability metric, it is determined to perform the second predicted action instead of the first predicted action, In response to determining to perform the second predicted action described above, A method comprising transmitting data that causes the second predicted action to be performed in a real-world environment in order to perform the aforementioned task.

29. The process further includes determining that the first suitability metric does not meet a threshold, Following and in response to determining that the first suitability metric does not meet the threshold, To generate the second request embedding, Processing the second request embedding to generate the second predicted action, Performing the second simulation, or The method according to claim 28, wherein one or more of the following are performed: determining the second suitability metric.

30. Before determining the first suitability metric, or in parallel therewith, To generate the second request embedding, Processing the second request embedding to generate the second predicted action, Performing the second simulation, or The method according to claim 28, wherein one or more of the following are performed: determining the second suitability metric.

31. A computer-readable storage medium that stores computer-executable instructions, wherein when the computer-executable instructions are executed by one or more processors, the computer-readable storage medium causes the one or more processors to execute the method according to any one of claims 1 to 30.

32. A system comprising one or more processors and a memory storing instructions that can be executed by the one or more processors, wherein when an instruction is executed, the system causes the one or more processors to execute the method according to any one of claims 1 to 30.

Images