Varying embedding and / or action model utilized in automatic generation of action sets in response to natural language requests

Abstract

In contrast to a rigid approach, implementations disclosed herein utilize a flexible approach to automatically determine a set of actions to utilize when attempting to perform a task requested by a user's natural language input. The method is flexible at least in the following aspects: at least selectively changing an embedding technique and / or an action model for generating an action set that determines the action set to be utilized. In other words, embodiments utilize a framework via which different embedding techniques and / or different action models can be at least selectively used to generate different candidate action sets for a given NL input by a user. Further, one of those sets of actions may be selected for actual use of attempting real-world execution of a given task reflected by a given NL input. The selection may be based on a suitability metric and / or other considerations for the selected action set.

Description

Background Art

[0001] Various model-based (e.g., machine learning model-based) techniques have been proposed for automatically generating actions that can be implemented when attempting to perform a task. However, such techniques can be rigid because the same model is always used when generating actions and / or the input processed by the model is always generated in the same way when generating actions. Therefore, such techniques may lack robustness in many cases, resulting in actions that fail to successfully perform the task in those cases. Such techniques may additionally and / or alternatively implement the generated actions without first considering whether the actions are suitable for performing the task and / or whether they are more suitable than alternative actions. This may result in the implementation of actions that fail to successfully perform the task in many cases. Summary of the invention

[0002] Implementations described herein relate to methods and apparatus for robustly automatically generating a set of actions for performing a task in response to a free-form natural language (NL) input (e.g., a spoken utterance) provided by a user and requesting that the task be performed. The generated set of actions may be provided for performing the task. For example, providing the generated set of actions may cause the set of actions to be automatically implemented, thereby facilitating automatic performance of the task.

[0003] The techniques disclosed herein may be utilized to generate action sets for various tasks across various domains. Some non-limiting examples of tasks include: automatically controlling a computer application; automatically monitoring a video feed for the occurrence of certain condition(s), and automatically performing an action in response; automatically converting source code in a first programming language to source code in a second programming language; automatically generating an application programming interface (API); and / or automatically monitoring changes to an inventory database and automatically performing an action in response; and / or automatically integrating with a new replacement subsystem via automatically adapting its API (e.g., automatically integrating with a new web application framework (e.g., a TypeScript-based framework) when switching from a previous web application framework (e.g., a JavaScript-based framework)).

[0004] In contrast to rigid methods, the implementation disclosed herein utilizes a flexible method to automatically determine the action set to be utilized when attempting to perform a task requested by a user's NL input. The method is flexible in at least the following aspects: at least selectively changing the embedding technology and / or action model, which is used to generate an action set that determines the action set to be utilized. In other words, the embodiment utilizes a framework, via which different embedding technologies and / or different action models can be at least selectively used to generate different candidate action sets for a given NL input of a user. In addition, one of those action sets can be selected for actual use in attempting the real-world execution of a given task reflected by a given NL input. The selection can be based on a fitness metric of the selected action set and / or other considerations. For example, the selection can be based on a fitness metric that satisfies an absolute threshold or a threshold relative to the fitness metric of (multiple) alternative action sets. Such flexible methods are more robust, making it possible to determine corresponding action sets that succeed for a variety of corresponding NL inputs and / or for a variety of corresponding states of a domain.

[0005] It is worth noting that, for the corresponding NL input and the corresponding state of the domain, which embedding technology and / or action model is used to generate the selected action will vary for different NL inputs and / or for different states of the domain. As a specific example, for a first NL request and a first state of the domain, the action set determined for utilization can be an action set generated based on processing an embedding generated using the first technology using a specific action model. On the other hand, for a second NL request and / or for a second state of the domain, the action set determined for utilization can be an action set generated based on processing an alternative embedding generated using the second technology using a specific action model. As another specific example, for a first NL request for a first state of the domain, the action set determined for utilization can be an action set generated based on processing an embedding generated using the first technology using the first action model. On the other hand, for a second NL request and / or for a second state of the domain, the action set determined for utilization can be an action set generated based on processing an embedding generated using the first technology using an alternative action model.

[0006] In addition or alternatively, the implementation disclosed herein at least selectively simulates the implementation of the generated action set when determining the action set for attempting to perform the task. As described herein, such simulations can help ensure that the determined action set is suitable for performing the task before the real-world implementation of the determined action set, thereby improving the accuracy of the determined action set. Such simulations can be used additionally or alternatively to determine whether additional action sets should be generated using alternative embedding techniques and / or alternative action models and / or can help guide the generation of additional action sets. In these and other ways, simulations can be used to determine whether and / or how to generate additional action sets for consideration, thereby balancing the expectations for robustness and accuracy with the expectations for efficient use of computing resources when generating and / or evaluating additional action sets.

[0007] As a non-limiting working example of embodiments disclosed herein, assume that a computer-aided design (CAD) application is executing on a client device and is displaying a "widget" (among other things), and a user provides a NL input as a spoken utterance of "make the widget 10% larger". In this working example, the domain of the task may be a specific CAD application, a class of applications (e.g., any CAD application, or more generally any visual manipulation application), or other domains. In addition, the tasks of the working example involve control of a computer application. However, as mentioned above and elsewhere herein, the techniques disclosed herein can be used to generate and / or evaluate action sets for various tasks across various domains using the techniques disclosed herein. In some embodiments or for some domains, the same embedding techniques and / or action models may be considered for each of multiple domains. In other embodiments or for some other domains, the sets of embedding techniques and / or action models considered for those domains can each be specific to that domain.

[0008] Continuing with the working example, a first request embedding can be generated using a first embedding technique that generates the request embedding based on NL input data that directly reflects the NL input. For example, the NL input data can include text of the NL input (e.g., text generated by automatic speech recognition of spoken utterances). Using the first technique, a large language model (LLM) or other machine learning (ML) model can be used to process the text to generate an NL embedding that is a lower dimensional semantic representation of the NL input data. The first request embedding can conform to the NL embedding. The LLM model or other machine learning model used to generate the NL embedding can optionally be domain specific.

[0009] A second request embedding may also be generated using a second embedding technique that generates the request embedding based on alternative NL input data that is generated by modifying and / or supplementing the NL input data based on domain-specific knowledge (DSK). For example, before the LLM model processes the NL input, the NL input may be modified and / or supplemented based on the DSK (i.e., the modified and / or supplemented NL input will be processed). For example, an item in the NL input may be replaced or supplemented by a domain-specific definition of the item.

[0010] A third request embedding may also be generated using a third embedding technique that processes the DSK associated with the NL input item separately using an LLM model or a separate machine learning model to generate a DSK embedding. The DSK embedding may then be connected or otherwise combined with the NL input embedding generated based on processing the NL input data, and the combined embedding is used as the request embedding. For example, the DSK embedding and the NL input embedding may be processed together by an additional neural network to generate a combined low-dimensional embedding. As a specific example, a domain-specific definition of an NL input item may be processed separately to generate a domain-specific knowledge embedding (e.g., a domain-specific definition of a "widget"). As another specific example, a domain-specific image of an NL input item may be processed separately to generate a domain-specific knowledge embedding (e.g., a domain-specific picture of a "wicket").

[0011] A fourth request embedding may also be generated using a fourth embedding technique that generates a request embedding based on additional alternative NL input data that is generated by modifying and / or supplementing the NL input data based on external, rather than specific domain knowledge. For example, a general web search may be performed based on some or all of the NL input data, and text from a response result (e.g., the best result) may be used to supplement or replace items in the NL input data. For example, a general web search for "widgets" may be performed, and a text fragment from the best result may be used to modify and / or supplement the NL input data. As another example, a nearest neighbor search may be performed additionally or alternatively, and the results from the nearest neighbor search may be used additionally or alternatively to modify and / or supplement the NL input data. A nearest neighbor search may be performed based on the NL input data and across at least one large text corpus to identify text (e.g., text from the nearest neighbors of all or part of the NL input data), and the identified text may be used to modify and / or supplement the NL input data.

[0012] The first request embedding may be processed using an action model to generate a first candidate action set. The second request embedding may be processed using an action model to generate a second candidate action set. The third request embedding may be processed using an action model to generate a third candidate action set. The fourth request embedding may be processed using an action model to generate a fourth candidate action set. Each of the first action set, the second action set, the third action set, and the fourth action set includes a corresponding action that can potentially be used alone or in combination with other actions to control an application based on a request input by the NL. The first, second, third, and fourth action sets may include different actions and / or different action orders, for example, generated based on the use of different request embeddings for each action.

[0013] Each action in the first action set, the second action set, the third action set, and the fourth action set may be implemented by interaction with an application, and when implemented, may result in a generated output (e.g., usable by another action) and / or may result in some control of the application. Control of the application may be through (one or more) simulated inputs (e.g., (one or more) simulated touch inputs) and / or via an API of the application. The actions may each include corresponding program code, such as code in JavaScript, Python, C++, or other programming languages, and / or may include API calls.

[0014] Some actions can be atomic or granular. An example of such an atomic action is "select the class <x>", where "X" is a variable that can be populated based on the item(s) of the NL input. Another example of an atomic action is "identify a <y>Attributes of the target location", where Y is a variable that can be filled based on the items in NL input. Another example of an atomic action is "drag the selected object to <target location>", where <target location> is a variable that can be filled based on "use <y>The variable that is populated by the output of the atomic action of "property identifies the target position". Further examples of atomic actions include "click the rotate right 90 degrees button", "select all", and "delete", each of which does not include any variables. Such actions that do not include any variables are also referred to as state-independent actions in this article. That is, state-independent actions will be performed in the same way regardless of the corresponding state of the domain. For example, "click the rotate right 90 degrees button" will cause the button to be "clicked" regardless of what else is being rendered in the CAD application. In contrast, some actions that include state-dependent variables are also referred to as state-dependent actions in this article. In other words, they can be state-dependent in that they are implemented based on the current state, resulting in different actions for different states. For example, for "select class <x>"Object" example, which object is selected will depend on the "Class <x>" and will depend on the state of the CAD application. For example, if " <x>" is "red" and there is a red circle at the top of the active screen of the CAD application, the implementation of the state-dependent action will cause the red circle to be selected. Conversely, if there is a red square at the bottom of the active screen, the implementation of the state-dependent action will cause the red square to be selected. As another example, for "on class <x>The result of the state-dependent action will depend on how many objects of the class exist in the state. <x>" As described herein, atomic actions can be grouped together into action sets, and applications can be controlled based on action sets. Some actions can optionally include coarser pre-composed sets of multiple atomic actions, such as "Select all and click the Rotate right 90 degrees button." An action set can also be composed of one or more "coarser" actions, optionally together with atomic actions.

[0015] In some implementations, alternative action models may be used to additionally or alternatively generate additional and / or alternative action sets. For example, an alternative action model may be used to process the first request embedding to generate a fifth action set. As another example, an alternative action model may be used to process the second request embedding to generate a sixth action set. The alternative action models may be of different types and / or trained in different ways. For example, the action model may be a model for generating a probability distribution over coarse and / or granular actions based on processing the request embedding and the embedding of the current state of the domain. A candidate action set may then be generated by combining the highest probability actions indicated by the generated probability distribution. For example, the action model may generate a probability distribution over 100 candidate actions A1-A100. The first combined action set may be an ordered sequence of the four highest probability actions, such as {A1, A25, A77, A42}, the second candidate set may be an alternative ordered sequence of the four highest probability actions, such as {A1, A77, A25, A42}, and the third candidate set may be an ordered sequence of five actions sampled from the ten highest probability actions, such as {A1, A25, A82, A29, A42}. As another example, the alternative action model may be an RL policy model, which is used by the RL agent to iteratively process a sequence of states of the domain and generate the next action based on the processing. The RL agent may then implement the next action, resulting in the next corresponding state in the sequence. Thus, in this other example, the action set may be a sequence of actions generated by the RL agent in the iterative process (i.e., a corresponding action generated at each iteration).

[0016] Each of the generated action sets may be evaluated to determine the suitability of the action set, and the action set that is most suitable (and optionally satisfies a suitability threshold) for actual real-world implementation is selected. In some implementations, the evaluation of the action set may include eliminating the action set if the action set violates one or more action rules defined for the domain. For example, for a domain, a given action rule may define that a given action is not allowed at all, or is not allowed if it occurs before or after certain other actions. If the action set includes a given action or includes a given action before or after certain other actions, the action set may violate the given action rule and be eliminated.

[0017] In some embodiments, the evaluation of the action set may additionally or alternatively include determining the suitability of the action set based on performing a simulation according to the action set. The simulation of the action set can be performed in a simulation environment that reflects the corresponding state of the domain. For example, the initial simulation environment can reflect the CAD application in its current state. Some action sets can be generated before the simulation and implemented during the corresponding simulation. Other action sets can be generated during the corresponding simulation. For example, the action set generated using the RL agent and the RL policy model can be generated during the corresponding simulation based on processing the simulation state data from the simulation. As described herein, some or all actions in the action set can be state-dependent, which means that their implementation will depend on the initial simulation environment and the modification of the simulation environment by any implementation or previous action in the action set.

[0018] In some implementations, determining the suitability of an action set based on a simulation of the action set may include determining whether the simulation violates one or more state rules defined for the domain. For example, a state rule for a domain may define that the domain should never encounter a certain state. If simulation data from a simulation indicates that a certain state was encountered, the action set may be determined to be unsuitable, or a measure of suitability of the action set may be negatively impacted.

[0019] In some implementations, determining the suitability of an action set based on a simulation of the action set may additionally or alternatively include causing simulation data from the simulation to be presented to a user who provided the NL request, and in response to the presentation, determining suitability based on feedback from the user. For example, a screenshot of the simulation environment in its final state from the simulation or other data reflecting the final state from the simulation may be presented to the user, and the user may provide (multiple) user interface inputs reflecting whether the screenshot reflects successful execution of the task. An instance of negative feedback may be used to eliminate a corresponding action set or negatively affect a fitness metric of a corresponding action set. Conversely, an instance of positive feedback may be used to select the most suitable corresponding action set, or to positively affect a fitness metric of a corresponding action set.

[0020] In some embodiments, determining the fitness of an action set based on a simulation of the action set may be additionally or alternatively based on how closely the simulation conforms to the request of the NL input. As an example, the final state of the simulation may be processed to generate an NL description of the final state, and the NL description (e.g., its embedding) is compared with the NL input when generating a fitness metric. For example, a "closer" embedding may correspond to a better fitness metric, which indicates more fitness. For example, an image annotation model may be used to process screenshots of the final state of the simulation to generate an NL description. As another example, a video from a simulation (e.g., a series of screenshots) may be processed to generate an NL description of the simulation, and the NL description may be compared with the NL input when generating a fitness metric. For example, each screenshot may be processed individually using a first machine learning model to generate a corresponding embedding, and the embedding may be processed in chronological order using a second machine learning model (e.g., a transformer) (e.g., an embedding corresponding to the earliest screenshot processed first) to generate an NL description of the simulation and / or generate an overall embedding of a semantically simulated description. As another example, non-visual (i.e., not images and not video) simulation data from a simulation can be processed to generate an NL description of the simulation, such as audio data from the simulation (e.g., generated by a simulation application during the simulation), textual data from the simulation (e.g., generated by a simulation application during the simulation), and / or other data from the simulation. More generally, one or more instances of simulation data can be processed to generate one or more corresponding embeddings and / or other representations, each instance reflecting a corresponding state of the simulation application at a corresponding time instance. Furthermore, the generated embeddings and / or other representations can be used to generate a fitness metric (e.g., by comparing to an NL input embedding and / or a request embedding).

[0021] In some implementations of the working embodiment, the first action set, the second action set, the third action set, the fourth action set and / or the additional action set are generated in parallel. This can reduce the delay of parsing the action set for real-world implementation. In other implementations, only a subset of the action set (e.g., only the first action set) is initially generated, and only when the action set of the subset is determined to be unsuitable, the additional action set is generated and evaluated. For example, only the first action set can be generated and evaluated initially, and only when the evaluation indicates that it is not suitable, the second action set will be generated and / or evaluated. In addition, only when the evaluation of the second action set indicates that it is not suitable, the third action set will be generated and / or evaluated. This can save computing resources by reducing the amount of processing performed at a given time and at least selectively eliminating the need to generate and / or evaluate subsequent action sets (e.g., when it is determined that the action set generated earlier is suitable).

[0022] In various implementations, determining whether to generate different action sets in parallel (or the degree of parallelization) may depend on one or more factors. For example, the domain of the request or the request itself may indicate whether the request is urgent (in which some degree of parallelism will be utilized) or non-urgent (in which case no parallel processing or a lesser degree of parallel processing will be utilized). As another example, determining whether to generate different action sets in parallel may additionally or alternatively be based on the current load on the server performing the action set generation and / or simulation.

[0023] The foregoing is provided as a non-limiting summary of some embodiments disclosed herein. These and other embodiments are described in further detail herein.

[0024] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause performance of any of the foregoing methods. Some embodiments include at least one transient or non-transitory computer-readable storage medium storing computer instructions executable by the one or more processors to perform any of the foregoing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram of an example environment in which implementations disclosed herein may be implemented.

[0026] Figure 2 Schematically shows Figure 1 An example of how components of the system may interact in automatically determining a set of actions to utilize when attempting to perform a task requested by a user's NL input.

[0027] Figure 3 is a flow chart illustrating an example method of practicing selected aspects of the present disclosure according to implementations disclosed herein.

[0028] Figure 4 is a flow chart illustrating another example method of practicing selected aspects of the present disclosure according to implementations disclosed herein.

[0029] Figure 5 An example architecture of a computing device is illustrated. DETAILED DESCRIPTION

[0030] Figure 1 Example environments are schematically depicted in which selected aspects of the present disclosure may be implemented according to various implementations. Figure 1 Any computing device depicted in or elsewhere in the figures may include logic, such as one or more microprocessors (e.g., a central processing unit or "CPU", a graphics processing unit or "GPU", a tensor processing unit or ("TPU")) that execute computer-readable instructions stored in a memory, or other types of logic, such as an application specific integrated circuit ("ASIC"), a field programmable gate array ("FPGA"), etc. Figure 1 Some of the systems depicted in (such as the NeuroArena system 120) may be implemented in whole or in part using one or more server computing devices forming what is sometimes referred to as a “cloud infrastructure,” although this is not required.

[0031] The NeuroArena system 120 may be operably coupled to one or more client computing devices (also referred to herein as clients), such as client computing device 110, via one or more computer networks 114. The NeuroArena system 120 may automatically determine, based on NL input provided by a user of the client 110, a set of actions to use in attempting to perform a task requested in the NL input. For example, a user may provide, via a microphone of the client 110, a free-form verbal NL input reflecting a task to be performed within a domain, and the NeuroArena system 120 may automatically determine, based on the NL input, a set of actions to use in attempting real-world performance of the task. Further, the NeuroArena system 120 may be operably coupled to an implementation system 140 via the network(s) 114.

[0032] The implementation system 140 can implement the action set determined by the neural arena system 120 in the corresponding domain when attempting real-world execution of the corresponding task. For example, if the task is to automatically monitor a video feed for the occurrence of certain conditions, the implementation system 140 can use the action set to process the video feed when monitoring the occurrence of certain conditions, and optionally, generate an alarm in response to detecting any certain conditions using the determined action set. In such an example, the implementation system 140 can be implemented in (multiple) servers that receive the video feed as a stream over the Internet and / or in (multiple) computing devices that are on a local network with a camera providing the video feed and receive the video feed as a stream of the local network. As another example, if the task is to automatically convert source code in a first programming language to source code in a second programming language, the implementation system 140 can be implemented in a server and / or client device, can receive the first programming language source code, and can automatically convert it into the second programming language source code using the determined action set.

[0033] The client device 110 may include one or more applications, such as the application 112, that interact with the Neural Arena system 120. For example, the application 112 may be an application via which a user's NL input may be provided (e.g., via verbal and / or typed input), and the application 112 may send the NL input to the Neural Arena system 120. Also, for example, the application 112 may be an application via which output generated by the Neural Arena system 120 may be presented to a user, such as output requesting user feedback (e.g., output reflecting the final state of a simulation of a candidate action set) and / or output reflecting an action set determined by the Neural Arena system 120 (e.g., for confirming an action set prior to automatic real-world implementation). In some embodiments, where the task includes control of a computer application executing on the client 110, the application 112 (or another application) may be an application that may be controlled using an action set determined by the Neural Arena system 120. In those embodiments, the implementation system 140 may interface with the application when implementing the action set and / or may be integrated (in whole or in part) as part of the application.

[0034] Although the Neural Arena system 120 and the implementation system 140 Figure 1 140 are depicted as being separate from one another and connected to the client 110 via the network(s) 114, but in various implementations, one or more aspects of the Neural Arena system 120 and the implementation system 140 may be combined and / or may be implemented locally at the client 110. For example, one or more of the engines of the Neural Arena system 120 may be implemented at the client 110 and / or one or more aspects of the implementation system 140 may be implemented at the client 110.

[0035] In various embodiments, the neuroarena system 120 includes an embedding engine 122 , an action engine 124 , a selection engine 126 , a simulation (SIM) engine 128 , and / or an evaluation engine 130 .

[0036] The embedding engine 122 can interface with one or more embedding ML models 152 when generating the embeddings described herein. Which embedding ML model(s) 152 the embedding engine 122 interfaces with, and / or the data it processes when interfacing with one or more embedding ML models 152 can depend on the embedding technology that the embedding engine 122 is utilizing. For example, for a given NL input, in one instance, the embedding engine 122 can generate a first embedding based on a first embedding technology that uses an LLM model of a specific domain of the embedding model 152 to process NL input data that reflects the NL input. In addition, for a given NL input, in another instance, the embedding engine 122 can generate a second embedding based on a second embedding technology that includes processing alternative NL input data using an LLM model of a specific domain. The embedding engine 122 can generate alternative NL input data by modifying and / or supplementing the NL input data using a DSK. In addition, for a given NL input, in another instance, the embedding engine 122 can generate a third embedding based on a third embedding technology that includes processing an image using an image embedding model of the embedding ML model 152. The image may be, for example, a DSK image associated with the NL input data or an external image associated with the NL input data.

[0037] The embedding technique utilized by the embedding engine 122 at a given instance may depend on various factors, and in some embodiments, may be specified by the selection engine 126. For example, the embedding technique utilized may depend on the domain of the task, the NL input for which the embedding is generated, and / or the action model utilized by the action engine 124 in generating the candidate action set. In addition, for example, the embedding technique used in a given instance of the NL input may be additionally or alternatively based on the embedding technique used in the previous instance when generating the candidate action set of the NL input and / or the evaluation of those candidate action sets. A variety of embedding ML models 152 may be provided. For example, the embedding ML models 152 may include those models that are specific to a particular domain, those models that are specific to a particular set of domains, and / or those models that are domain agnostic. As another example, the embedding ML models 152 may additionally or alternatively include those models that are specific to a first type of data (e.g., natural language data), those models that are specific to a second type of data (e.g., images), and / or those models that are specific to a third type of data (e.g., audio data).

[0038] The action engine 124 may interface with a plurality of action models 154 when generating candidate action sets as described herein. In some embodiments, the action engine 124 also interfaces with one or more action rules 164, which may be used to eliminate some of the generated candidate action sets from further consideration (e.g., from further consideration by the evaluation engine 130). The action rules 164 may be domain-specific and / or specific to a corresponding requesting entity, such as a user providing a corresponding NL request or an organization associated with a user providing a corresponding NL request. For example, for a particular domain and a particular organization, a given action rule may define that a given action is not allowed at all, or is not allowed if the given action occurs before or after certain other actions.

[0039] Which action model(s) 154 the action engine 124 interfaces with at a given instance may depend on various factors, and in some embodiments, may be specified by the selection engine 126. For example, the action model 154 utilized may depend on the domain of the task, the NL input for which the embedding is generated, and / or the embedding generated by the embedding engine 122. In addition, for example, the action model 154 used in a given instance of the NL input may be additionally or alternatively based on the action model used in the previous instance when generating the candidate action set of the NL input and / or the evaluation of those candidate action sets. Various action models 154 may be provided. For example, the action model 152 may include a machine learning model and / or a heuristic model. As another example, the action model 152 may include those action models that are specific to a particular domain, those action models that are specific to a particular set of domains, and / or those action models that are domain agnostic. As another example, the action models 152 may include: those action models that represent RL policies and are used to generate action sets by iteratively generating corresponding next actions in the action set based on applying updated state data at each iteration; those action models that are used to generate one or more candidate action sets in a single iteration; those action models that represent value functions and are used to generate metrics that reflect the value of the action set and the current state pair; and / or other action models. For example, the action model may include one or more of an RL policy ML model, an action sequence ML model, a constraint satisfaction model, a SAT solver, and / or other models.

[0040] In some embodiments, the selection engine 126 can interact with the embedding engine 122 to specify which embedding technology the embedding engine 122 utilizes at a given instance and / or can interact with the action engine 124 to specify which action model the action engine 124 utilizes at a given instance. For example, the selection engine 126 can specify that the embedding engine 122 initially utilizes a first embedding technology. Then, only when the evaluation engine 130 indicates that the corresponding candidate action set generated based on the first embedding technology is not suitable, can the selection engine 126 specify that the embedding engine 122 utilizes a second embedding technology when generating an additional candidate action set. As another example, the selection engine 126 can specify that the embedding engine 122 initially utilizes a first embedding technology and a second embedding technology. Then, only when the evaluation engine 130 indicates that the corresponding candidate action set generated based on the first embedding technology and the second embedding technology is not suitable, can the selection engine 126 specify that the embedding engine 122 utilizes a third and / or fourth embedding technology when generating an additional candidate action set.

[0041] In some embodiments, the selection engine 126 may optionally utilize one or more selection models 156 to determine which (or which) embedding techniques and / or actions to utilize at a given instance. For example, the selection model 156 may include a selection ML model that can be used to process the domain of the task and / or the NL input data (e.g., the embedding of the NL input data) of the request task and generate an output indicating the corresponding probability of each of the multiple embedding techniques and / or action models. The selection engine 126 can utilize the generated output when selecting which (or which) embedding techniques and / or action models to utilize. For example, the selection engine 126 can use the output to select the highest probability embedding technique and / or the highest probability action model for initial utilization. For example, such a selection ML model can be trained based on supervised training examples based on past NL inputs and action sets determined to be suitable (and optionally confirmed to be suitable after their real-world implementation). For example, assume that a given action set is generated for NL input using a first embedding technique and a first action model and is determined to be suitable. In response, a training example may be generated that includes the domain of the task for the NL input and / or the NL input data for the NL input as training example inputs and includes (one or more) positive values ​​for the first embedding technique and the first action model and negative values ​​for all other embedding techniques and action models as training example outputs.

[0042] The SIM engine 128 may be used to simulate, for each action set generated by the action engine 124, the implementation of the action set in a simulation environment (such as a simulation environment that reflects the current state of the domain). In addition, the SIM engine 128 generates simulation data for each simulation. In some cases, the action set may be generated independently of its simulation, and the SIM engine 128 is used to simulate the action set after the action set is generated. In some other cases, the action set may be generated via the SIM engine 128 during the simulation. For example, some RL policy model may be utilized in the simulation to generate candidate action sets, and the candidate action sets will be implemented during the simulation, and their generation will depend on the simulation states encountered during the simulation.

[0043] As a specific example, where the task is to control a computer application, the SIM engine 128 simulates the execution of the control application 112 by implementing a candidate action set. In such an example, the SIM engine 128 may use an emulator when performing the simulation, and may optionally start each simulation from the current state of the application. As another specific example, where the task is to convert code from a first programming language to a second programming language, the SIM engine 128 simulates the execution of the conversion by implementing a candidate action set, and optionally, further simulates the implementation of the resulting converted second programming language code. As another specific example, where the task is to automatically monitor changes in an inventory database and automatically perform actions in response, the SIM engine 128 simulates a dynamically changing inventory database and simulates monitoring of the simulated database by implementing a candidate action set. As another specific example, where the task is to automatically monitor a video feed for the occurrence of certain conditions and automatically perform actions in response, the SIM engine 128 simulates a video feed and / or replays a past real-world video feed, and simulates monitoring of the video feed by implementing a candidate action set.

[0044] The evaluation engine 130 may determine whether the candidate action set is suitable for performing the corresponding task and / or determine the most suitable action set in the candidate action set from multiple candidate action sets. In various implementations, when evaluating the action set, the evaluation engine 130 utilizes simulation data from the simulation of the action set by the SIM engine 128. In some of those embodiments, the evaluation engine 130 may compare the simulation data with one or more state rules 160, which may be used to determine that the candidate action set is not suitable and / or negatively affects the fitness score of the candidate action set, which is used to determine the fitness of the candidate action set. The state rule 160 may be domain-specific and / or specific to the corresponding requesting entity, such as the user providing the corresponding NL request or the organization associated with the user providing the corresponding NL request. For example, for a specific domain and a specific organization, a given state rule may define that a given state should never be encountered or that a specific state sequence should never be encountered. If the simulation data from the simulation of the candidate action set indicates that a given state and / or a specific state sequence is encountered, the evaluation engine 130 may determine that the candidate action set is not suitable. As another example, for a particular domain and a particular organization, a given state rule may define that a given state or a particular sequence of states is undesirable, but not prohibited. If simulation data from a simulation of a candidate action set indicates that the given state and / or the particular sequence of states is encountered, the evaluation engine 130 may negatively affect the suitability metric of the candidate action set.

[0045] In some additional or alternative embodiments of evaluating an action set using simulation data, the evaluation engine 130 may additionally or alternatively solicit and utilize user feedback based on the simulation data and / or analyze the degree of conformity of the simulation data with the request of the corresponding NL input. For example, the evaluation engine 130 may cause the simulation data from the simulation to be presented to the user who provides the NL request, and determine the suitability based on the feedback from the user in response to the presentation. For example, the evaluation engine 130 may cause a screenshot of the simulation environment in its final state from the simulation to be presented at the client 110, and in response, the user may provide (multiple) user interface inputs reflecting whether the screenshot reflects the successful execution of the task. The evaluation engine 130 may use an instance of negative feedback to eliminate the corresponding action set or negatively affect the fitness metric of the corresponding action set. Conversely, the evaluation engine 130 may use an instance of positive feedback to select the most suitable corresponding action set, or actively affect the fitness metric of the corresponding action set.

[0046] In some additional or alternative embodiments of evaluating an action set using simulated data, the evaluation engine 130 may additionally or alternatively determine the fitness of the action set based on the degree of conformity of the simulated data from the simulation of the action set with the request of the NL input. As an example, the evaluation engine 130 may process the final state of the simulation to generate an NL description of the final state, and compare the NL description with the NL input data when generating the fitness metric. For example, the embedding of the NL description of the final state may be compared with the embedding based on the NL description data (e.g., based only on the NL description data or based on the supplemented or modified NL description data as described herein). A comparison indicating a greater degree of similarity (e.g., a "closer" embedding) may correspond to a better fitness metric, which indicates more fitness (i.e., resulting in a better fitness score). For example, an image annotation model may be used to process a screenshot of the final state of the simulation to generate an NL description.

[0047] The machine learning models described herein can have various architectures and be trained in various ways. For example, one or more of the models can be a graph-based neural network (e.g., as a graph neural network (GNN), a graph attention neural network (GANN), or a graph convolutional neural network (GCN)), a sequence-to-sequence neural network, such as a transformer, an encoder-decoder, or a recurrent neural network ("RNN", such as a long short-term memory or "LSTM", a gated recurrent unit or "GRU", etc.), BERT (bidirectional encoder representation from a transformer). In addition, for example, reinforcement learning, supervised learning, and / or imitation learning can be utilized when training one or more of the machine learning models. Additional descriptions of some embodiments of various machine learning models are provided herein.

[0048] Go to Figure 2 , provides a description of examples of: engines 122 , 124 , 126 , 128 , and 130 of the neural arena system 120 ; interactions that can occur between those engines; and models 152 , 154 , and 156 and rules 160 and 164 that can be utilized by the neural arena system 120 .

[0049] exist Figure 2 In the example, embedding engine 122 processes at least NL input 101 when generating request embedding 123. NL input 101 is input by a user via a client device (e.g., Figure 1 The NL input 101 may be provided by interaction with a user interface input device of the client 110 of the client 110, and the NL input 101 includes a request to generate an action for the task. For example, the NL input 101 may be a spoken input of a user detected via a microphone of the client 110, and the embedding engine 122 may process the recognized text generated based on the spoken input when generating the request embedding 123 (e.g., using automatic speech recognition (ASR)). As another example, the NL input 101 may be a typed input provided via a virtual or hardware keyboard of the client 110, and the typed text may be processed by the embedding engine 122 when generating the request embedding 123. For example, the recognized text or the typed text may be processed using the NL ML model 152A of the request ML model 152 to generate an NL embedding. The NL ML model 152A may be, for example, an LLM. The request embedding 123 may be an NL embedding, or may be a function of the NL embedding and other NL embeddings.

[0050] In some embodiments, the request engine 122 additionally utilizes domain-specific knowledge (DSK) 102, context data 103, and / or external knowledge 104 when generating the request embedding 123. In those embodiments, whether the DSK 102, context data 103, and / or external knowledge 104 are utilized depends on the embedding technique utilized by the embedding engine 122. Furthermore, in some of those embodiments, the embedding technique utilized by the embedding engine 122 at a given instance can be specified by the selection engine 126. For example, for an initial request embedding 123 for a given NL input 101, the selection engine 126 can specify that the request embedding 123 be generated using a first embedding technique, wherein only the NL input 101 is utilized when generating the request embedding 123. Furthermore, for a next request embedding 123 for the same given NL input 101, the selection engine 126 can specify that the request embedding 123 be generated using a second embedding technique, wherein the NL input 101 is utilized and the DSK 102 is also utilized. For example, the request engine 122 may use the DSK 102 to change the NL input 101, and use the NL ML model 152 to process the change of the NL input 101 to generate the request embedding 123. As a specific example, the NL input 101 may be "make the side door 15% smaller", and the request engine 122 may use the DSK 102 to find the specific domain definition of "side door" of "small door next to the main door", and change the NL input 101 to "make the small door 15% smaller" or "make the side door (small door next to the main door) 15% smaller". Continuing with the specific example, the request embedding may be the NL embedding generated based on the change of the NL input 101. The selection engine 126 may utilize the selection ML model 156 and / or consider which embedding techniques have been used for a given NL input 101 when specifying which embedding technique to use in a given example.

[0051] As another example of an embedding technique that may be used by the embedding engine 122, the embedding engine 122 may identify specific domain-specific knowledge related to the terms of the NL input 101 from the DSK 102. Further, the embedding engine 122 may process the specific domain-specific knowledge using one or more of the request ML models 152 to generate a DSK embedding. The embedding engine 122 may then concatenate or otherwise combine the DSK embedding with the NL embedding and / or the context embedding (e.g., through additional neural network processing to generate a lower dimensional embedding), and use the combined embedding as the request embedding 123. For example, the NL input 101 may be "make the side door 15% smaller", and the embedding engine 122 may generate an NL embedding based on the NL input 101 (unchanged). Further, the embedding engine 122 may be able to identify a domain-specific NL definition of a "side door" from the DSK 102 and / or may be able to identify a domain-specific image of a "side door". The embedding engine 122 may use the NL ML model 152A to process the domain-specific NL definition of the "side door" to generate a domain-specific NL definition embedding, and / or may use the image ML model 152B to process the domain-specific "side door" image to generate a domain-specific image embedding. The embedding engine 122 may then generate the request embedding 123 based on the NL embedding and the domain-specific NL definition embedding and / or the domain-specific image embedding. The image ML model 152B may be, for example, a convolutional neural network (CNN) or other neural network that is trained to generate semantically rich image embeddings based on the processing of those images.

[0052] As described above, for one or more embedding techniques, the embedding engine 122 may additionally or alternatively utilize context data 103 and / or external knowledge 104 when generating the request embedding 123. For example, the context data 103 may include current state data of the domain of the task, and the embedding engine 122 may process the current state data to generate the context embedding, and may generate the request embedding 123 based on the context embedding. For example, the current state data may include an NL description of the current state of the domain. Additional or alternative context data 103 may be used to generate the context embedding, such as an indication of the currently executed application on the client device via which the NL input 101 is provided, the most recent NL input provided via the client device, the current time of the day or other current time data, the current location, and / or other context data. As another example, the external knowledge 104 may include data obtained via a general search engine or other general knowledge base and using the NL input 101. For example, the external knowledge 104 may include search results returned from a general search engine in response to a query formulated using the NL input 101. The embedding engine 122 may process the external knowledge 104 to generate a knowledge embedding, and may generate the request embedding 123 based on the knowledge embedding. In some embodiments, first context data may be utilized in a first embedding technique, and different second context data may be utilized in a second embedding technique. For example, the first context data may include an indication of an application currently executing on the client device, but exclude recent NL input, and the second context data may include recent NL input, but exclude indications of the application. As another example, the first context data may include a pixel-level abstraction (e.g., the pixels themselves) of an image of a current state (e.g., the current state of an application), but exclude indications of higher-level abstractions of the image (e.g., shapes or other objects derived from pixels), and the second context data may include higher-level abstractions but exclude pixel-level abstractions.

[0053] Action engine 124 processes request embedding 123 using one or more of the action ML models to generate one or more candidate action sets 125. For example, in a given instance, action engine 124 may use first RL policy model 154A, second RL policy model 154B, constraint satisfaction model 154C, action sequence model 154N, or other models of action models 154 (e.g., those generated by Figure 2 154 to process the request embedding. In some of those embodiments, which of the action models 154 is utilized by the action engine 124 at a given instance may be specified by the selection engine 126. For example: for an initial instance of a given NL input 101, the selection engine 126 may specify that the first RL strategy model 154A is used to generate the candidate action set 125; for a second instance of the same given NL input 101, the selection engine 126 may specify that the second RL strategy model 154B is used to generate the candidate action set 125; and for a third instance of the same given NL input 101, the selection engine 126 may specify that the action sequence model 154N is used to generate the candidate action set 125. Optionally, when generating the candidate action sets 125, the action engine 124 may eliminate one or more generated action sets based on those action sets that violate the action rules 164 as described herein.

[0054] SIM engine 128 may be used for each candidate action set 125 to simulate the implementation of the action set in a simulation environment (such as a simulation environment that reflects the current state of the domain). In addition, SIM engine 128 generates simulation data 129 for each simulation. In some cases, the action sets in action sets 125 may be generated independently of their simulation, and SIM engine 128 is used to simulate the action sets after the action sets are generated. In some other cases, action engine 124 may generate action sets via SIM engine 128 during simulation. This is reflected by the dashed double arrow line between action engine 124 and SIM engine 128.

[0055] The simulation data 129 is provided to the evaluation engine 130. The evaluation engine 130 may use the simulation data 129 to determine whether the corresponding candidate action set of the candidate action set 125 is suitable for performing the corresponding task and / or determine the most suitable candidate action set in the candidate action set from multiple candidate action sets. In some of those embodiments, the evaluation engine 130 may compare the simulation data with one or more state rules 160, which may be used to determine that the candidate action set is not suitable and / or negatively affect the suitability score of the candidate action set, which is used to determine the suitability of the candidate action set. In some additional or alternative embodiments of evaluating the action set using the simulation data 129, the evaluation engine 130 may additionally or alternatively solicit and utilize user feedback based on the simulation data 129 and / or analyze the degree of compliance of the simulation data 129 with the request of the NL input 101. In some additional or alternative embodiments of evaluating the action set using the simulation data 129, the evaluation engine 130 may additionally or alternatively determine the suitability of the action set based on the degree of compliance of the simulation data 129 from the simulation of the action set with the request of the NL input 101.

[0056] If the evaluation engine 130 determines that none of the candidate action sets 125 are suitable, it may output a non-suitability indication 131 to the selection engine 126. In response, the selection engine 126 may adjust the embedding technology utilized by the embedding engine 122 and / or the action model 154 utilized by the action engine 124. Additional candidate action sets 125 may then be generated based on different request embeddings 123 (e.g., generated using alternative embedding technologies) and / or based on different action models of the action model 154. For example, the selection engine 126 may adapt the utilized embedding technology but not the utilized action model 154. In response, the embedding engine 122 may generate different request embeddings 123 using different adapted embedding technologies, and the action engine 124 will process the different request embeddings 123 using the same action model as before. This may result in the generation of different candidate action sets 125 due to the different request embeddings 123. The SIM engine 128 may simulate the different action sets 125, and the evaluation engine 130 utilizes the resulting simulation data 129 when evaluating the different candidate action sets 125. Multiple iterations of this may occur until, for example, the evaluation engine 130 determines that the evaluated set of candidate actions is suitable.

[0057] If the evaluation engine 130 determines in a given iteration that one of the candidate action sets 125 is suitable, it can provide the action set 132 to the implementation system 140. The implementation system 140 can then cause the action set 132 to be implemented in a real-world environment. For example, the implementation system 140 can automatically implement the action set 132 without first prompting a user for verification. As another example, the implementation system 140 can first prompt a user for verification before implementation, and only implement the action set 132 if a positive user input is received in response to the prompt.

[0058] Figure 3 is a flowchart illustrating an example method 300 for practicing selected aspects of the present disclosure in accordance with implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of the NeuroArena system 120. Furthermore, while the operations of the method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0059] At block 302, the system receives NL input data reflecting a request for an action to automatically generate a task. For example, the request may be a verbal request, and the NL input data may be a transcription generated using automatic speech recognition.

[0060] At block 304, the system selects a request embedding technique and / or action model to utilize. In some implementations, the system selects a request embedding technique and / or action model to utilize based on the domain of the task and / or based on the NL input data of block 302. For example, the system may use a defined heuristic that indicates that for a particular domain and / or for NL input data that includes a particular term, a first embedding technique and a first action model should initially be utilized. As another example, the system may use a trained selection model (e.g., Figure 1 The system may process the NL input data (e.g., its embedding) and / or an indication of the domain using one of the selection models 156 to generate an output indicating which embedding technique and / or action model should be utilized. The system may select the embedding technique and / or action model to initially utilize based on the output.

[0061] At block 306, the system generates a request embedding based on processing the NL input data using the embedding ML model. In some embodiments, the embedding ML model is selected from a plurality of candidate embedding ML models according to the currently selected embedding technology. In some embodiments or iterations, block 306 includes sub-blocks 306A, sub-blocks 306B, and / or sub-blocks 306C. In some of those embodiments, which (if any) of the sub-blocks are executed in a given iteration of block 306 may depend on the currently selected embedding technology. For example, for the first embedding technology, no sub-blocks may be executed, for the second embedding technology, only sub-block 306A may be executed, for the third embedding technology, only sub-block 306B may be executed, for the fourth embedding technology, only sub-block 306C may be executed, for the fifth embedding technology, only sub-blocks 306A and 306B may be executed, and / or for the sixth embedding technology, all sub-blocks 306A, 306B, and 306C may be executed.

[0062] At subblock 306A, the system generates a request embedding based on the DSK (such as a pre-stored DSK). For example, the system can change the NL input using the DSK, and generate the request embedding based on processing the changed NL input. As another example, the system can separately process the DSK associated with the NL input to generate (multiple) domain-specific embeddings, and generate the request embedding based on the NL embedding from processing the NL input and based on the (multiple) domain-specific embeddings.

[0063] Sub-block 306A optionally includes further sub-block 306A1, in which the system interacts with the user to obtain at least some DSKs. For example, in response to determining that there are no pre-stored DSKs corresponding to one or more items of the received NL input, the system can interact with the user to obtain at least some DSKs. In some embodiments, performing sub-block 306A without performing sub-block 306A1 can be considered a first embedding technique, and performing sub-block 306A with performing sub-block 306A1 can be considered a different second embedding technique.

[0064] At sub-box 306B, the system generates a request embedding based on contextual data (such as contextual data that describes the context of the NL input but is not reflected by the content of the NL input itself). For example, the contextual data may include temporal contextual data (e.g., time of day, day of week, and / or day of year), previous NL input contextual data (e.g., NL inputs provided by the same user within the last N seconds and / or the last N NL inputs), client contextual data (e.g., images rendered on the client at or near the time the NL input was provided at the client and / or applications executed on the client at or near the time the NL input was provided). In some embodiments, at box 306B, the system may process the contextual data to generate a contextual embedding, and generate a request embedding based on the NL embedding from the processed NL input and based on the contextual embedding.

[0065] At subbox 306C, the system generates a request embedding based on external data from an external knowledge source. For example, a general web search may be performed based on some or all of the NL input data, text from a response result (e.g., a top result) may be used to supplement or replace items in the NL input data, and the system may generate a request embedding based on processing the changed NL input. As another example, the system may process text from the response result separately to generate an external data embedding, and generate a request embedding based on the NL embedding from the processed NL input and based on the external embedding. As yet another example, an image search may be performed based on some or all of the NL input data, an image from a response result (e.g., a top image result) may be processed using an image embedding ML model to generate an image embedding, and a request embedding may be generated based on the image embedding and based on the NL embedding from the processed NL input (e.g., an NL embedding generated using a domain-specific LLM).

[0066] At block 308, the system processes the request embedding generated from the most recent iteration of block 306 using the action model to generate one or more predicted action sets. For example, the system can process the generated request embedding using the action model to generate a single candidate action set or to generate multiple candidate action sets. In some implementations, the action model utilized is selected from the multiple candidate action models based on the currently selected action model.

[0067] In some implementations, block 308 includes sub-block 308A and / or sub-block 308B.

[0068] At sub-block 308A, if the candidate action set is determined to violate one or more defined action rules, the system may eliminate one or more of the candidate action sets.

[0069] At sub-frame 308B, the system may eliminate one or more of the one or more candidate action sets based on the previous iteration of sub-frame 314A and / or 314B (described below) input by NL. Generally speaking, sub-frame 314A and / or 314B may also be used to eliminate the candidate action set, but the candidate action set will be eliminated based on the fact that the candidate action set produces an undesirable simulation state when implemented. In some cases, the action that causes the simulation state in the action set may be identified in sub-frame 314A and / or 314B. In some of those specific implementations, sub-frame 308B may include eliminating the candidate action set that includes those actions. For example, if the candidate action set includes a specific action identified in sub-frame 314A and / or 314B, or if the candidate action set includes a specific action sequence identified in sub-frame 314A and / or 314B, the candidate action set may be eliminated.

[0070] Additional and / or alternative techniques may be utilized to eliminate action sets at block 308. For example, when utilizing an ML model to generate action sets and probabilities or other metrics indicating the likelihood that the action set is suitable for the requested embedding, implementations may eliminate action sets whose metrics fail to meet a threshold.

[0071] At block 310, the system determines whether there are any candidate action sets remaining from the most recent iteration of block 308. If not, the system proceeds to block 318 (described below). If yes, the system proceeds to block 312.

[0072] At block 312, for each of the action sets generated at the most recent iteration of block 308 (and not eliminated at block 308), the system implements the action set in simulation. As described herein, for some action models, the system may perform blocks 308 and 310 simultaneously. That is, for those action models, the system sequentially generates the actions of the action set by interacting with the simulation environment in simulation. For example, the RL agent may utilize the RL policy model of the action model(s) when selecting and implementing the actions in the action set in simulation. For other action models(s), the system may generate a completed action set in block 308 before implementing the action set in simulation at block 310.

[0073] At block 314, the system determines, for each of the simulated action sets and based on the simulation, whether the action set is suitable. The system determines whether the action set is suitable based on the simulation data from the simulation. For example, the system can generate a fitness metric for the action set based on the simulation data, and determine whether the action set is suitable based on the fitness metric. For example, if the fitness metric for the action set meets a threshold and indicates more fitness than any other action set considered at the iteration of block 314, the action set can be determined to be suitable. Otherwise, the action set can be determined to be unsuitable.

[0074] In some embodiments, when determining whether an action set is applicable, block 314 includes sub-block 314A, sub-block 314B, and / or sub-block 314C.

[0075] At subframe 314A, the system compares the simulation data of the action set with one or more state rules. For example, the state rule of a domain can define that the domain should never encounter a certain state. If the simulation data from the simulation indicates that a certain state is encountered, the system can determine that the action set is not suitable. As another example, additional or alternative state rules can define that additional specific states are undesirable for the domain. If the simulation data from the simulation indicates that a certain state is encountered, the system may negatively affect the fitness metric of the action set. The compared simulation data can include, for example, simulation state data reflecting the final simulation state and / or one or more intermediate simulation states.

[0076] At subframe 314B, the system solicits user feedback about the simulation data, and determines whether the action set is suitable based at least in part on the user feedback received in response to the solicitation. For example, the system can cause a representation of the final simulation state and / or the intermediate simulation state to be presented (e.g., visually and / or audibly) to the user. The representation can come from the simulation data or be generated from the simulation data. In addition, the system can process feedback received via one or more user interface inputs in response to presentation, and determine whether the action set is suitable based on the feedback. For example, an instance of negative feedback can be used to eliminate the corresponding action set or have a negative impact on the fitness metric of the corresponding action set. In contrast, an instance of positive feedback can be used to select the most suitable corresponding action set, or actively affect the fitness metric of the corresponding action set.

[0077] At sub-block 314C, the system compares the simulation data to the NL input data of block 302, and determines whether the action set is suitable based at least in part on the comparison. For example, a comparison indicating at least a positive threshold similarity can positively affect the fitness metric, and a comparison indicating less than a negative threshold similarity can negatively affect the fitness metric. As a specific example, the system can process the final state of the simulation to generate an NL description of the final state, and compare the NL description to the NL input data when generating the fitness metric. For example, an embedding of the NL description of the final state can be compared to an embedding based on the NL description data (e.g., based only on the NL description data or based on supplemented or modified NL description data as described herein).

[0078] At block 316, the system determines whether any of the action sets were suitable in the most recent iteration of block 314. If so, the system proceeds to block 320 and triggers real-world implementation of the most suitable action set (e.g., with the best fitness metric). In some embodiments or circumstances, block 320 may include automatically triggering the real-world implementation. In some other embodiments or circumstances, block 320 may include first prompting a user to confirm the action set, and only triggering the real-world implementation if a confirmation is received in response. In other embodiments or circumstances, block 320 may include transmitting data reflecting the action set to one or more computing devices for presentation to the user for implementation by the computing device and / or future implementation.

[0079] If, at block 316, the system determines that no actions in the most recent iteration of block 314 are suitable, the system proceeds to block 318. At block 318, the system selects an alternative request embedding technique and / or an alternative action model. The system then returns to block 306 and performs another iteration of blocks 306, 308, 310, and optionally 312, 314, and 316 using the alternative request embedding technique and / or the alternative action model. This can continue until a suitable set of actions is determined or until other conditions are met (e.g., a threshold number of iterations are performed and / or embedding techniques and / or action model variations are exhausted).

[0080] The alternative request embedding technique and / or alternative action model selected at each iteration of block 318 is unique relative to any alternative request embedding technique and / or alternative action model already used in the current execution of method 300 for instant NL input. In some embodiments, the system can utilize heuristics and / or trained selection models in selecting the next best possible embedding technique and / or action model. For example, the defined heuristics can indicate which embedding technique and action model to utilize initially, which embedding technique and action model to utilize next, which embedding technique and action model to utilize after that, etc. As another example, the system can utilize a trained selection model (e.g., Figure 1 The system can select the next embedding technique and / or action model to be utilized based on the probabilities.

[0081] In some embodiments, the system can additionally and / or alternatively utilize data from box 308 to determine whether to adjust the most recent embedding technique or the most recent action model being utilized. For example, if the action model used in the most recent iteration indicates a threshold confidence in the generated candidate action set, but is still found to be unsuitable, this may indicate that the most recent request embedding is inaccurate, but the most recent action model may be the correct action model to utilize. In response, the system may select an alternative embedding technique. On the other hand, if the action model utilized in the most recent iteration indicates less than a threshold confidence in the generated candidate action set, this may indicate that the most recent action model may not be the correct action model to utilize. In response, the system may select an alternative action model.

[0082] Figure 4 is a flowchart illustrating another example method 400 for practicing selected aspects of the present disclosure in accordance with implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of the NeuroArena system 120. Furthermore, while the operations of the method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0083] Initially, it should be noted that Figure 4 The method 400 includes Figure 3 However, in method 400, multiple variants of the request embedding technique, action model combinations are utilized in parallel when generating a set of candidate actions.

[0084] At block 402 , the system receives NL input data reflecting a request to automatically generate an action for a task.

[0085] At block 404, the system selects N variants, where N is an integer greater than 1. Each variant includes a unique requested embedding technique and / or a unique action model to be utilized. The requested embedding technique and / or action model is unique in that the combination of the two does not exist in any other variant for the current iteration of block 404 or for a previous iteration of block 404 for the same request of block 402. As an example, the system can generate a first variant including a first embedding technique and a first action model, a second variant including a second embedding technique and a second action model, and a third variant including a third embedding technique and a first action model.

[0086] At block 406A, the system generates a request embedding using a first variant of the variant selected at the most recent iteration of block 404 .

[0087] At block 408A, the system processes the request embedding of block 406A using the action model of the first variant to generate a set of predicted actions.

[0088] At block 410A, for each of the action sets, the system implements the action set in a simulation.

[0089] At block 412A, the system determines, for each of the action sets and based on the simulation data from the simulation, a fitness metric for the action set.

[0090] Similarly, at block 406N, the system generates a request embedding using the Nth variant of the variant selected at the most recent iteration of block 404. Further, at block 408N, the system processes the request embedding of block 406N using the action model of the Nth variant to generate a predicted action set. Further, at block 410N, for each of the action sets, the system implements the action set in a simulation. Further, at block 412N, the system determines a fitness metric for the action set for each of the action sets and based on the simulation data from the simulation.

[0091] like Figure 4 As shown by the horizontal ellipsis in , in a specific implementation where more than two variants are generated at an iteration of box 404, for each of these variants, the system may similarly use the variant to generate a request embedding and a predicted action set, implement each of the predicted action sets in simulation, and use the corresponding simulation data to determine a corresponding fitness metric for the predicted action set.

[0092] At block 414, the system determines whether any of the candidate action sets are suitable based on the suitability metrics generated at blocks 412A-N. If not, the system returns to block 404. If yes, the system proceeds to block 416 and enables real-world implementation of the suitable candidate action sets.

[0093] Figure 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, the client device 110, the NeuroArena system 120, and / or other components can include one or more components of the example computing device 510.

[0094] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0095] User interface input devices 522 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 510 or over a communication network.

[0096] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include devices and methods of all possible types of information output to a user or another machine or computing device from computing device 510.

[0097] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a Figure 3 Method 300, Figure 4 The logic of method 400 and / or selected aspects of other method(s) described herein.

[0098] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor 514.

[0099] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0100] Computing device 510 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 510 may have more Figure 5 The computing devices depicted may have greater or fewer components.

[0101] Although several embodiments have been described and illustrated herein, various other devices and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using experiments that do not exceed routine. Therefore, it should be understood that the foregoing embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, embodiments may be practiced in a manner different from that specifically described and claimed. Embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

[0102] In some implementations, a method implemented by one or more processors is provided, and the method includes receiving natural language input data, the natural language input data reflecting a user request for automatically generating an action for performing a task according to a corresponding state of a domain. The method also includes generating a request embedding based on processing the natural language input data. The method also includes a simulation of performing a task by implementing a predicted action set generated based on processing the request embedding using one or more trained action models in a simulation environment reflecting the corresponding state of the domain. The method also includes determining based on the simulation that the predicted action set is not suitable for performing the task. The method also includes generating an alternative predicted action set for performing the task in response to determining that the predicted action set is not suitable for performing the task. In some specific implementations, generating an alternative predicted action set includes: utilizing an alternative request embedding when generating the alternative predicted action set, and / or utilizing at least one alternative trained action model when generating the alternative predicted action set. The method also includes determining that the alternative predicted action set is suitable for performing the task, and in response to determining that the alternative predicted action set is suitable for performing the task, transmitting data so that the alternative predicted action set is implemented in a real-world environment to perform the task.

[0103] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0104] In some implementations, the method further comprises generating a set of predicted actions based on processing the request embedding using the one or more trained action models. In some of those implementations, generating the set of predicted actions occurs before performing the simulation.

[0105] In some embodiments, the method further includes generating a predicted action set based on processing the request embedding using one or more trained action models. In some of those embodiments, generating the predicted action set occurs during execution of the simulation, and is further based on processing simulation state data generated during execution of the simulation using one or more trained action models. The simulation state data processed can depend on the (one or more) trained action models utilized. For example, some trained action models can be configured to process simulation state data including pixels from simulated simulated images, and other trained action models can be trained to process shapes and / or other features detected from the simulated images without processing the simulated images themselves.

[0106] In some implementations, determining that the alternative predicted action is suitable for performing the task includes: performing additional simulations of the task by implementing the alternative predicted action in a simulation environment; and determining that the alternative predicted action is suitable for performing the task based on the additional simulations.

[0107] In some implementations, determining that the predicted action is not suitable for performing the task based on the simulation includes: processing simulation data from the simulation using the predicted action to generate a natural language output describing the processed simulation data; generating a metric based on comparing the natural language output to the natural language input data; and determining that the predicted action is not suitable for performing the task based on the metric failing to meet a threshold. In some versions of those embodiments, the natural language output is natural language text or a natural language embedding. In some additional and / or alternative versions of those implementations, the simulation data includes a final state of a simulation environment from a simulation using the predicted action.

[0108] In some embodiments, determining based on the simulation that the predicted action is not suitable for performing the task includes: using the predicted action to process simulation data from the simulation to determine whether one or more domain or specific task rules are violated, and in response to determining that at least one of the one or more domain or specific task rules is violated: determining that the predicted action is not suitable for performing the task. In some versions of those implementations, the method also includes determining a specific action in the predicted action, the implementation of which in the simulation results in a violation of a given rule in the one or more domain-specific rules or specific task rules. In some versions of those versions, determining that an alternative predicted action is suitable for performing the task includes determining that the alternative predicted action lacks the specific action.

[0109] In some implementations, determining that a predicted action is not suitable for performing a task based on a simulation includes: causing simulation data from a simulation using the predicted action to be presented at a client device via which a user request is received; in response to causing the simulation data to be presented at the client device, receiving a user interface input provided at the client device; and determining that the predicted action is not suitable for performing the task based on the user interface input. In some versions of those embodiments, the simulation data includes a final state of a simulation environment from a simulation using the predicted action. In some of those versions, the method also includes determining a specific action in the predicted action based on a specific feature of the final state pointed to by the user interface input, the implementation of which in the simulation results in the specific feature. In some versions, determining that an alternative predicted action is suitable for performing the task includes determining that the alternative predicted action lacks the specific action.

[0110] In some implementations, generating the alternative predicted action set includes utilizing the at least one alternative trained action model and not utilizing the one or more trained action models utilized in generating the predicted action set.

[0111] In some alternative request embedding implementations, generating the alternative predicted action set includes using the alternative request embedding in generating the alternative predicted action set.

[0112] In some alternative request embedding embodiments: generating the request embedding includes generating a natural language embedding based on processing natural language input data using a language model, and generating the request embedding based on the natural language embedding; and generating the alternative request embedding includes: generating alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplementary items from a domain-specific knowledge base for the task; generating an alternative natural language embedding based on processing the alternative natural language input data using a language model; and generating the alternative request embedding based on the alternative natural language embedding. In those embodiments, one or more supplementary items are not used to generate the request embedding.

[0113] In some alternative request embedding embodiments: generating the request embedding includes generating a natural language embedding based on processing the natural language input data using a language model, and generating the request embedding based on the natural language embedding; and generating the alternative request embedding includes: generating alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplementary items from an external knowledge source, the external knowledge source not specific to the domain or the task; generating an alternative natural language embedding based on processing the alternative natural language input data using a language model; and generating the alternative request embedding based on the alternative natural language embedding. In those embodiments, one or more supplementary items are not used to generate the request embedding.

[0114] In some alternative request embedding implementations: generating the alternative request embedding includes causing a clarification prompt to be presented at a client device via which the user request is received, receiving user feedback provided in response to the clarification prompt and via one or more user interface inputs at the client device, and generating the alternative request embedding based on processing the user feedback. In those embodiments, the user feedback is not utilized in generating the request embedding.

[0115] In some alternative request embedding implementations, generating the alternative request embedding includes generating a context embedding based on processing context data, and further generating the alternative request embedding based on the context embedding. In those implementations, the context data is not utilized when generating the request embedding.

[0116] In some alternative request embedding implementations, generating the request embedding includes: generating the request embedding based on processing natural language input data and processing first context data without processing second context data; and generating the alternative request embedding includes: generating the request embedding based on processing the natural language input data and processing the second context data without processing the first context data. In some versions of those implementations, the first context data represents a current state of the domain at a first level of abstraction, and the second context data represents a current state of the domain at a second level of abstraction. In some of those versions, the current state of the domain includes an image being rendered, and the first level of abstraction is a pixel-level abstraction, and the second level of abstraction is a shape-level abstraction.

[0117] In some implementations, generating the alternative predicted action set includes processing the alternative request embeddings using the at least one alternative trained action model when generating the alternative predicted action set.

[0118] In some embodiments, the method further includes determining, based on the one or more probabilities, whether to utilize an alternative request embedding or to alternatively utilize at least one alternative trained action model when generating an alternative predicted action set. In some of those embodiments, the one or more probabilities are used to predict the action set and are generated based on processing the request embedding using the one or more trained action models.

[0119] In some embodiments, determining whether to utilize the alternative request embedding or to alternatively utilize the at least one alternative trained action model in generating the alternative predicted action set based on the one or more probabilities includes: in response to the one or more probabilities satisfying one or more thresholds indicating at least a threshold confidence in the predicted action set, determining to utilize the alternative request embedding. In some of those embodiments, determining whether to utilize the alternative request embedding or to alternatively utilize the at least one alternative trained action model in generating the alternative predicted action set based on the one or more probabilities includes: in response to the one or more probabilities failing to satisfy one or more thresholds indicating at least a threshold confidence in the predicted action set, determining to utilize the alternative trained action model.

[0120] In some implementations, the predicted action set includes a first set of actions in a first sequence, and the alternative action set includes the first set of actions in a second sequence or the second set of actions in a third sequence.

[0121] In some embodiments, transmitting data so that an alternative set of predicted actions is implemented in a real-world environment to perform a task includes transmitting data so that the alternative set of predicted actions is automatically implemented in response to a user request and is automatically implemented without requiring any further user input after the user request is provided.

[0122] In some embodiments, a method implemented by one or more processors is provided, and the method includes receiving a natural language input, the natural language input including a request to automatically generate an action for performing a task based on a corresponding state of a domain. The method also includes generating a first request embedding based on performing a first process. The first process is based at least on the natural language input. The method also includes processing the first request embedding using at least one trained action model to generate a first predicted action for performing the task. The method also includes performing a first simulation of the task, the first simulation implementing the first predicted action in a simulated environment reflecting the corresponding state of the domain. The method also includes determining a first fitness metric for the first predicted action based on the first simulation. The method also includes generating a second request embedding different from the first request embedding based on performing a second process. The second process is based at least on the natural language input. The method also includes processing the second request embedding using at least one trained action model or at least one alternative trained action model to generate a second predicted action for performing the task. The method also includes performing a second simulation of the task, the second simulation implementing the second predicted action in a simulated environment reflecting the corresponding state of the domain. The method also includes determining a second fitness metric for the second action based on the second simulation. The method also includes determining to implement the second action in place of the first action based on comparing the first fitness metric with the second fitness metric. The method also includes, in response to determining to implement the second action, transmitting data to cause the second action to be implemented in the real-world environment to perform the task.

[0123] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0124] In some implementations, the method further includes determining that the first fitness metric does not satisfy a threshold. In some of these implementations, after determining that the first fitness metric does not satisfy the threshold and in response to determining that the first fitness metric does not satisfy the threshold, performing one or more of: generating a second request embedding; processing the second request embedding to generate the second predicted action; performing the second simulation; or determining the second fitness metric.

[0125] In some implementations, one or more of the following is performed prior to or in parallel with determining the first fitness metric: generating a second request embedding; processing the second request embedding to generate the second predicted action; performing the second simulation; and / or determining a second fitness metric.< / x> < / x> < / x> < / x> < / x> < / y> < / y> < / x>

Claims

1. A method implemented by one or more processors, the method comprising: receiving natural language input data reflecting a user request for automatically generating an action for performing a task according to a corresponding state of the domain; generating a request embedding based on processing the natural language input data; performing a simulation of the task by implementing, in a simulation environment reflecting a corresponding state of the domain, a set of predicted actions generated based on processing the request embedding using one or more trained action models; determining, based on the simulation, that the predicted action set is not suitable for performing the task; In response to determining that the predicted action set is not suitable for performing the task: Generating an alternative predicted action set for performing the task, wherein generating the alternative predicted action set comprises: utilizing alternative request embeddings in generating the alternative predicted action set, and / or utilizing at least one alternative trained motion model in generating the alternative set of predicted motions; determining that the set of alternative predicted actions is suitable for performing the task; In response to determining that the alternative set of predicted actions is suitable for performing the task: Data is transmitted such that the alternative set of predicted actions is implemented in a real-world environment to perform the task.

2. The method according to claim 1, further comprising: generating the set of predicted actions based on processing the request embedding using the one or more trained action models, Wherein, generating the predicted action set occurs before executing the simulation.

3. The method according to claim 1, further comprising: generating the set of predicted actions based on processing the request embedding using the one or more trained action models, Wherein generating the predicted action set occurs during execution of the simulation and is further based on processing simulation state data generated during execution of the simulation using the one or more trained action models.

4. A method according to any preceding claim, wherein: Determining that the alternative predicted action is suitable for performing the task includes: performing additional simulation of the task by implementing the alternative predicted actions in the simulation environment; and The alternative predicted action is determined to be suitable for performing the task based on the additional simulation.

5. A method according to any one of the preceding claims, wherein: Determining, based on the simulation, that the predicted action is not suitable for performing the task includes: processing simulation data from the simulation using the predicted actions to generate a natural language output describing the processed simulation data; generating a metric based on comparing the natural language output to the natural language input data; and Based on the metric failing to satisfy a threshold, it is determined that the predicted action is not suitable for performing the task.

6. The method according to claim 5, wherein: The natural language output is natural language text or natural language embedding.

7. The method according to claim 5, wherein: The simulation data includes a final state of the simulation environment from the simulation using the predicted action.

8. A method according to any one of the preceding claims, wherein: Determining, based on the simulation, that the predicted action is not suitable for performing the task includes: processing simulation data from the simulation using the predicted actions to determine whether one or more domain or task-specific rules are violated; In response to determining that at least one of the one or more domain or specific task rules is violated: A determination is made that the predicted action is inappropriate for performing the task.

9. The method according to claim 6, further comprising: determining a specific action among the predicted actions, implementation of the specific action in the simulation causing a given one of the one or more domain or task-specific rules to be violated; Wherein, determining that the alternative predicted action is suitable for performing the task includes determining that the alternative predicted action lacks the particular action.

10. A method according to any one of the preceding claims, wherein: Determining, based on the simulation, that the predicted action is not suitable for performing the task includes: causing simulation data from the simulation using the predicted action to be presented at a client device via which the user request was received; In response to causing the simulation data to be presented at the client device, receiving a user interface input provided at the client device; and Based on the user interface input, it is determined that the predicted action is not suitable for performing the task.

11. The method according to claim 10, wherein: The simulation data includes a final state of the simulation environment from the simulation using the predicted action.

12. The method according to claim 11, further comprising: Determine a specific action in the predicted actions based on the specific feature of the final state pointed to by the user interface input, the implementation of the specific action in the simulation leading to the specific feature; Wherein, determining that the alternative predicted action is suitable for performing the task includes determining that the alternative predicted action lacks the particular action.

13. A method according to any one of the preceding claims, wherein: Generating the alternative predicted action set includes utilizing the at least one alternative trained action model and not utilizing the one or more trained action models utilized in generating the predicted action set.

14. A method according to any one of the preceding claims, wherein: Generating the alternative predicted action set includes using the alternative request embeddings in generating the alternative predicted action set.

15. The method according to claim 14, in, Generating the request embedding includes: generating a natural language embedding based on processing the natural language input data using a language model; and generating the request embedding based on the natural language embedding; Wherein, generating the replacement request embedding includes: generating alternative natural language input data by modifying and / or supplementing the natural language input data using one or more supplementary terms from a task-specific domain knowledge base; generating an alternative natural language embedding based on processing the alternative natural language input data using a language model; and generating the alternative request embedding based on the alternative natural language embedding; Wherein, the one or more supplementary items are not used to generate the requested embedding.

16. The method according to claim 14, in, Generating the request embedding includes: generating a natural language embedding based on processing the natural language input data using a language model; and generating the request embedding based on the natural language embedding; Wherein, generating the replacement request embedding includes: generating alternative natural language input data by modifying and / or supplementing the natural language input data with one or more supplements from an external knowledge source that is not specific to the domain or the task; generating an alternative natural language embedding based on processing the alternative natural language input data using a language model; and generating the alternative request embedding based on the alternative natural language embedding; and wherein the one or more supplemental terms are not used to generate the request embedding.

17. The method according to claim 14, wherein: Generating the replacement request embedding includes: causing presentation of a clarification prompt at a client device via which the user request was received; receiving user feedback responsive to the clarification prompt and provided via one or more user interface inputs at the client device; and generating the alternative request embedding based on processing the user feedback; and Wherein, the user feedback is not used to generate the request embedding.

18. The method according to claim 14, wherein: Generating the replacement request embedding includes: generating a contextual embedding based on processing the contextual data; and generating the alternative request embedding further based on the context embedding; Wherein, the context data is not used to generate the request embedding.

19. The method according to claim 14, in, Generating the request embedding includes: generating the request embedding based on processing the natural language input data and processing the first context data without processing the second context data; and Wherein, generating the replacement request embedding includes: The request embedding is generated based on processing the natural language input data and processing the second context data and without processing the first context data.

20. The method according to claim 19, wherein: The first context data represents a current state of the domain at a first level of abstraction, and the second context data represents a current state of the domain at a second level of abstraction.

21. The method according to claim 20, wherein: The current state of the domain includes an image being rendered, and the first level of abstraction is a pixel level of abstraction, and the second level of abstraction is a shape level of abstraction.

22. The method according to claim 1, wherein: Generating the alternative predicted action set includes: In generating the set of alternative predicted actions, the alternative request embeddings are processed using the at least one alternative trained action model.

23. The method according to any preceding claim, further comprising: determining whether to utilize the alternative request embedding or alternatively utilize the at least one alternative trained action model in generating the alternative set of predicted actions based on one or more probabilities, Wherein the one or more probabilities are for the predicted action set and are generated based on processing the request embedding using the one or more trained action models.

24. The method according to claim 23, wherein: Determining whether to utilize the alternative request embedding or alternatively utilize the at least one alternative trained action model in generating the alternative set of predicted actions based on the one or more probabilities includes: Responsive to the one or more probabilities satisfying one or more thresholds indicating at least a threshold confidence in the set of predicted actions, a determination is made to utilize the alternative request embedding.

25. The method according to claim 24, wherein: Determining whether to utilize the alternative request embedding or alternatively utilize the at least one alternative trained action model in generating the alternative set of predicted actions based on the one or more probabilities includes: Responsive to the one or more probabilities failing to satisfy one or more thresholds indicative of at least a threshold confidence in the set of predicted actions, a determination is made to utilize the alternative trained action model.

26. A method according to any one of the preceding claims, wherein the set of predicted actions comprises a first group of actions in a first sequence, and wherein, The alternative set of actions includes the first set of actions in the second sequence or the second set of actions in the third sequence.

27. A method according to any one of the preceding claims, wherein: Transmitting the data so that the alternative set of predicted actions is implemented in the real-world environment to perform the task comprises: The data is sent such that the alternative predicted action is automatically implemented in response to the user request and is automatically implemented without requiring any further user input after the user request is provided.

28. A method implemented by one or more processors, the method comprising: receiving a natural language input, the natural language input comprising a request to automatically generate an action for performing a task according to a corresponding state of the domain; generating a first request embedding based on performing a first process, the first process being based on at least the natural language input; processing the first request embedding using at least one trained action model to generate a first predicted action for performing the task; executing a first simulation of the task, the first simulation implementing the first predicted action in a simulation environment reflecting a corresponding state of the domain; determining a first suitability measure for the first predicted action based on the first simulation; generating a second request embedding different from the first request embedding based on performing a second process, the second process being based on at least the natural language input; processing the second request embedding using the at least one trained action model or at least one alternative trained action model to generate a second predicted action for performing the task; executing a second simulation of the task, the second simulation implementing the second predicted action in the simulation environment reflecting the corresponding state of the domain; determining a second suitability metric for the second action based on the second simulation; determining to implement the second action in place of the first action based on comparing the first fitness metric and the second fitness metric; and In response to determining to implement the second action: Data is transmitted to cause the second action to be implemented in a real-world environment to perform the task.

29. The method according to claim 28, further comprising: determining that the first fitness metric does not satisfy a threshold, wherein, after determining that the first fitness metric does not satisfy the threshold and in response to determining that the first fitness metric does not satisfy the threshold, performing one or more of the following: Generate the second request embedding, processing the second request embedding to generate the second predicted action, Perform the second simulation, or The second fitness metric is determined.

30. The method according to claim 28 or 29, wherein: Prior to or in parallel with determining the first fitness metric, performing one or more of the following: Generate the second request embedding, processing the second request embedding to generate the second predicted action, Perform the second simulation, or The second fitness metric is determined.

31. A computer readable storage medium storing computer executable instructions which, when executed by one or more processors, cause the one or more processors to perform a method according to any preceding claim.

32. A system comprising one or more processors and a memory storing instructions executable by the one or more processors, the instructions when executed causing the one or more processors to perform the method according to any one of claims 1 to 30.

Images