Automating semantically related computing tasks across contexts

Abstract

The disclosed embodiments relate to automating semantically similar compute tasks across multiple contexts. In various embodiments, an initial natural language input and a first plurality of actions performed using a first computer application may be used to generate a first task embedding and a first action embedding in an action embedding space. An association between the first task embedding and the first action embedding may be stored. Later, a subsequent natural language input may be used to generate a second task embedding, which is then matched to the first task embedding. Based on the stored associations, the first action embedding may be identified and processed using the selected domain model to select an action to be performed using the second computer application. The selected domain model may be trained to transition between an action space and an action embedding space of the second computer application.

Description

Background Art

[0001] Individuals often operate computing devices to perform semantically similar tasks in different contexts. For example, an individual may use a first computer application to engage in a sequence of actions to perform a given task, such as setting various application preferences, retrieving / viewing specific data accessible by the first computer application, performing a sequence of operations within a specific domain (e.g., 3D modeling, graphics editing, word processing), etc. The same individual may later engage in a semantically similar but syntactically different sequence of actions to perform the same or semantically similar tasks in different contexts, such as when using different computer applications. Repeatedly performing actions that include these tasks may be cumbersome, prone to errors, and may unnecessarily consume computing resources and / or the individual's attention.

[0002] Many computer applications provide users with the option of recording action sequences so that these actions can be automated, for example, using a scripting language embedded in the computer application. Sometimes these recorded sequences are referred to as "macros". However, the action sequences of these records and / or the scripts they generate may suffer from various disadvantages. They are often limited to operations within a specific computer application and are usually narrowly customized to adapt to very specific contexts. In addition, the scripts that are their basis are often too complex to be understood by people who are not familiar with computer programming, let alone manipulated. Summary of the invention

[0003] Embodiments for automating semantically similar computing tasks across multiple contexts are described herein. More specifically, but not exclusively, embodiments for enabling individuals (commonly referred to as "users") to allow or request that they perform a sequence of actions to perform or complete a task in one context (e.g., in a given computer application, in a given domain, etc.) are captured (e.g., recorded) and seamlessly extended to other contexts without requiring programming knowledge are described herein. In various embodiments, the captured action sequence can be abstracted as an "action embedding" in a generalized "action embedding space". This domain-agnostic action embedding can abstractly represent a "semantic task", which can be converted into the action space of any number of domains using the corresponding domain model. In other words, a "semantic task" is a domain-agnostic higher-order task that looks for expressions of a sequence / multiple domain-specific actions as domain-specific actions within a specific domain.

[0004] Along with the captured action sequences (which are captured with the user's permission or at their request as described above), the individual may provide natural language input, e.g., spoken or typed, that provides additional semantic context to these captured action sequences. Natural language processing (NLP) may be performed on these natural language inputs to generate "task" or "policy" embeddings, which may then be associated with the action embeddings created simultaneously. The individual may then subsequently provide natural language input in different contexts that may be matched to one or more task / policy embeddings. The matched task / policy embeddings may be used to identify corresponding action embeddings in a generalized action embedding space. These corresponding action embeddings may be processed using a domain model associated with the current domain / context in which the individual operates to select multiple actions from the action space of the current domain that may be syntactically different from, but semantically equivalent to, the original action sequences captured in a previous domain.

[0005] In some embodiments, a method may be implemented using one or more processors and may include: obtaining an initial natural language input and a first plurality of actions performed using a first computer application; performing natural language processing (NLP) on the initial natural language input to generate a first task embedding, the first task embedding representing a first task conveyed by the initial natural language input; processing the first plurality of actions using a first domain model to generate a first action embedding representing the first plurality of actions performed using the first computer application, wherein the first domain model is trained to convert between an action space of the first computer application and an action embedding space including the first action embedding; storing an association between the first task embedding and the first action embedding in a memory; performing NLP on a subsequent natural language input to generate a second task embedding, the second task embedding representing a second task conveyed by the subsequent natural language input; determining, based on a similarity measure between the first task embedding and the second task embedding, that the second task semantically corresponds to the first task; in response to the determination, processing the first action embedding using a second domain model to select a second plurality of actions to be performed using a second computer application, wherein the second domain model is trained to convert between the action space of the second computer application and the action embedding space; and causing the second plurality of actions to be performed using the second computer application.

[0006] In various embodiments, at least one of the first computer application and the second computer application may be an operating system. In various embodiments, the first plurality of actions performed using the first computer application may be intercepted from data exchanged between the first computer application and the underlying operating system. In various embodiments, the exchanged data may include data indicative of keystrokes and pointing device input.

[0007] In various embodiments, the first plurality of actions performed using the first computer application may be captured from an application programming interface (API) of the first computer program. In various embodiments, the first plurality of actions performed using the first computer application may be captured from a domain-specific programming language associated with the first domain. In various embodiments, the first plurality of actions performed using the first computer application may be captured from a scripting language embedded in the first computer application.

[0008] In various implementations, a first plurality of actions performed using a first computer application may include interactions with a first graphical user interface (GUI) rendered by the first computer application. In various implementations, a second plurality of actions performed using a second computer application may include interactions with a second GUI rendered by the second computer application.

[0009] In various embodiments, the first computer application may be operable to exchange data with a first database having a first database schema, and the second computer application may be operable to exchange data with a second database having a second database schema different from the first database schema. In various embodiments, the first plurality of actions may interact with first data from the first database according to the first database schema, and the second plurality of actions may interact with second data from the second database according to the second database schema, and the second data may semantically correspond to the first data.

[0010] In various embodiments, the first computer application may be a first communication application that has been operated to communicate with a first plurality of contacts, and the second computer application may be a second communication application that has been operated to communicate with a second plurality of contacts. In various embodiments, the second task may seek past communications with one or more contacts included in the second plurality of contacts. In various embodiments, the second task may also seek past communications with one or more contacts included in the first plurality of contacts.

[0011] In various embodiments, the first computer application may be operable to exchange data with a first database having a first database schema, and the second computer application may be operable to exchange data with a second database having a second database schema different from the first database schema. In various embodiments, the first plurality of actions may interact with first data from the first database according to the first database schema, and the second plurality of actions may interact with second data from the second database according to the second database schema, and the second data may semantically correspond to the first data.

[0012] In another aspect, a method implemented using one or more processors may include: obtaining an initial natural language input and a first plurality of actions performed using a first input form configured for a first domain; performing NLP on the initial natural language input to generate a first policy embedding, the first policy embedding representing a first input policy conveyed by the initial natural language input; processing the first plurality of actions using a first domain model to generate a first action embedding, the first action embedding representing the first plurality of actions performed using the first input form, wherein the first domain model is trained to convert between an action space of the first domain and an action embedding space including the first action embedding; storing an association between the first policy embedding and the first action embedding in a memory; performing NLP on a subsequent natural language input to generate a second policy embedding, the second policy embedding representing a second policy conveyed by the subsequent natural language input; determining, based on a similarity measure between the first policy embedding and the second policy embedding, that the second policy semantically corresponds to the first policy; in response to the determination, processing the first action embedding using a second domain model to select a second plurality of actions to be performed using a second input form configured for a second domain, wherein the second domain model is trained to convert between the action space of the second domain and the action embedding space; and causing the second plurality of actions to be performed using the second input form. In various implementations, the first plurality of actions may include populating a first plurality of form fields with a first set of values, and the second plurality of actions may include populating a second plurality of form fields with at least some of the first set of values.

[0013] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause performance of any of the foregoing methods. Some embodiments include at least one non-transitory computer-readable storage medium storing computer instructions executable by the one or more processors to perform any of the foregoing methods.

[0014] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram of an example environment in which implementations disclosed herein may be implemented.

[0016] Figure 2 Examples of how data may be exchanged and / or processed to extend tasks performed in one domain to additional domains are schematically illustrated according to various embodiments.

[0017] Figure 3 Another example of how data may be processed to perform a single task across multiple domains in accordance with various embodiments is schematically illustrated.

[0018] Figure 4A and Figure 4B Examples of how the techniques described herein may be used to automatically fill in input forms are shown in accordance with various implementations.

[0019] Figure 5A and Figure 5B Another example of how the techniques described herein may be used to automatically fill in an input form is shown in accordance with various implementations.

[0020] Figure 6 is a flow chart illustrating an example method of practicing selected aspects of the present disclosure according to embodiments disclosed herein.

[0021] Figure 7 is a flow chart illustrating another example method of practicing selected aspects of the present disclosure according to embodiments disclosed herein.

[0022] Figure 8 is a flow chart illustrating another example method of practicing selected aspects of the present disclosure according to embodiments disclosed herein.

[0023] Fig. 9 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0024] Embodiments for automating semantically similar computing tasks across multiple contexts are described herein. More specifically, but not exclusively, embodiments for enabling individuals (commonly referred to as "users") to allow or request that they perform a sequence of actions to perform or complete a task in one context (e.g., in a given computer application, in a given domain, etc.) are captured (e.g., recorded) and seamlessly extended to other contexts without requiring programming knowledge are described herein. In various embodiments, the captured action sequence can be abstracted as an "action embedding" in a generalized "action embedding space". This domain-agnostic action embedding can abstractly represent a "semantic task", which can be converted into the action space of any number of domains using the corresponding domain model. In other words, a "semantic task" is a domain-agnostic higher-order task that looks for expressions of a sequence / multiple domain-specific actions as domain-specific actions within a specific domain.

[0025] Along with the captured action sequences (which are captured with the user's permission or at their request as described above), the individual may provide natural language input, e.g., spoken or typed, that provides additional semantic context to these captured action sequences. Natural language processing (NLP) may be performed on these natural language inputs to generate "task" or "policy" embeddings, which may then be associated with the action embeddings created simultaneously. The individual may then subsequently provide natural language input in different contexts that may be matched to one or more task / policy embeddings. The matched task / policy embeddings may be used to identify corresponding action embeddings in a generalized action embedding space. These corresponding action embeddings may be processed using a domain model associated with the current domain / context in which the individual operates to select multiple actions from the action space of the current domain that may be syntactically different from, but semantically equivalent to, the original action sequences captured in a previous domain.

[0026] As a non-limiting example, a user may authorize a local agent computer program (referred to herein as an "automation agent") to capture a series of operations performed by a user using a graphical user interface (GUI) of a first computer application to set various application parameters, such as setting visual parameters to "dark mode", setting application permissions (e.g., location, camera access, etc.), or other application preferences (e.g., Celsius vs. Fahrenheit, metric vs. imperial, preferred fonts, preferred sort order, etc.). Many of these various application parameters may not be unique to that particular computer application—other computer applications with similar functionality may have semantically similar application parameters. However, the semantically similar application parameters of other computer applications may be named, organized, and / or accessed differently (e.g., different submenus, command line input, etc.).

[0027] Using the techniques described herein, a user can provide natural language input to describe a sequence of actions performed using a GUI of a first computer application, e.g., while performing the action, or immediately before or after performing the action. A first task / policy embedding generated from the NLP of the input can be associated with (e.g., mapped to, combined with) a first action embedding generated from a captured sequence of actions using a first domain model. As previously described, the first domain model can convert between a general action embedding space and an action space of a first computer application.

[0028] Later, when operating a second computer application having similar functionality to the first computer application, the user may provide a semantically similar natural language input. A second task / strategy embedding generated from this subsequent natural language input may match the first task / strategy embedding, and therefore the first action embedding. The first action embedding may then be processed using a second domain model that converts between the general action embedding space and the action space of the second computer application to select actions to be performed at the second computer application. In some embodiments, these selected actions may be automatically performed, and the user may then be prompted to provide feedback on the resulting state of the second computer application. This feedback may be used, for example, to train the second domain model.

[0029] The technology described herein is not limited to automating semantically similar tasks across different computer applications. Other types of different contexts and domains are contemplated. For example, a sequence of actions performed by a user to fill in the input fields of a first input form (e.g., a webpage for ordering takeout) can be captured at the user's request and associated with an "input strategy" conveyed in a natural language input provided by the user. The task / strategy embedding generated from the user's natural language input can provide constraints, rules, and / or other data parameters that the user wishes to retain to extend to other domains. When the user later fills in another input form in a different domain (e.g., grocery delivery), the user can provide a natural language input that conveys the same strategy, which can cause at least some of the input fields of the new input form to be filled in with values ​​from the previous form fill. In this way, the user can, for example, create multiple different purchasing strategies or profiles that the user can select in different contexts (e.g., one for making personal purchases, another for making business purchases, another for making travel purchases, etc.).

[0030] Abstracting both the captured action sequences and the accompanying natural language input can provide many technical advantages. When the action sequences performed by an individual can be abstracted into semantically rich action embeddings that capture so much of the individual's intent, the individual does not have to provide long and detailed natural language input. Therefore, an individual can name an automated action with a word or a short phrase, and the association between the word / phrase and the corresponding action embedding still provides sufficient semantic context for cross-domain automation.

[0031] As with many artificial intelligence models, the more training data there is for training domain models, the more accurately they will translate between various domains and action embedding spaces. Human-provided feedback, such as previously described, can provide particularly valuable training data for supervised training, but may not be available in large quantities due to its cost. Therefore, in various embodiments, additional "synthetic" training data can be generated and used to train domain models in a process variably referred to herein as "self-supervised training" and "simulation." These synthetic training data can, for example, include automated variations and / or permutations of user recordings that are automatically generated and processed using domain models. The resulting "synthetic" results can be evaluated, for example, against automated "ground truth" results of the original user recordings and / or against natural language input provided by the user to determine errors. These errors can be used to train domain models, for example, using techniques such as backpropagation and gradient descent.

[0032] As an example, assume that an individual provides relatively simple and / or undetailed natural language input, such as words or short phrases, to describe a sequence of actions that they request to be recorded in a particular domain. Separate from the individual providing feedback on "ground truth" results that extend those recorded actions to a different domain, additional synthetic training data can be generated and used to generate synthetic results that extend those recorded actions to a different domain.

[0033] For example, short words / phrases provided by individuals can be used to generate and / or select longer, more detailed, and / or semantically similar synthetic natural language inputs. The process can then be reversed: the synthetic natural language inputs can be processed using NLP to generate synthetic task / policy embeddings, which in turn can be processed as described herein to select action embeddings in one or more domains and generate synthetic results. These synthetic results can be compared to baseline real results in the same domains, and / or feedback on these synthetic results can be solicited from individuals in order to train domain models for those domains.

[0034] As used herein, a "domain" may refer to a target subject area in which a computing component is intended to operate, e.g., the scope of knowledge, influence, and / or activity around which the logic of the computing component is centered. In some embodiments, the domain in which the task is to be extended may be identified by heuristically matching keywords in the user-provided input with domain keywords. In other embodiments, the user-provided input may be processed, for example, using NLP techniques such as word2vec, bidirectional encoder representations from transformers (BERT) transformers, various types of recurrent neural networks ("RNNs", e.g., long short-term memory or "LSTM", gated recurrent units or "GRU"), etc. to generate a semantic embedding representing a natural language input. In some embodiments, the natural language input semantic embedding (which, as previously described, may also be used as a "task" or "policy" embedding) may be used to identify one or more domains, e.g., based on the distance in the embedding space between the semantic embedding and other embeddings associated with various domains.

[0035] In various embodiments, one or more domain models may have been previously generated for each domain. For example, one or more machine learning models, such as RNNs (e.g., LSTM, GRU), BERT transformers, various types of neural networks, reinforcement learning strategies, etc., may be trained based on a document corpus associated with the domain. As a result of this training, the one or more domain models may be at least guided so that they can be used to process what will be referred to herein as "action embeddings" to select multiple candidate computational actions from the action space associated with the target domain for automation.

[0036] Figure 1 An example environment is schematically depicted in which selected aspects of the present disclosure may be implemented according to various implementations. Figure 1 Any computing device depicted in or elsewhere in the figures may include logic, such as one or more microprocessors (e.g., a central processing unit or “CPU,” a graphics processing unit or “GPU,” a tensor processing unit or (“TPU”)) that execute computer-readable instructions stored in a memory, or other types of logic, such as an application specific integrated circuit (“ASIC”), a field programmable gate array (“FPGA”), etc. Figure 1 Some of the systems depicted in the examples, such as the semantic task automation system 102, may be implemented using one or more server computing devices forming what is sometimes referred to as a "cloud infrastructure," but this is not required. In other implementations, aspects of the semantic task automation system 102 may be implemented on a client device 120, e.g., for purposes of protecting privacy, reducing latency, etc.

[0037] The semantic task automation system 102 may include a number of different components configured with selected aspects of the present disclosure, such as a domain module 104, an interface module 106, and a machine learning ( Figure 1 The semantic task automation system 102 may also include any number of databases for storing machine learning model weights and / or other data for performing selected aspects of the present disclosure. Figure 1 In, for example, the semantic task automation system 102 includes a database 110 storing a global domain model and another database 112 storing data indicating global action embeddings.

[0038] The semantic task automation system 102 may be operably coupled to any number of client computing devices operated by any number of users via one or more computer networks (114). Figure 1 In, for example, a first user 118-1 operates one or more client devices 120-1. A p-th user 118-P operates one or more client devices 120-P. As used herein, client devices 120 may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which in some cases may include a visual sensor and / or a touch screen display), a smart appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0039] The domain module 104 may be configured to determine various different information about domains that are relevant to a given user 118 at a given point in time, such as a domain in which the user 118 is currently operating, a domain into which the user would like to extend a semantic task, etc. To this end, the domain module 104 may collect contextual information about, for example, foreground and / or background applications executing on a client device 120 operated by the user 118, web pages currently / recently visited by the user 118, domains that the user 118 has visited and / or frequently visits, etc.

[0040] Using this collected context information, in some implementations, the domain module 104 can be configured to identify one or more domains associated with the natural language input provided by the user. For example, a request to record a task performed by a user 118 using a particular computer application and / or on a particular input form can be processed by the domain module 104 to identify the domain in which the user 118 performed the task to be recorded, which can be the domain of the particular computer application or input form. If the user 118 later requests to perform the same task in a different target domain, e.g., using a different computer application or a different input form, the domain module 104 can identify the target domain.

[0041] In some embodiments, the domain module 104 may also be configured to retrieve domain knowledge from various different sources associated with the identified domain. In some such embodiments, such as in addition to the previously mentioned natural language input or context information, this retrieved domain knowledge (and / or embeddings generated therefrom) may be provided to downstream components. This additional domain knowledge may allow downstream components (particularly machine learning models) to make predictions that are more likely to be satisfactory (e.g., to extend semantic tasks across different domains).

[0042] In some embodiments, the domain module 104 may apply the collected contextual information (e.g., current state) across one or more "domain selection" machine learning models 105 that are distinct from the domain models described herein. These domain selection machine learning models 105 may take various forms, such as various types of neural networks, support vector machines, random forests, BERT transformers, and the like. In various embodiments, the domain selection machine learning model 105 may be trained to select applicable domains based on attributes (or "context signals") of the current context or state of the user 118 and / or client device 120. For example, if the user 118 is operating an input form of a particular website to purchase goods or services, the uniform resource locator (URL) of the website or attributes of the underlying web page (such as keywords, tags, document object model (DOM) elements, and the like) may be applied as input across models in their native form or as a dimensionality reduction embedding. Other context signals that may be considered include, but are not limited to, the user's IP address (e.g., work vs. home vs. mobile IP addresses), time of day, social media status, calendar, email / text messaging content, and the like.

[0043] The interface module 106 may provide one or more graphical user interfaces (GUIs) that may be operated by various individuals, such as users 118-1 through 118-P, to perform various actions made available by the semantic task automation system. In various implementations, the user 118 may operate a GUI (e.g., a standalone application or web page) provided by the interface module 106 to opt-in or opt-out of the various techniques described herein. For example, before any tasks performed by the user 118-1 through 118-P using the client device 120-1 through 120-P are recorded and automated, the user 118-1 through 118-P may be required to provide explicit permission, as described herein.

[0044] The ML module 108 may access data indicative of various global domain / machine learning models / strategies in the database 110. These trained global domain / machine learning models / strategies may take various forms, including but not limited to graph-based networks (such as graph neural networks (GNNs), graph attention neural networks (GANNs), or graph convolutional neural networks (GCNs)), sequence-to-sequence models (such as encoder-decoders), various forms of recurrent neural networks (e.g., LSTM, GRU, etc.), BERT transformer networks, reinforcement learning strategies, and any other type of machine learning model that may be applied to facilitate selected aspects of the present disclosure. The ML module 108 may process various data based on these machine learning models at the request or command of other components such as the domain module 104 and / or the interface module 106.

[0045] Each client device 120 may operate at least a portion of what will be referred to herein as an "automation agent" 122. The automation agent 122 may be a computer application that may be operated by the user 118 to perform selected aspects of the present disclosure to facilitate extension of semantic tasks across different domains. For example, the automation agent 122 may receive a request and / or permission from the user 118 to record a sequence of actions performed by the user 118 using the client device 120 in order to complete a certain task. Without such explicit permission, the automation agent 122 may not be able to monitor the user's activities.

[0046] In some embodiments, the automated agent 122 may take the form of what is commonly referred to as a "virtual assistant" or "automated assistant" that is configured to engage in a human-machine natural language conversation with the user 118. For example, the automated agent 122 may be configured to semantically process natural language input provided by the user 118 to identify one or more intents. Based on these intents, the automated agent 122 may perform various tasks, such as operating smart appliances, retrieving information, performing tasks, and the like. In some embodiments, the conversation between the user 118 and the automated agent 122 (or a separate automated assistant accessible to / by the automated agent 122) may constitute a task sequence, which may be captured, abstracted into a domain-agnostic embedding, and then extended to other domains, as described herein.

[0047] For example, a human-machine conversation between a user 118 and an automated agent 122 (or a separate automated assistant, or even between an automated assistant and a third-party application) for ordering a pizza from a third-party agent at a first restaurant (and therefore, a first domain) can be captured and used to generate an "order pizza" action embedding. This action embedding can later be expanded to ordering pizza from a different restaurant, for example, via an automated assistant or via a separate interface.

[0048] exist Figure 1 , each of the client devices 120-1 may include an automation agent 122-1 that serves a first user 118-1. The first user 118-1 and his / her automation agent 122-1 may access and / or may be associated with a "profile" that includes various data related to performing selected aspects of the present disclosure on behalf of the first user 118-1. For example, the automation agent 122 may access one or more edge databases or data stores associated with the first user 118-1, including an edge database 124-1 that stores local domain models and action embeddings and / or another edge database 126-1 that stores recorded actions. Other users 118 may have similar arrangements. Any data stored in the edge databases 124-1 and 126-1 may be stored in part or in whole on the client device 120-1, for example, to protect the privacy of the first user 118-1. For example, the recorded actions 126 - 1 may be stored locally on the client device 120 - 1 in its original form, and the recorded actions 126 - 1 may include sensitive and / or personal information of the first user 118 - 1 , such as payment information, address, phone number, etc.

[0049] The local domain models stored in the edge database 124-1 may include, for example, local versions of global models stored in the global domain model database 110. For example, in some embodiments, global models may be propagated to the edges in order to guide the automated agents 122 to expand tasks into new domains associated with those propagated models; thereafter, the local models at the edges may or may not be trained locally based on the activities and / or feedback of the users 118. In some such embodiments, the local models (alternatively referred to as "local gradients" in the edge database 124) may be periodically used to train the global models (in the database 110), for example, as part of a federated learning framework. As the global model is trained based on the local models, the global model may be propagated back to other edge databases (124) in some cases, thereby keeping the local models up to date.

[0050] However, it is not required to employ federated learning in all embodiments. In some embodiments, the automation agent 122 can provide scrubbed data to the semantic task automation system 102, and the ML module 108 can remotely apply the model to the scrubbed data. In some embodiments, the "scrubbed" data can be data from which sensitive and / or personal information has been removed and / or obfuscated. In some embodiments, personal information can be scrubbed by the automation agent 122, for example, at the edge, based on various rules. In other embodiments, the scrubbed data provided to the semantic task automation system 102 by the automation agent 122 can be in the form of a reduced dimensionality embedding generated from the original data at the client device 120.

[0051] As previously described, the edge database 126-1 can store actions recorded by the automation agent 122-1. The automation agent 122-1 can record actions in a variety of different ways, depending on the level of access the automation agent 122-1 has to the computer applications executed on the client device 120-1 and the permissions granted by the user 118. For example, most smartphones include an operating system (OS) interface for providing or revoking permissions (e.g., location, access to the camera, etc.) to various computer applications. In various embodiments, such an OS interface can be operable to provide / revoke access to the automation agent 122, and / or select a specific level of access that the automation agent 122 will have to a specific computer application.

[0052] The automation agent 122-1 may have various levels of access to the work of the computer application, depending on the permissions granted by the user 118 and the cooperation from the software developer providing the computer application. Some computer applications may (e.g., with the permission of the user 118) provide the automation agent 122 with "under-the-hood" access to the application's API or to scripts written in a programming language (e.g., macros) embedded in the computer application. Other computer applications may not provide as much access. In such cases, the automation agent 122 may record actions in other ways, such as by capturing screenshots, performing optical character recognition (OCR) on those screenshots to identify menu items, and / or monitoring user input (e.g., interrupts captured by the OS) to determine which graphical elements are operated by the user 118 in which order. In some embodiments, the automation agent 122 may intercept actions performed using the computer application from data exchanged between the computer application and the underlying OS (e.g., via system calls). In some implementations, automation agent 122 may intercept and / or access data exchanged between or used by window managers and / or window systems.

[0053] Figure 2 Schematically depicts an example of how data may be processed by and / or using various components across domains. Starting from the top left, a user 118 operates a client device 120 to provide typed or spoken natural language input NLP-1. In the latter case, a speech-to-text (STT) engine (not depicted) may first be used to process the spoken utterance to generate speech recognition output. In either case, NLP-1 may be provided to an automated agent 122.

[0054] In addition, the user 118 operates the client device 120 to request and / or allow recording of actions performed by the user 118 using the client device 120. In various embodiments, the automation agent 122 cannot record the actions without receiving this permission. In some embodiments, this permission may be granted on an application-by-application basis, much like applications are granted permission to access GPS coordinates, local files, use of an onboard camera, etc. In other embodiments, this permission may only be granted until the user 118 speaks otherwise, such as by pressing a "stop recording" button similar to recording a macro or by providing voice input such as "stop recording" or "that's it."

[0055] Once the request / permission is received, in some embodiments, the automation agent 122 can confirm the request / permission. Next, a sequence of actions {A1, A2, ...} performed by the user 118 in domain A using the client device 120 can be captured and stored in the edge database 126. These actions {A1, A2, ...} can take various forms or combinations of forms, such as command line input, and interaction with one or more graphical elements of a GUI using various types of input (such as pointing device (e.g., mouse) input, keyboard input, voice input, line of sight input, and any other type of input capable of interacting with graphical elements of a GUI).

[0056] In various embodiments, the domain (A) in which these actions are performed may be identified, for example, by domain module 104 using any combination of NLP-1, computer applications operated by user 118, remote services accessed by the user (e.g., email, text messaging, social media), projects the user is working on, and the like. In some embodiments, a domain may be identified, at least in part, by an area of ​​a simulated digital world (sometimes referred to as a "metaverse") in which user 118 operates or visits virtually. For example, user 118 may record actions that cause their score and a brief video replay of their performance in a first metaverse game (i.e., a first domain) to be posted to their social media. User 118 may later wish to perform semantically similar tasks for a completely different metaverse game (i.e., a second domain) - the techniques described herein may allow user 118 to seamlessly extend actions previously recorded in the first domain to semantically corresponding or semantically equivalent actions in the second domain.

[0057] Return to reference Figure 2 Based at least in part on the natural language input, the automated agent 122 can generate a task / strategy embedding T'. For example, the automated agent 122 can perform STT processing (or cause STT processing to be performed) on the speech input provided by the user 118. The resulting speech recognition output can then be processed using various natural language processing techniques (including but not limited to techniques such as word2vec, BERT transformers, etc.) to generate a task / strategy embedding T' that represents the semantics of what the user 118 said.

[0058] Based on the captured domain-specific actions {A1, A2, ...}, the automation agent 122 can generate an action embedding A' that semantically represents the semantic task expressed by the domain-specific actions {A1, A2, ...}. The automation agent 122 can associate the action embedding A' and the task / strategy embedding T' in various ways. In some embodiments, these embeddings A', T' can be combined, for example, via cascading or by processing together to generate a joint embedding in a joint embedding space that captures the semantics of both the natural language input from the user 118 and the actions {A1, A2, ...}. In other embodiments, these embeddings A', T' can be in separate embedding spaces: a generalized action embedding space for the action embedding A', and a task / strategy embedding space for the task / strategy embedding T'. A mapping (e.g., a lookup table) between the two embeddings A', T' in the two embedding spaces can be stored.

[0059] Some time later, user 118 may issue another natural language input NLP-2 at client device 120 or at another computing device associated with user 118 (such as another computing device in a coordinated ecosystem of computing devices registered to user 118's online profile). NLP-2 may be the same as NLP-1 or at least semantically equivalent. However, user 118 may be operating in a different domain (domain B). Natural language processing may be performed on NLP-2 to generate another task / policy embedding T2'. Automation agent 122 may match T2' to a previous task / policy embedding T' ( Figure 2 The match can be based on, for example, a distance or similarity in the embedding space between T' and T2'. Various techniques can be used to calculate these distance and / or similarity measures, such as Euclidean distance, cosine similarity, dot product, etc.

[0060] Once the automated agent 122 has matched the task / strategy embedding T2' generated from NLP-2 to the task / strategy embedding T' generated from NLP-1, the automated agent 122 can process the action embedding A' using the domain model B (or provide it to another component for processing) based on the association previously created between A' and T'. The domain model B can be trained to convert between a general action embedding space and an action space associated with domain B. Therefore, processing the action embedding A' using the domain model B can generate a probability distribution over the action space across domain B. This probability distribution can be used, for example, by the automated agent 122 to select one or more domain-specific actions {B1, B2, ...} from the action space of domain B.

[0061] Actions such as {B1, B2, ...} may be selected from the action space of the domain in various ways. In some implementations, the actions may be selected in a random order or in an order of their probabilities. In some implementations, various sequences or permutations of the selected actions may be executed, e.g., as part of a real-time simulation, and the results (e.g., success or failure) may indicate which permutation is actually executed for the user 118. Once the domain model is sufficiently trained, it may be better able to predict the order in which the actions should be executed.

[0062] In any case, the selected actions {B1, B2, ...} may be provided by the automation agent 122 to the client device 120 so that the client device 120 may execute them. In some cases, this may cause the interactive elements of the GUI displayed on the client device 120 to be automatically operated, where the operations are rendered as they are performed. In other embodiments, these GUI operations may be performed without re-rendering.

[0063] In various embodiments, simulations may be performed, for example, by components of the automation agent 122 and / or the semantic task automation system 102 to further train the domain model. More specifically, various permutations of actions may be simulated to determine synthetic results. For example, these synthetic results may be compared to natural language inputs associated with the original set of actions from which the simulated permutations were selected. The success or failure of these synthetic results may be used as positive training examples and / or negative training examples for the domain model. In this way, the domain model may be trained based on far more actions and accompanying natural language inputs than those recorded by users.

[0064] Below the horizontal dashed line Figure 2 The bottom of the diagram depicts an example of a simulation that generates a synthetic result. At some point in time, for example, during the night when the server computer is under a relatively low computational burden, and without being specifically requested to do so, the automation agent 122 may run one or more simulations. For example, the automation agent 122 may process the action embedding A' based on the domain model C. As described above, this may generate a probability distribution over the action space of the domain C. Based on the probability distribution, the automation agent 122 may select actions {C1, C2, ...} to be performed to generate a synthetic result. In some embodiments, the automation agent 122 may simulate multiple different permutations of actions {C1, C2, ...} in order to generate multiple different synthetic results. The success or failure (or various intermediate metrics thereof) of each of these synthetic results may then be evaluated. The domain model C may be trained based on this evaluation.

[0065] Although the simulation Figure 2118 is depicted as being performed as part of a domain (C) in which user 118 is not yet operating, but this is not meant to be limiting. In various embodiments, simulations may be performed in other domains, such as in Figure 2 In the domain B of . For example, different permutations of the actions {B1, B2, …} can be executed to determine how resilient the domain-agnostic semantic task is to reordering of these actions. Furthermore, it is not necessary to execute all selected actions in any given domain to generate each synthetic result.

[0066] Figure 3 From Figure 2 Different angles schematically depict another example of how the techniques described herein can be used to perform semantic tasks across multiple different domains. Starting from the bottom left, user 118 operates client device 120 (in this example, a standalone interactive speaker) and speaks a natural language command "find the latest news from Redmond". STT module 330 can perform STT processing to generate speech recognition output. The speech recognition output can be processed by natural language processing (NLP) module 332 to generate text / policy embedding 334. Action Embedding Finder ( Figure 3 The STT module 330, the NLP module 332, and / or the AEF module 336 can match the text / policy embedding 334 to the action embedding (white star) in the action embedding space 338. In various embodiments, the STT module 330, the NLP module 332, and / or the AEF module 336 can be implemented as part of the automation agent 122, as part of the semantic task automation system 102, or any combination thereof.

[0067] The action embedding space 338 may include multiple action embeddings, each action embedding in Figure 3 338. These domain-agnostic action embeddings can be abstractions of domain-specific actions recorded in the action space of various domains. When processed based on the corresponding domain model, these recorded actions can be abstracted into embeddings shown as part of the embedding space 338. For the purpose of illustration and understanding only, the embedding space 338 is depicted as two-dimensional. It should be understood that the embedding space 338 will actually have as many dimensions as the individual embeddings, which can be hundreds or even thousands of dimensions.

[0068] The white stars represent the coordinates in the action embedding space 338 associated with the task / policy embedding 334. Figure 3, the white star is actually located between the two action embeddings enclosed by ellipse 340. In some embodiments, multiple action embeddings can match a single natural language input, for example, because those action embeddings are semantically similar to each other and / or are executed in response to semantically similar natural language inputs. In some embodiments, multiple matching action embeddings (such as the two within ellipse 340) can be combined into a unified representation, for example via concatenation or averaging, and the unified action embedding can be processed by downstream components.

[0069] The automation agent 122 may then process (or have processed) action embedding using multiple domain models AC, each domain model being associated with a different domain in which the user 118 communicates with others. Domain A may represent, for example, an email domain served by one or more email servers 342A. Domain B may represent, for example, an SMS or MMS domain served by one or more Simple Messaging Service (SMS) / Multimedia Messaging Service (MMS) servers 342B. Domain C may represent, for example, a social media domain served by one or more social media servers 342C. Any of the servers 342A-C may or may not be part of a cloud infrastructure, and therefore may not necessarily be bound to a specific server instance.

[0070] Processing selected action embeddings based on domain model A may generate actions {A1, A2, ...}, similar to those previously described. Similarly, processing selected action embeddings based on domain models B and C may generate actions {B1, B2, ...} and {C1, C2, ...}, respectively. These actions may be executed by servers 342A-C in their respective domains. As a result, email server 342A may retrieve and return recent emails to, for example, client device 120 (e.g., via automated agent 122), where someone named "Redmond" is the sender or recipient of the emails. SMS / MMS server 342B may retrieve and return recent text messages to, for example, client device 120 (e.g., via automated agent 122), where someone named "Redmond" is the sender or recipient of the text messages. And the social media server 342C can retrieve and return to, for example, the client device 120 (e.g., via the automated agent 122), the most recent social media posts or messages (e.g., "direct messages") by, from, or to someone named "Redmond" who is a friend of the user 118. In some embodiments, all of these returned messages can be collated and presented to the user 118. In other embodiments, these returned messages can be compared to identify the most recent message, and that message can be presented to the user 118 alone. For example, in the case where the client device 120 is a stand-alone interactive speaker without display capabilities, such as Figure 3 In situations where the user is in a mobile phone (or, for example, may be in a vehicle), it may be advantageous to minimize the amount of output to avoid overwhelming or distracting the user 118, in which case the most recent messages for any domain may be read aloud.

[0071] Figure 4A and Figure 4B An example of an input form that may be rendered, for example, by a Hypertext Markup Language (HTML) or Extensible Markup Language (XML) based web browser when a user (not depicted) visits the website "www.hyptotheticalair.com" to purchase an airline ticket is depicted. Depicted is a fairly standard input form that solicits the user's personal information and accompanying payment information. It may be assumed that the user operating the input form has created a number of payment profiles or policies, each corresponding to different payment information. For example, the user may have used the techniques described herein to record himself filling out (i.e., performing a sequence of actions) one form to create a "work profile" in which a "work" credit card is used, and filling out (i.e., performing another sequence of actions) another form to create a "marketing profile" in which a different "marketing" credit card is used.

[0072] exist Figure 4A , the input fields of the input form have been pre-populated, as indicated by the natural language output provided by the automated agent 122 (not depicted), which states: "I am using your primary work profile to fill out this form. Is that correct?", which may be because the domain module 104 has applied various contextual signals (such as the URL "hypotheticalair.com") or properties of the underlying web page (e.g., tags, keywords, DOM objects, subjects, etc.) as inputs to the cross-domain selection model 105 to select the domain to which the user's email work profile is associated.

[0073] The user responds "No, I want to use my marketing profile". Therefore, in Figure 4B , the values ​​used to pre-populate the input form have been changed to the user's marketing profile. The automated agent 122 says "Okay, I've updated the form with your marketing profile. Does this look correct?", and the user responds "Yes". In some embodiments, the user's affirmative response may cause the input form to be submitted, for example, to be completed or to advance to a different stage of the purchase. Additionally or alternatively, in some embodiments, the automated agent 122 may capture various features of the input form, such as the website URL, the layout of the input fields, or other contextual features of the input form. Using these extracted contextual features, the automated agent 122 may train one or more domain selection models 105 so that in the future, visits to the same website or to semantically similar websites in similar contexts will cause the user's marketing profile information rather than the user's primary work profile to be used to automatically populate the input form.

[0074] The domain model can also be trained based on user feedback (as opposed to the domain selection model 105). For example, if a user identifies a particular field that was incorrectly populated (e.g., the wrong expiration date for a credit card used), a domain-specific model can be trained based on that error (e.g., using gradient descent and / or back-propagation).

[0075] Figure 5A and Figure 5B Depicted are further examples of input forms that may be rendered, for example, by an HTML or XML-based web browser when a user (not depicted) visits the website "www.hypotheticalpizza.com" to order a pizza. In this example, it may be assumed that the user has not yet established a "personal" profile for purchasing / purchasing personal items (particularly food). Thus, in Figure 4A In the example, the user has manually filled in the input fields to include, among other things, the address and credit card information that the user wishes to use to purchase pizza in this case. In addition, the user has provided natural language input to the automated agent 122 (not depicted), "I want to use this personal profile when I buy food."

[0076] The automated agent 122 responds, “Okay, when I see you are ordering food, I will default to this profile.” Then, using the techniques described herein, the automated agent 122 can capture the actions performed by the user to populate these fields. The automated agent 122 can perform the techniques described herein to associate action embeddings that abstract these actions with all or part of the user's natural language input (such as “personal profile”). This domain-agnostic action embedding can later be extended to other domains, as described herein, such as other websites operable to order other types of food, such as groceries, different restaurants (e.g., other pizza restaurants or other types of restaurants), etc.

[0077] It is worth noting that the automation agent 122 in this example is able to retroactively record actions previously performed by the user, rather than recording actions that occur after the user's natural language input. In some embodiments, the automation agent 122 or another component can, with the user's explicit permission or opt-in, maintain a stack or buffer of actions performed by the user, such as when filling out an input form. If the user decides after performing these actions that they want to record them for extension across different domains, the user can make such a decision. Figure 5A The statements shown, and this stack or buffer can be used to retroactively retrieve actions that have been performed for automation of semantic tasks.

[0078] Figure 5B Another input form is depicted, this time for a website with the URL "hypotheticalgrocery.com". Similar to "hypotheticalpizza.com", "hypotheticalgrocery.com" is related to food, albeit with different entities (hypotheticalpizza vs. hypotheticalgrocery). Figure 5A As shown, when a user creates his or her "personal profile", the automated agent 122 has already used Figure 5A More specifically, domain module 104 has applied various contextual signals to domain selection model 105 to identify the domain of the input form. Next, automated agent 122 applies contextual signals across domain models associated with the identified domain (e.g., a general “food” or “grocery” domain) to automatically populate the input fields. Figure 5A The generated action embeddings are taken as input. A domain model can be trained to translate between the action embedding space and the identified domain. Therefore, the cross-domain model applies the action embeddings to generate a probability distribution over actions in the identified domain. These actions include filling Figure 5B , as shown in the figure.

[0079] It is worth noting that despite the fact Figure 5B The two name input fields in Figure 5A Their corresponding fields in the _name field—“given name” instead of “first name” and “surname” instead of “last name”—are Figure 5A The abstraction of the actions performed by the input form allows Figure 5A The resulting domain-agnostic action embedding abstracts away these details. Thus, when this domain-agnostic action embedding is then converted into concrete actions in the “hypotheticalgrocery” domain, the domain model associated with the “hypotheticalgrocery” domain can convert between these semantically equivalent terms accordingly.

[0080] The automated agent 122 notifies the user of this, saying "I used your personal profile to fill out this form. Is that correct?". The user responds in the affirmative. In some embodiments, this can be used as a positive training example to further train the automated fill function. Figure 5B Additionally or alternatively, in some embodiments, the automated agent 122 and / or the domain model 104 may train one or more domain selection machine learning models 105 based on Figure 5B In the context of , the user's "personal" payment profile should be applied. Figure 5B The “context” of a message may include the URL “hypotheticalgrocery.com,” tags or other attributes of the underlying web page, information about the user, information about the computing device operated by the user (e.g., home IP address versus work IP address), and the like.

[0081] Figure 6 600 for practicing selected aspects of the present disclosure according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of the semantic task automation system 102. In addition, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0082] At block 602, the system may record a first plurality of actions performed by the user 118 at a first computer application at the request of the user 118. At block 604, the system may receive, for example, through the automation agent 122, an initial natural language input that conveys information about a task performed or to be performed by the user 118. The natural language input may be received as typed text or spoken utterances. In some embodiments where the client device 120 being used also includes a camera, the user 118 may provide gestures or other visual cues (e.g., sign language) as additional input.

[0083] At block 606, the system may perform NLP on the initial natural language input to generate a first task (or policy) embedding representing the first task (or policy) conveyed by the initial natural language input, such as by the automated agent 122 or the ML module 108. For example, the automated agent 122 or the ML module 108 may process the natural language input using an NLP machine learning model and / or techniques such as word2vec, BERT transformer, etc., to generate the first task (or policy) embedding.

[0084] At block 608, the system may process the first plurality of actions (e.g., via the automation agent 122 and / or the ML module 108) using a first domain model (e.g., selected by the domain module 104 using one or more domain selection machine learning models 105) to generate first action embeddings. The first action embeddings may represent the first plurality of domain-specific actions performed using the first computer application in a reduced-dimensional form. To this end, the first domain model may be trained to convert between an action space of the first computer application and an action embedding space including the first action embeddings.

[0085] At block 610, the system may store the association between the first task embedding and the first action embedding in a memory, such as by the automation agent 122. For example, the automation agent 122 may store a single embedding in a joint task / action embedding space that includes both the task (or policy) embedding generated at block 606 and the action embedding generated at block 608 (e.g., as an average or concatenation of the two). Additionally or alternatively, in some implementations, the automation agent 122 may store a mapping between the two embeddings (e.g., as part of a lookup table).

[0086] When the user 118 initially wishes to record a semantic task, the user may perform the Figure 6 The operations of method 600 in the embodiment of the present invention are as follows. Subsequently, the automated semantic tasks can be extended to other domains. Figure 6 As indicated by box 700 in FIG. 1 , example operations for extending such semantic tasks to other domains are depicted as Figure 7 Part of method 700 in.

[0087] Reference now Figure 7 At block 702, the system may perform NLP on the subsequent natural language input, for example, via the automation agent 122, to generate a second task embedding representing a second task conveyed by the subsequent natural language input. As with block 606, other types of contextual signals and / or visual cues (e.g., gestures) may be used as input as part of the processing of block 702.

[0088] At block 704, the system can determine (e.g., via the automated agent 122) that the second task semantically corresponds to the first task based on a similarity measure between the first task embedding and the second task embedding. Such a similarity measure can be determined in various ways, such as via Euclidean distance, cosine similarity, dot product, etc.

[0089] In response to the determination of block 704, the system may identify (e.g., via the automation agent 122 or the domain module 104) one or more applicable domains in which the user wishes to perform a semantic task. In many cases, this may be a single domain, such as the domain of a computer application currently operated by the user in which the user wishes to perform a semantic task (e.g., apply a dark theme, set preferences, etc.). However, and as Figure 3 As shown, for example, there may be multiple applicable domains that a user wishes to extend / perform a semantic task at once.

[0090] Thus, at block 706, the system may determine (e.g., via the automation agent 122 or the domain module 104) whether there are more applicable domains to expand / perform the semantic task. If the answer is no, the method 700 ends. However, if the answer at block 706 is yes, the method 700 proceeds to block 708, where the next applicable domain is selected as the current domain.

[0091] At block 710, the system may (eg, via the automation agent 122 or the ML module 108) process the first action embedding using a domain model associated with the current domain to select a current plurality of domain-specific actions to be performed in the current domain. Figure 6 The first domain model described, the current domain model can be trained to convert between the action space of the current domain (e.g., the second computer application, the second input form, etc.) and the action embedding space.

[0092] At block 712, the system may cause (e.g., via the automation agent 122) a second plurality of actions to be performed in the current domain. If the current domain is a computer application having a GUI, the plurality of actions may be automatically performed on the GUI, for example, by the automation agent 122. In some implementations, the GUI may be visually updated at each step so that the user can see the actions being performed. In other implementations, the actions may be performed without updating the GUI so that the user only sees the final result of the plurality of actions.

[0093] At optional block 714, the system may receive feedback from the user (e.g., via the automation agent 122) regarding the performance of the semantic task in the current domain. The feedback may be solicited by the automation agent 122 or provided from the user without solicitation. At block 716, the system may train the current domain model based on the feedback, e.g., via the automation agent 122 or the ML module 108. If the current domain model is local to the client device (e.g., Figure 1 In the federated learning framework depicted in , stored in the edge database 124), a local model can be trained and may or may not be propagated (as a local gradient) to the global model database 110 so that the corresponding global model is trained.

[0094] From block 716 (or from block 712 if blocks 714-716 are omitted) method 700 may proceed back to block 706 where the system may again determine if there are any applicable domains. In some implementations, the applicable domains may be user configurable. For example, a user may register multiple domains such as email, SMS / MMS, and social media. Figure 3 In other embodiments, and with the user's permission or opt-in, the system may automatically determine applicable domains, such as through domain module 104. For example, domain module 104 may consult a user's digital contact list to identify communication modalities associated with individual contacts (e.g., email addresses, phone numbers, social media or instant messaging handles, etc.). Assume that a user requests the latest written communications from Delia Sue. If Delia Sue's digital contact card includes an email address, phone number, and gaming pseudonym, three domains may apply: email, SMS / MMS, and the domain where Delia Sue communicates with other players using her gaming pseudonym.

[0095] Figure 8 800 is a flowchart illustrating another example method 800 for practicing selected aspects of the present disclosure according to embodiments disclosed herein. Method 800 represents a variation of methods 600-700, which is applicable to, for example, FIG. 4A to FIG. 4B and FIG. 5A to FIG. 5B, where an action performed using one input form (first domain) is converted to a semantically equivalent action performed using another input form (second domain). For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of the semantic task automation system 102. In addition, although the operations of method 800 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0096] At block 802, the system may obtain initial natural language input, such as through an automated agent 122, and a first plurality of actions to be performed using a first input form configured for a first domain. Figure 5A In , the user first provides an action and then provides a natural language input "I want to use this personal profile when I buy food." At block 804, the system can perform NLP on the initial natural language input, for example, through the automated agent 122, to generate a first policy embedding representing a first input policy conveyed by the initial natural language input. Figure 5A For example, a user defines a policy where payment information associated with a "personal profile" is to be used when purchasing food.

[0097] At block 806, the system can process the first plurality of actions using the first domain model, such as by the automation agent 122, to generate first action embeddings representing the first plurality of actions performed using the first input form. As with the other domain models described herein, the first domain model can be trained to embed actions in the action space of the first domain (e.g., Figure 5A ) and the action embedding space including the first action embedding. At block 808, the system may store the association between the first policy embedding and the first action embedding in a memory, for example, via the automation agent 122.

[0098] At block 810, the system may perform NLP on the subsequent natural language input, for example, via the automated agent 122, to generate a second policy embedding representing a second policy conveyed by the subsequent natural language input. At block 812, the system may determine (for example, via the automated agent 122) that the second policy semantically corresponds to the first policy based on a similarity measure between the first policy embedding and the second policy embedding. For example, Figure 4A In , the user declines to use the payment information associated with his or her “primary work” profile and says “I want to use my marketing profile.” The speech recognition output generated from this utterance is processed and matched to the policy embedding (e.g., by Figure 3 The AEF module 336 in uses Euclidean distance or cosine similarity) and thus matches to action embeddings generated from actions performed by the user in the previous domain.

[0099] In response to the determination of block 812, at block 814, the system can process the first action embedding using the second domain model, e.g., via the automation agent 122 or the ML module 106, to select a second plurality of actions to be performed using a second input form configured for the second domain. The second domain model can be trained to convert between the action space and the action embedding space of the second domain. At block 816, the system can cause the second plurality of actions to be performed using the second input form, e.g., via the automation agent 122. For example, once the action embedding associated with the user's marketing profile is identified (e.g., via Figure 3 336 in the AEF module 336), is applied as input across the domain model associated with the user's marketing profile to automatically populate Figure 4B The input fields in the input form.

[0100] Fig. 9 910 is a block diagram of an example computing device 910 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing devices 120-1 through 120-P, the semantic task automation system 102, and / or other components can include one or more components of the example computing device 910.

[0101] The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924 (including, for example, a memory subsystem 925 and a file storage subsystem 926), a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0102] The user interface input device 922 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 910 or over a communication network.

[0103] The user interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 910 to a user or to another machine or computing device.

[0104] The storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include a Figures 6 to 8 The logic of selected aspects of methods 600, 700 and 800.

[0105] These software modules are typically executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 may include multiple memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 for storing fixed instructions. The file storage subsystem 926 can provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored by the file storage subsystem 926 in the storage subsystem 924, or stored in other machines accessible by the processor 914.

[0106] The bus subsystem 912 provides a mechanism for the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0107] Computing device 910 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Fig. 9 The description of computing device 910 depicted in FIG. 1 is intended only as a specific example for purposes of illustrating some embodiments. Fig. 9 Many other configurations of computing device 910 are possible with more or fewer components than the computing device depicted.

[0108] Although several embodiments have been described and illustrated herein, various other components and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using experiments that do not exceed routine. Therefore, it should be understood that the foregoing embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented using one or more processors, comprising: obtaining an initial natural language input and a first plurality of actions performed using a first computer application; performing natural language processing (NLP) on the initial natural language input to generate a first task embedding, the first task embedding representing a first task conveyed by the initial natural language input; processing the first plurality of actions using a first domain model of a first domain to generate first action embeddings, the first action embeddings representing the first plurality of actions performed using the first computer application, wherein the first domain model is trained to translate between an action space of the first computer application and an action embedding space including the first action embeddings; storing an association between the first task embedding and the first action embedding in a memory; performing NLP on a subsequent natural language input to generate a second task embedding representing a second task conveyed by the subsequent natural language input; determining, based on a similarity measure between the first task embedding and the second task embedding, that the second task semantically corresponds to the first task; In response to the determining, processing the first action embeddings using a second domain model to select a second plurality of actions to be performed using a second computer application, wherein the second domain model is trained to convert between an action space of the second computer application and the action embedding space; and The second plurality of actions are caused to be performed using the second computer application.

2. The method according to claim 1, wherein: At least one of the first computer application and the second computer application includes an operating system.

3. The method according to claim 1 or 2, wherein: The first plurality of actions performed using the first computer application are intercepted from data exchanged between the first computer application and an underlying operating system.

4. The method according to claim 2 or 3, wherein: The data exchanged includes data indicating keystrokes and pointing device input.

5. The method according to any one of claims 1 to 4, wherein: The first plurality of actions performed using the first computer application are captured from an application programming interface (API) of the first computer application.

6. The method according to any one of claims 1 to 5, wherein: The first plurality of actions performed using the first computer application are captured from a domain specific programming language associated with the first domain.

7. The method according to any one of claims 1 to 6, wherein: The first plurality of actions performed using the first computer application are captured from a scripting language embedded in the first computer application.

8. The method according to any one of claims 1 to 7, wherein: The first plurality of actions performed using the first computer application include interactions with a first graphical user interface (GUI) rendered by the first computer application.

9. The method according to any one of claims 1 to 8, wherein: The second plurality of actions performed using the second computer application include interactions with a second GUI rendered by the second computer application.

10. The method according to any one of claims 1 to 9, wherein: The first computer application is operable to exchange data with a first database having a first database schema, and the second computer application is operable to exchange data with a second database having a second database schema, the second database schema being different from the first database schema.

11. The method according to claim 10, wherein: The first plurality of actions interact with first data from the first database according to the first database schema, and the second plurality of actions interact with second data from the second database according to the second database schema, and the second data semantically corresponds to the first data.

12. The method according to any one of claims 1 to 11, wherein: The first computer application includes a first communication application that has been operated to communicate with a first plurality of contacts, and the second computer application includes a second communication application that has been operated to communicate with a second plurality of contacts.

13. The method according to claim 12, wherein: The second task seeks past communications with one or more contacts included in the second plurality of contacts.

14. The method according to claim 13, wherein: The second task also seeks past communications with one or more contacts included in the first plurality of contacts.

15. The method according to any one of claims 1 to 14, wherein: The first computer application is operable to exchange data with a first database having a first database schema, and the second computer application is operable to exchange data with a second database having a second database schema, the second database schema being different from the first database schema.

16. The method according to claim 15, wherein: The first plurality of actions interact with first data from the first database according to the first database schema, and the second plurality of actions interact with second data from the second database according to the second database schema, and the second data semantically corresponds to the first data.

17. A method implemented using one or more processors, comprising: obtaining initial natural language input and performing a first plurality of actions using a first input form configured for a first domain; performing natural language processing (NLP) on the initial natural language input to generate a first policy embedding representing a first input policy conveyed by the initial natural language input; processing the first plurality of actions using a first domain model to generate first action embeddings, the first action embeddings representing the first plurality of actions performed using the first input form, wherein the first domain model is trained to convert between an action space of the first domain and an action embedding space including the first action embeddings; storing in a memory an association between the first policy embedding and the first action embedding; performing NLP on a subsequent natural language input to generate a second policy embedding representing a second policy conveyed by the subsequent natural language input; determining, based on a similarity measure between the first policy embedding and the second policy embedding, that the second policy semantically corresponds to the first policy; In response to the determining, processing the first action embeddings using a second domain model to select a second plurality of actions to be performed using a second input form configured for a second domain, wherein the second domain model is trained to convert between an action space of the second domain and the action embedding space; and The second plurality of actions is caused to be performed using the second input form.

18. The method according to claim 17, wherein: The first plurality of actions includes populating a first plurality of form fields with a first set of values, and the second plurality of actions includes populating a second plurality of form fields with at least some of the first set of values.

19. A system comprising one or more processors and a memory storing instructions, which, in response to execution of the instructions, cause the one or more processors to: obtaining an initial natural language input and a first plurality of actions performed using a first computer application; performing natural language processing (NLP) on the initial natural language input to generate a first task embedding, the first task embedding representing a first task conveyed by the initial natural language input; processing the first plurality of actions using a first domain model to generate first action embeddings, the first action embeddings representing the first plurality of actions performed using the first computer application, wherein The first domain model is trained to convert between an action space of the first computer application and an action embedding space including the first action embedding; storing an association between the first task embedding and the first action embedding in a memory; performing NLP on a subsequent natural language input to generate a second task embedding representing a second task conveyed by the subsequent natural language input; determining, based on a similarity measure between the first task embedding and the second task embedding, that the second task semantically corresponds to the first task; responsive to the determining, processing the first action embeddings using a second domain model to select a second plurality of actions to be performed using a second computer application, wherein the second domain model is trained to convert between an action space of the second computer application and the action embedding space; as well as The second plurality of actions are caused to be performed using the second computer application.

20. The system of claim 19, wherein: At least one of the first computer application and the second computer application includes an operating system.

Images