Generating cross-domain guidance for navigating HCI

Abstract

The disclosed embodiments relate to automatically generating and providing guidance for navigating an HCI to perform semantically equivalent / similar computational tasks across different computer applications. In various embodiments, a domain of a first computer application operable using a first HCI may be used to select a domain model that transitions between an action space of the first computer application and another space. Based on the selected domain model, a domain-agnostic action embedding, which represents an action previously performed using a second HCI of the second computer application to perform the semantic task, may be processed to generate a probability distribution over the action in an action space of the first computer application. Based on the probability distribution, actions that may be performed using the first computer application may be identified that may be used to generate guidance for navigating the first HCI to perform the semantic task.

Description

Background Art

[0001] Individuals often operate computing devices to perform semantically similar tasks in different contexts. For example, an individual may use a first computer application to engage in a sequence of actions to perform a given semantic task, such as setting various application preferences, retrieving / viewing specific data accessible by the first computer application, performing a sequence of operations within a specific domain (e.g., 3D modeling, graphics editing, word processing), etc. The same individual may later engage in a semantically similar but syntactically different sequence of actions to perform a semantically equivalent task (e.g., the same semantic task) in a different context, such as when using a second computer application. However, the individual may not be very familiar with the second computer application and may therefore be unable to perform the semantic task. Summary of the invention

[0002] Embodiments are described herein for automatically generating and providing guidance for navigating a human-computer interface (HCI) to perform semantically equivalent and / or semantically similar computing tasks across different computer applications. More specifically, but not exclusively, embodiments are described herein for enabling individuals (generally referred to as "users") to leverage the actions they perform within one context (e.g., when performing a semantic task) in order to generate guidance for performing semantically equivalent or semantically similar tasks in other contexts. In various embodiments, the captured actions may be abstracted as "action embeddings" in a generalized "action embedding space". This domain-agnostic action embedding may abstractly represent a "semantic task" that may be converted into an action space of any number of domains using a corresponding domain model. In other words, a "semantic task" is a domain-agnostic higher-order task that finds expressions of sequences / multiple domain-specific actions as domain-specific actions within a specific domain.

[0003] In some embodiments, a method may be implemented using one or more processors and may include: identifying a first domain of a first computer application operable using a first human-computer interface (HCI); based on the identified domain, selecting a domain model that converts between an action space of the first computer application and another space; based on the selected domain model, processing action embeddings to generate one or more probability distributions over actions in the action space of the first computer program, wherein the action embeddings represent a plurality of actions previously performed using a second HCI of a second computer application to perform a semantic task; based on the one or more probability distributions, identifying a second plurality of actions that may be performed using the first computer application; and causing output to be presented at one or more output devices. In various embodiments, the output may include guidance for navigating the first HCI to perform the semantic task using the first computer application, and wherein the guidance is based on the identified second plurality of actions that may be performed using the first computer application.

[0004] In various implementations, the domain model may be trained to convert between the action space of the first computer program and the domain-agnostic action embedding space. In various implementations, the domain model may be trained to convert directly between the action space of the first computer program and the action space of the second computer program.

[0005] In various embodiments, the first HCI may take the form of a graphical user interface. In various embodiments, the guidance for navigating the first HCI may include one or more visual annotations covering the GUI. In various embodiments, one or more of the visual annotations may be rendered to draw attention to one or more graphical elements of the GUI.

[0006] In various embodiments, the guidance for navigating the first HCI may include one or more natural language outputs. In various embodiments, the method may further include: obtaining user input conveying a semantic task; and identifying an action embedding based on the semantic task. In various embodiments, the user input may be a natural language input, and the method may further include: performing natural language processing (NLP) on the natural language input to generate a first task embedding representing the semantic task; and determining a similarity measure between the first task embedding and the action embedding; wherein the action embedding is processed based on the similarity measure.

[0007] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause performance of any of the foregoing methods. Some embodiments include at least one non-transitory computer-readable storage medium storing computer instructions executable by the one or more processors to perform any of the foregoing methods.

[0008] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a schematic diagram of an example environment in which implementations disclosed herein may be implemented.

[0010] Figure 2 Examples of how data may be exchanged and / or processed to extend tasks performed in one domain to additional domains are schematically illustrated according to various embodiments.

[0011] Figure 3A , Figure 3B , Figure 3C and Figure 3D Examples of how the techniques described herein may be used to provide guidance for interacting with a computer application are shown in accordance with various implementations.

[0012] Figure 4 is a flow chart illustrating an example method of practicing selected aspects of the present disclosure according to embodiments disclosed herein.

[0013] Figure 5 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0014] Embodiments are described herein for automatically generating and providing guidance for navigating a human-computer interface (HCI) to perform semantically equivalent and / or semantically similar computing tasks across different computer applications. More specifically, but not exclusively, embodiments are described herein for enabling individuals (generally referred to as "users") to leverage the actions they perform within one context (e.g., when performing a semantic task) in order to generate guidance for performing semantically equivalent or semantically similar tasks in other contexts. In various embodiments, the captured actions may be abstracted as "action embeddings" in a generalized "action embedding space". This domain-agnostic action embedding may abstractly represent a "semantic task" that may be converted into an action space of any number of domains using a corresponding domain model. In other words, a "semantic task" is a domain-agnostic higher-order task that finds expressions of sequences / multiple domain-specific actions as domain-specific actions within a specific domain.

[0015] As a non-limiting working example, a user may authorize a local agent computer program (also referred to herein as an "agent" or "assistant") to monitor the user's interactions with one or more local computer applications. The monitoring may include, for example, capturing the user's interactions with an HCI (such as a graphical user interface (GUI)) of a first computer application. The user may interact with the HCI to perform various semantic tasks that may be performed using the first computer application. Each semantic task may include multiple individual or atomic interactions with the HCI of the first computer application.

[0016] For example, if the first computer application is a three-dimensional (3D) design application, the semantic task may include designing a 3D structure, and the atomic interactions may include, for example, navigating to specific menus, selecting specific tools from those menus, selecting specific settings for those tools, operating those tools on a canvas, etc. If the first computer application is a spreadsheet application, the semantic task may include, for example, creating a chart based on the underlying data. The atomic interactions may include, for example, sorting the data, adding columns (e.g., using equations that utilize existing column values ​​of operands), selecting a range for navigation menus, selecting specific items from those menus to create a desired chart, etc.

[0017] Referring back to the working example, a domain model associated with the domain of the first computer application can be used to abstract domain-specific actions captured in association with a semantic task performed using the first computer application into an action embedding. The action embedding can then be converted into any number of other domain action spaces, such as the action space of a second computer program. For example, a probability distribution can be generated over the actions in the action space of the second computer program. A plurality of domain-specific actions that can be performed using the second computer program can be selected from the action space, for example, based on their probabilities (e.g., generated using a softmax layer of a domain model). The selected domain-specific actions of the second computer program can then be used to generate guidance for navigating through the HCI provided by the second computer program.

[0018] Guidance for navigating the HCI may be generated and / or presented in various ways. In some embodiments where the HCI is a GUI, visual annotations may be presented, for example, covering all or part of the GUI. In some embodiments, these visual annotations may draw attention to graphical elements that may be operated (e.g., using a pointer device or finger if the display is a touch screen) to perform multiple domain-specific actions identified from the action space of the second computer program. Visual annotations may include, for example, arrows, animations, natural language text, shapes, etc. Additionally or alternatively, audible guidance may be presented to audibly guide the user to a specific graphical element corresponding to the identified domain-specific action from the action space of the second computer program. Audible guidance may include, for example, natural language output, noise accompanying visual annotations (e.g., animation), etc.

[0019] Thus, using the techniques described herein, a user can allow (e.g., by "opting in") an agent configured with selected aspects of the present disclosure to monitor these types of interactions with various HCIs in various domains. The knowledge gained by the agent can be captured (e.g., in various domain machine learning models) and leveraged to generate guidance for performing semantically similar actions in other domains. In some embodiments, the extent to which other users subsequently follow or deviate from such guidance can be used to train the domain models, e.g., so that they can select "better" actions in the future. In some cases, if enough users follow the same (or substantially similar) guidance in a given computer application, the guidance can be used to create a tool that can be automatically invoked, thereby freeing subsequent users from having to repeat the same atomic actions provided in the guidance.

[0020] In some embodiments, a user can provide natural language input to describe a sequence of actions performed using HCI in a first domain, e.g., while performing an action, or immediately before or after performing an action. For example, while operating a first spreadsheet application, a user can state "I am creating a bar chart to show the net loss for the last 90 days". A first task / strategy embedding generated from natural language processing (NLP) of this input can be associated with (e.g., mapped to, combined with) a first action embedding generated from the captured sequence of actions using a first domain model associated with the first spreadsheet application. As previously described, the first domain model can convert between the action space of the first spreadsheet and, for example, a general action embedding space and / or one or more other domain-specific action embedding spaces.

[0021] Later, when operating a second spreadsheet application with similar functionality to the first spreadsheet application, the user can provide semantically similar natural language input to learn how to perform a semantically equivalent (or at least semantically similar) task using the second spreadsheet application. For example, the user can say "How do I create a bar chart to show the net loss for the last 120 days". The second task / strategy embedding generated from this subsequent natural language input can be matched with the first task / strategy embedding, and therefore with the first action embedding. The first action embedding can then be processed using a second domain model that converts between a general action embedding space and the action space of the second spreadsheet application to identify actions that can be executed at the second spreadsheet application to perform the semantic task. These identified actions can be used to generate guidance for performing the semantic task in the second spreadsheet application.

[0022] For example, visual annotations and / or audible guidance may be provided to guide a user through the various menus, tables, cells, etc. of the second spreadsheet application to perform the creation of a bar chart showing the net losses for the last 120 days. Notably, the fact that the second chart will show 120 days of losses while the first chart shows 90 days of losses may be handled by the agent, for example, by retaining the number of days as a parameter associated with an action in the action space of the second spreadsheet application. In addition to capturing the semantics of the HCI itself, the domain model may also be trained to recognize where semantically equivalent data resides.

[0023] For example, when operating a first spreadsheet application to edit a first spreadsheet file, the data required to determine a net loss may be on a specific label that also includes various other data. In contrast, a second spreadsheet editable using a second spreadsheet application may include semantically similar data, i.e., data required to determine a net loss, on a different label with or without other data. However, if sufficient training examples are provided over time, the domain model used by an agent configured with selected aspects of the present disclosure may be able to locate appropriate data to determine a net loss. For example, different columns of different spreadsheets containing data related to a net loss may include semantically similar column headers. Additionally or alternatively, the actual data itself may share semantic features: it may be formatted similarly; have substantially similar values ​​(e.g., within the same order of magnitude, millions versus hundreds of millions, etc.); exhibit similar temporal patterns (e.g., higher sales during certain seasons), etc.

[0024] In addition to or in lieu of guidance, in some embodiments, the techniques described herein can be used to configure the HCI itself to conform to the behavior or capabilities of a particular user. For example, the visual settings of the GUI can be configured via a variety of different actions to make the GUI easier to operate for visually impaired users. This can include, for example, increasing font size, increasing contrast, reducing how many menu items are presented (e.g., based on frequency of use across user groups), increasing the size of operable graphical elements (such as sliders, buttons, etc.), activating user accessibility settings as sound prompts, etc. These actions can be captured in a given computer application and abstracted into action embeddings, for example, together with task / policy embeddings created from natural language input, such as "apply visually impaired settings."

[0025] Later, in a different context (e.g., when operating a different computer application), the user may provide natural language input, such as "I am visually impaired, please make this interface easier to operate". The previously generated action embeddings may be processed using the domain model associated with the new context to automatically make at least some of the above adjustments, and / or show the user how to make them. If any of the adjustments are not available or applicable, the user may be so notified, and / or the user may be provided with different recommendations that may meet similar needs.

[0026] The techniques described herein are not limited to generating guidance for performing semantic tasks across similar domains (e.g., from one spreadsheet application to another spreadsheet application). In various embodiments, guidance for performing semantic tasks can also be generated across semantically different domains / contexts. For example, semantically similar but domain-agnostic application parameters (e.g., different submenus, command line input, etc.) of various computer applications can be named, organized, and / or accessed differently. Such application parameters may include, for example, visual parameters that can be set to various modes, such as "dark mode"; application permissions (e.g., access to location, camera, files, other applications, etc.); or other application preferences (e.g., preferences for Celsius versus Fahrenheit, metric versus imperial, preferred fonts, preferred sorting order, etc.). Many of these various application parameters may not be unique to a particular computer application or domain. In fact, some application parameters, such as a "skin" applied to a GUI, can even be applied to an operating system (OS).

[0027] The above spreadsheet example includes two different spreadsheet applications. However, this is not meant to be limiting. The techniques described herein can be implemented to generate guidance for performing semantic tasks across multiple different use cases within a single domain. Assume that a user operates a first "docket" spreadsheet to organize summary and scheduling data in a particular way, and then generates a document report in a particular format. The actions performed by the user to create the report can be captured and abstracted into action embeddings as previously described, for example, using the domain model of whatever spreadsheet application the user is operating.

[0028] Later, the user may receive a second "summary" spreadsheet, created, for example, by a different summarization system or for a different entity. The second summary spreadsheet includes data that is semantically similar to the first summary spreadsheet, but may be organized differently and / or have a different schema. The columns may have different names and / or be in a different order. The data may be expressed using different syntaxes (e.g., "MM / DD / YY" versus "DD / MM / YYYY"). Nonetheless, the previously created action embeddings may be processed, for example, in conjunction with the second summary spreadsheet (e.g., as additional contextual input data) to generate guidance for performing the same semantic task using the second summary spreadsheet. For example, the same domain model or a separate domain model (e.g., reverse trained) that may also process contextual input data may be applied to identify actions that may be executed to perform the semantic task utilizing the second summary spreadsheet.

[0029] In various embodiments, domain models may be continuously trained based on how users interact with guidance generated using the techniques described herein. This in turn may affect how or whether various guidance is provided. Suppose that for a particular domain-agnostic semantic action, such as "set to dark mode," a particular suggested action is rarely or never performed in a particular domain, e.g., using a particular computer application for that domain. Perhaps the native settings of that computer application have already solved or presented a potential problem that requires that the suggested action be performed in another domain.

[0030] In such a scenario, the domain model associated with a particular domain may be further trained so that when processing the same (or similar) action embeddings, the resulting probability distribution over the action space of that domain will assign a lower probability to the suggested action. In contrast, in other domains where the underlying problem still exists, the suggested action may receive a larger probability. The assigned probabilities may indicate how the suggested action is presented to the user (e.g., how prominently, with an animation or a small visual annotation, audibly or visually), when the suggested action is presented to the user (e.g., relative to other suggestions), whether the suggested action should be presented to the user, or even if the action should be performed automatically without providing user guidance.

[0031] Continuous training of domain models is not limited to monitoring user feedback / reactions to HCI guidance provided in new domains. In some embodiments, the domain model can be trained without the user leaving the original domain in which the semantic task is performed. For example, when a user performs a sequence of actions using a computer application to complete a given semantic task, a domain model associated with the domain of the computer application can be used to process the action (or data indicating the action, such as an embedding) to generate a domain-agnostic embedding that semantically represents the given task. The domain-agnostic embedding can then be processed using a machine learning model (e.g., a sequence decoder) that is trained to generate a natural language output intended to describe a given semantic task performed by the user. For example, in response to changing a specific visual setting in an application or operating system, a natural language output such as "It looks like you changed your graphical interface to 'dark mode'" can be presented to the user.

[0032] This natural language output may be presented to the user auditorily or visually, along with a solicitation for user feedback ("Did you do anything?" or "Did I describe your action accurately?"). Positive feedback from the user ("Yes, that's correct") or negative feedback ("No, that's not what I did") may be used to train the domain model, for example, using techniques such as back-propagation and gradient descent. The user may have the ability to adjust or influence the frequency (or even whether) such solicitations for feedback are presented to them. In some cases, incentives may be provided to the user for being solicited for such feedback and / or for providing feedback. These incentives may come in various forms, such as monetary rewards, credits associated with the computer application (e.g., special items for a game), and the like. Additionally or alternatively, the agent itself may adjust the frequency of soliciting such feedback from the user based on signals such as the user's reaction (e.g., refusing to collaborate), a measure of accuracy associated with the domain model in question (a more accurate model may not be trained as frequently as a less accurate model), and the like.

[0033] As used herein, a "domain" may refer to a target subject area in which a computing component is intended to operate, e.g., the scope of knowledge, influence, and / or activity around which the logic of the computing component revolves. In some embodiments, a domain may be identified by heuristically matching keywords in a user-provided input with domain keywords. In other embodiments, the user-provided input may be processed, for example, using NLP techniques such as word2vec, BidirectionalEncoderRepresentationsfromTransformers (BERT) transformers, various types of recurrent neural networks ("RNNs", e.g., long short-term memory or "LSTM", gated recurrent units or "GRU"), etc.) to generate a semantic embedding representing the user input.

[0034] In various embodiments, one or more domain models may have been previously generated for each domain. For example, one or more machine learning models, such as RNNs (e.g., LSTM, GRU), BERT transformers, various types of neural networks, reinforcement learning strategies, etc., may be trained based on a document corpus associated with the domain. As a result of this training, one or more of the domain models may be at least guided so that it can be used to process what will be referred to herein as "action embeddings" to select a plurality of candidate computational actions from an action space associated with the target domain, which may then be used to provide guidance as described herein.

[0035] Figure 1 An example environment is schematically depicted in which selected aspects of the present disclosure may be implemented, according to various implementations. Figure 1 Any computing device depicted in or elsewhere in the figures may include logic, such as one or more microprocessors (e.g., a central processing unit or “CPU,” a graphics processing unit or “GPU,” a tensor processing unit or (“TPU”)) that execute computer-readable instructions stored in a memory, or other types of logic, such as an application specific integrated circuit (“ASIC”), a field programmable gate array (“FPGA”), etc. Figure 1 Some of the systems depicted in the examples, such as the semantic task guidance system 102, may be implemented using one or more server computing devices forming what is sometimes referred to as a "cloud infrastructure," although this is not required. In other implementations, aspects of the semantic task guidance system 102 may be implemented on a client device 120, e.g., for purposes of protecting privacy, reducing latency, etc.

[0036] The semantic task guidance system 102 may include a number of different components configured with selected aspects of the present disclosure, such as a domain module 104, an interface module 106, a machine learning ( Figure 1) module 108 and / or task identification ( Figure 1 The semantic task guidance system 102 may also include any number of databases for storing machine learning model weights and / or other data for performing selected aspects of the present disclosure. Figure 1 In, for example, the semantic task guidance system 102 includes a database 111 storing a global domain model and another database 112 storing data indicating global action embeddings.

[0037] The semantic task guidance system 102 may be operatively coupled to any number of client computing devices operated by any number of users via one or more computer networks (114). Figure 1 In, for example, a first user 118-1 operates one or more client devices 120-1. A p-th user 118-P operates one or more client devices 120-P. As used herein, client devices 120 may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which in some cases may include a visual sensor and / or a touch screen display), a smart appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0038] The domain module 104 may be configured to determine various different information about domains that are relevant to a given user 118 at a given point in time, such as the domains that the user 118 is currently operating on, the domains that the user has previously operated on, the domains that the user would like to extend a semantic task or receive guidance on how to perform a semantic task, etc. To this end, the domain module 104 may collect contextual information about, for example, foreground and / or background applications executing on a client device 120 operated by the user 118, web pages currently / recently visited by the user 118, domains that the user 118 may access and / or frequently visit, etc.

[0039] Using this collected context information, in some embodiments, the domain module 104 can be configured to identify one or more domains currently associated with the user. For example, a request to record or observe a task performed by a user 118 using a particular computer application and / or on a particular input form can be processed by the domain module 104 to identify the domain in which the user 118 performs the task to be recorded, which can be the domain of the particular computer application or input form. If the user 118 later requests guidance for performing the same task in a different target domain, for example, using a different computer application or a different input form, the domain module 104 can identify the target domain. The user does not need to request guidance in different target domains. In some embodiments, the techniques described herein can be implemented to provide a user with unsolicited guidance on how to perform a semantic task similar to a semantic task previously performed in another domain by simply operating a different computing application or input form.

[0040] In some embodiments, the domain module 104 may also be configured to retrieve domain knowledge from a variety of different sources associated with the identified domain. In some such embodiments, for example, in addition to the previously mentioned natural language input or context information, this retrieved domain knowledge (and / or embeddings generated therefrom) may be provided to downstream components. This additional domain knowledge may allow downstream components (particularly machine learning models) to make more likely to be satisfactory predictions (e.g., generate guidance to perform semantic tasks across different domains).

[0041] In some embodiments, the domain module 104 may apply the collected contextual information (e.g., current state) across one or more "domain selection" machine learning models 105 that are different from the domain models described herein. These domain selection machine learning models 105 may take various forms, such as various types of neural networks, support vector machines, random forests, BERT transformers, etc. In various embodiments, the domain selection machine learning model 105 may be trained to select applicable domains based on attributes (or "context signals") of the current context or state of the user 118 and / or client device 120. For example, if the user 118 is operating an input form of a particular website to obtain goods or services, the uniform resource locator (URL) of the website or attributes of the underlying web page (such as keywords, tags, document object model (DOM) elements, etc.) may be used as input to the model in its native form or as a dimensionality reduction embedding application. Other context signals that may be considered include, but are not limited to, the user's IP address (e.g., work vs. home vs. mobile IP addresses), time of day, social media status, calendar, email / text messaging content, etc.

[0042] The interface module 106 may provide one or more graphical user interfaces (GUIs) that may be operated by various individuals, such as users 118-1 through 118-P, to perform various actions made available by the semantic task guidance system 102. In various implementations, the user 118 may operate a GUI (e.g., a standalone application or web page) provided by the interface module 106 to opt-in or opt-out of using the various techniques described herein. For example, the users 118-1 through 118-P may be required to provide explicit permission before any tasks they perform using the client devices 120-1 through 120-P are observed and used to generate guidance as described herein.

[0043] In addition, the interface module 106 can be configured to practice selected aspects of the present disclosure to present or cause presentation of guidance for performing semantic tasks in different domains. For example, the interface module 106 can receive one or more sample actions from an action space of a particular domain from the ML module 108. The interface module 106 can then cause graphical and / or audio data indicating these actions to be presented to the user.

[0044] Assume that a designer is operating a new computer-aided design (CAD) computer application, and the designer previously operated an old CAD computer application, for example, as part of their work. The actions of a given task that the designer frequently performs using the old CAD computer application can be processed, for example, by the ML module 108 using a domain model associated with the old CAD computer application to generate a domain-agnostic action embedding. The action embedding can then be converted, for example, by the ML module 108 into the domain of the new CAD computer application to generate (e.g., sample) one or more actions that can be performed using the new CAD computer application. The interface module 106 can use these actions to generate audio and / or visual instructions to the user, thereby explaining how to use the new CAD computer application to perform a given task.

[0045] ML module 108 may access data indicative of various global domain / machine learning models / strategies in database 111. These trained global domain / machine learning models / strategies may take various forms, including but not limited to graph-based networks (such as graph neural networks (GNNs), graph attention neural networks (GANNs), or graph convolutional neural networks (GCNs)), sequence-to-sequence models (such as encoder-decoders), various forms of recurrent neural networks (e.g., LSTM, GRU, etc.), BERT transformer networks, reinforcement learning strategies, and any other type of machine learning model that may be applied to facilitate selected aspects of the present disclosure. ML module 108 may process various data based on these machine learning models at the request or command of other components such as domain module 104 and / or interface module 106.

[0046] The task ID module 110 may be configured to analyze interactions between individuals and computer applications collected by the semantic coordination agent 122 (described in more detail below). Based on these observations, the task ID module 110 may determine which self-contained semantic tasks performed by individuals in one domain may be performed by the same individual or other individuals in other domains. In other words, the task ID module 110 may selectively trigger the creation of domain-agnostic action embeddings (e.g., by the ML module 108), which may then use the domain-agnostic action embeddings to sample actions in different domains for use in providing semantic task guidance across these domains.

[0047] In some embodiments, the task ID module 110 can selectively trigger the creation of domain-agnostic action embeddings on an individual basis. If a particular individual appears to repeatedly perform the same semantic task in one domain, guidance for performing the semantic task in other domains can be specifically provided to that individual. Additionally or alternatively, the task ID module 110 can selectively trigger the creation of domain-agnostic action embeddings that are applicable across groups of individuals. If different individuals are observed (e.g., a threshold number of times, with a threshold frequency, etc.) to perform the same semantic task across one or more domains, the task ID module 110 can be triggered to generate domain-agnostic embeddings. The interface module 106 and / or the ML module 108 can then use these domain-agnostic embeddings to provide guidance for performing semantic tasks to any number of different individuals.

[0048] In various embodiments, the task ID module 110 and / or the semantic coordination agent 122 may observe the individual's interactions only with the individual's permission. For example, when installing (or updating) a computer application onto a particular client device 120, the semantic coordination agent 122 may ask the individual's permission to observe the individual's interactions with the new computer application.

[0049] Each client device 120 may operate at least a portion of the semantic coordination agent 122 described above. The semantic coordination agent 122 may be a computer application that may be operated by the user 118 to perform selected aspects of the present disclosure to facilitate extension of semantic tasks across different domains. For example, the semantic coordination agent 122 may receive a request and / or permission from the user 118 to observe / record a sequence of actions performed by the user 118 using the client device 120 in order to complete a certain task. In the absence of such an explicit request or permission, the semantic coordination agent 122 may not be able to observe the user's interactions.

[0050] In some embodiments, the semantic coordination agent 122 can take the form of what is commonly referred to as a "virtual assistant" or "automated assistant" that is configured to engage in a human-computer natural language conversation with the user 118. For example, the semantic coordination agent 122 can be configured to semantically process natural language input provided by the user 118 to identify one or more intents. Based on these intents, the semantic coordination agent 122 can perform various tasks, such as operating a smart device, retrieving information, performing tasks, etc. In some embodiments, the conversation between the user 118 and the semantic coordination agent 122 (or a separate automated assistant accessible to / by the semantic coordination agent 122) can constitute a task sequence, which can be captured, abstracted into a domain-agnostic embedding, and then extended to other domains as described herein.

[0051] For example, a human-machine conversation between a user 118 and a semantic coordination agent 122 (or a separate automated assistant, or even between an automated assistant and a third-party application) for ordering a pizza from a third-party agent at a first restaurant (and therefore, a first domain) can be captured and used to generate an "order pizza" action embedding. This action embedding can later be extended to ordering pizza from a different restaurant, for example, via an automated assistant or via a separate interface.

[0052] exist Figure 1 , each of the client devices 120-1 may include a semantic coordination agent 122-1 that serves the first user 118-1. The first user 118-1 and his / her semantic coordination agent 122-1 may have access to and / or may be associated with a "profile" that includes various data related to performing selected aspects of the present disclosure on behalf of the first user 118-1. For example, the semantic coordination agent 122 may access one or more edge databases or data stores associated with the first user 118-1, including an edge database 124-1 that stores local domain models and action embeddings and / or another edge database 126-1 that stores recorded actions. Other users 118 may have similar arrangements. Any data stored in the edge databases 124-1 and 126-1 may be stored in part or in whole on the client device 120-1, for example, to protect the privacy of the first user 118-1. For example, the recorded actions 126 - 1 may be stored locally on the client device 120 - 1 in its original form, and the recorded actions 126 - 1 may include sensitive and / or personal information of the first user 118 - 1 , such as payment information, address, phone number, etc.

[0053] The local domain models stored in the edge database 124-1 may include, for example, local versions of global models stored in the global domain model database 111. For example, in some embodiments, global models may be propagated to the edges in order to guide the semantic coordination agent 122 to expand tasks into new domains associated with those propagated models; thereafter, local models at the edges may or may not be trained locally based on the activities and / or feedback of users 118. In some such embodiments, local models (alternatively referred to as "local gradients" in the edge database 124) may be periodically used to train global models (in the database 111), for example, as part of a federated learning framework. When a global model is trained based on local models, the global model may be propagated back to other edge databases (124) in some cases to keep the local models up to date.

[0054] However, it is not required to employ federated learning in all embodiments. In some embodiments, the semantic coordination agent 122 may provide scrubbed data to the semantic task guidance system 102, and the ML module 108 may remotely apply the model to the scrubbed data. In some embodiments, the "scrubbed" data may be data from which sensitive and / or personal information has been removed and / or obfuscated. In some embodiments, personal information may be scrubbed by the semantic coordination automation agent 122 based on various rules, for example, at the edge. In other embodiments, the scrubbed data provided to the semantic task guidance system 102 by the semantic coordination agent 122 may be in the form of a dimensionality reduced embedding generated from the original data at the client device 120.

[0055] As previously described, the edge database 126-1 can store actions recorded by the semantic coordination agent 122-1. The semantic coordination agent 122-1 can observe and / or record actions in a variety of different ways, depending on the level of access the semantic coordination agent 122-1 has to the computer application executing on the client device 120-1 and the permissions granted by the user 118-1. For example, most smartphones include an operating system (OS) interface for providing or revoking permissions (e.g., location, access to the camera, etc.) to various computer applications. In various embodiments, such an OS interface can be operable to provide / revoke access to the semantic coordination agent 122, and / or select a particular level of access that the semantic coordination agent 122 will have to a particular computer application.

[0056] The semantic coordination agent 122-1 may have various levels of access to the workings of the computer application, depending on the permissions granted by the user 118 and the cooperation from the software developer providing the computer application. For example, with the permission of the user 118, some computer applications may provide the semantic coordination agent 122 with "under-the-hood" access to the application's API or scripts written in a programming language (e.g., macros) embedded in the computer application. Other computer applications may not provide as much access. In such cases, the semantic coordination agent 122 may record the actions in other ways, such as by capturing screenshots, performing optical character recognition (OCR) on those screenshots to identify menu items, and / or monitoring user input (e.g., interrupts captured by the OS) to determine which graphical elements are manipulated by the user 118 in what order. In some embodiments, the semantic coordination agent 122 may intercept actions performed using the computer application from data exchanged between the computer application and the underlying OS (e.g., via system calls). In some implementations, semantic coordination agent 122 may intercept and / or access data exchanged between or used by window managers and / or window systems.

[0057] Figure 2 Schematically depicts examples of how data may be processed and / or used by various components across multiple domains. Starting from the top left, a user 118 operates a client device 120 to request or provide permission for a semantic coordination agent 122 operating at least in part on the client device 120 to observe the user 118's interactions with a first computer application APP A. In various embodiments, the semantic coordination agent 122 cannot record the actions without receiving the permission. In some embodiments, the permission may be granted on an application-by-application basis, to the extent that applications are granted permission to access GPS coordinates, local files, use of an onboard camera, and the like. In other embodiments, the permission may only be granted until the user 118 specifies otherwise, for example, by pressing a "stop recording" button similar to recording a macro or by providing voice input such as "stop recording" or "that's it."

[0058] Once the request / permission is received, in some embodiments, the semantic coordination agent 122 can acknowledge (ACK) the request / permission, although this is not required. Some time later, the user 118 can launch APP A and use the client device 120 to perform a sequence of actions {A1, A2, ...} in domain A; these actions can be captured and stored in the edge database 126. These actions {A1, A2, ...} can take various forms or combinations of forms, such as command line input, and interaction with one or more graphical elements of the GUI using various types of input (such as pointer device (e.g., mouse) input, keyboard input, voice input, gaze input, and any other type of input capable of interacting with graphical elements of the GUI).

[0059] In some embodiments, groups of actions that are logically performed together (e.g., within a specific time interval, without interruption, etc.) can be grouped together as a semantic task, for example, by the task ID module 110. For example, a user can perform actions A1-A6 during one session, stop interacting with APP A for a period of time, and then perform actions A7-A15 later. In various embodiments, actions A1-A6 can be grouped together as one semantic task, and actions A7-A15 can be grouped together as another semantic task.

[0060] In various embodiments, the domain (A) in which these actions are performed can be identified, for example, by the domain module 104 using any number of signals, such as the fact that the user 118 launched APP A, as well as other signals that are available. These other signals can include, for example, natural language input (NLI) provided by the user, the user's calendar, the user's email, the user's social media posts, etc.

[0061] In various embodiments, the semantic coordination agent 122 may observe / record the actions {A1, A2, ...} and pass them (or data indicative thereof, such as reduced dimension embeddings) to another component, such as the ML module 108 ( Figure 2 ML module 108 may then process these actions using all or part of domain model A to generate domain-agnostic action embeddings A' (also referred to as "intermediate representations"). In various implementations, domain model A may include, for example, an encoder portion of a larger encoder-decoder architecture, which may be used to process sequences of tokens (e.g., actions {A1, A2, ...}) to generate intermediate representations, such as action embeddings A'.

[0062] In some embodiments, and as indicated by the dashed line, the user 118 may optionally provide an NLI-1 to describe what the user 118 is doing when performing the action {A1, A2, ...}. This NLI-1 may be captured by the semantic coordination agent 122, which may pass it to the ML module 108 for natural language processing to generate a task embedding T'. The task embedding T' may be used to provide additional context for the action {A1, A2, ...}. This additional context may be used in various ways, such as additional input for a domain model, or as an anchor to allow semantically similar actions to be requested in the future (by the user 118 or others). As indicated by the dashed line, in some embodiments, the task embedding T' and the action embedding A' may be associated with each other, for example in a database via a shared / joint embedding space, etc.

[0063] In some implementations, if the user 118 does not provide natural language input describing the action {A1, A2, ...}, the semantic coordination agent 122 may formulate (or cause formulation) a predicted description of the action, and then solicit feedback from the user 118 regarding the accuracy or quality of the description. Figure 2 , for example, the additional dashed arrow shows how the semantic coordination agent 122 uses the action embedding A' to generate a natural language output (NLO), for example, by processing (or causing processing) the embedding A' using a semantic decoder machine learning model trained to translate between a domain-agnostic action embedding space and a natural language vocabulary. The semantic coordination agent 122 can then use this natural language output as a solicitation for feedback ( Figure 2 FB) is presented to user 118. User 118 can provide feedback (e.g., "yes, that's correct" or "no, you're wrong"). Based on the feedback, semantic coordination agent 122 can train domain model A or cause domain model A to be trained.

[0064] As depicted by the horizontal dashed line, some time later, the user 118 launches APP B, which causes the semantic coordination agent 122 to identify domain B as the active domain. In various embodiments, the semantic coordination agent 122 can cause the action embedding A' to be processed, for example, by the ML module 108 using the domain model B. For example, the action embedding A' can be processed using a decoder portion of an encoder-decoder network that together forms or is associated with domain B. The decoding can, for example, generate a probability distribution over the action space across domain B. Based on these probability distributions, various actions {B1, B2, ...} selected from the action space of domain B can be generated and provided to the semantic coordination agent 122. The semantic coordination agent 122 can then communicate with the interface module 106 ( Figure 2 ) collaborate to generate HCI guidance for user 118.

[0065] In various embodiments, components such as the semantic coordination agent 122 and the ML module 108 can continuously train the domain model on an ongoing basis, for example, to improve the quality of the HCI guidance provided, to enable the HCI guidance to be more narrowly tailored to a particular context, etc. Assume that when presented with the HCI guidance, the user 118 follows some parts of the guidance, but not others, and in a different order. For example, assume that the HCI guidance performs actions {B1, B2, B3, B4, B5, B6, B7} in order, while Figure 2 In the example, user 118 performs fewer than all actions and performs them in a different order: {B1, B5, B4, B3}. In some implementations, the difference between the recommended actions {B1, B2, B3, B4, B5, B6, B7} and the actions {B1, B5, B4, B3} finally performed by user 118 can be used as an error, which can then be used, for example, by ML module 108 to train domain model B, for example, using techniques such as gradient descent, back propagation, etc.

[0066] Figure 3A-3D An example is depicted of how a component such as the semantic coordination agent 122 may perform or cause to be performed selected aspects of the present disclosure to provide guidance for performing semantic tasks in a new domain. For this example, it should be assumed that a user (not depicted) is operating the HCI 360 in the form of a GUI rendered by or on behalf of a "hypothetical CAD" computer application. It should be further assumed that the user previously worked using a different CAD computer application called "FakeCAD" (e.g., as part of the user's job) to repeatedly perform any number of tasks. Finally, it should be assumed that the user has recently switched from using FakeCAD to using hypothetical CAD (e.g., as a result of accepting a new job).

[0067] exist Figure 3A , the HCI is in the Home state, where the "HOME" menu item is active. Therefore, context specific menu items such as "New", "Open", "Save" and "Print" are active. These are merely examples of what may be presented and are not meant to be limiting. On the left are a number of design tools (1-8), which may generally include tools commonly found in CAD programs, such as tools for drawing lines, ellipses, circles, shape fill tools, different brushes, etc.

[0068] In this example, domain-specific actions previously performed by a user while operating a previous CAD software FakeCAD to perform various semantic tasks have been processed to generate domain-agnostic action embeddings. In particular, a domain model trained to convert to / from an action space associated with FakeCAD is used to process these domain-specific actions into domain-agnostic action embeddings. One or more of these domain-agnostic action embeddings are then processed, for example, by the ML module 108 using a domain model that is configured to convert to / from an action space associated with a new software (hypothetical CAD).

[0069] The output of the process using the domain model for the hypothetical CAD may include a probability distribution over actions in the action space of the hypothetical CAD. Based on these probability distributions, the ML module 108 or the semantic coordination agent 122 may select one or more actions in the action space of the hypothetical CAD. These selected actions may then be used, for example, by the interface module 106 to generate HCI guidance that helps the user navigate the HCI 360 to perform the tasks they previously performed using FakeCAD.

[0070] For example, in Figure 3A In , HCI guidance is presented in the form of an overlay annotation that points to the "View" menu and explains that "Several adjustments you make consistently can be found here." Once the user selects the "View" menu as suggested, HCI 360 can transition to Figure 3B in the state depicted in .

[0071] exist Figure 3B , the View menu is now active, which causes several context-specific menu items to appear. Zoom, Rulers, Mode, Wireframe, and Connect are merely illustrative examples of what may be presented and are not meant to be limiting. New HCI guidance is now provided urging the user to interact with the Rulers menu item and the Wireframe menu item. This may be because, for example, the user has a habit of frequently adjusting similar parameters when using FakeCAD in the past.

[0072] exist Figure 3C , the Home menu is active again. The user is now presented with additional HCI guidance related to Tool-2. In particular, the user is informed via an overlay annotation that "this is the tool that corresponds to Tool-x in FakeCAD, which is the FakeCAD tool you use most frequently." This can be done at various times, such as Figure 3A-3B ) presents the HCI guidance to the user after the HCI guidance presented in Figure 3A-3BThe HCI guidance presented in corresponds to actions that users typically perform earlier (e.g., setting common parameters). Figure 3C The HCI guidance presented in may correspond to actions that a user would traditionally perform later (e.g., after the user has set the general parameters to their liking and is ready to work).

[0073] Figure 3D Depicted is, for example, once the user has selected Tool-2 (eg Figure 3D ) may be presented. Here, the HCI guidance is again an overlay visual annotation that points to the "Format" menu and informs the user that "when using this tool, you typically use an 8pt stroke with anti-aliasing - you can find those settings here". If the user were to select the "Format" menu, one or more graphical elements would likely become available for the user to select a stroke size.

[0074] Figure 4 4 is a flowchart illustrating an example method 400 for practicing selected aspects of the present disclosure according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of the semantic task guidance system 102. In addition, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0075] At block 402, the system, such as through domain module 104, may identify a first domain of a first computer application operable using a first HCI. Figure 3A , the fact that the user initiated the hypothetical CAD may cause domain module 104 to identify the hypothetical CAD domain as active. As previously described, domain module 104 may use any number of signals in addition to or in lieu of a computer application to identify the current domain. For example, if the user is operating an HMD to explore a virtual reality universe (sometimes referred to as a "metaverse"), the region of the metaverse that the user is currently exploring may be used to identify the domain.

[0076] Based on the domain identified at block 402, at block 404, the system, such as via semantic coordination agent 122 or ML module 108, may select a first domain model that converts between an action space of the first computer application and another space. Figure 3A-3D In the example above, the ML module 108 selects a hypothetical CAD domain model.

[0077] Based on the selected first domain model, at block 406, the system, for example, via the ML module 108, can process the domain-agnostic action embeddings to generate one or more probability distributions over actions in the action space of the first computer program. The action embeddings can represent a plurality of actions previously performed using a second HCI of a second computer application to perform a semantic task. Figure 3A-3D , where a domain-agnostic action embedding is previously generated based on user interaction with the HCI of a previous software FakeCAD, and the domain-agnostic action embedding is processed using a domain model of a hypothetical CAD to generate a probability distribution over the action space of the hypothetical CAD.

[0078] Based on the one or more probability distributions generated at box 406, at box 408, the system, for example, through the ML module 108 or the semantic coordination agent 122, can identify a second plurality of actions that can be performed using the first computer application. At box 410, the system can cause an output to be presented at one or more output devices, for example, through the interface module 106. The output may include guidance for navigating the first HCI to perform the semantic task using the first computer application. The guidance can be generated, for example, by the interface module 106 based on the identified second plurality of actions that can be performed using the first computer application. The HCI guidance can appear in various forms. Visually, it can be presented as an overlaid annotation (e.g., such as Figure 3A-3D ), animation, video, written natural language output (e.g., text balloons), 3D rendering (e.g., in a virtual reality or metaverse setting), etc. Auditorily, HCI guidance can be presented as natural language output, various noises or sound effects, etc.

[0079] The examples described herein focus primarily on providing HCI guidance across semantically similar computer applications, such as between a CAD computer application FakeCAD and a hypothetical CAD, or between different spreadsheet applications. However, this is not meant to be limiting. To the extent that a particular semantic task is relatively agnostic to a particular domain, the task can be used to generate actions in any number of domains that are otherwise dissimilar. For example, setting a particular computer application to "dark mode" can be relatively generic, and thus can be exploited to provide HCI guidance across a variety of domains, such as other computer applications or even operating systems.

[0080] As another example, an automated assistant (sometimes referred to as a "virtual" assistant or "virtual agent") can interface with any number of third-party agents to allow the automated assistant to act as a liaison for performing tasks such as ordering goods or services, making reservations, booking ride sharing, and the like. Different companies that provide similar services (e.g., ride sharing) may require users to interact with their respective third-party agents using natural language conversations. However, the natural language conversation that can be used to interact with a first ride sharing agent may be different from the conversation used to interact with a second ride sharing agent. Nevertheless, the final parameters or "slot values" that are filled in to complete the ride sharing request can be semantically similar, even if they are named differently, requested at different points during the conversation, and the like. Therefore, the techniques described herein can be used to provide HCI guidance to users, for example, as natural language, audibly or visually, graphical elements on a display, and the like, which can help users who are accustomed to a first ride sharing agent interact more effectively with a second ride sharing agent. For example, domain-agnostic action embeddings can be processed to generate a script for an automated agent that acts as a liaison. The script can solicit necessary parameters or slot values ​​from the user in a domain-agnostic manner. The automated agent can then use these solicited values ​​to interact with any ride-sharing agent without requiring the user to learn the nuances of each agent.

[0081] Figure 5 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing devices 120-1 through 120-P, the semantic task guidance system 102, and / or other components can include one or more components of the example computing device 510.

[0082] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0083] User interface input device 522 may include a keyboard, a pointing device (such as a mouse, trackball, touch pad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 510 or over a communication network.

[0084] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include devices and methods of all possible types of information output from computing device 510 to a user or another machine or computing device.

[0085] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a Figure 4 The logic of selected aspects of method 400.

[0086] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor 514.

[0087] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0088] Computing device 510 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 510 may have more Figure 5 The computing devices depicted may have greater or fewer components.

[0089] Although several embodiments have been described and illustrated herein, various other components and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using experiments that do not exceed routine. Therefore, it should be understood that the foregoing embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, embodiments may be practiced in a manner different from that specifically described and claimed. Embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented using one or more processors, comprising: identifying a domain of a first computer application operable using a first human-computer interface (HCI); Based on the identified domain, selecting a domain model that converts between an action space of the first computer application and another space; processing the action embeddings to generate one or more probability distributions over actions in an action space of the first computer application based on the selected domain model, wherein the action embeddings represent a plurality of actions previously performed using a second HCI of a second computer application to perform a semantic task; identifying, based on the one or more probability distributions, a second plurality of actions that can be performed using the first computer application; and Causing output to be presented at one or more output devices, wherein the output includes instructions for navigating the first HCI to perform a semantic task using the first computer application, and wherein the instructions are based on the identified second plurality of actions that can be performed using the first computer application.

2. The method according to claim 1, wherein: The domain model is trained to convert between the action space of the first computer application and a domain-agnostic action embedding space.

3. The method according to claim 1 or 2, wherein: The domain model is trained to directly translate between the action space of the first computer application and the action space of the second computer application.

4. The method according to any one of claims 1 to 3, wherein: The first HCI includes a graphical user interface.

5. The method according to claim 4, wherein: The guidance for navigating the first HCI includes one or more visual annotations overlaying the GUI.

6. The method according to claim 5, wherein: One or more of the visual annotations are rendered to draw attention to one or more graphical elements of the GUI.

7. The method according to any one of claims 1 to 6, wherein: The guidance for navigating the first HCI includes one or more natural language outputs.

8. The method according to any one of claims 1 to 7, further comprising: Obtain user input that conveys a semantic task; as well as Recognizing action embeddings based on semantic tasks.

9. The method according to claim 8, wherein: The user input includes natural language input, and the method further includes: performing natural language processing (NLP) on the natural language input to generate a first task embedding representing a semantic task; and determining a similarity measure between the first task embedding and the action embedding; Among them, action embedding is processed based on similarity measurement.

10. A system comprising one or more processors and a memory storing instructions, wherein in response to execution of the instructions, the one or more processors: identifying a domain of a first computer application operable using a first human-computer interface (HCI); Based on the identified domain, selecting a domain model that converts between an action space of the first computer application and another space; Based on the selected domain model, the action embeddings are processed to generate one or more probability distributions over actions in an action space of the first computer application, wherein: The action embedding represents a plurality of actions previously performed using a second HCI of a second computer application to perform the semantic task; identifying, based on the one or more probability distributions, a second plurality of actions that can be performed using the first computer application; as well as Causing output to be presented at one or more output devices, wherein the output includes instructions for navigating the first HCI to perform a semantic task using the first computer application, and wherein the instructions are based on the identified second plurality of actions that can be performed using the first computer application.

11. The system according to claim 10, wherein: The domain model is trained to convert between the action space of the first computer application and a domain-agnostic action embedding space.

12. The system according to claim 10 or 11, wherein: The domain model is trained to directly translate between the action space of the first computer application and the action space of the second computer application.

13. A system according to any one of claims 10 to 12, wherein: The first HCI includes a graphical user interface.

14. The system according to claim 13, wherein: The guidance for navigating the first HCI includes one or more visual annotations overlaying the GUI.

15. The system of claim 14, wherein: One or more of the visual annotations are rendered to draw attention to one or more graphical elements of the GUI.

16. A system according to any one of claims 10 to 15, wherein: The guidance for navigating the first HCI includes one or more natural language outputs.

17. The system of any one of claims 10 to 16, further comprising instructions for: Obtaining user input conveying a semantic task; and Recognizing action embeddings based on semantic tasks.

18. The system of claim 17, wherein: The user input includes natural language input, and the system further includes instructions for: performing natural language processing (NLP) on the natural language input to generate a first task embedding representing a semantic task; and determining a similarity measure between the first task embedding and the action embedding; Among them, action embedding is processed based on similarity measurement.

19. A non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by a processor, cause the processor to: identifying a domain of a first computer application operable using a first human-computer interface (HCI); Based on the identified domain, selecting a domain model that converts between an action space of the first computer application and another space; Based on the selected domain model, the action embeddings are processed to generate one or more probability distributions over actions in an action space of the first computer application, wherein: The action embedding represents a plurality of actions previously performed using a second HCI of a second computer application to perform the semantic task; identifying, based on the one or more probability distributions, a second plurality of actions that can be performed using the first computer application; as well as Causing output to be presented at one or more output devices, wherein the output includes instructions for navigating the first HCI to perform a semantic task using the first computer application, and wherein the instructions are based on the identified second plurality of actions that can be performed using the first computer application.

20. The computer-readable medium of claim 19, wherein: The domain model is trained to convert between the action space of the first computer application and a domain-agnostic action embedding space.

Images