Encoding or decoding user interface interactions

Abstract

A method (600) for encoding and decoding user interface interactions includes receiving a query (132) from a user captured by an assistant-enabled device (102) associated with the user. The query includes a user intent to interact with an application (140). The method includes obtaining a set of categorized actions (142) for the application. Each categorized action in the set of categorized actions is associated with one or more parameters (144) and represents a high-level description of the user intent to interact with the application. For each categorized action in the set of categorized actions, the method includes selecting a respective script (402) associated with the respective categorized action that performs one or more low-level interactions with the application, and executing the respective script that represents the user intent to interact with the application.

Description

[Technical field]

[0001] The present disclosure relates to encoding and / or decoding user interface interactions and associated mobile application context. [Background technology]

[0002] In a speech-enabled environment (e.g., at home, work, school, automobile, etc.), a user can speak a query or command out loud to a computer-based system, which processes and responds to the query and / or performs a function based on the command. The speech-enabled environment may be implemented using a network of connected microphone devices located in various rooms or environmental areas. The speech-enabled environment may be controlled by a virtual assistant program or application. To respond to some queries or commands, the virtual assistant may execute or implement logic through zero-party (0P) interaction, first-party (1P) interaction, or third-party (3P) interaction. A 0P interaction is an interaction performed by the virtual assistant itself (i.e., by code or logic implemented within the virtual assistant application). A 1P interaction is an interaction with an application separate from the virtual assistant, but the separate application includes code or logic specifically designed for the operation of the virtual assistant application or for cooperation with the virtual assistant application. In contrast, a 3P interaction is an interaction with an application other than the virtual assistant that does not have code or logic designed to work with or cooperate with the virtual assistant application. Summary of the Invention

[0003] One aspect of the present disclosure provides a method for encoding and decoding user interface interactions. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations. The operations include receiving a query from a user captured by an assistant-enabled device associated with the user. The query includes a user intent to interact with the application. The operations include obtaining a set of categorized actions for the application. Each categorized action in the set of categorized actions is associated with one or more parameters and represents a high-level description of the user intent to interact with the application. For each categorized action in the set of categorized actions, the operations include selecting a respective script associated with the respective categorized action that performs one or more low-level interactions with the application, and executing the respective script that represents the user intent to interact with the application.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, obtaining the set of categorized actions includes generating, for each categorized action in the set of categorized actions, one or more contextual annotations using the application and converting the one or more contextual annotations to the respective categorized action. In some of these implementations, generating the one or more contextual annotations includes performing optical character recognition (OCR) or pattern recognition on image data corresponding to a captured screenshot of a user interface (UI) of the application. In some of these implementations, generating the one or more contextual annotations includes analyzing an accessibility tree associated with the application.

[0005] In some examples, the one or more low-level interactions include actuating a user input at a user interface (UI) of the application at a predetermined location on the UI of the application. Optionally, selecting the respective scripts is based on an identification of the application and a version of the application. Selecting the respective scripts may include passing to the respective script one or more parameters associated with the respective classified action.

[0006] In some implementations, when a query is received from a user, the application is in the foreground of a graphical user interface (GUI) executed by the assistant-enabled device. The assistant-enabled device may execute a digital assistant application configured to provide a digital assistant interface between the user and the assistant-enabled device, and an application that is separate and independent of the digital assistant application. In these examples, the assistant application may lack permission to execute the respective scripts for each categorized action of the set of categorized actions. The operating system of the assistant-enabled device, in some examples, executes the respective scripts for each categorized action of the set of categorized actions.

[0007] Another aspect of the present disclosure provides a system for encoding and decoding user interface interactions. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a query from a user captured by an assistant-enabled device associated with the user. The query includes a user intent to interact with the application. The operations include obtaining a set of categorized actions for the application. Each categorized action in the set of categorized actions is associated with one or more parameters and represents a high-level description of the user intent to interact with the application. For each categorized action in the set of categorized actions, the operations include selecting a respective script associated with the respective categorized action that performs one or more low-level interactions with the application, and executing the respective script that represents the user intent to interact with the application.

[0008] This aspect may include one or more of the following optional features: In some implementations, obtaining the set of categorized actions includes generating, for each categorized action in the set of categorized actions, one or more contextual annotations using the application and converting the one or more contextual annotations to the respective categorized action. In some of these implementations, generating the one or more contextual annotations includes performing optical character recognition (OCR) or pattern recognition on image data corresponding to a captured screenshot of a user interface (UI) of the application. In some of these implementations, generating the one or more contextual annotations includes analyzing an accessibility tree associated with the application.

[0009] In some examples, the one or more low-level interactions include actuating a user input at a user interface (UI) of the application at a predetermined location on the UI of the application. Optionally, selecting the respective scripts is based on an identification of the application and a version of the application. Selecting the respective scripts may include passing to the respective script one or more parameters associated with the respective classified action.

[0010] In some implementations, when a query is received from a user, the application is in the foreground of a graphical user interface (GUI) executed by the assistant-enabled device. The assistant-enabled device may execute a digital assistant application configured to provide a digital assistant interface between the user and the assistant-enabled device, and an application that is separate and independent of the digital assistant application. In these examples, the assistant application may lack permission to execute the respective scripts for each categorized action of the set of categorized actions. The operating system of the assistant-enabled device, in some examples, executes the respective scripts for each categorized action of the set of categorized actions.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1]FIG. 1 is a schematic diagram of an example system for encoding and decoding user interface interactions and associated mobile application context using classified intents and parameters. [Diagram 2] 2 is a schematic diagram of example components of the system of FIG. 1 for generating annotations. [Diagram 3] FIG. 2 is a schematic diagram of example actions and parameters of the system of FIG. 1. [Figure 4] FIG. 2 is a schematic diagram of an example script for the system of FIG. 1. [Diagram 5] FIG. 5 is a schematic diagram of the execution of the script of FIG. 4. [Figure 6] 1 is a flowchart of an exemplary arrangement of operations for encoding and decoding user interface interactions. [Figure 7] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0013] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In a speech-enabled environment (e.g., at home, work, school, automobile, etc.), a user can speak a query or command out loud to a computer-based system, which processes and responds to the query and / or performs a function based on the command. The speech-enabled environment may be implemented using a network of connected microphone devices located in various rooms or environmental areas. The speech-enabled environment may be controlled by a virtual assistant program or application. To respond to some queries or commands, the virtual assistant may execute or implement logic through zero-party (0P) interaction, first-party (1P) interaction, or third-party (3P) interaction. A 0P interaction is an interaction performed by the virtual assistant itself (i.e., by code or logic implemented within the virtual assistant application). A 1P interaction is an interaction with an application separate from the virtual assistant, but the separate application includes code or logic specifically designed for the operation of the virtual assistant application or for cooperation with the virtual assistant application. In contrast, a 3P interaction is an interaction with an application other than the virtual assistant that does not have code or logic designed to work with or cooperate with the virtual assistant application.

[0015] As virtual assistants become more capable, these assistants are increasingly being used to perform automated tasks for users. For example, a virtual assistant can be used to compose and send emails, perform Internet searches, make restaurant reservations, etc. To execute an automation query or command, a virtual assistant may implement its fulfillment logic using 0P, 1P, or 3P interactions. Although the implementation of automation using 0P or 1P relationships is straightforward, such implementations are not scalable. Instead, 3P is the most scalable way to improve the fulfillment capabilities of an automation product (such as a virtual assistant) because it does not require direct software development efforts (e.g., for a third-party application).

[0016] However, for the 3P fulfillment approach, most operating systems (OS) cannot directly perform low-level user interface (UI) interactions (e.g., activating a button or entering text into a text field) on behalf of the automation application because the automation application does not have system privileges or full trust over the user device. Some automation techniques rely on recording and playing back complex macros, but these macros tend to be difficult to understand, prone to runtime errors, and generally difficult to develop and / or debug.

[0017] Embodiments herein include systems and methods for encoding and decoding complex user interactions and associated application context that categorize user intent into high-level abstractions so that the abstractions cannot be used in security attacks (e.g., using the abstractions to perform arbitrary actions via a 3P application, such as repeatedly clicking a button at a particular location) when implemented as a programming interface.

[0018] 1, in some implementations, an exemplary system 100 includes user devices 102 associated with one or more users 10 and communicating with a remote system 111 via a network 104. The user devices 102 may correspond to computing devices such as mobile phones, computers (laptop or desktop), tablets, smart speakers / displays, smart appliances, smart headphones, wearables, vehicle infotainment systems, etc., and include data processing hardware 103 and memory hardware 105. The user devices 102 may include or communicate with one or more microphones 106 for capturing speech 119 from the respective users 10. The remote system 111 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources 112 (e.g., data processing hardware) and / or storage resources 114 (e.g., memory hardware).

[0019] The user device 102 at least partially executes an assistant application 110 (e.g., a virtual assistant application or a digital assistant application) or any other automated application. At least a portion of the assistant application 110 may execute on a remote system 111. The assistant application 110 receives a query 132 from the user 10. For example, the assistant application 110 obtains streaming audio 118 captured by the microphone(s) 106 from the user 10's speech 119. The user 10 communicates the query 132 to the user device 102 (i.e., the assistant application 110) via the speech 119. The user 10 may provide the query 132 via other user inputs (e.g., a touch screen, a keyboard, a mouse, etc.). The query 132 requests or commands the user device 102 (e.g., the assistant application 110) to perform one or more actions within or using the target application 140. The target application executes on the user device 102 (e.g., simultaneously with the assistant application 110). The target application 140 is, for example, running in the foreground of the user device 102. In some examples, the query 132 specifies the target application 140. In yet other examples, the assistant application 110 determines the target application 140 from contextual clues in the query 132 and other context information (e.g., applications installed on the user device 102, etc.).

[0020] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0021] The target application 140 is, in some implementations, a third-party application (i.e., 3P) that is separate and independent of the assistant application 110. That is, the target application 140 does not explicitly include programming that facilitates collaboration between the assistant application 110 and the target application 140. The target application 140 is, in some examples, created, managed, and / or distributed by a vendor or entity that is separate from the creator, manager, or distributor of the assistant application 110. The user device 102 may run the assistant application 110 and the target application 140 simultaneously. Here, the query 132 includes the utterance 119, “Send an email to James that says ‘Approved,’” and thus the target application 140 is an email application. For example, the target application 140 is a third-party application that the user 10 downloaded via an online application repository. In this example, the query 132 requests or commands the assistant application 110 to create an email using the target application 140. The email is sent to James, and the email includes the text, “Approved.”

[0022] In some implementations, the assistant application 110 includes or communicates with an automatic speech recognition (ASR) engine 130. The ASR engine 130 performs speech recognition on the query 132 (e.g., using a Markov model, a neural network, end-to-end ASR, etc.). The ASR engine 130 can execute in whole or in part on the user device 102 and / or the remote system 111. In some examples, the user device 102 processes the query 132 locally using the ASR engine 130. In other examples, the remote system 111 executes at least in part the ASR engine 130, and the user device 102 transmits data derived from the query 132 to the remote system 111 for some or all of the processing of the query 132. The query 132, in some implementations, includes a user intent to interact with the target application 140. Here, the query 132 includes "send an email to James that says 'approved'". In this example, the user intent is to compose an email using a target application 140 (eg, an email client).

[0023] In some examples, as described in more detail below, the assistant application 110 obtains a set of actions 142, 142a-n for the target application 140. Each action 142 may be classified according to a high-level description of the user's 10 intent to interact with the target application 140 (i.e., classified actions 142). Each action 142 may be associated with one or more parameters 144, 144a-n. In some examples, each action 142 and associated parameters 144 form a key-value pair to capture the user intent. The actions 142 may be similar to the classification granularity of user intent in natural language understanding (NLU) and fulfillment. That is, each set of actions 142 provides a high-level description of the user intent when using the target application 140. For example, if the target application 140 is an email client, one set of actions 142 describes composing an email (e.g., set recipient, set subject, set body, send, etc.), another set of actions 142 describes deleting the email, and a third set of actions describes forwarding the email. Each of these sets of actions 142 describes, at a high level, potential intents that the user 10 may have when interacting with the target application 140. In this example, an email client is included as the target application 140, but it is understood that the target application 140 may be any type of application having corresponding associated actions 142. For example, the target application 140 may be a navigation application, a social media application, a booking application, a calendar application, a shopping application, etc.

[0024] Each set of actions 142 may be categorized to represent a different primary user intent (e.g., sending an email versus deleting an email). Each set of actions 142 may include any number of actions 142 (i.e., one or more actions 142). The assistant application 110 and / or the ASR engine 130, or any other module executing on the user device 102 and / or the remote system 111 based on the query 132, selects one or more of the sets of actions 142 that best reflect the intent of the user 10 present in the query 132 (e.g., from the utterance 119). In this example, the categorized set of actions 142 for sending an email (FIG. 3) best reflects the user intent.

[0025] The user device 102 at least partially executes the intent controller 200. Some parts of the intent controller 200 may execute on the remote system 111. The intent controller 200 receives a set of categorized actions 142 and parameters 144, or alternatively receives a user intent from the ASR engine 130 and / or the assistant application 110, and selects a set of categorized actions 142 and associated parameters 144. For each action 142 in the set of actions 142, the intent controller 200 selects one or more scripts 402, 402a-n, which are stored, for example, in a script storage 404. The script storage 404 stores the scripts 402 at the user device 102 and / or the remote system 111. As described in more detail below, each selected script 402 executes one or more low-level interactions 206 with the target application 140. For example, the script 402 translates a high-level description of a set of categorized actions 142 into an application-specific UI interaction command, such as entering text in a text field. The intent controller 200 may pass the parameters 144 directly to the script 402 or may derive new parameters using additional context information for the value (eg, foreground application identification information).

[0026] The intent controller 200 executes each respective script 402 (i.e., scripts 402 that reflect a set of low-level behaviors 206 of actions 142) used to express a user intent to interact with a target application 140. In this example, a script 402 is selected and executed to compose an email to James with the body text "Approved." (FIG. 5). The intent controller 200 uses the low-level behaviors 206 (based on the execution of the scripts 402) to interact with the target application 140.

[0027] In some implementations, the intent controller 200 has a level of permission, access, or privileges that exceeds the level of permission of the assistant application 110. For example, the intent controller 200 is part of an operating system (OS) that runs on the user device 102. Due to trust or permission issues, the assistant application 110 may not be able or capable of performing the low-level operations 206 directly with the target application 140 (e.g., via executing the script 402). For example, direct interaction with the UI of the user device 102 (e.g., reproducing user input) requires a high level of trust or permission that is not given to the assistant application 110 and instead requires the trusted part (i.e., the intent controller 200) to perform. However, allowing the intent controller 200 to perform the low-level operations 206 directly on behalf of the assistant application 110 and / or the ASR engine 130 creates security vulnerabilities. For example, a malicious application may abuse that functionality to perform actions that are inconsistent with the user intent.

[0028] Instead, by restricting less trusted applications (e.g., assistant application 110) to use only high-level descriptions or abstractions of user intents to which, for example, user 10 has already agreed, system 100 mitigates any security risks. Using the illustrated example, user 10 may agree to create an email but not to delete an email. Thus, intent controller 200 can verify that user 10 has agreed to an action 142 proposed or selected by assistant application 110. For example, when assistant application 110 selects an action 142 related to deleting an email, intent controller 200 rejects such action 142. On the other hand, when assistant application 110 selects an action 142 related to creating an email, intent controller 200 proceeds to convert action 142 into appropriate script 402, with or without additional validation checks.

[0029] 2, in some implementations, the annotation generator 210 of the intent controller 200 obtains, determines, or generates a set of categorized actions 142 by using the target application 140 to generate one or more context annotations 220, 220a-n. The annotation generator 210 may convert the one or more context annotations 220 into the respective categorized action(s) 142. For example, the intent controller 200 generates the context annotations 220 using captured screenshots of the user interface 146 of the target application 140 via optical character recognition (OCR), image recognition, pattern recognition, etc. Additionally or alternatively, the intent controller 200 generates the context annotations 220 by parsing the accessibility tree 148 of the target application 140.

[0030] That is, since 3P applications such as the target application 140 do not usually declare the actions that the application can support, the intent controller 200 performs a step of checking the user intents that the target application 140 may support. For this purpose, the intent controller 200 (e.g., with the user's consent) generates contextual annotations 220 by using structural analysis and on-screen understanding of different graphical elements of the UI 146 of the target application 140 (e.g., via OCR or an application programming interface (API) 149). For example, the intent controller 200 and / or the user 10 move the target application 140 to the foreground of a graphical user interface (GUI) 107 running on the user device 102 such that the UI 146 of the target application 140 is visible. The annotation generator 210 converts the annotations 220 into classification actions 142.

[0031] Here, the UI 146 of the target application 140 has three context annotations 220, 220a-c. The first context annotation 220a represents a "To:" text field, the second context annotation 220b represents a "Subject:" text field, and the third context annotation 220c represents a "Compose email" text field. Based on these context annotations 220a-c, the intent controller 200 determines (e.g., via semantic processing) that the composition action 142 associated with the target application 140 is to compose and send an email. Note that in this example, the context annotations 220 are primarily English words, but the context annotations can be in any language and / or can be graphical representations (e.g., icons or other indicia).

[0032] In some implementations, the annotation generator 210 determines the required parameters 144 and parameter types for each action 142 determined from the context annotations 220. For example, based on the context annotations 220a, the annotation generator 210 determines that the action 142 associated with completing the "To:" field of an email requires a string as a parameter 144 and makes an appropriate association with the action 142. The intent controller 200 may sort or categorize the actions 142 together based on the intent that the different actions 142 represent. The intent controller 200 may share the actions 142 and parameters 144 with the assistant application 110 and / or the remote system 111.

[0033] 3, in some implementations, the assistant application 110 (or intent controller 200) selects a set of appropriate classification actions 142 based on the user intent determined by the ASR engine 130 from the query 132 (e.g., the user intent expressed in the query 132 via the utterance 119). In some examples, the target application 140 is in the foreground of the GUI 107 running on the user device 102. In other examples, the target application 140 is in the background or is not currently running on the user device 102. In either case, as shown by the schematic diagram 300, the assistant application 110 (or intent controller 200) receives target application information 310 that provides the assistant application 110 information about the target application 140. For example, the target application information 310 includes an identification of the target application 140 (i.e., an identification of the application currently in the foreground of the GUI 107), or other identification information (e.g., an identification of all email clients installed on the user device 102). Based on the query 132 and the target application information 310, the assistant application 110 selects an appropriate set of actions 142 that best reflects the user intent derived from the utterance 119.

[0034] In the illustrated example, three sets of categorized actions 142 are shown. A first set of actions 142a is directed to responding to the email, a second set of actions 142 is directed to sending the email, and a third set of actions 142n is directed to deleting the email. From the available options, the ASR engine 130 selects a set of actions 142b that represents the user intent to send the email (based on the utterance 119 in this example). Here, the parameters 144 associated with the selected set of actions 142b and provided by the query 132 include a first parameter 144a that indicates that the recipient of the email is "jjones@gmail.com" and a second parameter 144b that the body of the email contains the text "approved." Here, the second parameter 144b was extracted directly from the query 132. However, the first parameter 144a was derived from "James" instead. For example, the assistant application 110 retrieves all contacts with the first name James from the contacts of the user 10. Assistant application 110 may derive the value of parameter 144 from any other contextual information available to assistant application 110 and / or remote system 111 (e.g., frequency of messages from different contacts, recent messages, geographic location, name of foreground application, etc.).

[0035] The set of actions 142 may be associated with parameters 144 that cannot be extracted or derived from the query 132. In these cases, the assistant application 110 may apply default values ​​or no values ​​at all. The set of selected actions 142b (i.e., the “send email” action 142) and associated parameters 144a,b are then sent to the intent controller 200.

[0036] 4, in some implementations, the intent controller 200 includes a script flow controller 410. The script flow controller 410 selects an appropriate script (e.g., from the script storage 404) based on the selected action 142 and the target application information 310, where the target application information 310 may include information such as an identification of the target application 140, a package name associated with the target application 140, a version of the target application 140, etc. Because UI elements of an application vary between different applications (e.g., two different email clients) and between different versions of the same application (e.g., buttons and text boxes in different positions), the script flow controller 410 determines a script 402 that is compatible with the particular target application 140 based at least in part on the target application information 310. As shown in the schematic diagram 400, in this example the action 142 revolves around sending an email, but the script storage 404 may include many scripts for sending email based on different email clients and different versions of those email clients.

[0037] Each script 402, when executed, implements one or more low-level interactions with the target application 140. Here, low-level interactions 206 refer to interactions or actions with minimal abstraction between a description of the interaction and the actual implementation of the interaction. In some examples, a low-level interaction includes actuating a user input at a predetermined location on the UI 146 of the target application 140 on the UI 146 of the target application 140. That is, a script 402 may cause a button to be pressed or toggled, text to be entered, a slider to be moved, etc. on the UI 146 of the target application 140. For example, one script 402 may cause the target application 140 to register a UI selection at a particular location (to replicate a user 10 selecting a button on the UI 146). If the target application 140 includes an API 149 (FIG. 2), the intent controller 200 may additionally or alternatively interact with the target application 140 via the API 149.

[0038] The script flow controller 410 generates an execution flow 412 for the selected script 402 using the selected script 402 and, for example, logical operators (IF-THEN-ELSE, AND, OR, etc.). In some examples, the script flow controller 410 passes to each script 402 one or more parameters 144 associated with the script 402 and / or the action 142. Some scripts 402 may include parameters 144, while others may not. Here, the script flow controller 410 has three scripts 402 selected. The first script 402a "Create Email" is not associated with any parameters passed or derived from the query 132. For example, this action requires the selection of the "Create Email" button or other input that does not require parameters 144 from the user 10. The script flow controller 410 also selected a second script 402b "Email Recipients" and a third script 402c "Email Body". These two scripts are each associated with a first parameter 144a ("jjones@gmail.com") and a second parameter 144b ("Approved."). For example, the second script 402b uses the first parameter 144a to populate the recipient text boxes 510, 510a (FIG. 5).

[0039] In this example, the three scripts 402a-c are combined with an AND logical operator. However, it is understood that the script flow controller 410 may combine any number of scripts 402 with any number of operators (OR, IF-THEN-ELSE, etc.) depending on the complexity of the query 132 and the target application 140. The script flow controller 410 may pass the execution flow 412 to the script executor 420.

[0040] 5, as shown in schematic diagram 500, the script executor 420, in some examples, executes each script 402 according to an execution flow 412 to interact with the target application 140 using low-level actions 206 (e.g., via the script 402 that replicates the user 10 interacting with user inputs of the user device 102, such as a touch screen, keyboard, mouse, or via the API 149). In this example, the low-level actions 206 include entering "jjones@gmail.com" in the text box 510a and "approved" in the body text boxes 510, 510c. As shown in the examples of FIGS. 1-4, the user's utterance (i.e., "send an email to James saying 'approved'") did not include a parameter 144 associated with the subject text boxes 510, 510a. In some examples (such as the examples shown here), the intent controller 200 may rely on default values ​​instead of values ​​provided by the user 10. Here, the default value "approved" may depend on some or all of the body text. In other examples, the intent controller 200 prompts the user 10 for additional information or attempts to derive the parameter 144 from other sources.

[0041] 6 is a flowchart of an exemplary arrangement of operations of a method 600 for encoding and decoding user interface interactions. The computer-implemented method 600, when executed by the data processing hardware 510, causes the data processing hardware 103, 112 to perform operations. The method 600 includes, in operation 602, receiving a query 132 from the user 10 captured by an assistant-enabled device 102 (i.e., the user device 102) associated with the user 10. The query 132 includes a user intent to interact with the application 140. The method 600 includes, in operation 604, obtaining a set of categorized actions 142 for the application 140. Each categorized action 142 in the set of categorized actions 142 is associated with one or more parameters 144 and represents a high-level description of the user intent of the user 10 to interact with the application 140. At operation 606, the method 600 includes, for each respective categorized action 142 in the set of categorized actions 142, selecting a respective script 402 associated with the respective categorized action 142. The respective script 402 performs one or more low-level operations or interactions 206 with the application 140. At operation 608, the method 600 includes, for each respective categorized action 142 in the set of categorized actions 142, executing the respective script 402 to represent a user intent to interact with the application 140.

[0042] 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. The computing device 700 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are meant to be illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0043] Computing device 700 includes a processor 710, memory 720, storage device 730, high-speed interface / controller 740 that connects to memory 720 and high-speed expansion port 750, and low-speed interface / controller 760 that connects to low-speed bus 770 and storage device 730. Each component 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or otherwise present as desired. Processor 710 (i.e., data processing hardware) may include data processing hardware 103 of user device 102 and / or data processing hardware 112 of a remote system. Data processing hardware 710 may process instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 connected to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and memory types. Also, multiple computing devices 700 may be connected together (eg, as a bank of servers, a group of blade servers, or a multi-processor system) with each device providing a portion of the required operations.

[0044] The memory (i.e., memory hardware) 720 stores information non-transiently within the computing device 700. The memory hardware 720 may include the memory hardware 105 of the user device 102 and / or the memory hardware 114 of the remote system 111. The memory 720 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0045] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0046] The high-speed controller 740 manages the bandwidth-intensive operations of the computing device 700, and the low-speed controller 760 manages the bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is coupled to a memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and can connect, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0047] Computing device 700 may be implemented in a number of different forms, as shown in the figure: for example, it may be implemented as a standard server 700a, or multiple times within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0048] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system.

[0049] These computer programs (also known as programs, software, software applications, or codes) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0050] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data to generate output. The processors and logic flows can also be executed by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operatively coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0051] To provide for interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0052] Several implementations have been described. Of course, it is understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (600), when executed by data processing hardware (710), causing the data processing hardware (710) to: receiving a query (132) from a user captured by an assistant-enabled device (102) associated with the user, the query (132) including a user intent to interact with an application (140); obtaining a set of categorized actions (142) for the application (140), each categorized action (142) in the set of categorized actions (142) being associated with one or more parameters (144) and representing a high level description of the user's intent to interact with the application (140); For each categorized action (142) in the set of categorized actions (142), selecting a respective script (402) associated with each of the classified actions (142), the respective script (402) performing one or more low-level interactions (206) with the application (140); executing the respective script (402) to represent the user's intent to interact with the application (140); The method further comprises:

2. Obtaining the set of categorized actions (142) includes, for each categorized action (142) in the set of categorized actions (142), generating one or more contextual annotations (220) using said application (140); converting said one or more contextual annotations (220) into said respective classified actions (142); 10. The method of claim 1, comprising:

3. 3. The method (600) of claim 2, wherein generating the one or more contextual annotations (220) comprises performing optical character recognition (OCR) or pattern recognition on image data corresponding to a captured screenshot of a user interface (146) (UI) of the application (140).

4. 3. The method (600) of claim 2, wherein generating the one or more contextual annotations (220) comprises parsing an accessibility tree (148) associated with the application (140).

5. The method (600) of any of claims 1 to 4, wherein performing the one or more low-level interactions (206) comprises actuating, in a user interface (146) (UI) of the application (140), a user input at a predetermined location on the UI of the application (140).

6. The method (600) according to any of claims 1 to 5, wherein selecting the respective script (402) is based on an identification of the application (140) and a version of the application (140).

7. 7. The method (600) of any of claims 1 to 6, wherein selecting the respective script (402) comprises passing to the respective script (402) the one or more parameters (144) associated with the respective classified action (142).

8. The method (600) of any one of claims 1 to 7, wherein the application (140) is in the foreground of a graphical user interface (107) (GUI) executed by the assistant-enabled device (102) when the query (132) is received from the user.

9. The assistant-enabled device (102) A digital assistant application (110) configured to provide a digital assistant interface between the user and the assistant-enabled device (102); The application (140) is separate from and independent of the digital assistant application (110); The method (600) according to any one of claims 1 to 8, further comprising:

10. 10. The method of claim 9, wherein the digital assistant application lacks permission to execute the respective script for each categorized action of the set of categorized actions.

11. 11. The method of claim 10, wherein an operating system of the assistant-enabled device executes the respective script for each categorized action in the set of categorized actions.

12. Data processing hardware (710); memory hardware (720) in communication with the data processing hardware (710), storing instructions that, when executed on the data processing hardware (710), cause the data processing hardware (710) to: Receiving a query (132) from a user captured by an assistant-enabled device (102) associated with the user, the query (132) including a user intent to interact with an application (140); obtaining a set of categorized actions (142) for the application (140), each categorized action (142) in the set of categorized actions (142) being associated with one or more parameters (144) and representing a high level description of the user's intent to interact with the application (140); For each categorized action (142) in the set of categorized actions (142), selecting a respective script (402) associated with each of the classified actions (142), the respective script (402) performing one or more low-level interactions (206) with the application (140); executing the respective scripts (402) to represent the user's intent to interact with the application (140); A system (100) comprising:

13. Obtaining the set of categorized actions (142) includes, for each categorized action (142) in the set of categorized actions (142), generating one or more contextual annotations (220) using said application (140); converting said one or more contextual annotations (220) into said respective classified actions (142); The system (100) of claim 12, comprising:

14. 14. The system (100) of claim 13, wherein generating the one or more contextual annotations (220) comprises performing optical character recognition (OCR) or pattern recognition on image data corresponding to a captured screenshot of a user interface (146) (UI) of the application (140).

15. 14. The system (100) of claim 13, wherein generating the one or more contextual annotations (220) comprises parsing an accessibility tree (148) associated with the application.

16. The system (100) of any of claims 12 to 15, wherein performing the one or more low-level interactions (206) includes actuating a user input at a user interface (146) (UI) of the application at a predetermined location on the UI of the application.

17. The system (100) of any of claims 12 to 16, wherein the selecting of the respective script (402) is based on an identity of the application and a version of the application.

18. The system (100) of any of claims 12 to 17, wherein selecting the respective script (402) comprises passing to the respective script (402) the one or more parameters (144) associated with the respective classified action (142).

19. The system (100) of any one of claims 12 to 18, wherein the application is in the foreground of a graphical user interface (107) (GUI) executed by the assistant-enabled device when the query (132) is received from the user.

20. the assistant-enabled device, A digital assistant application (110) configured to provide a digital assistant interface between the user and the assistant-enabled device; The application (140) is separate from and independent of the digital assistant application (110); The system (100) according to any one of claims 12 to 19, further comprising:

21. 21. The system of claim 20, wherein the digital assistant application lacks permission to execute the respective script for each categorized action of the set of categorized actions.

22. 22. The system of claim 21, wherein an operating system of the assistant-enabled device executes the respective scripts for each categorized action in the set of categorized actions.

Images