Encoding or decoding of user interface interactions

Abstract

This invention provides a method for encoding and decoding user interface interactions. [Solution] The method includes receiving a query (132) from a user captured by an assistant-enabled device (102). The query includes a user intent to interact with an application (140). The method obtains a set of categorized actions (142) for the application. Each categorized action is associated with one or more parameters (144) and represents a high-level description of the user intent of the user to interact with the application. For each categorized action, the method selects the respective script (402) associated with each categorized action that performs a low-level interaction with the application and executes the respective script that represents the user intent to interact with the application.

Description

Technical Field

[0003]

[0001] The present disclosure relates to encoding and / or decoding user interface interactions and related mobile application contexts.

Background Art

[0002] In a voice interaction environment (e.g., home, workplace, school, automobile, etc.), a user can speak a query or command to a computer-based system, and the computer-based system processes the query and responds, and / or executes a function based on the command. The voice interaction environment can be implemented using a network of connected microphone devices arranged in various rooms or environmental areas. The voice interaction environment may be controlled by a virtual assistant program or application. To respond to some queries or commands, the virtual assistant may execute or implement logic via zero-party (0P) interaction, first-party (1P) interaction, or third-party (3P) interaction. 0P interaction is an interaction executed by the virtual assistant itself (i.e., by code or logic implemented within the virtual assistant application). 1P interaction is an interaction between the virtual assistant and another application, but the other application includes code or logic specially designed for the operation of the virtual assistant application or for cooperation with the virtual assistant application. In contrast, 3P interaction is an interaction between the virtual assistant and another application that does not have code or logic designed for the operation of the virtual assistant application or for cooperation with the virtual assistant application.

Summary of the Invention

[0003] One aspect of this disclosure provides a method for encoding and decoding user interface interactions. When executed by data processing hardware, the computer implementation method causes the data processing hardware to perform an operation. The operation includes receiving a query from the user, captured by an assistant-enabled device associated with the user. The query includes the user's intent to interact with the application. The operation includes obtaining a set of categorized actions for the application. Each categorized action in the set of categorized actions is associated with one or more parameters and represents a high-level description of the user's intent to interact with the application. For each categorized action in the set of categorized actions, the operation includes selecting the respective script associated with each categorized action that performs one or more low-level interactions with the application, and executing the respective script that represents the user's intent to interact with the application.

[0004] Embodiments of this disclosure may include one or more of the following optional features. In some embodiments, obtaining a set of classified actions includes, for each classified action in the set of classified actions, using the application to generate one or more contextual annotations and converting one or more contextual annotations to their respective classified actions. In some of these embodiments, generating one or more contextual annotations includes performing optical character recognition (OCR) or pattern recognition on image data corresponding to captured screenshots of the application's user interface (UI). In some of these embodiments, generating one or more contextual annotations includes parsing an accessibility tree associated with the application.

[0005] In some examples, one or more low-level interactions involve the application's user interface (UI) and triggering user input at a given location on the UI. Optionally, the selection of each script is based on the application's identification information and version. Selecting each script may involve passing one or more parameters to the script that relate to its respective categorized action.

[0006] In some embodiments, when an application receives a query from a user, it resides in the foreground of a graphical user interface (GUI) run by an assistant-enabled device. The assistant-enabled device may run a digital assistant application configured to provide a digital assistant interface between the user and the device, and applications separate from and independent of the digital assistant application. In these examples, the assistant application may lack permission to execute the respective scripts for each categorized action in a set of categorized actions. In some examples, the operating system of the assistant-enabled device executes the respective scripts for each categorized action in a set of categorized actions.

[0007] Another aspect of this disclosure provides a system for encoding and decoding user interface interactions. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform an action when executed by the data processing hardware. An action includes receiving a query from the user, captured by an assistant-enabled device associated with the user. The query includes the user's intent to interact with the application. An action includes obtaining a set of categorized actions for the application. Each categorized action in the set of categorized actions is associated with one or more parameters and represents a high-level description of the user's intent to interact with the application. For each categorized action in the set of categorized actions, an action includes selecting the respective script associated with each categorized action that performs one or more low-level interactions with the application, and executing the respective script that represents the user's intent to interact with the application.

[0008] This embodiment may include one or more of the following optional features. In some embodiments, obtaining a set of classified actions includes, for each classified action in the set of classified actions, using the application to generate one or more contextual annotations and converting one or more contextual annotations to their respective classified actions. In some of these embodiments, generating one or more contextual annotations includes performing optical character recognition (OCR) or pattern recognition on image data corresponding to captured screenshots of the application's user interface (UI). In some of these embodiments, generating one or more contextual annotations includes parsing an accessibility tree associated with the application.

[0009] In some examples, one or more low-level interactions involve the application's user interface (UI) and triggering user input at a given location on the UI. Optionally, the selection of each script is based on the application's identification information and version. Selecting each script may involve passing one or more parameters to the script that relate to its respective categorized action.

[0010] In some embodiments, when an application receives a query from a user, it resides in the foreground of a graphical user interface (GUI) run by an assistant-enabled device. The assistant-enabled device may run a digital assistant application configured to provide a digital assistant interface between the user and the device, and applications separate from and independent of the digital assistant application. In these examples, the assistant application may lack permission to execute the respective scripts for each categorized action in a set of categorized actions. In some examples, the operating system of the assistant-enabled device executes the respective scripts for each categorized action in a set of categorized actions.

[0011] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings and the claims. [Brief explanation of the drawing]

[0012] [Figure 1]This is a schematic diagram of an exemplary system for encoding and decoding user interface interactions and associated mobile application contexts using classified intents and parameters. [Figure 2] This is a schematic diagram of the exemplary components of the system shown in Figure 1 for generating annotations. [Figure 3] Figure 1 is a schematic diagram illustrating the exemplary actions and parameters of the system. [Figure 4] Figure 1 is a schematic diagram of an example script for the system shown. [Figure 5] This is a schematic diagram showing the execution of the script in Figure 4. [Figure 6] This is a flowchart illustrating an exemplary arrangement of operations for encoding and decoding user interface interactions. [Figure 7] This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0013] Similar reference symbols in various drawings refer to the same elements. [Modes for carrying out the invention]

[0014] In a voice-enabled environment (e.g., home, workplace, school, car), a user can voice queries or commands to a computer-based system, which then processes and responds to the queries and / or performs functions based on the commands. A voice-enabled environment can be implemented using a network of connected microphone devices placed in various rooms or environmental areas. The voice-enabled environment may be controlled by a virtual assistant program or application. To respond to certain queries or commands, the virtual assistant may execute or implement logic through zero-party (0P) interactions, first-party (1P) interactions, or third-party (3P) interactions. 0P interactions are interactions performed by the virtual assistant itself (i.e., by code or logic implemented within the virtual assistant application). 1P interactions are interactions with an application other than the virtual assistant, but the other application contains code or logic specifically designed for the operation of the virtual assistant application or for interaction with the virtual assistant application. In contrast, 3P interaction is an interaction with an application other than the virtual assistant that does not have any code or logic designed for operation with or interaction with the virtual assistant application.

[0015] As virtual assistants become more sophisticated, they are increasingly being used to perform automated tasks for users. For example, virtual assistants can be used to compose and send emails, perform internet searches, make restaurant reservations, and more. To execute automated queries or commands, virtual assistants may use 0P, 1P, or 3P interactions to implement their fulfillment logic. While embodiments of automation using 0P or 1P relationships are straightforward, such embodiments are not scalable. Instead, 3P is the most scalable way to improve the fulfillment capabilities of automated products (such as virtual assistants) because it does not require direct software development work (e.g., for third-party applications).

[0016] However, in the 3P fulfillment approach, most operating systems (OS) cannot directly perform low-level user interface (UI) interactions (e.g., activating a button or entering text into a text field) on behalf of the automated application. This is because the automated application does not have system privileges or complete trust in the user device. Some automation techniques rely on recording and playing back complex macros, but these macros tend to be difficult to understand, prone to runtime errors, and generally difficult to develop and / or debug.

[0017] Embodiments of this specification include systems and methods for encoding and decoding complex user interactions and associated application contexts. The systems and methods classify user intent into high-level abstractions so that when the abstractions are implemented as programming interfaces, the abstractions are not used for security attacks (for example, to perform arbitrary actions via a 3P application, such as repeatedly clicking a button at a specific location).

[0018] Referring to Figure 1, in some embodiments, the exemplary system 100 includes a user device 102 associated with one or more users 10, which communicates with a remote system 111 via a network 104. The user device 102 may correspond to a computing device such as a mobile phone, a computer (laptop or desktop), a tablet, a smart speaker / display, a smart appliance, smart headphones, a wearable, or a vehicle infotainment system, and comprises data processing hardware 103 and memory hardware 105. The user device 102 may include one or more microphones 106 for capturing utterances 119 from each user 10, or may communicate with one or more microphones. The remote system 111 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) and / or storage resources 114 (e.g., memory hardware) having scalable / elastic computing resources 112 (e.g., data processing hardware).

[0019] User device 102 runs at least partially an assistant application 110 (e.g., a virtual assistant application or a digital assistant application) or any other automation application. At least a portion of the assistant application 110 may run on a remote system 111. The assistant application 110 receives a query 132 from user 10. For example, the assistant application 110 obtains streaming audio 118 captured by a microphone 106 from user 10's utterance 119. User 10 communicates the query 132 to user device 102 (i.e., assistant application 110) via utterance 119. User 10 may provide the query 132 via other user input (e.g., a touchscreen, keyboard, mouse, etc.). The query 132 requests or commands user device 102 (e.g., assistant application 110) to perform one or more actions within or using a target application 140. The target application runs on user device 102 (e.g., concurrently with assistant application 110). The target application 140 is running, for example, in the foreground on user device 102. In some examples, query 132 specifies the target application 140. In yet another example, the assistant application 110 determines the target application 140 from contextual clues in query 132 and other contextual information (for example, applications installed on user device 102, etc.).

[0020] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0021] In some embodiments, the target application 140 is a third-party application (i.e., a 3P) that is separate from and independent of the assistant application 110. That is, the target application 140 does not explicitly include any programming that facilitates cooperation between the assistant application 110 and the target application 140. In some examples, the target application 140 is created, managed, and / or distributed by a vendor or operator different from the creator, administrator, or distributor of the assistant application 110. The user device 102 may run the assistant application 110 and the target application 140 simultaneously. Here, query 132 includes the utterance 119 “Send an email to James saying ‘Approved’,” and therefore the target application 140 is an email application. For example, the target application 140 is a third-party application downloaded by user 10 via an online application repository. In this example, query 132 requests or commands the assistant application 110 to create an email using the target application 140. The email is sent to James and contains the text “Approved.”

[0022] In some embodiments, the assistant application 110 includes or communicates with an automatic speech recognition (ASR) engine 130. The ASR engine 130 performs speech recognition on a query 132 (e.g., using a Markov model, neural network, end-to-end ASR, etc.). The ASR engine 130 can be executed wholly or partially on the user device 102 and / or the remote system 111. In some examples, the user device 102 uses the ASR engine 130 to locally process the query 132. In other examples, the remote system 111 executes the ASR engine 130 at least partially, and the user device 102 sends data derived from the query 132 to the remote system 111 for some or all of the processing of the query 132. The query 132 includes, in some embodiments, a user intent that interacts with the target application 140. Here, the query 132 includes "sending an email with 'approved' to James". In this example, the user intent is to create an email using the target application 140 (e.g., an email client, etc.).

[0023] In some examples, as will be described in more detail below, the assistant application 110 retrieves a set of actions 142, 142a-n for the target application 140. Each action 142 may be categorized according to a high-level description of the user intent of user 10 interacting with the target application 140 (i.e., categorized action 142). Each action 142 may be associated with one or more parameters 144, 144a-n. In some examples, each action 142 and associated parameter 144 form a key-value pair to capture the user intent. The actions 142 may be analogous to the classification granularity of user intent in natural language understanding (NLU) and fulfillment; that is, each set of actions 142 provides a high-level description of the user intent when using the target application 140. For example, if the target application 140 is an email client, one set of actions 142 describes creating an email (e.g., setting recipients, setting a subject, setting a body, sending, etc.), another set of actions 142 describes deleting an email, and a third set of actions describes forwarding an email. Each of these sets of actions 142, at a higher level, describes the potential intentions that user 10 may have when interacting with the target application 140. In this example, the target application 140 is an email client, but it is understood that the target application 140 could be any type of application with corresponding related actions 142. For example, the target application 140 could be a navigation application, a social media application, a booking application, a calendar application, a shopping application, etc.

[0024] Each set of actions 142 can be classified to represent different major user intentions (e.g., sending an email vs. deleting an email). Each set of actions 142 can include any number of actions 142 (i.e., one or more actions 142). The assistant application 110 and / or the ASR engine 130, or any other module that executes based on a query on the user device 102 and / or the remote system 111, selects one or more of the sets of actions 142 that best reflect the intention of the user 10 present in the query 132 (e.g., from utterance 119). In this example, the set of classified actions 142 for sending an email (Figure 3) best reflects the user intention.

[0025] The user device 102 runs the intent controller 200 at least partially. Some parts of the intent controller 200 can run on the remote system 111. The intent controller 200 receives a set of classified actions 142 and parameters 144, or alternatively, receives user intent from the ASR engine 130 and / or the assistant application 110, and selects a set of classified actions 142 and associated parameters 144. For each action 142 in the set of actions 142, the intent controller 200 selects one or more scripts 402, 402a~n, which are stored, for example, in the script storage 404. The script storage 404 stores the scripts 402 on the user device 102 and / or the remote system 111. Each selected script 402 performs one or more low-level interactions 206 with the target application 140, as will be described in more detail below. For example, a script 402 translates a high-level description of the set of classified actions 142 into a purpose-specific UI interaction command, such as entering text into a text field. The intent controller 200 can pass parameter 144 directly to script 402, or it can derive a new parameter using additional contextual information (e.g., foreground application identification information) for the value.

[0026] The intent controller 200 executes each script 402 (i.e., a set of low-level actions 142, each reflecting a specific low-level action 206). These are used to represent the user intent to interact with the target application 140. In this example, the selection and execution of script 402 creates an email to James with the body text “Approved.” (Figure 5). The intent controller 200 then interacts with the target application 140 using low-level action 206 (based on the execution of script 402).

[0027] In some embodiments, the intent controller 200 has a level of permission, access, or privilege that exceeds the permission level of the assistant application 110. For example, the intent controller 200 is part of the operating system (OS) running on the user device 102. Due to trust or permission issues, the assistant application 110 may be unable or impossible to perform low-level operations 206 directly with the target application 140 (e.g., by executing script 402). For example, direct interaction with the UI of the user device 102 (e.g., reproducing user input) requires a high level of trust or permission that is not granted to the assistant application 110, and instead requires that a trusted part (i.e., the intent controller 200) perform it. However, allowing the intent controller 200 to perform low-level operations 206 directly on behalf of the assistant application 110 and / or the ASR engine 130 creates a security vulnerability. For example, a malicious application could exploit its functionality to perform actions inconsistent with user intent.

[0028] Instead, the system 100 mitigates any security risks by restricting less reliable applications (e.g., assistant application 110) to use only high-level descriptions or abstractions of user intent that user 10 has already agreed to. Using the illustrated example, user 10 may agree to create an email but not to delete it. Therefore, the intent controller 200 can verify that user 10 has agreed to the action 142 proposed or selected by the assistant application 110. For example, if the assistant application 110 selects an action 142 related to deleting an email, the intent controller 200 will reject such action 142. On the other hand, if the assistant application 110 selects an action 142 related to creating an email, the intent controller 200 will proceed to translate action 142 into an appropriate script 402, with or without additional verification checks.

[0029] Referring here to Figure 2, in some embodiments, the annotation generator 210 of the intent controller 200 obtains, determines, or generates a set of classified actions 142 by generating them using the target application 140 and one or more contextual annotations 220a-n. The annotation generator 210 may convert one or more contextual annotations 220 into their respective classified actions 142. For example, the intent controller 200 generates contextual annotations 220 using captured screenshots of the user interface 146 of the target application 140 via optical character recognition (OCR), image recognition, pattern recognition, etc. Additionally or alternatively, the intent controller 200 generates contextual annotations 220 by parsing the accessibility tree 148 of the target application 140.

[0030] In other words, since 3P applications such as the target application 140 do not typically declare the actions the application can support, the intent controller 200 performs a step to determine the user intent that the target application 140 may support. For this purpose, the intent controller 200 (e.g., with the user's consent) generates contextual annotations 220 by using structural analysis and on-screen understanding of different graphic elements of the target application 140's UI 146 (e.g., via OCR or an application programming interface (API) 149). For example, the intent controller 200 and / or user 10 move the target application 140 to the foreground of a graphical user interface (GUI) 107 running on the user device 102 so that the UI 146 of the target application 140 is visible. The annotation generator 210 translates the annotations 220 into classification actions 142.

[0031] Here, the UI 146 of the target application 140 has three contextual annotations 220, 220a-c. The first contextual annotation 220a represents the "To:" text field, the second contextual annotation 220b represents the "Subject:" text field, and the third contextual annotation 220c represents the "Compose email" text field. Based on these contextual annotations 220a-c, the intent controller 200 decides (e.g., via semantic processing) that the action 142 in this configuration associated with the target application 140 should compose and send an email. Note that in this example, the contextual annotations 220 are mainly English words, but contextual annotations can be in any language and / or graphical representations (e.g., icons or other symbols).

[0032] In some embodiments, the annotation generator 210 determines the required parameters 144 and parameter types for each action 142 determined from the context annotation 220. For example, based on context annotation 220a, the annotation generator 210 determines that action 142 associated with completing the "To:" field of an email requests a string as parameter 144 and makes the appropriate association with action 142. The intent controller 200 may sort or categorize the actions 142 together based on the intent they represent. The intent controller 200 may share the actions 142 and parameters 144 with the assistant application 110 and / or remote system 111.

[0033] Referring here to Figure 3, in some embodiments, the assistant application 110 (or intent controller 200) selects an appropriate set of classification actions 142 based on the user intent determined by the ASR engine 130 from the query 132 (e.g., the user intent expressed in query 132 via utterance 119). In some examples, the target application 140 is in the foreground of the GUI 107 running on the user device 102. In other examples, the target application 140 is in the background or not currently running on the user device 102. In any case, as shown by schematic Figure 300, the assistant application 110 (or intent controller 200) receives target application information 310 that provides assistant application information about the target application 140. For example, the target application information 310 includes identification information for the target application 140 (i.e., identification information for the application currently in the foreground of the GUI 107) or other identification information (e.g., identification information for all email clients installed on the user device 102). Based on query 132 and target application information 310, the assistant application 110 selects the appropriate set action 142 that best reflects the user intent derived from utterance 119.

[0034] The illustrated example shows three sets of categorized actions 142. The first set of actions 142a deals with responding to emails, the second set of actions 142 deals with sending emails, and the third set of actions 142n deals with deleting emails. From the available options, the ASR engine 130 selects a set of actions 142b that represents the user's intention to send an email (based on utterance 119 in this example). Here, the parameters 144 associated with the selected set of actions 142b, provided by query 132, include a first parameter 144a indicating that the recipient of the email is "jjones@gmail.com" and a second parameter 144b indicating that the body of the email contains the text "Approved." Here, the second parameter 144b is extracted directly from query 132. However, the first parameter 144a is derived instead from "James". For example, the assistant application 110 retrieves all contacts from user 10's contacts whose first name is James. The assistant application 110 may derive the value of parameter 144 from any other contextual information available to the assistant application 110 and / or the remote system 111 (e.g., frequency of messages from different contacts, recent messages, geographical location, name of foreground application, etc.).

[0035] The set of actions 142 may be associated with parameters 144 that cannot be extracted or derived from query 132. In these cases, the assistant application 110 may apply default values ​​or no values ​​at all. Here, the selected set of actions 142b (i.e., “Send Email” action 142) and associated parameters 144a,b are sent to the intent controller 200.

[0036] Referring here to Figure 4, in some embodiments, the intent controller 200 includes a script flow controller 410. The script flow controller 410 selects an appropriate script (for example, from script storage 404) based on the selected action 142 and target application information 310. Here, the target application information 310 may include information such as the identification information of the target application 140, the package name associated with the target application 140, and the version of the target application 140. Because the UI elements of an application differ between different applications (e.g., two different email clients) and between different versions of the same application (e.g., buttons and text boxes in different positions), the script flow controller 410 determines a script 402 that is compatible with a particular target application 140, at least in part, based on the target application information 310. As shown in schematic Figure 400, in this example, action 142 revolves around sending an email, but script storage 404 may include many scripts for sending emails based on different email clients and different versions of those email clients.

[0037] Each script 402, when executed, implements one or more low-level interactions with the target application 140. Here, a low-level interaction 206 refers to an interaction or action with minimal abstraction between the description of the interaction and its actual implementation. In some examples, a low-level interaction involves activating user input at a given location on the UI 146 of the target application 140. That is, a script 402 may cause a button to be pressed or toggled, text to be entered, a slider to be moved, etc., on the UI 146 of the target application 140. For example, one script 402 may cause the target application 140 to register a UI selection at a specific location (to simulate user 10 selecting a button on UI 146). If the target application 140 includes API 149 (Figure 2), the intent controller 200 may, additionally or alternatively, interact with the target application 140 via API 149.

[0038] The script flow controller 410 generates an execution flow 412 for the selected script 402 using the selected script 402 and logical operators (such as IF-THEN-ELSE, AND, OR, etc.). In some examples, the script flow controller 410 passes each script 402 one or more parameters 144 associated with the script 402 and / or action 142. Some scripts 402 may include parameters 144, while others may not. Here, the script flow controller 410 has three selected scripts 402. The first script 402a, "Compose Email," is not associated with any parameters passed from or derived from query 132. For example, this action requires the selection of a "Compose Email)" button or other input from user 10 that does not require parameter 144. The script flow controller 410 also selects a second script 402b, "Email Recipients," and a third script 402c, "Email Body." These two scripts are associated with the first parameter 144a ("jjones@gmail.com") and the second parameter 144b ("Approved."), respectively. For example, the second script 402b uses the first parameter 144a to populate the recipient text boxes 510, 510a (Figure 5).

[0039] In this example, the three scripts 402a to c are joined by the AND logical operator. However, it is understood that the script flow controller 410 may join any number of scripts 402 containing any number of operators (OR, IF-THEN-ELSE, etc.) depending on the complexity of the query 132 and the target application 140. The script flow controller 410 may pass the execution flow 412 to the script execution unit 420.

[0040] Referring here to Figure 5, as shown in schematic Figure 500, the script execution unit 420, in some examples, executes each script 402 according to the execution flow 412 to interact with the target application 140 using low-level operations 206 (e.g., via scripts 402 that replicate user input from user devices 102 such as a touchscreen, keyboard, or mouse, or via API 149). In this example, the low-level operation 206 includes typing "jjones@gmail.com" in text box 510a and "Approved." in body text boxes 510, 510c. As shown in the examples in Figures 1-4, the user's utterance (i.e., "Send an email to James saying 'Approved'") did not include the parameter 144 associated with the subject text boxes 510, 510a. In some examples (such as those shown here), the intent controller 200 may rely on default values ​​instead of values ​​provided by user 10. Here, the default value "approved" may depend on some or all of the body text. In another example, the intent controller 200 attempts to prompt user 10 for additional information or derive parameter 144 from another source.

[0041] Figure 6 is a flowchart illustrating an exemplary configuration of the operation of method 600 for encoding and decoding user interface interactions. The computer implementation method 600 causes data processing hardware 103, 112 to perform operations when executed by data processing hardware 510. In operation 602, method 600 includes receiving a query 132 from user 10, captured by an assistant-enabled device 102 (i.e., user device 102) associated with user 10. The query 132 includes a user intent to interact with application 140. In operation 604, method 600 includes obtaining a set of classified actions 142 for application 140. Each classified action 142 in the set of classified actions is associated with one or more parameters 144 and represents a high-level description of user intent of user 10 to interact with application 140. In operation 606, method 600 includes selecting a script 402 associated with each classified action 142 in a set of classified actions 142. Each script 402 performs one or more low-level actions or interactions 206 with application 140. In operation 608, method 600 includes executing a script 402 to represent a user intent to interact with application 140 for each classified action 142 in a set of classified actions 142.

[0042] Figure 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are illustrative only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0043] The computing device 700 includes a processor 710, memory 720, storage device 730, a high-speed interface / controller 740 connected to memory 720 and high-speed expansion port 750, and a low-speed bus 770 and a low-speed interface / controller 760 connected to storage device 730. Each component 710, 720, 730, 740, 750, and 760 is interconnected using various buses and may be mounted on a common motherboard or otherwise present as needed. The processor 710 (i.e., data processing hardware) may include data processing hardware 103 for user device 102 and / or data processing hardware 112 for remote systems. The data processing hardware 710 processes instructions for execution within the computing device 700, including instructions stored in memory 720 or storage device 730, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 780 connected to the high-speed interface 740. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Alternatively, multiple computing devices 700 may be connected (for example, as a server bank, a group of blade servers, or a multiprocessor system), with each device providing the necessary part of the operation.

[0044] Memory (i.e., memory hardware) 720 stores information non-temporarily within the computing device 700. Memory hardware 720 may include memory hardware 105 of the user device 102 and / or memory hardware 114 of the remote system 111. Memory 720 may be computer-readable media, volatile memory units, or non-volatile memory units. Non-temporarily memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disks or tapes.

[0045] The storage device 730 can provide high-capacity storage to the computing device 700. In some embodiments, the storage device 730 is a computer-readable medium. In various different embodiments, the storage device 730 may be an array of devices including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices of a storage area network or other configuration. In additional embodiments, a computer program product is visibly embodied in an information carrier. The computer program product includes instructions that perform one or more of the above-described methods at runtime. The information carrier is a computer-readable or machine-readable medium, such as memory 720, the storage device 730, or memory on the processor 710.

[0046] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages bandwidth-intensive operations. This assignment of roles is merely an example. In some embodiments, the high-speed controller 740 is coupled to memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet) that can connect to one or more input / output devices, such as a keyboard, pointing device, scanner, or network devices such as switches or routers, for example, via a network adapter.

[0047] The computing device 700 may be implemented in several different forms, as shown in the figure. For example, it may be implemented as a standard server 700a, or multiple times within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0048] Various embodiments of the systems and technologies described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be specific or general-purpose and may include implementations in one or more computer programs that are executable and / or interpretable in a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to a storage system.

[0049] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-transient computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signals” means any signals used to provide machine instructions and / or data to a programmable processor.

[0050] The processes and logic flows described herein, also known as data processing hardware, can be executed by one or more programmable processors that perform functions by executing one or more computer programs, operating on input data, and producing outputs. Processors and logic flows can also be executed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, by example, both general-purpose and dedicated microprocessors, as well as one or more processors in either type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be complemented or incorporated by dedicated logic circuits.

[0051] To provide user interaction, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor) or a touchscreen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) on which the user can provide input to the computer. Other types of devices may also be used to provide user interaction, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and the input from the user may be received in any form, including acoustic, spoken language, or haptic input. Furthermore, the computer may interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0052] Several embodiments are described. Needless to say, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer implementation method (600), which is executed by data processing hardware (710), wherein the data processing hardware (710) includes: Receiving a query (132) from the user, captured by an assistant-enabled device (102) associated with the user, wherein the query (132) includes the user's intent to interact with the application (140), The acquisition of a set of classified actions (142) for the application (140), wherein each classified action (142) in the set of classified actions (142) is associated with one or more parameters (144) and represents a high-level description of the user intent of the user interacting with the application (140), For each of the classified actions (142) in the set of classified actions (142), The selection involves selecting a script (402) associated with each of the classified actions (142), wherein each of the scripts (402) performs one or more low-level interactions (206) with the application (140), Executing each of the scripts (402) to represent the user's intent to interact with the application (140), The method for causing an operation including the operation described above.

2. Obtaining the set of classified actions (142) means that for each classified action (142) in the set of classified actions (142), Using the aforementioned application (140), generate one or more context annotations (220), Converting one or more context annotations (220) into the respective classified actions (142) The method according to claim 1 (600), including the method according to claim 1.

3. The method (600) of claim 2, wherein generating the one or more contextual annotations (220) includes performing optical character recognition (OCR) or pattern recognition on image data corresponding to a captured screenshot of the user interface (146) (UI) of the application (140).

4. The method according to claim 2 (600), wherein generating one or more contextual annotations (220) includes analyzing an accessibility tree (148) associated with the application (140).

5. The method (600) according to any one of claims 1 to 4, wherein performing one or more low-level interactions (206) includes activating user input at a predetermined location on the user interface (146) (UI) of the application (140).

6. The method (600) according to any one of claims 1 to 5, wherein the selection of each of the scripts (402) is based on the identification information of the application (140) and the version of the application (140).

7. The method (600) according to any one of claims 1 to 6, wherein selecting each of the scripts (402) includes passing to each of the scripts (402) one or more parameters (144) associated with each of the classified actions (142).

8. The method (600) according to any one of claims 1 to 7, wherein when the application (140) receives the query (132) from the user, the application (140) is in the foreground of a graphical user interface (107) (GUI) executed by the assistant-enabled device (102).

9. The aforementioned assistant-compatible device (102) A digital assistant application (110) configured to provide a digital assistant interface between the user and the assistant-enabled device (102), The aforementioned application (140) is separate from and independent of the aforementioned digital assistant application (110). The method according to any one of claims 1 to 8 (600), which performs the following:

10. The method according to claim 9 (600), wherein the digital assistant application (110) lacks permission to execute each of the scripts (402) for each of the classified actions (142) in the set of classified actions (142).

11. The method according to claim 10 (600), wherein the operating system of the assistant-enabled device (102) executes the respective script (402) for each of the classified actions (142) in the set of classified actions (142).

12. Data processing hardware (710), Memory hardware (720) that communicates with the data processing hardware (710), storing instructions, and when an instruction is executed on the data processing hardware (710), the memory hardware (720) communicates with the data processing hardware (710). Receiving a query (132) from the user, captured by an assistant-enabled device (102) associated with the user, wherein the query (132) includes the user's intent to interact with the application (140), The acquisition of a set of classified actions (142) for the application (140), wherein each classified action (142) in the set of classified actions (142) is associated with one or more parameters (144) and represents a high-level description of the user intent of the user interacting with the application (140), For each of the classified actions (142) in the set of classified actions (142), The selection involves selecting a script (402) associated with each of the classified actions (142), wherein each of the scripts (402) performs one or more low-level interactions (206) with the application (140), The memory hardware (720) is made to perform an operation that includes executing each of the scripts (402) to represent the user intent to interact with the application (140), A system (100) comprising the above.

13. Obtaining the set of classified actions (142) means that for each classified action (142) in the set of classified actions (142), Using the aforementioned application (140), generate one or more context annotations (220), Converting one or more context annotations (220) into the respective classified actions (142) The system (100) according to claim 12, including the system described in claim 12.

14. The system (100) according to claim 13, wherein generating one or more contextual annotations (220) includes performing optical character recognition (OCR) or pattern recognition on image data corresponding to a captured screenshot of the user interface (146) (UI) of the application (140).

15. The system (100) according to claim 13, wherein generating one or more contextual annotations (220) includes analyzing an accessibility tree (148) related to the application.

16. The system (100) according to any one of claims 12 to 15, wherein performing one or more of the low-level interactions (206) includes activating user input at a predetermined location on the user interface (146) (UI) of the application.

17. The system (100) according to any one of claims 12 to 16, wherein the selection of each of the scripts (402) is based on the application identification information and the version of the application.

18. The system (100) according to any one of claims 12 to 17, wherein selecting each of the scripts (402) includes passing to each of the scripts (402) one or more parameters (144) associated with each of the classified actions (142).

19. The system (100) according to any one of claims 12 to 18, wherein when the application receives the query (132) from the user, the application is in the foreground of a graphical user interface (107) (GUI) executed by the assistant-enabled device.

20. The aforementioned assistant-enabled device A digital assistant application (110) configured to provide a digital assistant interface between the user and the assistant-enabled device, The aforementioned application (140) is separate from and independent of the aforementioned digital assistant application (110), A system (100) according to any one of claims 12 to 19, which performs the following:

21. The system (100) according to claim 20, wherein the digital assistant application (110) lacks permission to execute each of the scripts (402) for each of the classified actions (142) in the set of classified actions (142).

22. The system (100) according to claim 21, wherein the operating system of the assistant-enabled device executes the respective script (402) for each of the classified actions (142) in the set of classified actions (142).

Images