Privacy-preserving automation with local grounding and remote generation

Abstract

Methods, systems, and apparatus for responding to an automation query. A user device receives an automation query. The user device generates a proxy automation query and proxy starting screen by replacing private data included in the automation query and a starting screen with respective proxy data. The user device sends the proxy automation query and proxy screen to a server, where the server processes the received data in a user device emulator and using a generation model to obtain actions that are responsive to the proxy automation query. The user device generates corresponding actions that are responsive to the automation query by replacing proxy data included in the actions that are responsive to the proxy automation query with respective private data. The user device processes the actions that are responsive to the automation query using a local grounding model to obtain an executable action responsive to the automation query.

Description

[0001] Attorney Docket No.: 56113-0778WO1

[0002] PRIVACY-PRESERVING AUTOMATION WITH LOCAL GROUNDING AND

[0003] REMOTE GENERATION

[0004] BACKGROUND

[0005] Automation refers to the process of using artificial intelligence to perform tasks or execute commands on behalf of a user without requiring manual intervention. The tasks can range from straightforward actions such as setting reminders or sending messages, to more complex operations such as navigating software interfaces or extracting information from documents or applications.

[0006] Example automation systems include large language model (LLM)-based assistants. In these examples, a user of a user device provides an input query or command, e.g., in natural language. An LLM interprets the query, generates an appropriate response or action plan, and determines the steps needed to complete the query or command. The assistant then interacts with relevant systems or applications on the user device to execute the steps to complete the query or command. Some LLM-based assistants use two models to complete an automation task - a generation model that generates the appropriate response or action plan and a grounding model that maps the appropriate response or action plan to a current state of the user device.

[0007] SUMMARY

[0008] This specification describes systems and methods for privacy-preserving automation with local grounding and remote generation.

[0009] In general, one innovative aspect of the subject matter described in this specification may be embodied in methods that include the actions of receiving, by a user device, an automation query; determining, by the user device, a starting screen for the automation query; generating, by the user device, a proxy automation query and data representing a proxy screen, comprising replacing private data included in the automation query and data representing the starting screen with respective proxy data; sending, by the user device, the proxy automation query and data representing the proxy screen to a server, wherein the server processes the proxy automation queiy and the data representing the proxy screen in an emulator of the user device and using a generation model to obtain a list of natural language actions that are responsive to the proxy automation queiy; receiving, by the user device and from the server, the list of natural language actions that are responsive to the proxy automation query; generating, by the user device, a corresponding list of natural language Attorney Docket No.: 56113-0778WO1 actions that are responsive to the automation query, comprising replacing proxy data included in the list of natural language actions that are responsive to the proxy automation query with respective private data; processing, by the user device, the list of natural language actions that are responsive to the automation query using a user-side copy of a grounding model to obtain an executable action responsive to the automation query; and executing, by the user device, the executable action to respond to the automation query.

[0010] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination thereof installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus (e.g., one or more computers or computer processors), cause the apparatus to perform the actions.

[0011] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. In some implementations the starting screen is a current screen displayed on the user device at a time the automation query is received.

[0012] In some implementations the method further comprises storing, by the user device, a mapping between the proxy data and the private data included in the automation query and the data representing the starting screen, wherein replacing the proxy data included in the list of natural language actions that are responsive to the proxy automation query with respective data of the private data comprises using the mapping.

[0013] In some implementations the method further comprises sending, by the user device, information identifying the user device type and operating system version to the server, wherein the server uses the information to select the emulator of the user device from a plurality of emulators stored by the server.

[0014] In some implementations the server further uses the information to select a server-side copy of the grounding model from a plurality of grounding models stored by the server.

[0015] In some implementations the server uses a server-side copy of the grounding model to process the proxy automation query and the data representing the proxy screen in the emulator of the user device.

[0016] In some implementations the generation model comprises a multi-modal Large Language Model stored by the server. Attorney Docket No.: 56113-0778WO1

[0017] In some implementations to process the proxy automation query and the data representing the proxy screen in an emulator of the user device and using a generation model to obtain a list of natural language actions that are responsive to the proxy automation query, the server processes the proxy automation query and the data representing the proxy screen using the generation model to obtain a first natural language action that is responsive to the proxy automation query; processes the first natural language action using the grounding model to obtain a corresponding first executable action that is responsive to the proxy automation query; executes the first executable action in the emulator of the user device; determines whether execution of the first executable action in the emulator of the user device updates the screen on the user device or completes the automation queiy; in response to determining that execution of the first executable action in the emulator of the user device completes the automation query, sends the first natural language action to the user device; or in response to determining that execution of the first executable action in the emulator of the user device updates the screen, determining whether the emulator has sufficient information about the updated screen; and in response to determining that the emulator has sufficient information about the updated screen, processes the proxy automation query and the data representing the updated screen to obtain a second natural language action; or in response to determining that the emulator has insufficient information about the updated screen, sends a request to the user device for additional data representing the updated screen.

[0018] In some implementations the method further comprises, in response to receiving, by the user device, the request for additional data representing the updated screen: generating, by the user device, data representing a proxy updated screen, comprising replacing private data included in data representing the updated screen with respective proxy data; and sending, by the user device, the data representing the proxy updated screen to the server.

[0019] In some implementations the method further comprises sending, by the user device, batched data representing multiple proxy screens of the user device, wherein the multiple proxy screens of the user device correspond to predicted follow-up actions required to complete the automation query to the server, wherein the server processes the batched data in the emulator of the user device and using the generation model to obtain the list of natural language actions that are responsive to the proxy automation queiy.

[0020] In some implementations the predicted follow-up actions required to complete the automation query comprise scrolling on the user device.

[0021] In some implementations the method further comprises predicting, by the user device and based on the automation query and data representing the starting screen, actions required Attorney Docket No.: 56113-0778WO1 to complete the automation query; batching, by the user device, data representing screens of the user device that correspond to a subset of the actions, wherein the size of the subset is dependent on a predetermined data transmission threshold; and replacing, by the user device, private data included in the screens of the user device that correspond to a subset of the actions with respective proxy data.

[0022] In some implementations predicting the actions required to complete the automation query comprises: processing, by the user device, the automation query and data representing the starting screen using a predefined rule-based system to determine the actions required to complete the automation query.

[0023] In some implementations predicting the actions required to complete the automation query comprises: processing, by the user device, the automation query and data representing the starting screen using a trained machine learning model to determine the actions required to complete the automation query.

[0024] In some implementations the method further comprises sending, by the user device, batched data representing multiple proxy screens of the user device, wherein the multiple proxy screens of the user device correspond to candidate follow-up actions required to complete the automation query to the server, wherein the server processes the batched data in the emulator of the user device and using the generation model to obtain the list of natural language actions that are responsive to the proxy automation query.

[0025] In some implementations the method further comprises identifying by the user device and based on the automation query and data representing the starting screen, candidate follow-up actions required to complete the automation query; determining whether the number of candidate follow-up actions is less than a predetermined threshold; and in response to determining that the number of candidate follow-up actions is less than the predetermined threshold, generating the batched data representing multiple proxy screens of the user device.

[0026] In some implementations the automation query comprises an instruction to perform a task on the user device without manual intervention.

[0027] Some implementations of the subject matter described herein may realize, in certain instances, one or more of the following advantages.

[0028] Examples of the presently described methods and systems increase automation performance, e.g., accuracy, whilst preserving user privacy and data security. For example, the presently described automation system splits the generation and grounding steps of automation between a server and user device. The server runs a generation model that would be too large for the user to run on-device, since the user device will typically have a Attorney Docket No.: 56113-0778WO1 constrained memory space and / or limited processing power. Actions output by the generation model are grounded using a grounding model local to the user device. This distribution of local and remote compute enables the user device to profit from the large generation model without having to run the generation model itself.

[0029] To preserve privacy and prevent unauthorized access to user private data, only proxy data is streamed between the server and user device. In addition, communication costs are reduced since structured data that provides the content of an application (absent, e.g., formatting-specific information) is streamed from the user device to the server instead of raw screen data that renders the content of the application, e.g., pictures / images or XML / HTML (which would include a significant amount of formatting-specific information). In particular, the structured data that is streamed from the user device to the server is more efficient because it does not include data that would be included in the virtual environment provided by the server, e.g., the structured data could be XML but a more efficient XML representation than a direct XML representation of the screen.

[0030] In addition, in some implementations, streamed data can be batched to reduce the number of round trips of communication between the server and user device. In addition, latency is reduced since the server can interact with the generation model locally within a virtual environment without needing to request additional proxy data, e.g., corresponding to a next screen from the user device.

[0031] To address the problem of automation with large models while preserving privacy and preventing unauthorized access to user private data, the presently described automation system uses a combination of local and remote compute with privacy -preserving streaming of data.

[0032] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0033] DESCRIPTION OF DRAWINGS

[0034] FIG. 1 is a conceptual diagram of automation using privacy-preserving streaming of data, remote generation, and local grounding.

[0035] FIG. 2 is a block diagram of an example automation system. Attorney Docket No.: 56113-0778WO1

[0036] FIG. 3 is a diagram of the example automation system 200 of FIG. 2 during an example process for privacy-preserving automation with local grounding and remote generation.

[0037] FIG. 4 is a flow chart of an example process for responding to an automation query.

[0038] FIG. 5 is a flow chart of an example process for generating natural language actions for responding to an automation query.

[0039] Like reference symbols in the various drawings indicate like elements.

[0040] DETAILED DESCRIPTION

[0041] This specification describes systems and methods for automation using privacypreserving streaming of data, remote generation, and local grounding.

[0042] FIG. 1 is a conceptual diagram 100 of automation using privacy-preserving streaming of data, remote generation, and local grounding. A user 102 of a user device 104 invokes a LLM-based automation system on the user device 104 and poses an automation queiy 106. The automation query 106 requires the automation system to interact with the user device 104 and perform a task on behalf of the user 102 without requiring manual intervention. In the example shown in FIG. 1, the user 102 asks the automation system to disable all notifications for the next hour. To complete this automation query, the automation system must navigate to settings then locate and select a disable notifications option (and in some cases, an hour later, repeat the process to enable notifications again). Another example automation query includes asking the automation system a question such as “what restaurant did Tom mention in his recent WhatsApp message to me?” To complete this automation query, the automation system must navigate to WhatsApp, locate and select the contact “Tom,” scroll to identify the relevant message, and extract the restaurant name from the message.

[0043] Many automation systems complete automation queries using two models - a generation model and a grounding model. The generation model takes as input an automation query and a user device screen. The generation model processes the input and outputs a natural language action. The grounding model then takes the natural language action output by the generation model as input and processes the input to output an executable action that can be performed on the user device screen. For example, continuing the example shown in FIG. 1, the generation model can take as input the automation query “disable all notifications for the next hour” and a user device home screen, then output a natural language action “navigate to settings button.” The grounding model can then take as Attorney Docket No.: 56113-0778WO1 input the natural language action “navigate to settings button” and process the input to output an executable action “CLICK button[settings].”

[0044] Since the role of the generation model is to reason about the user’s request , the generation model is typically a general-purpose model and not specific to a current interface of the user device. For example, the generation model can be a GPT4V multi-modal LLM or multi-modal Gemini model. Conversely, the grounding model is highly specific to the current interface of the user device. For example, the grounding model can be different based on whether the user is on a website or in a particular application. Therefore, the generation model is typically significantly larger than the grounding model, and often too large to ran on-device. However, on-device processing is often preferable since the processing of an automation query often involves private user data. Transmission of such private user data off-device can reduce data security, and allow unauthorized third parties to access the private user data.

[0045] To address the problem of automation with large models while preserving privacy and data security, the presently described automation system uses a combination of local and remote compute with privacy-preserving streaming of data. When the user 102 invokes the automation system on the user device 104 and asks an automation query 106, the automation system generates proxy data for the automation query. The proxy data replaces private user data in the automation query and user device screen. The proxy data for the automation query is provided to a remote server 108. The server 108 runs a virtual environment emulator 1 10 of the user device and performs action generation on the proxy data in the emulator 1 10 using a generation model. The emulator 110 allows the server 108 to dynamically interact with a proxy version of the automation query received at the user device 104, avoiding the need to send the generation model to the user device 104. The server 108 then returns an action description for the proxy data, as output by the generation model, to the user device 104. The user device 104 then performs action grounding on the original private user data using the action description for the proxy data and a local copy of a grounding model. The grounded actions can then be executed on the user device 104 to complete the automation query 106.

[0046] FIG. 2 is a block diagram of an example automation system 200. The example automation system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described herein can be implemented. The example system 200 includes a server Attorney Docket No.: 56113-0778WO1

[0047] 202 and a user device 204. The server 202 and user device 204 can be connected via a network, e.g., a local area network (LAN), wide area network (WLAN), the Internet, or a combination thereof, which can be accessed over a wired and / or a wireless communications link.

[0048] The server 202 includes a generation model 206, multiple grounding models 208, and multiple emulators 210. The generation model 206, multiple grounding models 208, and multiple emulators 210 can, in turn, be connected via a network, e.g., a local area network (LAN), wide area network (WLAN), the Internet, or a combination thereof, which can be accessed over a wired and / or a wireless communications link.

[0049] The generation model 206 is a multi-modal machine learning model, e.g., LLM, that is configured, through training, to interpret an automation query and generate a plan or sequence of actions necessary to complete the automation request. In particular, the generation model 206 can be configured to receive, as input, an automation query and data representing a user device screen. The data can include a visual representation of the user device screen, e.g., an image or screenshot of the user device screen, or an intermediate representation of the user device screen, e.g., HTML data or Android view hierarchy data. The generation model 206 is configured to process the received input and generate a natural language output that describes actions that need to be performed to complete the automation request. The natural language output does not detail specific actions to be taken on the user device, e.g., in terms of specific user interface elements or screen configurations. The generation model 206 is typically larger than the grounding models 208 described below or the grounding model 212 included in the user device 204, i.e., is larger in terms of the number of parameters included in the model and / or the complexity and scope of tasks it is configured to handle. Operations performed by the server 202 using the generation model 206 are described in more detail below with reference to FIGS. 3-5.

[0050] The grounding models 208 are machine learning models that are configured, through training, to map or “ground” natural language outputs output by the generation model 206 to specific actions that are executable on respective user devices and / or user device operating systems. In particular, each grounding model is configured to receive, as input, a natural language output from the generation model 206. The grounding model is configured to process the received input and generate executable actions that need to be performed on a respective user device and / or operating system to complete the automation request. For example, if the generation model 206 generates a natural language output “select the disable Attorney Docket No.: 56113-0778WO1 notifications button to disable notifications,” a grounding model can identify an exact button on a user device screen that corresponds to the natural language output.

[0051] Since the grounding models 208 are responsible for mapping high-level commands generated by the larger generation model 206 to specific, actionable commands on different user devices and operating systems, each grounding model of the multiple grounding models 208 corresponds to a respective user device and / or operating system. For example, different user devices, e.g., smartphones, tablets, desktops, have different screen sizes, resolutions, and user interface layouts. As another example, different operating systems have distinct user interface component styles or interaction patterns. A grounding model for a respective user device and / or operating system therefore needs to be trained or fine-tuned to recognize and interact with these specific features. In some implementations the server 202 can be configured to select an appropriate grounding model from the grounding models 208 when generating a response to an automation query, e.g., using user device information that identifies the device type and / or operating system version. Operations performed by the server 202 using the grounding models 208 are described in more detail below with reference to FIGS. 3-5.

[0052] The emulators 210 are configured to provide virtual environments that mimic the functionality of different user devices and / or operating systems. That is, each emulator of the emulators 210 corresponds to a respective user device and / or operating system. When the server 202 receives an automation query from a user device, the server 202 can select an appropriate emulator from the emulators 210 to run a virtual environment that mimics the user device and / or its operating system version. The server 202 then processes the automation query within the emulator and using the generation model 206 and a corresponding grounding model of the grounding models 208, e.g., testing different executable actions until a list of executable actions that best complete the automation query is identified, without needing physical access to the user device. Operations performed by the server 202 using the emulators 210 are described in more detail below with reference to FIGS. 3-5.

[0053] The user device 204 is a computing system, e.g., a smartphone, tablet, desktop, or an end-user component, e.g., a browser, that a user interacts with. The user device 204 includes a grounding model 206, a proxy map 208, and a batching module 210. The grounding model 206, proxy map 208, and batching module 210 can, in turn, be connected via a network, e.g., a local area network (LAN), wide area network (WLAN), the Internet, or a combination thereof, which can be accessed over a wired and / or a wireless communications link. Attorney Docket No.: 56113-0778WO1

[0054] The grounding model 212 is a grounding model that is specific to the user device 204. That is, the grounding model 212 is a machine learning model that is configured, through training, to ground natural language outputs output by the generation model 206 to specific actions that are executable on the user device 204. The grounding model 212 corresponds to at least one grounding model included in the server-side grounding models 208. For example, in some implementations the grounding model 212 can be a copy of one of the server-side grounding models 208. In other implementations the grounding model 212 can be similar to one of the server-side grounding models 208, e.g., has also been trained to process generation model outputs to generate specific actions that are executable on user devices with a same type and / or operating system as the user device 204.

[0055] The proxy map 214 is configured to store data that maps private user data included in an automation queiy or screens of the user device to proxy data generated by the user device 204, as described in more detail below with reference to FIGS. 3-5. In some implementations, data stored in the proxy map 214 can be deleted after a predetermined amount of time or after an automation query has been completed by the user device 204.

[0056] The batching module 216 is configured to batch data to send to the server 202 when processing an automation query. For example, the batching module 216 can include a rulebased system or machine learning model that has been trained to identify or predict a sequence of screens that might be required by the server 202 when processing an automation query from a particular starting screen. This avoids having to stream new data to the server 202 each time the server 202 is confronted with a new screen when processing an automation query in a respective user device emulator. Operations performed by the batching module 216 are described in more detail below with reference to FIGS. 3-5.

[0057] FIG. 3 is a diagram 300 of the example automation system 200 of FIG. 2 during an example process for privacy-preserving automation with local grounding and remote generation. The block diagram 200 illustrates the example process as including stages (A)- (J). However, in some implementations the process can include fewer or more stages.

[0058] During stage (A) of the example process, a user of the user device 204 invokes an automation query. During stage (B), the user device 204 detects private data included in the automation query and the screen shown by the user device 204 at the time the automation query was received (referred to as a “current screen”). The private data can include, e.g., names, addresses, contact information, account information, etc. The user device 203 generates a proxy Atorney Docket No.: 56113-0778WO1 version of the automation query and a proxy version of the current screen by replacing the detected private data with proxy data. For example, private data can be masked out or replaced with anonymized data. The user device 204 stores a mapping between the private data included in the automation query and the current screen and the proxy data used to replace the private data.

[0059] During stage (C). the user device sends the proxy version of the automation query’ (referred to hereafter as a proxy automation query) and data representing the proxy version of the current screen (referred to hereafter as a proxy screen) to the server 202. The data representing the proxy screen can include structured data that provides the content of the current screen, e.g., messages in a messenger application that are displayed on the current screen. In some implementations the data representing the proxy screen can be an intermediate representation of the screen, e.g., HTML data or Android view hierarchy data. In other implementations the data can represent a screenshot of the current screen.

[0060] During stage (D), the server 202 uses the proxy automation query and the data representing the proxy screen to run an emulator of the user device 204. For example, the server 202 can use the proxy automation query and data representing the proxy screen to identify the user device type and an operating system version used by the user device. Alternatively, this information can be included in the data sent to the server 202 during stage (C). The server then uses this information to select a suitable emulator from multiple emulators stored by the server. The emulate allows the server to dynamically’ interact with the user device within a virtual environment that mimics the user device and process the proxy automation query’ to identify a best response to the proxy automation query’, without having to repeatedly communicate w ith the user device 204 and with higher fidelity' (compared to systems that rely on screenshots). The server also uses this information to select a suitable grounding model from multiple grounding models stored by the server, e.g., a grounding model that has been trained on training data from the user device or similar user devices.

[0061] During stage (E). the server 202 runs the proxy automation query and the data representing the proxy screen against a generation model to obtain a natural language action that is responsive to the proxy automation query’. The server 202 then runs the output of the generation model against the grounding model selected during stage (D) to obtain an executable action that corresponds to the natural language action.

[0062] During stage (F). the server 202 executes the executable action within the emulator and starting from the current screen. The execution of the executable action can change the Atorney Docket No.: 56113-0778WO1 screen from the current screen to a subsequent screen, e.g., a subsequent screen obtained by scrolling up or down from the cunent screen or by opening a new application. The server 202 can therefore repeat stages (E) and (F) using the proxy automation query and each subsequent screen until execution of an executable action completes the proxy automation query . Then, during stage (H), server 202 sends the generation model outputs (the natural language actions) generated during the one or more repetitions of stage (E) and needed to complete the proxy automation query to the user device.

[0063] In some cases, during one or more of the repetitions of stages (E) and (F), execution of the executable action for the repetition may lead to a subsequent screen that the emulator does not have enough information about. For example, the subsequent screen may be incomplete or missing more than a predetermined threshold amount of information. In these cases, during stage (G), the server 202 can request that the user device 204 send the server 202 data representing a proxy version of the subsequent screen. In response, the user device 204 can generate the data representing a proxy version of the subsequent screen, e.g., byreplacing private data included in the subsequent screen with proxy data and storing the replacements made in the mapping, and send the data to the server 202. The server 202 can then continue performing stages (E) and (F), until a sequence of executable actions output from the grounding model that completes the proxy automation query' is obtained.

[0064] During stage (I), the user device 204 receives the generation model outputs from the server 202 and uses the mapping generated during stage (B) to replace the proxy data replacements with corresponding private data. This maps the generation model outputs to natural language actions on real user data. During stage (J), the user device runs the natural language actions on real user data against a local copy of a grounding model to obtain a list of executable actions that, when executed by the user device, complete the original automation query invoked during stage (A). In some implementations the local copy of the grounding model can be the same as the grounding model used by the server during stage (E). In other implementations the local copy of the grounding model can be similar to the grounding model used by the server during stage (E). Two models can be defined as similar if, given the same inputs, the models generate outputs that are semantically similar. For example, two outputs can be classified as semantically similar if a vector-based similarity- metric between the generated outputs satisfy a predetermined threshold, e.g., if the cosine similarity- of their embeddings satisfies a predetermined threshold.

[0065] During stage (J), the user device 204 executes the list of executable actions to complete the automation query-. Atorney Docket No.: 56113-0778WO1

[0066] FIG. 4 is a flow chart of an example process 400 for responding to an automation query. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a user device, e.g., the user device 204 of FIG. 2, appropriately programmed, can perform example process 400.

[0067] The system receives an automation query (step 402). The automation query is an instruction received from a user of the user device system to perform a task on the user device without manual intervention.

[0068] In response to receiving the automation query, the system determines a starting screen for the automation query. In some implementations, the starting screen is a current screen displayed on the user device at the time the automation query is received. In other implementations, the starting screen can be a screen not currently displayed on the user device but a default screen that can be accessed by the user device operating system and used to run the automation query from, e.g., a home screen.

[0069] The system generates a proxy automation query and data representing a proxy screen (step 404). The data that represents the proxy screen is data that enables the starting screen to be recreated in a virtual environment provided by a server-side emulator of the user device, as described in more detail below with reference to FIG. 5. The system detects and replaces private data included in the automation query and data representing the starting screen with respective proxy data. For example, the system can mask out private data and / or replace private data with anonymized data. The system stores a mapping between the proxy data and the private data in a local cache.

[0070] In some implementations the system can generate batched data the represents multiple proxy screens of the user device at step 404. For example, the system can use the automation query and starting screen of the user device to predict one or more follow-up actions required to complete the automation query7and screens that correspond to the one or more follow-up actions, e.g., a sequence of subsequent screens required to complete the automation query. For example, the sequence of subsequent screens can correspond to screens obtained by scrolling through an application or website. The system can then generate a batch of data that represents proxy versions of some or all of the predicted subsequent screens by replacing private data included in the predicted subsequent screens with proxy data, as described above. The size of the batch of data can depend on a predetermined acceptable data transmission threshold. For example, the size of the batch of data can be limited to include data that Atorney Docket No.: 56113-0778WO1 represents a proper subset of the predicted subsequent screens, e.g., the first screen or first few screens in the predicted sequence of screens.

[0071] As another example, the system can use the automation query and starting screen of the user device to identify candidate follow-up actions or paths of follow-up actions that can be implemented to complete the automation query . The system can determine whether the number of candidate follow-up actions is less than a predetermined threshold and, if so, the system can generate batched data that includes data representing screens for each candidate follow-up action or path of follow-up actions by replacing private data included in the screens for the candidate follow-up action or path of follow-up actions with proxy data, as described above. Again, the size of the predetermined threshold can depend on a predetermined acceptable data transmission threshold so that transmission of the batch data to the server is not too costly.

[0072] In either of the examples described above, the system can use a predefined rule-based system to predict a sequence of subsequent screens required to complete the automation query or to identify’ screens for candidate paths of follow-up actions that can be implemented to complete the automation query. Alternatively or in addition, the system can use atained machine learning model to predict the sequence of subsequent screens required to complete the automation query or to identify the screens for candidate paths of follow-up actions that can be implemented to complete the automation query.

[0073] The system sends the proxy automation query and data representing the proxy screen (or batched data representing multiple proxy screens) to a server (step 406). The system can also send information identifying the user device type and operating system version to the serv er. The server processes the proxy automation query and the data representing the proxy screen (or batched data representing multiple proxy screens) in an emulator of the user device and using a local copy of a grounding model and a generation model to obtain a list of natural language actions that are responsive to the proxy automation query. The emulator is selected by the server based on the information identifying the user device type and operating system version. Additional operations performed by the server at step 406 is described in more detail below with reference to FIG. 5.

[0074] In some implementations, the system can receive a request from the server for additional information about a screen of the user device, e.g., a subsequent or updated screen identified by the server whilst generating the list of natural language actions that are responsive to the proxy automation query. In these implementations, the system can generate data representing a proxy version of the requested screen and send the data to the server. Atorney Docket No.: 56113-0778WO1

[0075] The system receives the list of natural language actions that are responsive to the proxy automation query from the server (step 408). The system then generates a corresponding list of natural language actions that are responsive to the original automation query, e.g., using the mapping stored at step 404 to replace proxy data included in the list of natural language actions with respective private data (step 410).

[0076] The system processes the list of natural language actions that are responsive to the automation query using a user-side copy of the grounding model to obtain an executable action responsive to the automation query (step 412). The system then executes the executable action to respond to the automation query (step 414).

[0077] FIG. 5 is a flow chart of an example process 500 for generating natural language actions for responding to an automation query. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a server, e.g., the server 202 of FIG. 2, appropriately programmed, can perform example process 400.

[0078] The system receives an automation query and data representing a starting screen for the automation query from a user device (step 502).

[0079] The system processes the automation query and data representing a starting screen for the automation query to identify information that identifies the user device and the current version of its operating system. The system then uses the information to select an emulator of the user device from multiple emulators stored by the system and to select a local copy of a grounding model that has been trained for the user device or similar uses devices from multiple grounding models stored by the system (step 504).

[0080] The system runs the emulator and processes the automation query and the data representing the starting screen for the automation query in the emulator to generate a list of natural language actions that are responsive to the automation query (step 506). The system provides the automation query and the data representing the starting screen for the automation query as input to a generation model to obtain a first natural language action that is responsive to the automation query (step 508). The generation model is a large general- purpose model, e.g., a multi-modal Large Language Model stored by the system and is larger than the grounding model selected at step 504, e.g., includes a larger number of parameters, has a larger complexity, and / or has a larger task scope.

[0081] The system then processes the first natural language action using the selected grounding model to obtain a corresponding first executable action that is responsive to the Atorney Docket No.: 56113-0778WO1 automation query (step 510). The system executes the first executable action in the emulator of the user device and determines whether the execution completes the automation query or leads to an updated screen for the user device (step 512). In some implementations, an automation query can be considered complete if the associated task has been completed, e.g., a message sent, alarm set, or application opened. In other implementations, an automation query can be considered complete if the associated task has been completed and the user device has returned to a final screen, e.g., the home screen or screen that was shown at the time of the automation query.

[0082] In response to determining that execution of the first executable action completes the automation query, the system sends the first natural language action output from the generation model to the user device (step 514). In response to determining that execution of the first executable action does not complete the automation query and leads to a subsequent screen for the user device, the system repeats steps 508-512 using the automation query and data representing the subsequent screen until the system obtains a list executable actions that, when executed by the user device, completes the automation query. The system then sends a corresponding list of natural language actions (the outputs of the generation model) to the user device.

[0083] In some implementations, the system may repeat steps 508-512 using the automation query and data representing a subsequent screen and determine that the virtual environment provided by the emulator of the user device has insufficient information about the subsequent screen to obtain a complete or correct output from the generation model. In these implementations, the system can send a request to the user device for additional information about the subsequent screen and resume processing of the automation query' when the additional information is received.

[0084] In some implementations, the system can receive a batch of data representing multiple screens of the user device from the user device at step 502. The multiple screens of the user device can be screens that the user device predicted as being required to complete the automation query’, e.g., screens for performing follow-up actions required to complete the automation query. In these implementations, the system can use the batched data when repeating steps 508-512.

[0085] Embodiments and all of the functional operations described in this specification may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural Atorney Docket No.: 56113-0778WO1 equivalents, or in combinations of one or more of them. Embodiments may be implemented as one or more computer program products, i.e.. one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium may be a machine- readable storage device, a machine-readable storage substrate, a memory device, a composition of mater effecting a machine-readable propagated signal, or a combination of one or more of them. The term ‘"data processing apparatus’7encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.

[0086] A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0087] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0088] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors Atorney Docket No.: 56113-0778WO1 of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory’ or both.

[0089] The essential elements of a computer are a processor for performing instructions and one or more memory' devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g, magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer may be embedded in another device, e.g. , a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory’ devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0090] To provide for interaction with a user, embodiments may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user may provide input to the computer. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may' be any form of sensory' feedback, e.g., visual feedback, auditory' feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input.

[0091] Embodiments may be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web brow ser through which a user may interact with an implementation, or any combination of one or more such back end, middleware, or front end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network (“WAN”), e.g. , the Internet.

[0092] The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. Atorney Docket No.: 56113-0778WO1

[0093] The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0094] While this specification contains many specifics, these should not be construed as limitations on the scope of the disclosure or of what may be claimed, but rather as descriptions of features specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0095] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0096] In each instance where an HTML file is mentioned, other file types or formats may be substituted. For instance, an HTML file may be replaced by an XML, JSON, plain text, or other types of files. Moreover, where a table or hash table is mentioned, other data structures (such as spreadsheets, relational databases, or structured files) may be used.

[0097] Thus, particular embodiments have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results.

[0098] What is claimed is:

Claims

1. Atorney Docket No.: 56113-0778WO1CLAIMS1. A computer implemented method comprising: receiving, by a user device, an automation query: determining, by the user device, a starting screen for the automation uery: generating, by the user device, a proxy automation query and data representing a proxy screen, comprising replacing private data included in the automation query and data representing the starting screen with respective proxy data; sending, by the user device, the proxy automation query' and data representing the proxy screen to a server, wherein the server processes the proxy automation query and the data representing the proxy screen in an emulator of the user device and using a generation model to obtain a list of natural language actions that are responsive to the proxy automation query; receiving, by the user device and from the server, the list of natural language actions that are responsive to the proxy automation query; generating, by the user device, a corresponding list of natural language actions that are responsive to the automation query', comprising replacing proxy data included in the list of natural language actions that are responsive to the proxy automation query' with respective private data; processing, by the user device, the list of natural language actions that are responsive to the automation query using a user-side copy of a grounding model to obtain an executable action responsive to the automation query; and executing, by the user device, the executable action to respond to the automation query.

2. The method of claim 1, wherein the starting screen is a current screen displayed on the user device at a time the automation query is received.

3. The method of any of claims 1 or 2, further comprising storing, by the user device, a mapping between the proxy data and the private data included in the automation query and the data representing the starting screen, wherein replacing the proxy data included in the list of natural language actions that are responsive to the proxy automation query with respective data of the private data comprises using the mapping.Atorney Docket No.: 56113-0778WO14. The method of any preceding claim, further comprising sending, by the user device, information identifying the user device type and operating system version to the server, wherein the server uses the information to select the emulator of the user device from a plurality of emulators stored by the server.

5. The method of claim 4, wherein the server further uses the information to select a server-side copy of the grounding model from a plurality of grounding models stored by the server.

6. The method of any preceding claim, wherein the server uses a server-side copy of the grounding model to process the proxy automation query' and the data representing the proxy screen in the emulator of the user device.

7. The method of any preceding claim, wherein the generation model comprises a multimodal Large Language Model stored by' the server.

8. The method of any preceding claim, wherein to process the proxy automation query and the data representing the proxy screen in an emulator of the user device and using a generation model to obtain a list of natural language actions that are responsive to the proxy automation query, the server processes the proxy automation query and the data representing the proxy screen using the generation model to obtain a first natural language action that is responsive to the proxy automation query; processes the first natural language action using the grounding model to obtain a corresponding first executable action that is responsive to the proxy automation query7; executes the first executable action in the emulator of the user device; determines whether execution of the first executable action in the emulator of the user device updates the screen on the user device or completes the automation query; in response to determining that execution of the first executable action in the emulator of the user device completes the automation query, sends the first natural language action to the user device; or in response to determining that execution of the first executable action in the emulator of the user device updates the screen, determining whether the emulator has sufficient information about the updated screen; andAtorney Docket No.: 56113-0778WO1 in response to determining that the emulator has sufficient information about the updated screen, processes the proxy automation query and the data representing the updated screen to obtain a second natural language action; or in response to determining that the emulator has insufficient information about the updated screen, sends a request to the user device for additional data representing the updated screen.

9. The method of claim 8, further comprising, in response to receiving, by the user device, the request for additional data representing the updated screen: generating, by the user device, data representing a proxy updated screen, comprising replacing private data included in data representing the updated screen with respective proxy data; and sending, by the user device, the data representing the proxy updated screen to the server.

10. The method of any preceding claim, further comprising sending, by the user device, batched data representing multiple proxy screens of the user device, wherein the multiple proxy screens of the user device correspond to predicted follow-up actions required to complete the automation query to the server, wherein the server processes the batched data in the emulator of the user device and using the generation model to obtain the list of natural language actions that are responsive to the proxy automation query.

11. The method of claim 10, wherein the predicted follow-up actions required to complete the automation query comprise scrolling on the user device.

12. The method of claim 10 or claim 11, further comprising: predicting, by the user device and based on the automation query7and data representing the starting screen, actions required to complete the automation query; batching, by the user device, data representing screens of the user device that correspond to a subset of the actions, wherein the size of the subset is dependent on a predetermined data transmission threshold; and replacing, by the user device, private data included in the screens of the user device that correspond to a subset of the actions with respective proxy data.Atorney Docket No.: 56113-0778WO113. The method of claim 12, wherein predicting the actions required to complete the automation query comprises: processing, by the user device, the automation query and data representing the starting screen using a predefined rule-based system to determine the actions required to complete the automation query.

14. The method of claim 12 or claim 13. wherein predicting the actions required to complete the automation query comprises: processing, by the user device, the automation query' and data representing the starting screen using a trained machine learning model to determine the actions required to complete the automation query.

15. The method of any preceding claim, further comprising sending, by the user device, batched data representing multiple proxy screens of the user device, wherein the multiple proxy screens of the user device correspond to candidate follow-up actions required to complete the automation query to the server, wherein the server processes the batched data in the emulator of the user device and using the generation model to obtain the list of natural language actions that are responsive to the proxy automation query7.

16. The method of claim 15. further comprising: identifying by the user device and based on the automation query and data representing the starting screen, candidate follow-up actions required to complete the automation query7; determining whether the number of candidate follow-up actions is less than a predetermined threshold; and in response to determining that the number of candidate follow-up actions is less than the predetermined threshold, generating the batched data representing multiple proxy screens of the user device.

17. The method of any preceding claim, wherein the automation query comprises an instruction to perform a task on the user device without manual intervention.Attorney Docket No.: 56113-0778WO118. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising the method of any preceding claim.

19. One or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising the method of any preceding claim.

Images