Learning computer control using a pointing device and keyboard actions

Abstract

A computer implementation method for controlling a particular computer to perform a task is described. The method includes: receiving control input including visual input, the visual input including one or more screen frames of a computer display representing at least the current state of the particular computer; processing the control input using a neural network to generate one or more control outputs for controlling the particular computer to perform a task, wherein the one or more control outputs include action type outputs specifying at least one of a pointing device action or a keyboard action to be performed to control the particular computer; determining one or more actions from the one or more control outputs; and performing the one or more actions to control the particular computer.

Description

[0001] Cross-references to related applications

[0002] This application is a non-provisional application filed on January 28, 2022, in the United States, namely, Provisional Patent Application No. 63 / 304,549, and claims priority thereto, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification relates to a system for controlling a particular computer to perform tasks using a neural network. Background Technology

[0004] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations, which controls a particular computer to perform a task.

[0006] According to a first aspect, a method for controlling a specific computer to perform a task is provided. The method includes: receiving control input including visual input, the visual input including one or more screen frames of a computer display representing at least the current state of the specific computer; processing the control input using a neural network to generate one or more control outputs for controlling the specific computer to perform a task, wherein the one or more control outputs include action type outputs specifying at least one of a pointing device action or a keyboard action to be performed to control the specific computer; determining one or more actions from the one or more control outputs; and performing the one or more actions to control the specific computer.

[0007] In some implementations, a pointing device action is one of a plurality of actions of a pointing device associated with a particular computer, which includes one or more of the following: input submitted using a computer mouse, input submitted via a touch screen device, or input submitted via a touchpad device.

[0008] In some implementations, a keyboard action is either a key press or text output.

[0009] In some implementations, screen frames in visual input are images representing steps in the process of performing a task on a particular computer.

[0010] In some implementations, the control inputs also include one or more language inputs, one or more prior controls, or both.

[0011] In some implementations, one or more language inputs include at least one of the following: i) task instruction input or ii) task field input including one or more task fields.

[0012] In some implementations, one or more language inputs include Document Object Model (DOM) element inputs, which include one or more DOM elements.

[0013] In one implementation, task instruction input includes text instruction input or voice instruction input.

[0014] In some implementations, one or more control outputs may further include at least one of the following: i) cursor coordinate output, ii) keyboard key index output, or iii) task field index output.

[0015] In some implementations, the cursor coordinate output includes x and y coordinates representing the desired cursor position on the display of a particular computer.

[0016] In some implementations, a task is one of a plurality of tasks performed on a particular computer, including clicking a canvas, dragging items on the screen, filling out forms, editing text, highlighting text, forwarding emails, making appointments, and booking tickets.

[0017] In some implementations, the neural network includes a visual processing subnetwork, one or more language processing subnetworks, a prior control processing subnetwork, a multimodal transformer subnetwork, and an output subnetwork, wherein processing control inputs using the neural network to generate one or more control outputs includes: processing visual inputs using the visual processing subnetwork to generate one or more visual embeddings; processing each of the one or more language inputs using the language processing subnetwork to generate a corresponding language embedding; processing one or more prior controls using the prior control processing subnetwork to generate prior control embeddings; combining one or more visual embeddings and one or more language embeddings using the multimodal transformer subnetwork to generate a transformed embedding; and processing the transformed embedding and the prior control embedding using the output subnetwork to generate one or more control outputs.

[0018] In some implementations, the visual processing subnetwork includes an image encoder subnetwork and a flattening subnetwork, and processing visual input using the visual processing subnetwork to generate one or more visual embeddings includes: processing visual input using the image encoder subnetwork to generate one or more encoded visual feature vectors, and flattening one or more encoded visual feature vectors using the flattening subnetwork to generate one or more visual embeddings.

[0019] In some implementations, the image encoder subnetwork is a ResNet with four ResNet blocks.

[0020] In some implementations, each language processing subnetwork includes a lexicalization layer and a language encoder subnetwork, and processing language inputs from one or more language inputs using the language processing subnetwork to generate corresponding language embeddings includes: splitting the corresponding language inputs into lexicals using the lexicalization layer and generating corresponding lexical embeddings for each lexical in the lexicals, and focusing on the lexical embeddings by using the language encoder subnetwork to generate corresponding language embeddings.

[0021] In some implementations, the language encoder subnetwork is a 1-layer transformer with 4 heads.

[0022] In some implementations, focusing on lexical embeddings to generate corresponding language embeddings includes: processing lexical embeddings to generate key vectors and value vectors, processing additional embeddings to generate query vectors, and using a cross-attention mechanism to process key vectors, value vectors, and query vectors to generate corresponding language embeddings.

[0023] In some implementations, generating transform embeddings using a multimodal transformer subnetwork includes: processing one or more visual embeddings to generate visual query vectors, visual key vectors, and visual value vectors; processing one or more linguistic embeddings to generate linguistic query vectors, linguistic key vectors, and linguistic value vectors; processing one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors; and using a self-attention mechanism to process the visual query vectors, visual key vectors, visual value vectors, linguistic query vectors, linguistic key vectors, linguistic value vectors, additional query vectors, additional key vectors, and additional value vectors to generate transform embeddings.

[0024] In some implementations, the output subnetwork includes one or more Long Short-Term Memory (LSTM) layers.

[0025] In some implementations, the output subnetwork also includes an attention policy subnetwork, which is configured to process the output of one or more LSTM layers to generate task field index output.

[0026] In some implementations, the attention strategy subnetwork is configured to use the corresponding outputs of two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing subnetwork to generate key and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate task field index outputs.

[0027] In some implementations, the method also includes: using i) imitation learning to co-train the neural network to mimic one or more behaviors demonstrated by a human, and ii) reinforcement learning.

[0028] In some implementations, the method also includes using imitation learning to train a neural network to mimic the behavior of a human demonstration as shown in the visual input, and using reinforcement learning to train the neural network.

[0029] In some implementations, during the training of the neural network, visual input demonstrates how a human would perform a task on a specific computer.

[0030] In some implementations, using the output subnetwork to process the transform embedding and the previous control embedding to generate one or more control outputs includes: using the output subnetwork to process the connection between the transform embedding and the previous control embedding to generate one or more control outputs.

[0031] In some implementations, the neural network has been trained using imitation learning and optionally reinforcement learning.

[0032] In some implementations, the action type output specifies an action taken by another operator to take control of a particular computer as part of the process of performing a task.

[0033] In some implementations, a particular computer receives one or more inputs from a human user to control the computer to perform a first part of a task, and upon receiving a command from the user, a neural network controls the computer, thereby generating one or more actions to control the computer to perform a second part of the task following the first part.

[0034] According to a second aspect, a system is provided that includes one or more computers and one or more storage devices storing instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform operations according to the method of the first aspect.

[0035] According to a third aspect, one or more computer storage media encoded with instructions that, when executed by one or more computers, cause one or more computers to perform operations according to the method of the first aspect.

[0036] According to a third aspect, a system is provided, the system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to implement a neural network configured to process control inputs to generate one or more control outputs for controlling a particular computer to perform a task, the control inputs including (i) visual inputs, which include one or more screen frames of a computer display representing at least the current state of the particular computer, (ii) one or more language inputs, and (iii) one or more prior controls, the neural network comprising: a visual processing subnetwork configured to process the visual inputs to generate one or more visual embeddings; one or more language processing subnetworks, each language processing subnetwork configured to process a language input from the one or more language inputs to generate a corresponding language embedding; a prior control processing subnetwork configured to process one or more prior controls to generate a prior control embedding; a multimodal transformer subnetwork configured to combine one or more visual embeddings and one or more language embeddings to generate a transformed embedding; and an output subnetwork configured to process the transformed embedding and the prior control embedding to generate one or more control outputs.

[0037] It should be understood that the aspects can be implemented in any convenient form. For example, the aspects and implementation methods can be implemented by a suitable computer program, which can be carried on a suitable carrier medium, which can be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). The aspects can also be implemented using suitable means that can take the form of a programmable computer that can run computer programs.

[0038] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Unlike existing systems that require access to Document Object Model (DOM) elements and output actions based on DOM elements, the described neural network system typically receives input available to a human user and generates outputs intuitive to humans (e.g., outputs specifying mouse / touchscreen and keyboard actions). The actions output by the described neural network system can be directly observed from logs of human interactions with the computer, as they are keyboard and pointing device actions, thus allowing for the easy collection of large amounts of data. Furthermore, since privileged information is not required in the training data, collecting large training datasets with inputs and outputs in the same space as the training data processed and generated by the described neural network system is greatly simplified. Therefore, the described system can be efficiently trained using imitation learning (and optionally, reinforcement learning) with large amounts of data to achieve state-of-the-art performance. Moreover, the described system can be used to perform tasks requiring pointing device / keyboard actions (e.g., click or drag actions) that cannot be achieved using DOM element-based actions.

[0039] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0040] Figure 1 The illustration shows a neural network system configured to control a specific computer to perform a task.

[0041] Figure 2 It shows that in relation to Figure 1 Example architectures of neural network systems described in the associated description.

[0042] Figure 3 Examples of tasks that require different forms of interaction with a specific computer are shown.

[0043] Figure 4 This is a flowchart of an example process used to control a specific computer to perform a task.

[0044] Figure 5 This is a flowchart illustrating an example process for using a neural network to process control inputs to generate one or more control outputs.

[0045] The same figure terms and terms in each figure represent the same elements. Detailed Implementation

[0046] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations, which controls a particular computer to perform a task.

[0047] A specific computer can be, for example, a desktop computer, a laptop computer, a remote computer residing in a cloud computing system, a mobile device (e.g., a mobile phone, a tablet, etc.), or any type of computing system.

[0048] This task requires a set of computer interactions (i.e., interactions between a specific computer and an operator, which can be a human operator or the neural network system itself). For example, such as... Figure 3As illustrated, the task could be clicking on a canvas (302). In this task, the computer, under the operator's control, needs to create a line that evenly divides the angle into two, and then press the submit button. As another example, the task could be dragging an item on the screen (304). As yet another example, the task could be a multi-step task (306), such as using a text box to enter a term and pressing "search," then finding and clicking the fifth search result. Other examples of tasks include, but are not limited to, filling out forms, editing text, highlighting text, forwarding emails, making appointments by navigating the internet using a web browser or other application, and booking tickets by navigating the internet.

[0049] Figure 1 The illustration depicts a neural network system 100 configured to control a specific computer 102 to perform a task. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, where the systems, components, and techniques described below can be implemented.

[0050] In some implementations, the neural network system 100 is implemented as one or more computer programs on a particular computer 102. In other implementations, the neural network system 100 is implemented as one or more computer programs on a cloud computing system or as a collection of one or more other computers located away from the particular computer 102.

[0051] The neural network system 100 receives a task instruction specifying a task to be performed. The system 100 then controls a particular computer 102 to perform the task by performing the following actions at each of a series of time steps during task execution. First, the neural network system 100 is configured to receive control input 106, including visual input. The visual input includes one or more screen frames from the computer display that at least represent the current state of the particular computer 102. The screen frames in the visual input are images representing steps in the process of performing a task on the particular computer. For example, a screen frame could be an image that the display device of the particular computer 102 would display to a human operator interacting with the particular computer to perform the task at that stage of task execution.

[0052] The control input 106 may also include one or more language inputs, one or more prior controls, or both. The prior control is the control output generated by the neural network system 100 at a previous time step.

[0053] One or more language inputs include at least one of the following: i) task instruction input or ii) task field input including one or more task fields. Task instruction input includes text instruction input (e.g., text entered by a user using a keyboard) or voice instruction input (e.g., sound data captured from a human user's speech using a microphone; optionally, the captured sound data may be processed by a natural language processing system to generate the voice instruction input). Optionally, one or more language inputs may include Document Object Model (DOM) element inputs, which include one or more DOM elements. For example, DOM element inputs may be documents according to the DOM4 standard or WHATWG (Web Hypertext Applications Technology Working Group).

[0054] The neural network system 100 is configured to use a neural network to process control input 106 to generate one or more control outputs 110 for controlling a specific computer 102 to perform a task (e.g., a task 104 of booking a flight). The one or more control outputs 110 are based on actions of a pointing device or keyboard (i.e., actions based on a pointing device / keyboard), rather than actions based on DOM elements.

[0055] The terms "keyboard" and "pointing device" refer to any type of keyboard and pointing device that can be used to operate a computing device. Examples of keyboards include, but are not limited to, laptop keyboards, desktop keyboards, dedicated keyboards that may be used in point-of-sale devices, such as in restaurants or stores; keys / buttons on remote controls; keys / buttons on mobile phones; and / or keys / buttons on game controllers. Examples of pointing devices include, but are not limited to, mice, analog sticks, properly calibrated accelerometers, infrared pointers, or eye-tracking devices.

[0056] One or more control outputs 110 include at least one of action type output, cursor coordinate output, keyboard key index output, or task field index output.

[0057] The action type output specifies at least one of a pointing device action or a keyboard action to be performed to control a particular computer. For example, when the pointing device is a mouse, the pointing device action can include moving the cursor and clicking various buttons on the mouse. As another example, when the pointing device has a touchscreen display, the pointing device action can include any kind of touch input on the display, such as input submitted by a user's finger or stylus.

[0058] The action type output can be selected from a set of possible actions, including no operation (indicating no action), mouse or touchscreen actions, and keyboard actions. Mouse or touchscreen actions can include: move, click, double-click, press, release, scroll up, and scroll down.

[0059] Keyboard actions can be, for example, key presses or text output. Keyboard actions as key presses are used to output one of a set of keyboard keys or macros (such as CTRL+C) (i.e., to generate and output data from one of a specified set of keyboard keys or macros). Keyboard actions as text outputs are used to output the string given by the task field index.

[0060] The task field index output allows the neural network system 110 to select the task field string to issue. The cursor coordinate output includes the x and y coordinates representing the desired cursor position on the monitor of a specific computer.

[0061] The following is for reference. Figure 5 A more detailed description of processing control inputs to generate one or more control outputs.

[0062] After generating one or more control outputs, the neural network system 100 determines one or more actions 112 from the one or more control outputs 110. Each of the one or more control outputs 110 can be modeled by a probability distribution (e.g., a single discrete probability distribution, in addition to the cursor coordinates modeled by two discrete distributions (for height and width coordinates)). As discussed above, the one or more control outputs 110 include at least one of action type output, cursor coordinate output, keyboard key index output, or task field index output. To determine one or more actions 112, the system 100 first selects an action type from a set of possible actions, including no action (indicating no action), mouse or touchscreen actions, and keyboard actions, based on the action type output (which is a probability distribution). One or more actions of the selected action type will be performed as specified in the remainder of the one or more control outputs 110. For example, if the mouse action "move mouse" is selected, the cursor coordinate output included in the one or more control outputs 110 specifies the location where the cursor should be moved. If the keyboard action "send text" is selected, the task field index output included in the one or more control inputs specifies which task field string to send. If the keyboard action "key press" is selected, the keyboard key index output included in the control input specifies which key to press. If control output 110 does not include one of the cursor coordinate output, task field index output, or keyboard key index output, the action type output should specify zero probability for the corresponding action type.

[0063] System 100 performs one or more actions 112 to control a specific computer 102. In particular, system 100 automatically issues commands using a computer control application programming interface (API) or another interface available to system 100 to instruct the specific computer 102 to perform one or more actions 112.

[0064] The neural network system 100 can have any suitable architecture that allows the neural network to map control inputs 106 to one or more control outputs 110 to control a particular computer 102 to perform a task. Figure 2 The diagram illustrates an example of the architecture of a neural network system 100.

[0065] Figure 2 The above reference is shown. Figure 1 An example of the architecture of the described neural network system 100. Figure 2 The neural network system 200 shown is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0066] The neural network system 200 includes a neural network 201 with multiple subnetworks: a visual processing subnetwork (204), a language processing subnetwork (212), a prior control processing subnetwork (232), a multimodal converter subnetwork (230), and an output subnetwork (238).

[0067] In order to control a particular computer to perform a task, at each of a series of time steps during task execution, system 200 receives control input including visual input 202. Visual input 202 includes one or more screen frames from a computer display that at least represents the current state of the particular computer. For example, the screen frames in the visual input are images representing steps in the process of performing a task on the particular computer (e.g., screenshots of the computer display).

[0068] In some implementations, the computer display is a display of a particular computer, and the system 200 performs one or more actions on the particular computer to control the particular computer.

[0069] In some implementations, when the specific computer is a remote computer, system 100 can receive control input from a local computer (e.g., a thin client) and process the control input to generate one or more control outputs to control the remote computer. In these implementations, the computer monitor is the monitor of the local computer, and the actions are performed on the local computer. These actions can enable the local computer to interact with the specific computer (e.g., via a communication network) to control the specific computer.

[0070] The neural network system 200 uses a neural network 201 to process control inputs to generate one or more control outputs 250 for controlling a particular computer to perform a task.

[0071] The visual processing subnetwork 204 includes an image encoder subnetwork 206 and a flattening subnetwork 208. The image encoder neural network can be a convolutional neural network, such as ResNet. For example, the image encoder neural network can be a ResNet with one or more ResNet blocks (e.g., four, five, or seven ResNet blocks). The image encoder neural network 206 is configured to process visual input 202 to generate one or more encoded visual feature vectors. The flattening subnetwork 208 is configured to flatten the one or more encoded visual feature vectors to generate one or more visual embeddings 210.

[0072] In some implementations, the control inputs also include one or more language inputs (e.g., language input 214), one or more prior controls, or both. Prior controls are previously performed actions, i.e., actions sampled from a probability distribution defined by the prior control output generated by the neural network system 200 at a previous time step. The one or more language inputs include task instruction inputs. Alternatively or additionally, the one or more language inputs may include task field inputs, which include one or more task fields. The one or more task fields include information organized to describe different categories of the task to be solved. Examples of task instruction inputs and task fields are given below.

[0073] Task command input

[0074] Book the shortest one-way flight from HVR to Fayetteville, AR on 12 / 27 / 2016.

[0075] Task fields

[0076] Standard: 'shortest'

[0077] Date: '12 / 27 / 2016'

[0078] From: 'HVR'

[0079] To: 'Fayetteville, AR'

[0080] In some implementations, one or more language inputs may optionally include Document Object Model (DOM) element inputs, which include one or more DOM elements.

[0081] Task instruction input is input from a user specifying what the task is from the system. Task instruction input can include text instruction input or voice instruction input. For example, task instruction input could be text instructing the system to organize items by, for example, dragging items. For example, the text might state, "Sort the following numbers in ascending order: 13, -39, 9, 43." As another example, task instruction input includes a transcription of a verbal instruction instructing the system to book a ticket. For example, the verbal instruction might say, "Book the shortest one-way flight from: Santa Rosa, CA to: SAT on 1 / 31 / 2022."

[0082] The language processing subnetwork 212 includes a lexicalization layer 216 and a language encoder subnetwork 220. In some embodiments, the language encoder subnetwork 220 is a 1-layer transformer with 4 heads. The lexicalization layer 216 is configured to split the language input 214 into lexical units and generate a corresponding lexical embedding 218 for each lexical unit. The language encoder subnetwork 220 is configured to focus on the lexical embeddings 218 to generate a corresponding language embedding 222 for the language input 214. Specifically, the language encoder subnetwork 220 processes the lexical embeddings 218 to generate a key vector K and a value vector V, processes additional embeddings to generate a query vector Q, and uses a cross-attention mechanism 126 to process the key vector K, the value vector V, and the query vector Q to generate a corresponding language embedding 229 for the language input 214. The additional embedding can be constant, for example, independent of the control input. It can be learned as part of the training of the neural network system 200 or pre-set (e.g., randomly) before training.

[0083] The previous control processing subnetwork 232 is configured to process one or more previous controls to generate a previous control embedding 234. The previous control is the control output generated by the neural network system 200 at a previous time step.

[0084] The multimodal transformer subnetwork 230 is configured to combine one or more visual embeddings 210 generated by the visual processing subnetwork 204 and one or more language embeddings 222 generated by the language encoder subnetwork 220 to generate a transform embedding 236.

[0085] Specifically, the multimodal transformer subnetwork 230 is configured to process one or more visual embeddings 210 to generate a visual query vector Q1, a visual key vector K1, and a visual value vector V1. The multimodal transformer subnetwork 230 is configured to process one or more linguistic embeddings 222 to generate a linguistic query vector Q2, a linguistic key vector K2, and a linguistic value vector V2. The multimodal transformer subnetwork 230 is configured to process one or more additional embeddings 240 to generate additional query vectors Q3, additional key vectors K3, and additional value vectors V3. The one or more additional embeddings can be constant, for example, independent of the control input. They can be learned as part of the training of the neural network system 200, or pre-set (e.g., randomly) before training. The multimodal transformer subnetwork 230 uses a self-attention mechanism 242 to process the visual query vector Q1, visual key vector K1, visual value vector V1, linguistic query vector Q2, linguistic key vector K2, linguistic value vector V2, additional query vector Q3, additional key vector K3, and additional value vector V3 to generate a transform embedding 236.

[0086] The output subnetwork 238 processes the transformation embedding 236 and the previous control embedding 234 to generate one or more control outputs 250 for controlling a specific computer to perform a task. The one or more control outputs 250 include at least one of action type output, cursor coordinate output, keyboard key index output, or task field index output. The action type output, cursor coordinate output, and keyboard key index output can be generated via linear transformation, while the task field index output can be generated via an attention-based policy.

[0087] Specifically, to generate the task field index output, the output subnetwork 238 includes one or more Long Short-Term Memory (LSTM) layers. The output subnetwork 238 also includes an attention policy subnetwork (not shown), which is configured to process the outputs of the one or more LSTM layers to generate the task field index output. In particular, the attention policy subnetwork is configured to use the corresponding outputs of two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing subnetwork to generate key and value vectors, and use a cross-attention mechanism to process the query vector, key vector, and value vector to generate the task field index output.

[0088] System 200 determines one or more actions from one or more control outputs 250. Each of the one or more control outputs 250 can be modeled by a probability distribution (e.g., a single discrete probability distribution, in addition to cursor coordinates modeled by two discrete distributions (for height and width coordinates)). As discussed above, the one or more control outputs 250 include at least one of action type output, cursor coordinate output, keyboard key index output, or task field index output. To determine one or more actions, system 200 first selects an action type from a set of possible actions, including no action (indicating no action), mouse or touchscreen actions, and keyboard actions, based on the action type output (which is a probability distribution). One or more actions of the selected action type will be performed as specified in the remainder of the one or more control outputs 250. For example, if moving the mouse is selected, the cursor coordinate output specifies the position where the cursor should be moved. If issuing text is selected, the task field index output specifies which task field string to issue. If key press is selected, the keyboard key index output specifies which key to issue.

[0089] System 200 then performs one or more actions to control a specific computer. In particular, the system uses a computer control application programming interface (API) to automatically issue commands to instruct the specific computer to perform one or more actions.

[0090] Neural network 201, or any neural network that generates control outputs from control inputs, can be trained on a training dataset by system 200 or another training system. Learning may be at least partially based on a training dataset of human demonstrations, i.e., real-world observations of human user actions registered by user-operated data input devices (e.g., keyboards, pointing devices, and / or touchscreens). Control inputs for corresponding time steps of these demonstrations may include images displayed to the human user on a display device of a particular computer during the execution of the demonstration, i.e., a display based on the actions taken by the human user. Many techniques for training neural networks to mimic human demonstrations (“imitation learning”) are known.

[0091] For example, imitation learning and optionally reinforcement learning can be used to train neural network 201. In some implementations, system 200 uses both i) imitation learning to mimic one or more behaviors demonstrated by a human and ii) reinforcement learning to co-train neural network 101. “Co-training” refers to a known semi-supervised learning technique that trains two classifiers based on two different types of data: “automatic trajectories” (sequences) of actions automatically selected by the semi-trained neural network system 200 (i.e., without human intervention). The classifiers (e.g., the classifier for human demonstrations) can be used to generate labels for the automatic trajectories to generate rewards for the automatically selected actions, thereby enabling reinforcement learning based on the automatic trajectories.

[0092] A training dataset can be used to train a neural network 201, where two types of data are mixed and used interchangeably. Human demonstrations are input-output sequences generated by human interaction with a specific computer (or another equivalent computer). The fact that the inputs and outputs of the neural network lie in the same space observable from human interaction allows the system to be trained on a large dataset through imitation learning.

[0093] In some other implementations, system 200 first trains neural network 201 using imitation learning to mimic human-demonstrated behavior, and then trains neural network 201 using reinforcement learning. Specifically, system 200 may receive a reward after neural network 201 performs each action and / or after a specific computer controlled by neural network 201 completes a task. System 200 can use the received rewards to train neural network 201 using reinforcement learning.

[0094] In some implementations, system 200 can co-train neural network 201 on multiple tasks. This is for two reasons. First, there is a significant transfer effect, where training per frame for each task is more efficient for co-training. Second, the ultimate goal is a generally useful computer / agent, and therefore it is desirable to have a computer / agent with as many capabilities as possible.

[0095] Specifically, in order to train the neural network 201 jointly on multiple tasks, the system 200 trains the neural network 201 to solve multiple tasks simultaneously by randomly sampling the tasks to solve and training the mixture of tasks. This contrasts with training separate neural networks to solve each task individually.

[0096] Figure 4 This is a flowchart of an example process 400 for controlling a specific computer to perform a task. A task is one of several daily tasks performed on a specific computer (i.e., tasks that statistically represent the types of tasks that most users occasionally perform using the computer). These tasks may include, but are not limited to, clicking on a canvas, dragging items on the screen, filling out forms, converting text, forwarding emails, making appointments, and booking tickets.

[0097] For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification—for example, Figure 1 The neural network system 100 is capable of executing process 400.

[0098] Process 400 is executed at each of a series of time steps during task execution, that is, from the time the system receives the task instruction for the specified task until the task is completed.

[0099] The system receives control input, including visual input (step 402). The visual input includes one or more screen frames from a computer display that at least represents the current state of a particular computer. The screen frames in the visual input are images representing steps in the process of performing a task on the particular computer.

[0100] In some implementations, the control input also includes one or more language inputs, one or more prior controls, or both. In some implementations, the one or more language inputs include at least one of the following: i) a task instruction input or ii) a task field input including one or more task fields. The task instruction input may include text instruction input or voice instruction input. In some implementations, the one or more language inputs include document object model (DOM) element input, which includes one or more DOM elements.

[0101] The system uses a neural network to process control inputs to generate one or more control outputs for controlling a specific computer to perform a task (step 404). The one or more control outputs include action type outputs, which specify at least one of a point-and-click device action or a keyboard action to be performed to control the specific computer. In some implementations, the action type output specifies an action for obtaining control of the specific computer from another operator as part of the process of performing the task. This facilitates situations where a trained neural network system can collaborate with a human user to control a specific computer, where the trained neural network obtains control of the specific computer from a human user in certain circumstances (e.g., to perform operations that would be cumbersome for a human user to control the specific computer). Voice command inputs can specify that the neural network system should obtain control of the specific computer for a part of the process of performing the task.

[0102] A pointing device action is one of several actions performed by a pointing device associated with a specific computer. These actions can include one or more of the following: input submitted using a computer mouse, input submitted via a touchscreen device, or input submitted via a touchpad device. Keyboard actions can be either key presses or text output.

[0103] In some implementations, one or more control outputs include at least one of the following: i) cursor coordinate output, ii) keyboard key index output, or iii) task field index output. Cursor coordinate output includes the x and y coordinates representing the desired cursor position on the monitor of a particular computer.

[0104] In some implementations, the neural network includes a visual processing subnetwork, one or more language processing subnetworks, a prior control processing subnetwork, a multimodal converter subnetwork, and an output subnetwork. References are made below. Figure 5The use of neural networks to process control inputs to generate one or more control outputs is described in more detail in these embodiments.

[0105] The system determines one or more actions from one or more control outputs (step 406). Each of the one or more control outputs can be modeled by a probability distribution (e.g., a single discrete probability distribution, in addition to the cursor coordinates modeled by two discrete distributions (for height and width coordinates)). As discussed above, the one or more control outputs include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine one or more actions, the system first selects an action type from a set of possible actions, including no action (indicating no action), mouse or touchscreen actions, and keyboard actions, based on the action type output (which is a probability distribution). One or more actions of the selected action type will be performed as specified in the remainder of the one or more control outputs. For example, if moving the mouse is selected, the cursor coordinate output specifies the location where the cursor should be moved. If issuing text is selected, the task field index output specifies which task field string to issue. If key press is selected, the keyboard key index output specifies which key to issue.

[0106] The system performs one or more actions to control a specific computer (step 408). Specifically, the system uses a computer control application programming interface (API) to automatically issue commands to instruct the specific computer to perform one or more actions. For example, if one or more actions are mouse actions, such as moving the cursor to an object on the monitor, clicking and holding the left mouse button, and dragging an object to another location, the system uses the computer control API to automatically issue commands to instruct the specific computer to perform the mouse actions to drag the object to the desired location. Possible actions may not include any actions that a human user would not be able to perform using a mouse, touchscreen, or keyboard.

[0107] As described above, method 400 may include taking over control of a specific computer from a user. Therefore, method 400 provides a user interface that authorizes, for example, the automatic execution of a portion of a task according to user commands. In other words, the specific computer can receive one or more inputs from a human user to control the specific computer to perform a first part of a task. Subsequently, the user can issue instructions to the neural network system (e.g., in the form of task instructions, such as by speaking commands recorded by a microphone), specifying that the neural network will perform a second part of the task following the first part. That is, the neural network system will control the specific computer. During this phase, Figure 4The method can be executed at each of one or more consecutive time steps, whereby one or more actions performed by a particular computer (when step 408 is executed in each time step) control the computer to perform the second part of the task. In the first few executions of method 400, the prior control used in method 400 can be based on actions specified by the user when performing the first part of the task.

[0108] Figure 5 This is a flowchart of an example process 500 for processing control inputs using a neural network to generate one or more control outputs. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, such as... Figure 1 The neural network system 100 is capable of executing process 500.

[0109] Process 500 is executed at each of a series of time steps during task execution, that is, from the time the system receives the task instruction for the specified task until the task is completed.

[0110] The system uses a visual processing subnetwork to process the visual input to generate one or more visual embeddings (step 502). In some implementations, the visual processing subnetwork includes an image encoder subnetwork and a flattening subnetwork. The image encoder subnetwork can be a ResNet. The ResNet can have four ResNet blocks.

[0111] The system processes visual input using an image encoder subnetwork to generate one or more encoded visual feature vectors, and then processes visual input using a visual processing subnetwork to generate one or more visual embeddings. It also flattens the one or more encoded visual feature vectors using a flattening subnetwork to generate the same visual embeddings. For each of the one or more language processing subnetworks, the system processes language input from one or more language inputs using the language processing subnetwork to generate the corresponding language embedding (step 504).

[0112] In some implementations, each language processing subnetwork includes a lexicalization layer and a language encoder subnetwork. The language encoder subnetwork can be a language transformer. The language transformer can be a one-layer transformer with four heads.

[0113] In these implementations, the system uses a language processing subnetwork to process language inputs from one or more language inputs to generate corresponding language embeddings by: (i) splitting the corresponding language inputs into lexical units using a lexicalization layer and generating a corresponding lexical embedding for each lexical unit; and (ii) focusing on the lexical embeddings using a language encoder subnetwork to generate the corresponding language embeddings. To focus on the lexical embeddings to generate the corresponding language embeddings, the system processes the lexical embeddings to generate key and value vectors, processes additional embeddings to generate query vectors, and uses a cross-attention mechanism to process the key vectors, value vectors, and query vectors to generate the corresponding language embeddings.

[0114] The system uses the previous control processing subnetwork to process one or more previous controls to generate a previous control embedding (step 506).

[0115] The system uses a multimodal transformer subnetwork to combine one or more visual embeddings and one or more linguistic embeddings to generate a transformed embedding (step 508). Specifically, in some implementations, the system processes one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector. The system processes one or more linguistic embeddings to generate a linguistic query vector, a linguistic key vector, and a linguistic value vector. The system processes one or more additional embeddings to generate an additional query vector, an additional key vector, and an additional value vector. Then, the system uses a self-attention mechanism to process the visual query vector, visual key vector, visual value vector, linguistic query vector, linguistic key vector, linguistic value vector, additional query vector, additional key vector, and additional value vector to generate the transformed embedding.

[0116] The system uses an output subnetwork to process the transform embedding and the previous control embedding to generate one or more control outputs (step 510). The output subnetwork may include one or more Long Short-Term Memory (LSTM) layers. The system can use the output subnetwork to process the connection between the transform embedding and the previous control embedding to generate one or more control outputs.

[0117] The output subnetwork may also include an attention policy subnetwork, which is configured to process the outputs of one or more LSTM layers to generate task field index outputs. The attention policy subnetwork may be configured to use the corresponding outputs of two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing subnetwork to generate key and value vectors, and use a cross-attention mechanism to process the query vector, key vector, and value vector to generate the task field index outputs.

[0118] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof have been installed on the system, which causes the system to perform the operation or action in operation. For one or more computer programs to be configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform an operation or action.

[0119] Embodiments of the subject matter and functional operation described herein can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed herein and their structural equivalents, or in combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals—e.g., machine-generated electrical, optical, or electromagnetic signals—that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0120] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0121] A computer program, also referred to or described as a program, software, software application, application, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored as a portion of a file containing other programs or data, such as one or more scripts stored in a lexical language document, in a single file dedicated to the program in question, or in multiple harmonizing files, such as a file storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on a single computer located at a single site or on multiple computers distributed across multiple sites and interconnected via a data communication network.

[0122] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way or at all, and which can be stored on storage devices in one or more locations. Thus, for example, an indexed database can comprise multiple collections of data, each of which can be organized and accessed differently.

[0123] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines can be installed and run on the same computer or multiple computers.

[0124] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs or ASICs, or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0125] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data—e.g., disks, magneto-optical disks, or optical disks—to receive data from, transfer data to, or both. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0126] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0127] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device—for example, a smartphone running a messaging application—and receiving response messages from the user.

[0128] The data processing apparatus for implementing machine learning models can also include, for example, dedicated hardware accelerator units for handling the common and computationally intensive portions of machine learning training or production—i.e., inference—workloads.

[0129] Machine learning frameworks, such as TensorFlow or Jax, can be used to implement and deploy machine learning models.

[0130] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer having a graphical user interface, a web browser, or an application through which a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication through any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0131] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device as a client and to receive user input from that user. It is possible to receive data generated on the user device, such as the result of user interaction, from the device at the server.

[0132] While this specification contains numerous details of specific implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Some features described in this specification within the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0133] Similarly, although operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0134] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for controlling a specific computer to perform a task, the method comprising: Receive control input including visual input, the visual input including one or more screen frames of a computer display that at least represents the current state of the particular computer; The control input is processed using a neural network to generate one or more control outputs for controlling the particular computer to perform the task, wherein the one or more control outputs include action type outputs specifying at least one of a point device action or a keyboard action to be performed to control the particular computer; Determine one or more actions from the one or more control outputs; and Perform one or more of the aforementioned actions to control the specific computer. The control inputs also include one or more language inputs and one or more previous controls. The neural network includes a visual processing subnetwork, one or more language processing subnetworks, a prior control processing subnetwork, a multimodal converter subnetwork, and an output subnetwork, wherein processing the control input using the neural network to generate the one or more control outputs includes: The visual input is processed using the visual processing subnetwork to generate one or more visual embeddings; The language processing subnetwork is used to process each of the one or more language inputs to generate a corresponding language embedding; The previous control is processed using the previous control processing subnetwork to generate a previous control embedding; The one or more visual embeddings and the one or more language embeddings are combined using a multimodal transformer subnetwork to generate transform embeddings; and The output subnetwork is used to process the transform embedding and the previous control embedding to generate the one or more control outputs.

2. The method according to claim 1, wherein, The visual processing subnetwork includes an image encoder subnetwork and a flattening subnetwork, and The use of the visual processing subnetwork to process the visual input to generate one or more visual embeddings includes: The visual input is processed using the image encoder subnetwork to generate one or more encoded visual feature vectors, and The one or more encoded visual feature vectors are flattened by using the flattening subnetwork to generate the one or more visual embeddings.

3. The method of claim 2, wherein, The image encoder subnetwork is a ResNet with four ResNet blocks.

4. The method of claim 1, wherein, Each language processing subnetwork includes a lexicalization layer and a language encoder subnetwork. The language processing subnetwork is used to process the language inputs in one or more language inputs to generate corresponding language embeddings, including: The lexicalization layer is used to decompose the corresponding language input into lexical units, and a corresponding lexical embedding is generated for each lexical unit. The corresponding language embeddings are generated by focusing on the lexical embeddings using a language encoder subnetwork.

5. The method of claim 4, wherein, The language encoder subnetwork is a 1-layer transformer with 4 heads.

6. The method of claim 4, wherein, Focusing on the lexical embeddings to generate the corresponding language embeddings includes: The lexical embeddings are processed to generate key vectors and value vectors. Process additional embeddings to generate query vectors, and The key vector, the value vector, and the query vector are processed using a cross-attention mechanism to generate the corresponding language embedding.

7. The method of claim 1, wherein, Generating the transform embedding using the multimodal transformer subnetwork includes: Process the one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector. Process the one or more language embeddings to generate language query vectors, language key vectors, and language value vectors. Process one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors. A self-attention mechanism is used to process the visual query vector, the visual key vector, the visual value vector, the language query vector, the language key vector, the language value vector, the additional query vector, the additional key vector, and the additional value vector to generate the transform embedding.

8. The method of claim 1, wherein, The output subnetwork includes one or more Long Short-Term Memory (LSTM) layers.

9. The method of claim 8, wherein, The output subnetwork also includes an attention policy subnetwork, which is configured to process the output of the one or more LSTM layers to generate task field index output.

10. The method of claim 9, wherein, The attention strategy subnetwork is configured as follows: The corresponding outputs of the two LSTM layers are used as query vectors. The task field embeddings generated by the task field language processing sub-network are processed to generate key vectors and value vectors, and A cross-attention mechanism is used to process the query vector, the key vector, and the value vector to generate the task field index output.

11. The method according to any one of claims 1 to 10, further comprising: The neural network is trained using i) imitation learning to mimic one or more behaviors demonstrated by humans and ii) reinforcement learning.

12. The method according to any one of claims 1 to 10, further comprising: The neural network is trained using imitation learning to mimic the behavior demonstrated by a human as shown in the visual input, and The neural network is trained using reinforcement learning.

13. The method according to any one of claims 1 to 10, wherein, During the training of the neural network, the visual input demonstrates how a human would perform the task on the specific computer.

14. The method of any one of claims 1 to 10, wherein, Processing the transform embedding and the previous control embedding using the output sub-network to generate the one or more control outputs includes: processing the connection of the transform embedding and the previous control embedding using the output sub-network to generate the one or more control outputs.

Images