Learned computer control using pointing device and keyboard actions

Abstract

To control a particular computer to execute a task.SOLUTION: A computer-implemented method includes the steps of: receiving a control input including a visual input, the visual input including one or more screen frames of a computer display representing at least a current state of a particular computer; processing the control input using a neural network to generate one or more control outputs used to control the particular computer to execute a task, the one or more control outputs including an action type output specifying at least one of a pointing device action or a keyboard action to be performed to control the particular computer; determining one or more actions from the one or more control outputs; and executing the one or more actions to control the particular computer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a non-provisional application based on and claims priority to U.S. Provisional Patent Application No. 63 / 304,549, filed January 28, 2022, the entire contents of which are hereby incorporated by reference herein.

[0002] This specification relates to a system for controlling a particular computer to perform a task using a neural network. [Background technology]

[0003] A neural network is a machine learning model that utilizes one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input for the next layer of the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of its respective set of parameters. Summary of the Invention [Means for solving the problem]

[0004] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations that controls the particular computers to perform tasks.

[0005] According to a first aspect, there is provided a method for controlling a specific computer to perform a task, the method comprising the steps of receiving a control input including a visual input, the visual input including one or more screen frames of a computer display representing at least a current state of the specific computer, processing the control input using a neural network to generate one or more control outputs used to control the specific computer to perform the task, the one or more control outputs including an action type output specifying at least one of a pointing device action or a keyboard action to be taken to control the specific computer, determining one or more actions from the one or more control outputs, and performing the one or more actions to control the specific computer.

[0006] In some implementations, the pointing device action is one of multiple actions of a pointing device associated with a particular computer, the multiple actions including one or more of input submitted using a computer mouse, input submitted through a touchscreen device, or input submitted through a touchpad device.

[0007] In some implementations, the keyboard action is one of a key press or a text emission.

[0008] In some implementations, the screen frames of the visual input are images that represent steps in a process of performing a task on a particular computer.

[0009] In some implementations, the control input further includes one or more linguistic inputs, one or more previous controls, or both.

[0010] In some implementations, the one or more linguistic inputs include at least one of i) a task instruction input, or ii) a task field input including one or more task fields.

[0011] In some implementations, the one or more language inputs include a Document Object Model (DOM) element input that includes one or more DOM elements.

[0012] In one implementation, the task instruction input includes a text instruction input or a voice instruction input.

[0013] In some implementations, the one or more control outputs further include at least one of i) a cursor coordinate output, ii) a keyboard key index output, or iii) a task field index output.

[0014] In some implementations, the cursor coordinate output includes x and y coordinates that represent the desired cursor position on the display of a particular computer.

[0015] In some implementations, the task is one of multiple tasks performed on a particular computer, including clicking a canvas, dragging an item on the screen, filling in a form, editing text, highlighting text, forwarding an email, making a reservation, and booking a ticket.

[0016] In some implementations, the neural network includes a visual processing sub-network, one or more language processing sub-networks, a previous control processing sub-network, a multimodal transformer sub-network, and an output sub-network, and the step of processing the control input using the neural network to generate one or more control outputs includes: processing the visual input using the visual processing sub-network to generate one or more visual embeddings; processing each linguistic input in the one or more linguistic inputs using the language processing sub-network to generate a respective linguistic embedding; processing one or more previous control embeddings using the previous control processing sub-network to generate a previous control embedding; combining the one or more visual embeddings and the one or more linguistic embeddings using the multimodal transformer sub-network to generate a transformed embedding; and processing the transformed embeddings and the previous control embeddings using the output sub-network to generate one or more control outputs.

[0017] In some implementations, the visual processing sub-network includes an image encoder sub-network and a flattening sub-network, and processing the visual input using the visual processing sub-network to generate one or more visual embeddings includes processing the visual input using the image encoder sub-network to generate one or more encoded visual feature vectors, and flattening the one or more encoded visual feature vectors by using the flattening sub-network to generate the one or more visual embeddings.

[0018] In some implementations, the image encoder sub-network is a ResNet with four ResNet blocks.

[0019] In some implementations, each language processing sub-network includes a tokenization layer and a language encoder sub-network, and processing one or more language inputs using the language processing sub-network to generate a respective language embedding includes splitting each language input into tokens using the tokenization layer and generating a corresponding token embedding for each token in the tokens, and attending across the token embeddings by using the language encoder sub-network to generate the corresponding language embedding.

[0020] In some implementations, the language encoder sub-network is a one-layer transformer with four heads.

[0021] In some implementations, attending across token embeddings to generate corresponding linguistic embeddings includes processing the token embeddings to generate key vectors and value vectors, processing the additional embeddings to generate query vectors, and processing the key vectors, value vectors, and query vectors using a cross-attention mechanism to generate corresponding linguistic embeddings.

[0022] In some implementations, generating a transformed embedding using a multimodal transformer sub-network includes processing one or more visual embeddings to generate visual query vectors, visual key vectors, and visual value vectors; processing one or more linguistic embeddings to generate linguistic query vectors, linguistic key vectors, and linguistic value vectors; processing one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors; and processing the visual query vectors, visual key vectors, visual value vectors, linguistic query vectors, linguistic key vectors, linguistic value vectors, additional query vectors, additional key vectors, and additional value vectors using a self-attention mechanism to generate the transformed embedding.

[0023] In some implementations, the output sub-network includes one or more long short-term memory (LSTM) layers.

[0024] In some implementations, the output sub-network further includes an attention policy sub-network configured to process the output of the one or more LSTM layers to generate a task field index output.

[0025] In some implementations, the attention policy sub-network is configured to use corresponding outputs of the two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing sub-network to generate key vectors and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate a task field index output.

[0026] In some implementations, the method further includes co-training the neural network using i) imitation learning to mimic one or more behaviors of human demonstrations and ii) reinforcement learning.

[0027] In some implementations, the method further includes training the neural network using imitation learning to mimic the behavior of a human performer shown in the visual input, and training the neural network using reinforcement learning.

[0028] In some implementations, during training of the neural network, the visual input demonstrates how a human would perform a task on a particular computer.

[0029] In some implementations, processing the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs includes processing a concatenation of the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs.

[0030] In some implementations, the neural network is trained using imitation learning, and possibly reinforcement learning.

[0031] In some implementations, the action type output specifies an action to obtain control of a particular computer from another operator for part of the process of performing a task.

[0032] In some implementations, the particular computer receives one or more inputs from a human user to control the particular computer to perform a first portion of the task, and upon receiving a command from the user, the neural network takes control of the particular computer, whereby the neural network generates one or more actions to control the particular computer to perform a second portion of the task subsequent to the first portion.

[0033] According to a second aspect, there is provided a system including one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform operations of a method according to the first aspect.

[0034] According to a third aspect, there is provided one or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations of a method according to the first aspect.

[0035] According to a third aspect, there is provided a system including one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to process control inputs to generate one or more control outputs for controlling the particular computer to perform a task, the control inputs including (i) visual inputs including one or more screen frames of a computer display representing at least a current state of the particular computer, (ii) one or more linguistic inputs, and (iii) one or more previous controls, the neural network including: a visual processing sub-network configured to process the visual inputs to generate one or more visual embeddings; one or more language processing sub-networks, each language processing network configured to process a linguistic input in the one or more linguistic inputs to generate a respective linguistic embedding; a previous control processing sub-network that processes the one or more previous controls to generate a previous control embedding; a multimodal transformer sub-network configured to combine the one or more visual embeddings and the one or more linguistic embeddings to generate transformed embeddings; and an output sub-network configured to process the transformed embeddings and the previous control embeddings to generate one or more control outputs.

[0036] It will be appreciated that aspects may be implemented in any convenient form. For example, aspects and implementations may be implemented by a suitable computer program that may be carried on a suitable carrier medium, which may be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). Aspects may also be implemented using a suitable apparatus, which may take the form of a programmable computer that runs a computer program.

[0037] Particular embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: Unlike existing systems that require access to Document Object Model (DOM) elements and output actions via DOM elements, the described neural network systems generally accept inputs available to human users and generate outputs that are intuitive to humans (e.g., outputs specifying mouse / touchscreen and keyboard actions). The actions output by the described neural network systems are keyboard and pointing device actions, and therefore, large amounts of data can be easily collected, making them directly observable from logs of human interactions with computers. Additionally, because confidential information need not be included in the training data, collecting large training data sets with inputs and outputs in the same space as those processed and generated by the described neural network systems is greatly simplified. Thus, the described systems can be effectively trained through simulated learning (and potentially reinforcement learning) using large amounts of data to achieve state-of-the-art performance. Furthermore, the described systems can be used to perform tasks that require pointing device / keyboard actions (e.g., click or drag actions) that cannot be achieved via actions via DOM elements.

[0038] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0039] [Figure 1] FIG. 1 illustrates a neural network system configured to control a particular computer to perform a task. [Figure 2] FIG. 2 illustrates an example architecture of the neural network system described in the description associated with FIG. 1. [Figure 3] 1A-1C illustrate examples of tasks that require different forms of interaction with a particular computer. [Figure 4] 1 is a flow diagram of an exemplary process for controlling a particular computer to perform a task. [Figure 5] 1 is a flow diagram of an example process for processing a control input using a neural network to generate one or more control outputs. DETAILED DESCRIPTION OF THE INVENTION

[0040] Like reference numbers and designations in the various drawings indicate like elements.

[0041] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations that controls the particular computers to perform tasks.

[0042] The particular computer may be, for example, a desktop, a laptop, a remote computer in a cloud computing system, a mobile device (e.g., a cell phone, a tablet, etc.), or any type of computing system.

[0043] A task is a task that requires a set of computer interactions (i.e., interactions between a particular computer and an operator, which may be a human operator or the present neural network system). For example, as shown in FIG. 3, a task may be clicking on a canvas (302). In this task, a computer under the control of an operator is required to create a line that evenly bisects an angle and press a submit button. As another example, a task may be dragging an item on the screen (304). As yet another example, a task may be a multi-step task (306), such as entering a term using a text box, pressing "search," finding the fifth search result, and clicking. Other examples of tasks include, but are not limited to, filling out a form, editing text, highlighting text, forwarding an email, making a reservation by navigating the Internet using a web browser or other application, and booking tickets by navigating the Internet.

[0044] 1 illustrates a neural network system 100 configured to control a particular computer 102 to perform a task. Neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0045] In some implementations, the neural network system 100 is implemented as one or more computer programs on a particular computer 102. In some other implementations, the neural network system 100 is implemented as one or more computer programs on a cloud computing system or other set of one or more computers that are remote from the particular computer 102.

[0046] The neural network system 100 receives task instructions that specify a task to be performed. The system 100 then controls a particular computer 102 to perform the task by performing the following actions at each of a series of time steps during execution of the task: Initially, the neural network system 100 is configured to receive control input 106, including visual input. The visual input includes one or more screen frames of a computer display that represent at least the current state of the particular computer 102. A screen frame of the visual input is an image that represents a step in a process of performing the task on the particular computer. For example, a screen frame may be an image that a display device of the particular computer 102 displays to a human operator interacting with the particular computer to perform the task, at that stage in the task.

[0047] The control input 106 may further include one or more linguistic inputs, one or more previous controls, or both. A previous control is a control output generated by the neural network system 100 at a previous time step.

[0048] The one or more linguistic inputs include at least one of i) task instruction inputs or ii) task field inputs including one or more task fields. The task instruction inputs include text instruction inputs (e.g., text entered by a user using a keyboard) or voice instruction inputs (e.g., sound data captured using a microphone from a human user's speech; in some cases, the captured sound data may be processed by a natural language processing system to generate the voice instruction inputs). In some cases, the one or more linguistic inputs may include Document Object Model (DOM) element inputs including one or more DOM elements. For example, the DOM element input may be a document conforming to the DOM4 specification or the Web Hypertext Application Technology Working Group (WHATWG).

[0049] The neural network system 100 is configured to process the control inputs 106 using a neural network to generate one or more control outputs 110 that are used to control a particular computer 102 to perform a task (e.g., task 104 of booking a flight). The one or more control outputs 110 are pointing device or keyboard-based actions (i.e., pointing device / keyboard actions), rather than DOM element actions.

[0050] The terms "keyboard" and "pointing device" refer to any type of keyboard and pointing device that may be used to operate a computing device. Examples of keyboards include, but are not limited to, laptop keyboards, desktop keyboards, special-purpose keyboards that may be used in point-of-sale devices, e.g., in restaurants or stores, keys / buttons on a remote control, keys / buttons on a mobile phone, and / or keys / buttons on a game controller. Examples of pointing devices include, but are not limited to, a mouse, an analog stick, a properly calibrated accelerometer, an infrared pointer, or an eye-tracking device.

[0051] The one or more control outputs 110 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output.

[0052] The action type output specifies at least one of a pointing device action of a pointing device or a keyboard action taken to control a particular computer. For example, when the pointing device is a mouse, the pointing device action may include moving a cursor and clicking various buttons on the mouse. In another example, when the pointing device has a touchscreen display, the pointing device action may include any type of touch input on the display, for example, transmitted by a user's finger or by a stylus pen.

[0053] The action type may be selected from a set of possible actions including no-op (indicating no action), mouse or touchscreen actions, and keyboard actions based on the action type output. Mouse or touchscreen actions may include move, click, double-click, press, release, wheel up, and wheel down.

[0054] A keyboard action can be, for example, a key press or a text send. A keyboard action that is a key press is used to emit (i.e., generate and output data that specifies) one of a set of keyboard keys or small macros (such as CTRL+C). A keyboard action that is a text send is used to emit a string given by the task field index output.

[0055] The task field index output allows selection of the task field string to be sent by the neural network system 100. The cursor coordinate output includes x and y coordinates representing the desired cursor position on the display of a particular computer.

[0056] The step of processing the control inputs to generate one or more control outputs is described in more detail below with reference to FIG.

[0057] After generating one or more control outputs 110, the neural network system 100 determines one or more actions 112 from the one or more control outputs 110. Each control output of the one or more control outputs 110 may be modeled by a probability distribution (e.g., a single discrete probability distribution, except for the cursor coordinate, which is modeled by two discrete distributions (for the height and width coordinates)). As described above, the one or more control outputs 110 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine the one or more actions 112, the system 100 first selects an action type from a set of possible actions, including no-op (indicating no action), a mouse or touchscreen action, and a keyboard action, based on the action type output (which is a probability distribution). The one or more actions of the selected action type are to be performed as specified in the remainder of the one or more control outputs 110. For example, if the mouse action "move mouse" is selected, a cursor coordinate output included in the one or more outputs 110 specifies where the cursor should be moved. If the keyboard action "emit text" is selected, the task field index output included in one or more control inputs specifies which task field string to emit. If the keyboard action "key press" is selected, the keyboard key index output included in the control input specifies which key to emit. If the control output 110 does not include one of the cursor coordinate output, task field index output, or keyboard key index output, the action type output specifies a zero probability for the corresponding action type.

[0058] The system 100 performs one or more actions 112 to control a particular computer 102. In particular, the system 100 automatically issues commands using a computer control application programming interface (API) or another interface made available to the system 100 to instruct the particular computer 102 to perform one or more actions 112.

[0059] Neural network system 100 can have any suitable architecture that allows the neural network to map control inputs 106 to one or more control outputs 110 to control a particular computer 102 to perform a task. An example of an architecture for neural network system 100 is shown in FIG. 2.

[0060] Figure 2 illustrates an example architecture for the neural network system 100 described above in connection with Figure 1. The neural network system 200 illustrated in Figure 2 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0061] The neural network system 200 includes a neural network 201 having multiple sub-networks: a visual processing sub-network (204), a language processing sub-network (212), a front control processing sub-network (232), a multimodal transformer sub-network (230), and an output sub-network (238).

[0062] To control a particular computer to perform a task, at each successive time step during the execution of the task, system 200 receives control input, including visual input 202. Visual input 202 includes one or more screen frames of a computer display that represent at least the current state of the particular computer. For example, a screen frame of visual input is an image (e.g., a screenshot of a computer display) that represents a step in a process of performing the task on the particular computer.

[0063] In some implementations, the computer display is the display of a particular computer, and the system 200 performs one or more actions on the particular computer to control the particular computer.

[0064] In some implementations, when the particular computer is a remote computer, system 100 may receive control inputs from a local computer (e.g., a thin client) and process the control inputs to generate one or more control outputs for controlling the remote computer. In these implementations, the computer display is the display of the local computer, and actions are performed on the local computer. These actions may cause the local computer to interact with the particular computer (e.g., over a communications network) to control the particular computer.

[0065] The neural network system 200 processes control inputs using a neural network 201 to generate one or more control outputs 250 that are used to control a particular computer to perform a task.

[0066] The visual processing subnetwork 204 includes an image encoder subnetwork 206 and a flattening subnetwork 208. The image encoder neural network may be a convolutional neural network such as ResNet. For example, the image encoder neural network may be a ResNet having one or more ResNet blocks (e.g., 4, 5, or 7 ResNet blocks). The image encoder neural network 206 is configured to process the visual input 202 to generate one or more encoded visual feature vectors. The flattening subnetwork 208 is configured to flatten the one or more encoded visual feature vectors to generate one or more visual embeddings 210.

[0067] In some implementations, the control input further includes one or more linguistic inputs (e.g., linguistic input 214), one or more previous controls, or both. A previous control is a previously taken action, i.e., an action sampled from a probability distribution defined by a previous control output generated by neural network system 200 at a previous time step. The one or more linguistic inputs include a task instruction input. Alternatively or additionally, the one or more linguistic inputs may include a task field input including one or more task fields. The one or more task fields include information organized into different categories that describe the task to be solved. The following is an example of a task instruction input and a task field: Task command input "Book the shortest one-way flight from: HVR to: Fayetteville, AR on 12 / 27 / 2016." Task Field criterion: 'shortest' date: '12 / 27 / 2016' from: 'HVR' to: 'Fayetteville, AR'

[0068] In some implementations, the one or more language inputs may optionally include a Document Object Model (DOM) element input that includes one or more DOM elements.

[0069] Task instruction input is input from a user of the system that specifies what the task is. Task instruction input may include text instruction input or voice instruction input. For example, task instruction input may be text instructing the system to organize items, such as by dragging items. For example, the text may indicate, "Sort the following numbers in increasing order: 13, -39, 9, 43." As another example, task instruction input includes a transcription of a user's verbal instruction instructing the system to book a ticket. For example, the verbal instruction may say, "Book the shortest one-way flight from: Santa Rosa, CA to: SAT on 1 / 31 / 2022."

[0070] The linguistic processing subnetwork 212 includes a tokenization layer 216 and a linguistic encoder subnetwork 220. In some implementations, the linguistic encoder subnetwork 220 is a one-layer transformer with four heads. The tokenization layer 216 is configured to divide the linguistic input 214 into tokens and, for each token in the tokens, generate a corresponding token embedding 218. The linguistic encoder subnetwork 220 is configured to perform attention across the token embeddings 218 to generate a respective linguistic embedding 222 for the linguistic input 214. In particular, the linguistic encoder subnetwork 220 processes the token embeddings 218 to generate a key vector K and a value vector V, processes the additional embeddings to generate a query vector Q, and processes the key vector K, value vector V, and query vector Q using a cross-attention mechanism 126 to generate a respective linguistic embedding 229 for the linguistic input 214. The additional embeddings may be constant, e.g., independent of a control input. The additional embeddings may be learned as part of training the neural network system 200, or may be preset (e.g., randomly) before training.

[0071] The previous control processing sub-network 232 is configured to process one or more previous controls to generate a previous control embedding 234. A previous control is a control output generated by the neural network system 200 at a previous time step.

[0072] The multimodal transformer subnetwork 230 is configured to combine one or more visual embeddings 210 generated by the visual processing subnetwork 204 and one or more linguistic embeddings 222 generated by the linguistic encoder subnetwork 220 to generate transformed embeddings 236.

[0073] In particular, the multimodal transformer subnetwork 230 is configured to process one or more visual embeddings 210 to generate a visual query vector Q1, a visual key vector K1, and a visual value vector V1. The multimodal transformer subnetwork 230 is configured to process one or more linguistic embeddings 222 to generate a linguistic query vector Q2, a linguistic key vector K2, and a linguistic value vector V2. The multimodal transformer subnetwork 230 is configured to process one or more additional embeddings 240 to generate an additional query vector Q3, an additional key vector K3, and an additional value vector V3. The one or more additional embeddings may be constant, e.g., independent of control inputs. The one or more additional embeddings may be learned as part of training the neural network system 200 or may be preset (e.g., randomly) prior to training. The multimodal transformer sub-network 230 uses a self-attention mechanism 242 to process the visual query vector Q1, the visual key vector K1, the visual value vector V1, the linguistic query vector Q2, the linguistic key vector K2, the linguistic value vector V2, the additional query vector Q3, the additional key vector K3, and the additional value vector V3 to generate a transformed embedding 236.

[0074] An output subnetwork 238 processes the transformed embedding 236 and the previous control embedding 234 to generate one or more control outputs 250 used to control a particular computer to perform a task. The one or more control outputs 250 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. The action type output, cursor coordinate output, and keyboard key index output may be generated by a linear transformation, and the task field index output may be generated by an attention-based policy.

[0075] In particular, the output subnetwork 238 includes one or more long short-term memory (LSTM) layers to generate a task field index output. The output subnetwork 238 further includes an attention policy subnetwork (not shown) configured to process the outputs of the one or more LSTM layers to generate a task field index output. In particular, the attention policy subnetwork is configured to use corresponding outputs of the two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing subnetwork to generate key vectors and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate a task field index output.

[0076] The system 200 determines one or more actions from one or more control outputs 250. Each control output of the one or more control outputs 250 may be modeled by a probability distribution (e.g., a single discrete probability distribution, except for the cursor coordinate, which is modeled by two discrete distributions (for the height and width coordinates)). As described above, the one or more control outputs 250 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine the one or more actions, the system 200 first selects an action type from a set of possible actions, including no-op (indicating no action), a mouse or touchscreen action, and a keyboard action, based on the action type output (which is a probability distribution). The one or more actions of the selected action type are to be performed as specified in the remainder of the one or more control outputs 250. For example, if move mouse is selected, the cursor coordinate output specifies where the cursor should be moved. If emit text is selected, the task field index output specifies which task field string to emit. If key press is selected, the keyboard key index output specifies which key to emit.

[0077] The system 200 then performs one or more actions to control the particular computer. In particular, the system automatically issues commands using a computer control application programming interface (API) to instruct the particular computer to perform one or more actions.

[0078] Neural network 201, or any neural network that generates control outputs from control inputs, may be trained on a training data set by system 200 or another training system. Learning may be based, at least in part, on a training data set of human demonstrations, i.e., real-world observations of human user actions registered by a data input device (e.g., a keyboard, a pointer device, and / or a touchscreen) operated by the user. The control inputs for corresponding time steps of these demonstrations may include images displayed to the human user on a particular computer display device, i.e., a display on which the human user acted while the demonstration was taking place. Many techniques are known for training neural networks to imitate human demonstrations ("imitation learning").

[0079] For example, the neural network 201 may be trained using imitation learning and possibly reinforcement learning. In some implementations, the system 200 co-trains the neural network 101 using both i) imitation learning to mimic one or more behaviors of human demonstrations and ii) reinforcement learning. "Co-training" refers to a known semi-supervised learning technique that trains two classifiers based on two different types of data (e.g., human demonstrations and "auto-trajectories" (sequences) of actions automatically selected (i.e., without human involvement) by the semi-trained neural network system 200). The classifiers (e.g., the classifiers for the human demonstrations) may be used to generate labels for the auto-trajectories, generate rewards for the automatically selected actions, and enable reinforcement learning based on the auto-trajectories.

[0080] The neural network 201 may be trained using a training data set in which the two types of data are mixed and used interchangeably. A human demonstration is an input-output sequence produced as a result of human interaction with a particular computer (or another equivalent computer). The fact that the inputs and outputs of a neural network are in the same space that can be observed from human interaction enables the system to train the neural network with large data sets through imitation learning.

[0081] In some implementations, system 200 first trains neural network 201 using imitation learning to mimic the behavior of a human performer, and then uses reinforcement learning to train neural network 201. In particular, system 200 may receive a reward after neural network 201 performs each action and / or after a particular computer controlled by neural network 201 completes a task. System 200 may use the received reward to train neural network 201 using reinforcement learning.

[0082] In some implementations, the system 200 may co-train the neural network 201 on multiple tasks. This is for two reasons. First, training for each task frame by frame is more efficient in co-training, and there is a significant transfer effect. Second, the ultimate goal is a generally useful computer / agent, and therefore it is desirable to have one computer / agent with as many capabilities as possible.

[0083] In particular, to co-train neural network 201 on multiple tasks, system 200 trains neural network 201 to solve multiple tasks simultaneously by randomly sampling the tasks to be solved and training on this mix of tasks, as opposed to training a separate neural network to solve each task individually.

[0084] 4 is a flow diagram of an exemplary process 400 for controlling a particular computer to perform a task. The task is one of a plurality of daily tasks performed on the particular computer (i.e., a task that is statistically representative of the types of tasks that most users occasionally perform using a computer). The plurality of tasks may include, but is not limited to, clicking a canvas, dragging items on a screen, filling in forms, converting text, forwarding email, making an appointment, and booking tickets.

[0085] For convenience, process 400 is described as being performed by one or more computer systems at one or more locations. For example, a neural network system appropriately programmed in accordance with this specification, such as neural network system 100 of FIG. 1, can perform process 400.

[0086] Process 400 is performed during the execution of a task, ie, after the system receives a task instruction specifying the task, at each of a series of time steps until the task is completed.

[0087] The system receives control input (step 402), including visual input. The visual input includes one or more screen frames of a computer display that represent at least the current state of a particular computer. The screen frames of the visual input are images that represent steps in a process of performing a task on a particular computer.

[0088] In some implementations, the control input further includes one or more linguistic inputs, one or more previous controls, or both. In some implementations, the one or more linguistic inputs include at least one of i) a task instruction input, or ii) a task field input including one or more task fields. The task instruction input may include a text instruction input or a voice instruction input. In some implementations, the one or more linguistic inputs include a Document Object Model (DOM) element input including one or more DOM elements.

[0089] The system processes the control input using the neural network to generate one or more control outputs used to control a specific computer to perform a task (step 404). The one or more control outputs include an action-type output that specifies at least one of a pointing device action or a keyboard action to be taken to control the specific computer. In some implementations, the action-type output specifies an action to obtain control of the specific computer from another operator for a portion of the process of performing the task. This enables situations in which the trained neural network system may collaborate with a human user to control the specific computer, and the trained neural network may take over control of the specific computer from a human user in some circumstances (e.g., to perform an action that would be tedious and cumbersome for a human user to control the specific computer). Voice command input may specify that the neural network system should obtain control of the specific computer for a portion of the process of performing the task.

[0090] A pointing device action is one of a number of actions of a pointing device associated with a particular computer. The number of actions may include one or more of input sent using a computer mouse, input sent through a touchscreen device, or input sent through a touchpad device. A keyboard action may be one of a key press or text submission.

[0091] In some implementations, the one or more control outputs include at least one of: i) a cursor coordinate output, ii) a keyboard key index output, or iii) a task field index output. The cursor coordinate output includes x and y coordinates representing a desired cursor position on a display of a particular computer.

[0092] In some implementations, the neural network includes a visual processing sub-network, one or more language processing sub-networks, a previous control processing sub-network, a multimodal transformer sub-network, and an output sub-network. Processing control inputs using a neural network to generate one or more control outputs in these implementations is described in more detail below with reference to FIG.

[0093] The system determines one or more actions from the one or more control outputs (step 406). Each control output of the one or more control outputs may be modeled by a probability distribution (e.g., a single discrete probability distribution, except for the cursor coordinate, which is modeled by two discrete distributions (for the height and width coordinates)). As described above, the one or more control outputs include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine the one or more actions, the system first selects an action type from a set of possible actions, including no-op (indicating no action), a mouse or touchscreen action, and a keyboard action, based on the action type output (which is a probability distribution). The one or more actions of the selected action type are to be executed as specified in the remainder of the one or more control outputs. For example, if move mouse is selected, the cursor coordinate output specifies where the cursor should be moved. If emit text is selected, the task field index output specifies which task field string to emit. If key press is selected, the keyboard key index output specifies which key to emit.

[0094] The system performs one or more actions to control the particular computer (step 408). Specifically, the system automatically issues commands using a computer-controlled application programming interface (API) to instruct the particular computer to perform one or more actions. For example, if the one or more actions are mouse actions such as moving a cursor to an object on a display, clicking and holding the left mouse button, and dragging the object to another position, the system automatically issues commands using the computer-controlled API to instruct the particular computer to perform the mouse action to drag the object to the desired position. The possible actions should not include any actions that a human user cannot perform using a mouse, touchscreen, or keyboard.

[0095] As described above, method 400 may include a step of taking over control of a particular computer from a user. Thus, method 400 provides a user interface that allows a portion of a task to be performed automatically, for example, at the user's command. In other words, the particular computer may receive one or more inputs from a human user to control the particular computer to perform a first portion of a task. The user may then issue an instruction to the neural network system (e.g., in the form of a task instruction, e.g., by speaking a command that is registered by a microphone) specifying that the neural network perform a second portion of the task following the first portion of the task. That is, the neural network system assumes control of the particular computer. At that stage, the method of FIG. 4 may be performed at each of one or more successive time steps, whereby one or more actions performed by the particular computer (when step 408 is performed at each of the time steps) control the computer to perform the second portion of the task. In the first few implementations of method 400, the previous control used in method 400 may be based on the action specified by the user in performing the first portion of the task.

[0096] 5 is a flow diagram of an exemplary process 500 for processing control inputs using a neural network to generate one or more control outputs. For convenience, process 500 is described as being performed by one or more computer systems at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 of FIG. 1, may perform process 500.

[0097] Process 500 is performed during the execution of a task, ie, after the system receives a task instruction specifying the task, at each of a series of time steps until the task is completed.

[0098] The system processes the visual input using a visual processing sub-network to generate one or more visual embeddings (step 502). In some implementations, the visual processing sub-network includes an image encoder sub-network and a flattening sub-network. The image encoder sub-network may be a ResNet. The ResNet may have four ResNet blocks.

[0099] The system processes the visual input using a visual processing sub-network to generate one or more visual embeddings by processing the visual input using an image encoder sub-network to generate one or more encoded visual feature vectors, and flattening the one or more encoded visual feature vectors by using a flattening sub-network to generate one or more visual embeddings.

[0100] For each of the one or more linguistic processing sub-networks, the system processes a linguistic input in the one or more linguistic inputs using the linguistic processing sub-network to generate a respective linguistic embedding (step 504).

[0101] In some implementations, each language processing sub-network includes a tokenization layer and a language encoder sub-network. The language encoder sub-network may be a language transformer. The language transformer may have one layer of a transformer with four heads.

[0102] In these implementations, the system processes one or more linguistic inputs using a linguistic processing subnetwork to generate each linguistic embedding by (i) splitting each linguistic input into tokens using a tokenization layer and generating a corresponding token embedding for each token in the tokens, and (ii) attending across the token embeddings by using a linguistic encoder subnetwork to generate the corresponding linguistic embedding. To attend across the token embeddings to generate the corresponding linguistic embeddings, the system processes the token embeddings to generate key vectors and value vectors, processes the additional embeddings to generate query vectors, and processes the key vectors, value vectors, and query vectors using a cross-attention mechanism to generate the corresponding linguistic embeddings.

[0103] The system processes one or more previous controls using a previous control processing sub-network to generate a previous control embedding (step 506).

[0104] The system combines one or more visual embeddings and one or more linguistic embeddings using a multimodal transformer subnetwork to generate a transformed embedding (step 508). Specifically, in some implementations, the system processes one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector. The system processes one or more linguistic embeddings to generate a linguistic query vector, a linguistic key vector, and a linguistic value vector. The system processes one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors. The system then processes the visual query vector, visual key vector, visual value vector, linguistic query vector, linguistic key vector, linguistic value vector, additional query vector, additional key vector, and additional value vector using a self-attention mechanism to generate the transformed embedding.

[0105] The system processes the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs (step 510). The output sub-network may include one or more long short-term memory (LSTM) layers. The system may process the concatenation of the transformed embedding and the previous control embedding using the output sub-network to generate one or more control outputs.

[0106] The output subnetwork may further include an attention policy subnetwork configured to process the outputs of the one or more LSTM layers to generate a task field index output. The attention policy subnetwork may be configured to use corresponding outputs of the two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing subnetwork to generate key vectors and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate the task field index output.

[0107] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when run, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0108] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware containing the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus.

[0109] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can sometimes include code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0110] A computer program, sometimes referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored as part of a file with other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0111] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0112] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed on and operate on the same computer.

[0113] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0114] A computer suitable for running a computer program can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from, transfer data to, or both of, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash device, to name just a few.

[0115] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0116] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone running a messaging application, and receiving a response message from the user in return.

[0117] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling common and computationally intensive parts of the machine learning training or production, i.e., inference, workload.

[0118] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0119] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components such as an application server, or includes front-end components, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0120] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., to display the data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of a user interaction, may be received from the device at the server.

[0121] While the specification contains details of many specific implementations, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as functioning in a combination and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and a claimed combination may be directed to a subcombination or variations of a subcombination.

[0122] Similarly, while operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed to achieve desirable results. In some environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may typically be integrated into a single software product or packaged into multiple software products.

[0123] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0124] 100 Neural Network Systems 102 Computer 104 Tasks 106 Control Input 110 Control Output 112 Action 126 Cross-Attention Mechanism 200 Neural Network System 201 Neural Networks 202 Visual Input 204 Visual Processing Subnetwork 206 Image Encoder Sub-Network 208 Flattened Subnetwork 210 Visual Embedding 212 Language Processing Subnetwork 214 Language Input 216 Tokenization Layer 218 Token Embedding 220 Language Encoder Subnetwork 222 Language Embedding 229 Language Embedding 230 Multimodal Transformer Subnetwork 232 Previous Control Processing Subnetwork 234 Previous Control Embed 236 Transformed Embedding 238 output subnetworks 240 Additional Embedding 242 Self-Attention Mechanism 250 control output

Claims

1. 1. A computer-implemented method for controlling a specific computer to perform a task, comprising: receiving a control input comprising a visual input, said visual input comprising one or more screen frames of a computer display representing at least a current state of said particular computer; processing the control inputs using a neural network to generate one or more control outputs used to control the particular computer to perform the task, the one or more control outputs including an action type output specifying at least one of a pointing device action or a keyboard action to be taken to control the particular computer; determining one or more actions from the one or more control outputs; performing the one or more actions to control the particular computer; A method comprising:

2. 2. The method of claim 1, wherein the pointing device action is one of a plurality of pointing device actions associated with the particular computer, the plurality of actions including one or more of input transmitted using a computer mouse, input transmitted through a touchscreen device, or input transmitted through a touchpad device.

3. 3. The method of claim 1, wherein the keyboard action is one of a key press or a text submission.

4. 4. The method of claim 1, wherein the visual input screen frames are images representing steps in a process of performing the task on the particular computer.

5. 5. The method of claim 1, wherein the control input further comprises one or more linguistic inputs, one or more previous controls, or both.

6. The method of claim 5 , wherein the one or more linguistic inputs include at least one of: i) a task instruction input; or ii) a task field input including one or more task fields.

7. The method of claim 5 , wherein the one or more language inputs include a Document Object Model (DOM) element input that includes one or more DOM elements.

8. The method of claim 6 , wherein the task instruction input comprises a text instruction input or a voice instruction input.

9. 9. The method of claim 1, wherein the one or more control outputs further include at least one of: i) a cursor coordinate output; ii) a keyboard key index output; or iii) a task field index output.

10. 10. The method of claim 9, wherein the cursor coordinate output comprises x and y coordinates representing a desired cursor position on the display of the particular computer.

11. 11. The method of claim 1, wherein the task is one of a plurality of tasks performed on the particular computer, the plurality of tasks including clicking a canvas, dragging an item on the screen, filling in a form, editing text, highlighting text, forwarding an email, making a reservation, and booking a ticket.

12. the neural network includes a visual processing sub-network, one or more language processing sub-networks, a prior control processing sub-network, a multimodal transformer sub-network, and an output sub-network, and processing the control input using the neural network to generate the one or more control outputs comprises: processing the visual input using the visual processing sub-network to generate one or more visual embeddings; processing each linguistic input in the one or more linguistic inputs using the linguistic processing sub-network to generate a respective linguistic embedding; processing the one or more previous controls using the previous control processing sub-network to generate a previous control embedding; combining the one or more visual embeddings and the one or more linguistic embeddings using a multimodal transformer sub-network to generate a transformed embedding; processing the transformed embedding and the previous control embedding using the output sub-network to generate the one or more control outputs; 6. The method of claim 5, comprising:

13. the visual processing sub-network includes an image encoder sub-network and a flattening sub-network; processing the visual input using the visual processing sub-network to generate one or more visual embeddings, processing the visual input using the image encoder sub-network to generate one or more encoded visual feature vectors; flattening the one or more encoded visual feature vectors by using the flattening sub-network to generate the one or more visual embeddings; 13. The method of claim 12, comprising:

14. The method of claim 13 , wherein the image encoder sub-network is a ResNet having four ResNet blocks.

15. each linguistic processing sub-network includes a tokenization layer and a linguistic encoder sub-network; processing the linguistic inputs in the one or more linguistic inputs using the linguistic processing sub-network to generate the respective linguistic embeddings; using the tokenization layer to split the respective linguistic input into tokens and generate a corresponding token embedding for each token in the tokens; Attentioning across the token embeddings by using the linguistic encoder sub-network to generate the corresponding linguistic embeddings.

15. The method of any one of claims 12 to 14, comprising:

16. 16. The method of claim 15, wherein the language encoder sub-network is a one-layer transformer with four heads.

17. Attentioning across the token embeddings to generate the corresponding linguistic embeddings; processing the token embeddings to generate key vectors and value vectors; processing the additive embedding to generate a query vector; processing the key vectors, the value vectors, and the query vector using a cross-attention mechanism to generate the corresponding language embeddings; 16. The method of claim 15, comprising:

18. generating the transformed embedding using the multimodal transformer sub-network, processing the one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector; processing the one or more linguistic embeddings to generate a linguistic query vector, a linguistic key vector, and a linguistic value vector; processing one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors; processing the visual query vector, the visual key vector, the visual value vector, the linguistic query vector, the linguistic key vector, the linguistic value vector, the additional query vector, the additional key vector, and the additional value vector using a self-attention mechanism to generate the transformed embedding; 18. The method of any one of claims 12 to 17, comprising:

19. 18. The method of claim 12, wherein the output sub-network comprises one or more long short-term memory (LSTM) layers.

20. 20. The method of claim 19, wherein the output sub-network further comprises an attention policy sub-network configured to process the output of the one or more LSTM layers to generate a task field index output.

21. the attention policy subnetwork: using the corresponding outputs of the two LSTM layers as query vectors; processing the corresponding task field embeddings generated by the task field language processing sub-network to generate key vectors and value vectors; using a cross-attention mechanism to process the query vector, the key vector, and the value vector to generate the task field index output; The method of claim 20 configured to:

22. 22. The method of claim 1, further comprising co-training the neural network using i) imitation learning to mimic one or more behaviors of a human performance, and ii) reinforcement learning.

23. training the neural network using imitation learning to mimic the behavior of a human demonstration shown in the visual input; training the neural network using reinforcement learning; 23. The method of any one of claims 1 to 22, further comprising:

24. 24. The method of claim 1, wherein during training of the neural network, the visual input demonstrates how a human would perform the task on the particular computer.

25. 25. The method of claim 12, wherein processing the transformed embedding and the previous control embedding using the output sub-network to generate the one or more control outputs comprises processing a concatenation of the transformed embedding and the previous control embedding using the output sub-network to generate the one or more control outputs.

26. 26. The method of any one of claims 1 to 25, wherein the neural network is trained using imitation learning and optionally reinforcement learning.

27. 27. The method of claim 1, wherein the action type output specifies an action to obtain control of the particular computer from another operator for part of a process of performing the task.

28. 28. The method of any one of claims 1 to 27, wherein the specific computer is configured to receive one or more inputs from a human user to control the specific computer to perform a first part of a task, and wherein the neural network is configured to take control of the specific computer upon receiving a command from the user, whereby the neural network is configured to generate one or more actions to control the specific computer to perform a second part of the task subsequent to the first part.

29. 1. A system comprising: one or more computers; one or more computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 28; A system comprising:

30. 29. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 28.

31. 1. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to process control inputs to generate one or more control outputs for controlling a particular computer to perform a task, the control inputs including (i) visual inputs including one or more screen frames of a computer display representing at least a current state of the particular computer, (ii) one or more linguistic inputs, and (iii) one or more previous controls, and wherein the neural network: a visual processing sub-network configured to process the visual input to generate one or more visual embeddings; one or more language processing sub-networks, each configured to process a language input in the one or more language inputs to generate a respective language embedding; and a previous control processing sub-network for processing the one or more previous controls to generate a previous control embedding; a multimodal transformer sub-network configured to combine the one or more visual embeddings and the one or more linguistic embeddings to generate a transformed embedding; an output sub-network configured to process the transformed embedding and the previous control embedding to generate the one or more control outputs; A system comprising:

Images