Learned computer control using pointing devices and keyboard actions

Abstract

A computer-implemented method for controlling a specific computer to perform a task is described, the method including the steps of receiving a control input including a visual input, the visual input including one or more screen frames of a computer display representing at least a current state of the specific computer, processing the control input using a neural network to generate one or more control outputs used to control the specific computer to perform the task, the one or more control outputs including an action type output that specifies at least one of a pointing device action or a keyboard action to be taken to control the specific computer, determining one or more actions from the one or more control outputs, and performing the one or more actions to control the specific computer.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application based on and claims priority to U.S. Provisional Patent Application No. 63 / 304,549, filed January 28, 2022, the entire contents of which are hereby incorporated by reference herein.

[0002] This specification relates to a system for controlling a particular computer to perform tasks using neural networks. [Background technology]

[0003] A neural network is a machine learning model that utilizes one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer of the network, i.e. the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. Summary of the Invention [Means for solving the problem]

[0004] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations that controls the particular computers to perform tasks.

[0005] According to a first aspect, there is provided a method for controlling a specific computer to perform a task, the method comprising the steps of receiving a control input including a visual input, the visual input including one or more screen frames of a computer display representing at least a current state of the specific computer, processing the control input using a neural network to generate one or more control outputs used to control the specific computer to perform the task, the one or more control outputs including an action type output specifying at least one of a pointing device action or a keyboard action to be taken to control the specific computer, determining one or more actions from the one or more control outputs, and performing the one or more actions to control the specific computer.

[0006] In some implementations, the pointing device action is one of multiple actions of a pointing device associated with a particular computer, the multiple actions including one or more of input submitted using a computer mouse, input submitted through a touch screen device, or input submitted through a touch pad device.

[0007] In some implementations, the keyboard action is one of a key press or a text emission.

[0008] In some implementations, the screen frames of the visual input are images that represent steps in a process of performing a task on a particular computer.

[0009] In some implementations, the control input further includes one or more linguistic inputs, one or more previous controls, or both.

[0010] In some implementations, the one or more linguistic inputs include at least one of i) a task instruction input, or ii) a task field input including one or more task fields.

[0011] In some implementations, the one or more language inputs include a Document Object Model (DOM) element input that includes one or more DOM elements.

[0012] In one implementation, the task instruction input includes a text instruction input or a voice instruction input.

[0013] In some implementations, the one or more control outputs further include at least one of i) a cursor coordinate output, ii) a keyboard key index output, or iii) a task field index output.

[0014] In some implementations, the cursor coordinate output includes x and y coordinates that represent the desired cursor position on a particular computer's display.

[0015] In some implementations, a task is one of multiple tasks performed on a particular computer, including clicking a canvas, dragging an item on a screen, filling in a form, editing text, highlighting text, forwarding an email, making a reservation, and booking a ticket.

[0016] In some implementations, the neural network includes a visual processing sub-network, one or more language processing sub-networks, a prior control processing sub-network, a multimodal transformer sub-network, and an output sub-network, and the step of processing the control input using the neural network to generate one or more control outputs includes: processing the visual input using the visual processing sub-network to generate one or more visual embeddings; processing each linguistic input in the one or more linguistic inputs using the language processing sub-network to generate a respective linguistic embedding; processing one or more prior controls using the prior control processing sub-network to generate a prior control embedding; combining the one or more visual embeddings and the one or more linguistic embeddings using the multimodal transformer sub-network to generate a transformed embedding; and processing the transformed embedding and the prior control embedding using the output sub-network to generate one or more control outputs.

[0017] In some implementations, the visual processing subnetwork includes an image encoder subnetwork and a flattening subnetwork, and the step of processing the visual input using the visual processing subnetwork to generate one or more visual embeddings includes the steps of processing the visual input using the image encoder subnetwork to generate one or more encoded visual feature vectors, and flattening the one or more encoded visual feature vectors by using the flattening subnetwork to generate the one or more visual embeddings.

[0018] In some implementations, the image encoder sub-network is a ResNet with four ResNet blocks.

[0019] In some implementations, each linguistic processing sub-network includes a tokenization layer and a linguistic encoder sub-network, and processing one or more linguistic inputs using the linguistic processing sub-network to generate a respective linguistic embedding includes splitting the respective linguistic input into tokens using the tokenization layer and generating a corresponding token embedding for each token in the tokens, and attending across the token embeddings by using the linguistic encoder sub-network to generate the corresponding linguistic embedding.

[0020] In some implementations, the linguistic encoder sub-network is a one-layer transformer with four heads.

[0021] In some implementations, the step of attending across the token embeddings to generate the corresponding linguistic embeddings includes processing the token embeddings to generate key vectors and value vectors, processing the additional embeddings to generate query vectors, and processing the key vectors, value vectors, and query vector using a cross-attention mechanism to generate the corresponding linguistic embeddings.

[0022] In some implementations, generating a transformed embedding using the multimodal transformer sub-network includes processing one or more visual embeddings to generate visual query vectors, visual key vectors, and visual value vectors; processing one or more linguistic embeddings to generate linguistic query vectors, linguistic key vectors, and linguistic value vectors; processing one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors; and processing the visual query vectors, visual key vectors, visual value vectors, linguistic query vectors, linguistic key vectors, linguistic value vectors, additional query vectors, additional key vectors, and additional value vectors using a self-attention mechanism to generate the transformed embedding.

[0023] In some implementations, the output sub-network includes one or more long short-term memory (LSTM) layers.

[0024] In some implementations, the output sub-network further includes an attention policy sub-network configured to process the output of the one or more LSTM layers to generate a task field index output.

[0025] In some implementations, the attention policy sub-network is configured to use corresponding outputs of the two LSTM layers as query vectors, process the corresponding task field embeddings generated by the task field language processing sub-network to generate key vectors and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate a task field index output.

[0026] In some implementations, the method further includes co-training the neural network using i) imitation learning to mimic one or more behaviors of human performance, and ii) reinforcement learning.

[0027] In some implementations, the method further includes training the neural network using imitation learning to mimic human demonstration behavior shown in the visual input, and training the neural network using reinforcement learning.

[0028] In some implementations, during training of a neural network, visual input demonstrates how a human would perform a task on a particular computer.

[0029] In some implementations, processing the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs includes processing a concatenation of the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs.

[0030] In some implementations, the neural network is trained using imitation learning, and possibly reinforcement learning.

[0031] In some implementations, the action type output specifies an action to obtain control of a particular computer from another operator for a portion of a process to perform a task.

[0032] In some implementations, the particular computer receives one or more inputs from a human user to control the particular computer to perform a first portion of the task, and upon receiving a command from the user, the neural network takes control of the particular computer, whereby the neural network generates one or more actions for controlling the particular computer to perform a second portion of the task subsequent to the first portion.

[0033] According to a second aspect, there is provided a system including one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform operations of a method according to the first aspect.

[0034] According to a third aspect, there is provided one or more computer storage media encoded with instructions which, when executed by one or more computers, cause the one or more computers to perform operations of a method according to the first aspect.

[0035] According to a third aspect, there is provided a system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to process control inputs to generate one or more control outputs for controlling a particular computer to perform a task, the control inputs including (i) a visual input including one or more screen frames of a computer display representing at least a current state of the particular computer, (ii) one or more linguistic inputs, and (iii) one or more previous controls, the neural network including: a visual processing sub-network configured to process the visual inputs to generate one or more visual embeddings; one or more language processing sub-networks, each language processing network configured to process a linguistic input in the one or more linguistic inputs to generate a respective linguistic embedding; a previous control processing sub-network that processes the one or more previous controls to generate a previous control embedding; a multimodal transformer sub-network configured to combine the one or more visual embeddings and the one or more linguistic embeddings to generate a transformed embedding; and an output sub-network configured to process the transformed embeddings and the previous control embeddings to generate one or more control outputs.

[0036] It will be appreciated that aspects may be implemented in any convenient form. For example, aspects and implementations may be implemented by a suitable computer program that may be carried on a suitable carrier medium, which may be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). Aspects may also be implemented using a suitable apparatus, which may take the form of a programmable computer that executes a computer program.

[0037] Particular embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: Unlike existing systems that require access to Document Object Model (DOM) elements and output actions via DOM elements, the described neural network systems generally receive inputs available to human users and generate outputs that are intuitive to humans (e.g., outputs specifying mouse / touch screen and keyboard actions). The actions output by the described neural network systems are keyboard and pointing device actions, and thus they are directly observable from logs of human interaction with computers, since a large amount of data can be easily collected. In addition, since no confidential information needs to be included in the training data, the collection of large training data sets with inputs and outputs in the same space as those processed and generated by the described neural network systems is greatly simplified. Thus, the described systems can be effectively trained through simulated learning (and possibly reinforcement learning) using large amounts of data to achieve state-of-the-art performance. Furthermore, the described systems can be used to perform tasks that require pointing device / keyboard actions (e.g., click or drag actions) that cannot be achieved via actions via DOM elements.

[0038] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 illustrates a neural network system configured to control a particular computer to perform a task. [Diagram 2] FIG. 2 illustrates an example architecture of the neural network system described in the description associated with FIG. 1. [Diagram 3] 1A-1C illustrate examples of tasks that require different forms of interaction with a particular computer. [Figure 4] 1 is a flow diagram of an exemplary process for controlling a particular computer to perform a task. [Diagram 5] 1 is a flow diagram of an example process for processing a control input using a neural network to generate one or more control outputs. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] Like reference numbers and designations in the various drawings indicate like elements.

[0041] This specification describes a neural network system implemented as a computer program on one or more computers at one or more locations that controls the particular computers to perform tasks.

[0042] The particular computer may be, for example, a desktop, a laptop, a remote computer in a cloud computing system, a mobile device (e.g., a cell phone, a tablet, etc.), or any type of computing system.

[0043] A task is a task that requires a set of computer interactions (i.e., interactions between a particular computer and an operator, which may be a human operator or the present neural network system). For example, as shown in FIG. 3, a task may be to click on a canvas (302). The task requires a computer under the control of an operator to create a line that evenly bisects an angle and press a submit button. As another example, a task may be to drag an item on the screen (304). As yet another example, a task may be a multi-step task (306), such as entering a term using a text box, pressing "search," finding the fifth search result, and clicking. Other examples of tasks include, but are not limited to, filling out a form, editing text, highlighting text, forwarding an email, making a reservation by navigating the Internet using a web browser or other application, and booking tickets by navigating the Internet.

[0044] 1 illustrates a neural network system 100 configured to control a particular computer 102 to perform tasks. Neural network system 100 is one example of a system, implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0045] In some implementations, the neural network system 100 is implemented as one or more computer programs on a particular computer 102. In some other implementations, the neural network system 100 is implemented as one or more computer programs on a cloud computing system or other set of one or more computers that are remote from the particular computer 102.

[0046] The neural network system 100 receives task instructions that specify a task to be performed. The system 100 then controls the particular computer 102 to perform the task by performing the following actions at each of a series of time steps during execution of the task: Initially, the neural network system 100 is configured to receive a control input 106 that includes a visual input. The visual input includes one or more screen frames of a computer display that represent at least a current state of the particular computer 102. A screen frame of the visual input is an image that represents a step of a process of performing the task on the particular computer. For example, a screen frame may be an image that a display device of the particular computer 102 displays to a human operator interacting with the particular computer to perform the task, at that stage of performing the task.

[0047] The control input 106 may further include one or more linguistic inputs, one or more previous controls, or both. A previous control is a control output generated by the neural network system 100 at a previous time step.

[0048] The one or more linguistic inputs include at least one of i) a task instruction input, or ii) a task field input including one or more task fields. The task instruction input includes a text instruction input (e.g., text entered by a user using a keyboard), or a voice instruction input (e.g., sound data captured using a microphone from a human user's speech, in some cases the captured sound data may be processed by a natural language processing system to generate the voice instruction input). In some cases, the one or more linguistic inputs may include a Document Object Model (DOM) element input including one or more DOM elements. For example, the DOM element input may be a document according to the DOM4 specification, or the Web Hypertext Application Technology Working Group (WHATWG).

[0049] The neural network system 100 is configured to process the control inputs 106 using a neural network to generate one or more control outputs 110 that are used to control a particular computer 102 to perform a task (e.g., task 104 of booking a flight). The one or more control outputs 110 are pointing device or keyboard based actions (i.e., pointing device / keyboard actions) and not DOM element actions.

[0050] The terms "keyboard" and "pointing device" refer to any type of keyboard and pointing device that may be used to operate a computing device. Examples of keyboards include, but are not limited to, laptop keyboards, desktop keyboards, purpose-built keyboards that may be used at a point-of-sale device, e.g., in a restaurant or store, keys / buttons on a remote control, keys / buttons on a cell phone, and / or keys / buttons on a game controller. Examples of pointing devices include, but are not limited to, a mouse, an analog stick, a properly calibrated accelerometer, an infrared pointer, or an eye-tracking device.

[0051] The one or more control outputs 110 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output.

[0052] The action type output specifies at least one of a pointing device action of a pointing device or a keyboard action to be taken to control a particular computer. For example, when the pointing device is a mouse, the pointing device action may include moving a cursor and clicking various buttons on the mouse. In another example, when the pointing device has a touch screen display, the pointing device action may include any type of touch input on the display, for example, transmitted by a user's finger or by a stylus pen.

[0053] The action type may be selected from a set of possible actions including no-op (indicating no action), mouse or touch screen actions, and keyboard actions based on the action type output. Mouse or touch screen actions may include move, click, double click, press, release, wheel up, and wheel down.

[0054] A keyboard action can be, for example, a key press or a text send. A keyboard action that is a key press is used to emit (i.e. generate and output data that specifies) one of a set of keyboard keys or small macros (such as CTRL+C). A keyboard action that is a text send is used to emit a string given by the task field index output.

[0055] The task field index output allows one to select a task field string to be sent out by the neural network system 100. The cursor coordinate output includes x and y coordinates that represent the desired cursor position on the display of a particular computer.

[0056] The step of processing the control inputs to generate one or more control outputs is described in more detail below with reference to FIG.

[0057] After generating the one or more control outputs, the neural network system 100 determines one or more actions 112 from the one or more control outputs 110. Each control output of the one or more control outputs 110 may be modeled by a probability distribution (e.g., a single discrete probability distribution, except for the cursor coordinate, which is modeled by two discrete distributions (for height and width coordinates)). As explained above, the one or more control outputs 110 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine the one or more actions 112, the system 100 first selects an action type from a set of possible actions including no-op (indicating no action), a mouse or touch screen action, and a keyboard action based on the action type output (which is a probability distribution). The one or more actions of the selected action type are to be executed as specified in the remainder of the one or more control outputs 110. For example, if a mouse action "move mouse" is selected, a cursor coordinate output included in the one or more outputs 110 specifies where the cursor should be moved. If the keyboard action "emit text" is selected, the task field index output in one or more control inputs specifies which task field string to emit. If the keyboard action "key press" is selected, the keyboard key index output in the control inputs specifies which key to emit. If the control output 110 does not include one of the cursor coordinate output, task field index output, or keyboard key index output, the action type output specifies a zero probability for the corresponding action type.

[0058] The system 100 performs one or more actions 112 to control a particular computer 102. In particular, the system 100 automatically issues commands using a computer control application programming interface (API) or another interface made available to the system 100 to instruct the particular computer 102 to perform one or more actions 112.

[0059] The neural network system 100 may have any suitable architecture that enables the neural network to map control inputs 106 to one or more control outputs 110 in order to control a particular computer 102 to perform a task. An example of an architecture for a neural network system 100 is shown in FIG.

[0060] Figure 2 illustrates an example of the architecture of the neural network system 100 described above in relation to Figure 1. The neural network system 200 illustrated in Figure 2 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0061] The neural network system 200 includes a neural network 201 having multiple sub-networks: a visual processing sub-network (204), a language processing sub-network (212), a front control processing sub-network (232), a multimodal transformer sub-network (230), and an output sub-network (238).

[0062] To control a particular computer to perform a task, at each of a series of time steps during the execution of the task, the system 200 receives control inputs including visual inputs 202. The visual inputs 202 include one or more screen frames of a computer display that represent at least a current state of the particular computer. For example, a screen frame of the visual input is an image (e.g., a screenshot of a computer display) that represents a step of a process of performing the task on the particular computer.

[0063] In some implementations, the computer display is a display of a particular computer, and the system 200 performs one or more actions on the particular computer to control the particular computer.

[0064] In some implementations, when the particular computer is a remote computer, system 100 may receive control inputs from a local computer (e.g., a thin client) and process the control inputs to generate one or more control outputs for controlling the remote computer. In these implementations, the computer display is the display of the local computer, and actions are performed on the local computer. These actions may cause the local computer to interact with the particular computer (e.g., over a communications network) to control the particular computer.

[0065] The neural network system 200 processes control inputs using a neural network 201 to generate one or more control outputs 250 that are used to control a particular computer to perform a task.

[0066] The visual processing sub-network 204 includes an image encoder sub-network 206 and a flattening sub-network 208. The image encoder neural network may be a convolutional neural network such as ResNet. For example, the image encoder neural network may be a ResNet having one or more ResNet blocks (e.g., 4, 5, or 7 ResNet blocks). The image encoder neural network 206 is configured to process the visual input 202 to generate one or more encoded visual feature vectors. The flattening sub-network 208 is configured to flatten the one or more encoded visual feature vectors to generate one or more visual embeddings 210.

[0067] In some implementations, the control input further includes one or more linguistic inputs (e.g., linguistic input 214), one or more previous controls, or both. The previous controls are previously taken actions, i.e., actions sampled from a probability distribution defined by previous control outputs generated by the neural network system 200 at previous time steps. The one or more linguistic inputs include a task instruction input. Alternatively or additionally, the one or more linguistic inputs may include a task field input including one or more task fields. The one or more task fields include information organized into different categories that describe the task to be solved. The following is an example of a task instruction input and a task field: Task command input "Book the shortest one-way flight from: HVR to: Fayetteville, AR on 12 / 27 / 2016." Task Fields criterion: 'shortest' date: '12 / 27 / 2016' from: 'HVR' to: 'Fayetteville, AR'

[0068] In some implementations, the one or more language inputs may optionally include a Document Object Model (DOM) element input that includes one or more DOM elements.

[0069] A task instruction input is an input from a user of the system that specifies what the task is. A task instruction input may include a text instruction input or a voice instruction input. For example, a task instruction input may be text that instructs the system to organize items, for example, by dragging the items. For example, the text may indicate, "Sort the following numbers in increasing order: 13, -39, 9, 43." As another example, a task instruction input includes a transcription of a user's verbal instruction instructing the system to book a ticket. For example, the verbal instruction may say, "Book the shortest one-way flight from: Santa Rosa, CA to: SAT on 1 / 31 / 2022."

[0070] The linguistic processing sub-network 212 includes a tokenization layer 216 and a linguistic encoder sub-network 220. In some implementations, the linguistic encoder sub-network 220 is a one-layer transformer with four heads. The tokenization layer 216 is configured to split the linguistic input 214 into tokens and generate a corresponding token embedding 218 for each token in the tokens. The linguistic encoder sub-network 220 is configured to attend across the token embeddings 218 to generate a respective linguistic embedding 222 for the linguistic input 214. In particular, the linguistic encoder sub-network 220 processes the token embeddings 218 to generate a key vector K and a value vector V, processes the additional embeddings to generate a query vector Q, and processes the key vector K, value vector V, and query vector Q using the cross-attention mechanism 126 to generate a respective linguistic embedding 229 for the linguistic input 214. The additional embeddings may be constant and independent of the control input, for example. The additional embeddings may be learned as part of the training of the neural network system 200, or may be preset (e.g., randomly) prior to training.

[0071] The previous control processing sub-network 232 is configured to process one or more previous controls to generate a previous control embedding 234. A previous control is a control output generated by the neural network system 200 at a previous time step.

[0072] The multimodal transformer subnetwork 230 is configured to combine one or more visual embeddings 210 generated by the visual processing subnetwork 204 and one or more linguistic embeddings 222 generated by the linguistic encoder subnetwork 220 to generate a transformed embedding 236.

[0073] In particular, the multimodal transformer sub-network 230 is configured to process one or more visual embeddings 210 to generate a visual query vector Q1, a visual key vector K1, and a visual value vector V1. The multimodal transformer sub-network 230 is configured to process one or more linguistic embeddings 222 to generate a linguistic query vector Q2, a linguistic key vector K2, and a linguistic value vector V2. The multimodal transformer sub-network 230 is configured to process one or more additional embeddings 240 to generate an additional query vector Q3, an additional key vector K3, and an additional value vector V3. The one or more additional embeddings may be constant, e.g., independent of control inputs. The one or more additional embeddings may be learned as part of training the neural network system 200, or may be preset (e.g., randomly) prior to training. The multimodal transformer sub-network 230 uses a self-attention mechanism 242 to process the visual query vector Q1, the visual key vector K1, the visual value vector V1, the linguistic query vector Q2, the linguistic key vector K2, the linguistic value vector V2, the additional query vector Q3, the additional key vector K3, and the additional value vector V3 to generate a transformed embedding 236.

[0074] An output sub-network 238 processes the transformed embeddings 236 and the previous control embeddings 234 to generate one or more control outputs 250 that are used to control a particular computer to perform a task. The one or more control outputs 250 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. The action type output, cursor coordinate output, and keyboard key index output may be generated by a linear transformation, and the task field index output may be generated by an attention-based policy.

[0075] In particular, the output sub-network 238 includes one or more long short-term memory (LSTM) layers to generate a task field index output. The output sub-network 238 further includes an attention policy sub-network (not shown) configured to process the output of the one or more LSTM layers to generate a task field index output. In particular, the attention policy sub-network is configured to use corresponding outputs of the two LSTM layers as query vectors, process corresponding task field embeddings generated by the task field language processing sub-network to generate key vectors and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate a task field index output.

[0076] The system 200 determines one or more actions from the one or more control outputs 250. Each control output of the one or more control outputs 250 may be modeled by a probability distribution (e.g., a single discrete probability distribution, except for the cursor coordinate, which is modeled by two discrete distributions (for the height and width coordinates)). As explained above, the one or more control outputs 250 include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine the one or more actions, the system 200 first selects an action type from a set of possible actions including no-op (indicating no action), a mouse or touch screen action, and a keyboard action based on the action type output (which is a probability distribution). The one or more actions of the selected action type are to be performed as specified in the remainder of the one or more control outputs 250. For example, if move mouse is selected, the cursor coordinate output specifies where the cursor should be moved. If emit text is selected, the task field index output specifies which task field string to emit. If key press is selected, the keyboard key index output specifies which key to emit.

[0077] The system 200 then performs one or more actions to control the particular computer. In particular, the system automatically issues commands using a computer control application programming interface (API) to instruct the particular computer to perform one or more actions.

[0078] Neural network 201 or any neural network that generates control outputs from control inputs may be trained on a training data set by system 200 or another training system. Learning may be based at least in part on a training data set of human demonstrations, i.e., real-world observations of human user actions registered by a data input device (e.g., a keyboard, a pointer device, and / or a touch screen) operated by the user. The control inputs for corresponding time steps of these demonstrations may include images displayed to the human user on a particular computer's display device, i.e., a display on which the human user acted while the demonstration was taking place. Many techniques are known for training neural networks to mimic human demonstrations ("imitation learning").

[0079] For example, the neural network 201 may be trained using imitation learning and possibly reinforcement learning. In some implementations, the system 200 co-trains the neural network 101 using both i) imitation learning to mimic one or more behaviors of human demonstrations and ii) reinforcement learning. "Co-training" refers to a known semi-supervised learning technique that trains two classifiers based on two different types of data (e.g., human demonstrations and "auto-trajectories" (sequences) of actions automatically selected (i.e., without human involvement) by the semi-trained neural network system 200). The classifiers (e.g., classifiers for human demonstrations) may be used to generate labels for the auto-trajectories, generate rewards for the automatically selected actions, and enable reinforcement learning based on the auto-trajectories.

[0080] The neural network 201 may be trained using a training data set in which the two types of data are mixed and used interchangeably. A human demonstration is an input-output sequence produced as a result of human interaction with a particular computer (or another equivalent computer). The fact that the inputs and outputs of a neural network are in the same space that can be observed from human interaction enables the system to train the neural network with large data sets through imitation learning.

[0081] In some implementations, the system 200 first trains the neural network 201 using imitation learning to mimic human performance behavior, and then trains the neural network 201 using reinforcement learning. In particular, the system 200 may receive a reward after the neural network 201 performs each action and / or after a particular computer controlled by the neural network 201 completes a task. The system 200 may use the received reward to train the neural network 201 using reinforcement learning.

[0082] In some implementations, the system 200 may co-train the neural network 201 on multiple tasks. This is for two reasons. First, training on a frame-by-frame basis for each task seen in co-training is more efficient and has a significant transfer effect. Second, the ultimate goal is a generally useful computer / agent, and therefore it is desirable to have one computer / agent with as many capabilities as possible.

[0083] In particular, to co-train neural network 201 on multiple tasks, system 200 trains neural network 201 to solve multiple tasks simultaneously by randomly sampling the tasks to be solved and training on this mix of tasks. This is in contrast to training a separate neural network to solve each task individually.

[0084] 4 is a flow diagram of an example process 400 for controlling a particular computer to perform a task. The task is one of a number of daily tasks performed on the particular computer (i.e., a task that is statistically representative of the types of tasks that most users occasionally perform using a computer). The number of tasks may include, but is not limited to, clicking a canvas, dragging items on a screen, filling in forms, converting text, forwarding email, making reservations, and booking tickets.

[0085] For convenience, process 400 is described as being performed by one or more computer systems at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 of FIG. 1, may perform process 400.

[0086] Process 400 is performed during the execution of a task, ie, after the system receives a task instruction specifying a task, at each of a series of time steps until the task is completed.

[0087] The system receives control input (step 402), which includes visual input. The visual input includes one or more screen frames of a computer display that represent at least a current state of a particular computer. The screen frames of the visual input are images that represent steps in a process of performing a task on a particular computer.

[0088] In some implementations, the control input further includes one or more linguistic inputs, one or more previous controls, or both. In some implementations, the one or more linguistic inputs include at least one of i) a task instruction input, or ii) a task field input including one or more task fields. The task instruction input may include a text instruction input or a voice instruction input. In some implementations, the one or more linguistic inputs include a Document Object Model (DOM) element input including one or more DOM elements.

[0089] The system processes the control input using the neural network to generate one or more control outputs used to control the specific computer to perform the task (step 404). The one or more control outputs include an action type output that specifies at least one of a pointing device action or a keyboard action to be taken to control the specific computer. In some implementations, the action type output specifies an action to obtain control of the specific computer from another operator for a portion of the process to perform the task. This allows for a situation in which the trained neural network system may collaborate with a human user to control the specific computer, and the trained neural network takes over control of the specific computer from a human user in some circumstances (e.g., to perform an operation for which it would be tedious and cumbersome for a human user to control the specific computer). A voice command input may specify that the neural network system is to obtain control of the specific computer for a portion of the process to perform the task.

[0090] A pointing device action is one of a number of actions of a pointing device associated with a particular computer. The number of actions may include one or more of input sent using a computer mouse, input sent through a touch screen device, or input sent through a touch pad device. A keyboard action may be one of a key press or a text submission.

[0091] In some implementations, the one or more control outputs include at least one of i) a cursor coordinate output, ii) a keyboard key index output, or iii) a task field index output. The cursor coordinate output includes x and y coordinates that represent a desired cursor position on a display of a particular computer.

[0092] In some implementations, the neural network includes a visual processing sub-network, one or more language processing sub-networks, a previous control processing sub-network, a multimodal transformer sub-network, and an output sub-network. Processing a control input using a neural network to generate one or more control outputs in these implementations is described in more detail below with reference to FIG.

[0093] The system determines one or more actions from the one or more control outputs (step 406). Each control output of the one or more control outputs may be modeled by a probability distribution (e.g., a single discrete probability distribution, except for the cursor coordinate, which is modeled by two discrete distributions (for the height and width coordinates)). As explained above, the one or more control outputs include at least one of an action type output, a cursor coordinate output, a keyboard key index output, or a task field index output. To determine the one or more actions, the system first selects an action type from a set of possible actions including no-op (indicating no action), a mouse or touch screen action, and a keyboard action based on the action type output (which is a probability distribution). The one or more actions of the selected action type are to be performed as specified in the remainder of the one or more control outputs. For example, if move mouse is selected, the cursor coordinate output specifies where the cursor should be moved. If emit text is selected, the task field index output specifies which task field string to emit. If key press is selected, the keyboard key index output specifies which key to emit.

[0094] The system performs one or more actions to control the particular computer (step 408). In particular, the system automatically issues commands using a computer-controlled application programming interface (API) to instruct the particular computer to perform one or more actions. For example, if the one or more actions are mouse actions such as moving a cursor to an object on a display, clicking and holding the left mouse button, dragging the object to another position, etc., the system automatically issues commands using the computer-controlled API to instruct the particular computer to perform a mouse action to drag the object to the desired position. The possible actions should not include actions that a human user cannot perform using a mouse, touch screen, or keyboard.

[0095] As mentioned above, the method 400 may include a step of taking over control of the particular computer from a user. Thus, the method 400 provides a user interface that allows a portion of the task to be performed automatically, for example, at the command of the user. In other words, the particular computer may receive one or more inputs from a human user to control the particular computer to perform a first portion of the task. The user may then issue an instruction (e.g., in the form of a task instruction, e.g., by speaking a command that is registered by a microphone) to the neural network system that specifies that the neural network is to perform a second portion of the task following the first portion of the task. That is, the neural network system will assume control of the particular computer. At that stage, the method of FIG. 4 may be performed at each of one or more successive time steps, whereby one or more actions performed by the particular computer (when step 408 is performed at each of the time steps) control the computer to perform the second portion of the task. In the first few implementations of the method 400, the previous control used in the method 400 may be based on the action specified by the user in performing the first portion of the task.

[0096] 5 is a flow diagram of an exemplary process 500 for processing a control input using a neural network to generate one or more control outputs. For convenience, process 500 is described as being performed by one or more computer systems at one or more locations. For example, a neural network system suitably programmed in accordance with this specification, such as neural network system 100 of FIG. 1, may perform process 500.

[0097] Process 500 is performed during the execution of a task, ie, after the system receives a task instruction specifying a task, at each of a series of time steps until the task is completed.

[0098] The system processes the visual input using a visual processing sub-network to generate one or more visual embeddings (step 502). In some implementations, the visual processing sub-network includes an image encoder sub-network and a flattening sub-network. The image encoder sub-network may be a ResNet. The ResNet may have four ResNet blocks.

[0099] The system processes the visual input using a visual processing subnetwork to generate one or more visual embeddings by processing the visual input using an image encoder subnetwork to generate one or more encoded visual feature vectors, and flattening the one or more encoded visual feature vectors by using a flattening subnetwork to generate one or more visual embeddings.

[0100] For each of the one or more linguistic processing sub-networks, the system processes a linguistic input in the one or more linguistic inputs using the linguistic processing sub-network to generate a respective linguistic embedding (step 504).

[0101] In some implementations, each linguistic processing sub-network includes a tokenization layer and a linguistic encoder sub-network. The linguistic encoder sub-network may be a linguistic transformer. The linguistic transformer may have one layer of a transformer with four heads.

[0102] In these implementations, the system processes each linguistic input in one or more linguistic inputs using a linguistic processing sub-network to generate each linguistic embedding by (i) splitting each linguistic input into tokens using a tokenization layer and generating a corresponding token embedding for each token in the tokens, and (ii) attending across the token embeddings by using a linguistic encoder sub-network to generate the corresponding linguistic embedding. To attend across the token embeddings to generate the corresponding linguistic embeddings, the system processes the token embeddings to generate key vectors and value vectors, processes the additional embeddings to generate query vectors, and processes the key vectors, value vectors, and query vectors using a cross-attention mechanism to generate the corresponding linguistic embeddings.

[0103] The system processes one or more previous controls using a previous control processing sub-network to generate a previous control embedding (step 506).

[0104] The system combines the one or more visual embeddings and the one or more linguistic embeddings using a multimodal transformer sub-network to generate a transformed embedding (step 508). In particular, in some implementations, the system processes the one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector. The system processes the one or more linguistic embeddings to generate a linguistic query vector, a linguistic key vector, and a linguistic value vector. The system processes the one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors. The system then processes the visual query vector, visual key vector, visual value vector, linguistic query vector, linguistic key vector, linguistic value vector, additional query vector, additional key vector, and additional value vector using a self-attention mechanism to generate the transformed embedding.

[0105] The system processes the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs (step 510). The output sub-network may include one or more long short-term memory (LSTM) layers. The system may process the concatenation of the transformed embedding and the previous control embedding using an output sub-network to generate one or more control outputs.

[0106] The output sub-network may further include an attention policy sub-network configured to process the output of the one or more LSTM layers to generate a task field index output. The attention policy sub-network may be configured to use corresponding outputs of the two LSTM layers as query vectors, process corresponding task field embeddings generated by the task field language processing sub-network to generate key vectors and value vectors, and use a cross-attention mechanism to process the query vectors, key vectors, and value vectors to generate a task field index output.

[0107] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform an operation or action.

[0108] The subject matter and embodiments of the functional operations described herein may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0109] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may possibly include code that creates an execution environment for a computer program, e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0110] A computer program, sometimes referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in part of a file with other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or parts of code. A computer program may be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0111] As used herein, the term "database" is used broadly to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed differently.

[0112] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed on and operate on the same computer.

[0113] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., FPGAs or ASICs, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0114] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory, or a random access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated in special-purpose logic circuitry. Typically, the computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to the one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash device, to name just a few examples.

[0115] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks.

[0116] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, for example, the feedback provided to the user can be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback, and the input from the user can be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer can interact with the user by sending documents to a device used by the user and receiving documents from the device, for example, by sending a web page to the web browser in response to a request received from the web browser on the user's device. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, for example, a smartphone running a messaging application, and receiving a response message from the user in return.

[0117] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling common and computationally intensive parts of the machine learning training or production, i.e. inference, workload.

[0118] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0119] An embodiment of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, such as an application server, or includes front-end components, e.g., a client computer having a graphical user interface, a web browser, or an app, through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0120] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., to display data to a user interacting with the device acting as a client and to receive user input from the user. Data generated at the user device, e.g., a result of a user interaction, may be received at the server from the device.

[0121] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Some features described in the context of separate embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as functioning in a combination and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0122] Similarly, although operations are illustrated in the figures and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed to achieve a desired result. In some environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0123] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0124] 100 Neural Network System 102 Computer 104 Tasks 106 Control Input 110 Control Output 112 Action 126 Cross Attention Mechanism 200 Neural Network System 201 Neural Networks 202 Visual Input 204 Visual Processing Sub-Network 206 Image Encoder Sub-Network 208 Flattened Subnetwork 210 Visual Embedding 212 Language Processing Sub-network 214 Language Input 216 Tokenization Layer 218 Token Embedding 220 Language Encoder Sub-Network 222 Language Embedding 229 Language Embedding 230 Multimodal Transformer Subnetwork 232 Previous Control Processing Subnetwork 234 Previous Control Embed 236 Transformed embedding 238 Output Subnetworks 240 Additional embedding 242 Self-Attention Mechanism 250 Control Output

Claims

1. 1. A computer-implemented method for controlling a specific computer to perform a task, comprising: receiving a control input comprising a visual input, said visual input comprising one or more screen frames of a computer display representing at least a current state of said particular computer; processing the control inputs using a neural network to generate one or more control outputs used to control the particular computer to perform the task, the one or more control outputs including an action type output specifying at least one of a pointing device action or a keyboard action to be taken to control the particular computer, the neural network including a visual processing sub-network, one or more language processing sub-networks, a previous control processing sub-network, a multimodal transformer sub-network, and an output sub-network; determining one or more actions from the one or more control outputs; performing the one or more actions to control the particular computer; Including, the control input further comprises one or more linguistic inputs, one or more previous controls, or both; processing the control inputs using the neural network to generate the one or more control outputs; processing the visual input using the visual processing sub-network to generate one or more visual embeddings; processing each linguistic input in the one or more linguistic inputs using the linguistic processing sub-network to generate a respective linguistic embedding; processing the one or more previous controls using the previous control processing sub-network to generate a previous control embedding; combining the one or more visual embeddings and the one or more linguistic embeddings using a multimodal transformer sub-network to generate a transformed embedding; processing the transformed embedding and the previous control embedding using the output sub-network to generate the one or more control outputs; Including, method.

2. 2. The method of claim 1, wherein the pointing device action is one of a plurality of pointing device actions associated with the particular computer, the plurality of actions including one or more of input transmitted using a computer mouse, input transmitted through a touch screen device, or input transmitted through a touch pad device.

3. The method of claim 2 , wherein the keyboard action is one of a key press or a text submission.

4. The method of claim 3 , wherein the visual input screen frames are images representing steps in a process of performing the task on the particular computer.

5. The method of claim 1, wherein the one or more linguistic inputs include at least one of i) a task instruction input, or ii) a task field input including one or more task fields.

6. The method of claim 1 , wherein the one or more language inputs comprise a Document Object Model (DOM) element input comprising one or more DOM elements.

7. The method of claim 5 , wherein the task command input comprises a text command input or a voice command input.

8. The method of claim 7 , wherein the one or more control outputs further include at least one of: i) a cursor coordinate output; ii) a keyboard key index output; or iii) a task field index output.

9. 9. The method of claim 8, wherein the cursor coordinate output includes x and y coordinates representing a desired cursor position on the computer display of the particular computer.

10. 10. The method of claim 9, wherein the task is one of a plurality of tasks performed on the particular computer, the plurality of tasks including clicking a canvas, dragging an item on a screen, filling in a form, editing text, highlighting text, forwarding an email, making a reservation, and booking a ticket.

11. The visual processing sub-network comprising an image encoder sub-network and a flattening sub-network, processing the visual input using the visual processing sub-network to generate one or more visual embeddings; processing the visual input using the image encoder sub-network to generate one or more encoded visual feature vectors; flattening the one or more encoded visual feature vectors by using the flattening sub-network to generate the one or more visual embeddings.

2. The method of claim 1, comprising:

12. The method of claim 11 , wherein the image encoder sub-network is a ResNet having four ResNet blocks.

13. each linguistic processing sub-network includes a tokenization layer and a linguistic encoder sub-network; processing the linguistic inputs in the one or more linguistic inputs using the linguistic processing sub-network to generate the respective linguistic embeddings; using the tokenization layer to split each of the linguistic inputs into tokens and generate, for each token in the tokens, a corresponding token embedding; Attention across the token embeddings by using the linguistic encoder sub-network to generate the corresponding linguistic embeddings.

13. The method of claim 12, comprising:

14. The method of claim 13 , wherein the language encoder sub-network is a one-layer transformer with four heads.

15. Attentioning across the token embeddings to generate the corresponding linguistic embeddings; processing the token embeddings to generate key vectors and value vectors; processing the additive embedding to generate a query vector; processing the key vectors, the value vectors, and the query vector using a cross-attention mechanism to generate the corresponding linguistic embeddings; 14. The method of claim 13, comprising:

16. generating the transformed embedding using the multimodal transformer sub-network, processing the one or more visual embeddings to generate a visual query vector, a visual key vector, and a visual value vector; processing the one or more linguistic embeddings to generate a linguistic query vector, a linguistic key vector, and a linguistic value vector; processing one or more additional embeddings to generate additional query vectors, additional key vectors, and additional value vectors; processing the visual query vector, the visual key vector, the visual value vector, the linguistic query vector, the linguistic key vector, the linguistic value vector, the additional query vector, the additional key vector, and the additional value vector using a self-attention mechanism to generate the transformed embedding; 16. The method of claim 15, comprising:

17. 16. The method of claim 15, wherein the output sub-network includes one or more long short-term memory (LSTM) layers.

18. 20. The method of claim 17, wherein the output sub-network further comprises an attention policy sub-network configured to process an output of the one or more LSTM layers to generate a task field index output.

19. The attention policy sub-network, Using the corresponding outputs of the two LSTM layers as query vectors; processing the corresponding task field embeddings generated by the task field language processing sub-network to generate key vectors and value vectors; using a cross-attention mechanism to process the query vector, the key vector, and the value vector to generate the task field index output; The method of claim 18 , configured to:

20. 20. The method of claim 19, further comprising co-training the neural network using i) imitation learning to mimic one or more behaviors of a human performance, and ii) reinforcement learning.

21. training the neural network using imitation learning to mimic human demonstration behaviors shown in the visual input; training the neural network using reinforcement learning; 21. The method of claim 20, further comprising:

22. 22. The method of claim 21, wherein during training of the neural network, the visual input demonstrates how a human would perform the task on the particular computer.

23. 20. The method of claim 19, wherein processing the transformed embedding and the previous control embedding using the output sub-network to generate the one or more control outputs comprises processing a concatenation of the transformed embedding and the previous control embedding using the output sub-network to generate the one or more control outputs.

24. 24. The method of claim 23, wherein the neural network is trained using imitation learning, and optionally reinforcement learning.

25. 25. The method of claim 24, wherein the action type output specifies an action to obtain control of the particular computer from another operator for a portion of a process of performing the task.

26. 26. The method of claim 25, wherein the particular computer is configured to receive one or more inputs from a human user to control the particular computer to perform a first portion of a task, and wherein the neural network is configured to take control of the particular computer upon receiving a command from the user, whereby the neural network is configured to generate one or more actions for controlling the particular computer to perform a second portion of the task subsequent to the first portion.

27. 1. A system comprising: One or more computers; one or more computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 26; A system comprising:

28. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the method recited in any one of claims 1 to 26.

29. 1. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, cause the one or more computers to implement a neural network configured to process control inputs to generate one or more control outputs for controlling a particular computer to perform a task, the control inputs including (i) visual inputs including one or more screen frames of a computer display representing at least a current state of the particular computer, (ii) one or more linguistic inputs, and (iii) one or more previous controls, the neural network: a visual processing sub-network configured to process the visual input to generate one or more visual embeddings; one or more linguistic processing sub-networks, each configured to process a linguistic input in the one or more linguistic inputs to generate a respective linguistic embedding; and a previous control processing sub-network for processing the one or more previous controls to generate a previous control embedding; a multimodal transformer sub-network configured to combine the one or more visual embeddings and the one or more linguistic embeddings to generate a transformed embedding; an output sub-network configured to process the transformed embedding and the previous control embedding to generate the one or more control outputs; A system comprising:

Images