Reinforcement learning method for selecting a software policy network and controlling a software client

Abstract

Techniques are disclosed that enable automation of user interface inputs by generating action sequences to perform tasks using a multi-agent reinforcement learning framework. Various implementations process an intent associated with a received user interface input using an overall reinforcement policy network to select a software reinforcement learning policy network. An action sequence can be generated by processing the intent using the selected software reinforcement learning policy network and a sequence of software client state data. The action sequence is used to control a software client corresponding to the selected software reinforcement learning policy network.

Description

TECHNICAL FIELD

[0001] This application relates to reinforcement learning methods for selecting software policy networks and controlling software clients. BACKGROUND

[0002] Many users repeat certain tasks on client devices (e.g., smartphones), such as ordering pizza, requesting a taxi / car service, downloading a report, controlling a smart device, etc. For each of these tasks, the user repeats a corresponding sequence of actions when the user performs the task. The client device can additionally include an automated assistant (also referred to as a “personal assistant,” “mobile assistant,” etc.) that can interact with the user. The client device can include various client devices, such as a smartphone, a tablet computer, a wearable device, a car system, a standalone personal assistant device, etc. The automated assistant receives input from the user, including spoken natural language input (i.e., an utterance), and can respond by performing an action, by controlling another device, and / or by providing responsive content (e.g., visual and / or audible natural language output). The automated assistant that interacts through the client device can be implemented through the client device itself and / or through one or more remote computing devices (e.g., computing devices in the cloud) that are in network communication with the client device. SUMMARY

[0003] The implementations described herein relate to holistic multi-agent reinforcement learning techniques for: (a) selecting a software policy network to perform a task specified by received user input, and (b) utilizing the selected software policy network to generate a sequence of actions for controlling a corresponding software client to automatically (i.e., without further user input) complete the task. Various implementations can reduce the amount of user input required to complete a task because, for example, an appropriate software policy network can be automatically selected and / or a sequence of actions can be automatically generated utilizing the selected software policy network and one or more (e.g., all) actions of the sequence can be automatically performed to control the corresponding software client and complete the task. As a result, consumption of various client device resources, such as prompting the user to select an appropriate software client, rendering content from the software client, and waiting for user input to control the software client and / or processing such user input, is reduced. Additionally, the amount of user input to complete a particular task can be reduced, which can be advantageous for users with limited flexibility and / or other users. Moreover, various implementations automatically select and / or control a software client based on received user input, thereby automatically guiding human / assistant interactions when performing a technical task specified by the user input.

[0004] Users can use various software applications to perform various tasks (i.e., perform sequences of actions to accomplish tasks) at a client device. For example, a sequence of actions applied in a music application can cause the music application to play music through the client device, a sequence of actions applied in a restaurant application can cause the restaurant application to order a pizza, a sequence of actions applied in a lighting control application can cause the lighting control application to control one or more smart lights, and so on. The implementations disclosed herein present an overall reinforcement learning framework for selecting an appropriate software policy network to resolve an intent specified by a user input, and to control a corresponding software client for resolving the intent. In many implementations, the overall reinforcement learning framework can effectively extend automated assistant functionality to various software applications of a client device.

[0005] Generally, the goal of reinforcement learning is to control an agent that seeks to maximize a reward function, in the context of a client device task, the reward function represents a user-provided definition of what the client device should attempt to accomplish. Reinforcement learning finds an optimal policy from an initial state distribution that maximizes the expected sum of rewards. The reward is based on a reward function that depends on the task to be accomplished. Thus, reinforcement learning seeks to learn an optimal policy for performing a given task.

[0006] The overall reinforcement learning framework is a multi-agent framework and includes an overall agent (and a corresponding overall policy network model) and one or more software client agents (each with a corresponding software policy neural network), where each software application agent corresponds to a particular software application of the client device. For example, a user can provide a natural language spoken input of "play rock music" to the client device, and the overall policy network can be used to select a software policy network that corresponds to the intent of the spoken input. For example, the overall agent can utilize the overall policy network to select a software policy network that corresponds to a hypothetical music client, where the hypothetical music client can play music. The software policy network that corresponds to the hypothetical music client can then be used (e.g., by the corresponding agent) to generate and apply (to the hypothetical music client) a sequence of actions to "play rock music."

[0007] The intent corresponding to the user interface input can be determined using additional components of the client device and / or remote devices in communication with the client device, such as a natural language understanding (NLU) engine of an automated assistant. The determined intent can be processed using a policy network model of the overall reinforcement learning agent (i.e., an overall policy neural network). In many implementations, the policy network model can be a neural network, such as a feedforward neural network, a convolutional neural network, a recurrent neural network, a sequence-to-sequence network, a pointer network, and / or an additional type of neural network model.

[0008] Similarly, each software client can have a corresponding software policy network model that can process the intent, the sequence of state data, one or more portions of the user interface input, and / or additional information to generate a sequence of actions to complete the task. The sequence of actions can be applied to the software clients in turn to complete the task. Note that the sequence of state data (used to generate the sequence of actions) is based on the corresponding state of the software client, which is affected by the actions already applied in the sequence of actions.

[0009] In various implementations, the user interface input can include a portion corresponding to the intent and additional portions corresponding to additional intents. For example, the user interface input can include“turn off the lights and turn down the air conditioning” or“turn off the lights and order a large cheese pizza.” In some such implementations, the overall policy network can process the intent associated with the first portion (e.g., the intent corresponding to turning off the lights) to select a software policy network associated with a first software client for controlling the lights. Similarly, the overall policy network can process the intent associated with the second portion (e.g., the intent corresponding to turning down the air conditioning or the intent corresponding to ordering a large cheese pizza) to select an additional software policy network associated with a second software client for controlling the air conditioning system or ordering a pizza. Each selected software policy neural network can be used to generate a corresponding sequence of actions to complete the corresponding task. For example, the first software policy neural network would be used to generate a first sequence of actions to control the first software client, and the second software policy neural network would be used to generate a second sequence of actions to control the second software client.

[0010] If the software policy network cannot be selected using the overall policy network, the system can prompt the user to provide the software client associated with completing the task, and the software client can be used to update one or more parameters (such as one or more weights) in the overall policy network. Additionally or alternatively, the system can prompt the user for the entire sequence of actions to perform the task, and the portions of the sequence of actions (e.g., the actions where the user selected the appropriate software client) can be used to update one or more parameters of the overall policy network. Further, the sequence of actions can be used as a training episode to update one or more parameters (such as one or more weights) of the corresponding software policy network. Similarly, if the software policy network cannot generate the sequence of actions (despite the correct software policy network being selected using the overall policy network), the system can prompt the user for the sequence of actions to be used as a training episode to update one or more parameters in the software policy network.

[0011] In many implementations, one or more weights in the overall policy network and / or one or more weights in the selected software policy network can be updated based on the reward signal. For example, the reward signal for the overall policy network can indicate success in selecting a particular software application policy network. The overall reward signal can be based on content generated by the selected particular software client, further actions performed by the user (e.g., user actions after the client device renders output), and / or additional reward signals. For example, the user can provide the natural language input "turn on the kitchen light." If after the client device renders output, the user repeats the request "turn off the kitchen light," a negative reward signal can be determined, indicating that the light was not turned off. Similarly, if after the client device renders output, the user provides additional user interface input with a different intent than the previous user interface input, a positive reward signal can be determined, indicating that the task was completed. In other words, the user providing the additional task provides an indication that the previous task was successfully completed. Additionally or alternatively, a message such as a confirmation message (or lack of the confirmation message when the confirmation message is expected) can provide a reward signal indicating that the task was successfully completed (or not successfully completed if the confirmation message was missing, if the confirmation message indicated an alternative task was completed, and so on). For example, the confirmation message "Your order of three pizzas has been placed" after the user interface input "please order me three cheese pizzas" can provide a positive reward signal that the task of ordering the pizzas was completed. In contrast, the confirmation message "Your order of three brownies has been placed" after the user interface input "please order me three cheese pizzas" can provide a negative reward signal that the task of ordering the pizzas was not successfully completed. Further, the client device can prompt the user to provide a reward signal after performing one or more actions (e.g., prompt the user to provide a spoken response indicating whether the task was successfully completed).

[0012] In many implementations, the trained overall policy network and / or the trained software policy network can be transmitted to the client device from a server remote from the client device. In some such implementations, the trained overall policy network and / or the trained software policy network can be further trained and refined for the user of the client device based on a sequence of actions by the user of the client device.

[0013] The various implementations disclosed herein set forth techniques for automating the performance of technical tasks corresponding to received user interface inputs, e.g., to control technical systems. By automating tasks using holistic policy networks and software policy networks, the user interface inputs required to perform the tasks are reduced. For example, the user interface input“decrease the temperature by three degrees” corresponds to a technical task of adjusting the temperature on a smart thermostat. A sequence of actions by which a user performs this task can include: navigating to a software client corresponding to the smart thermostat; opening the software client corresponding to the smart thermostat; selecting the thermostat; decreasing the temperature by one degree; decreasing the temperature by another degree; decreasing the temperature by another degree. By replacing the sequence of user actions with a single instance of user input (i.e., the spoken input“decrease the temperature by three degrees”), automating this task reduces the inputs required. Reducing the inputs required to complete a task can conserve resources, including battery power, processor cycles, and / or memory consumption (i.e., the client device does not need to render output corresponding to each state of the software client, which can reduce the total screen on time, the client device receives fewer user interface inputs, so it can not need to process as many user inputs, etc.).

[0014] The various implementations disclosed herein reduce the action space corresponding to a software policy network in generating one or more iterations of actions in a sequence of actions using the software policy network. For example, not every action of a software client is available in every state of the software client. For example, an initial launch screen of a software client can have actions A, B, C, and D available, while another screen of the software client can have actions C, D, and E available. The actions available in a current state can be determined using one or more of various techniques. For example, the actions available can be determined using an accessibility application programming interface (API) based on an analysis of rendered content (e.g., graphical buttons, text input fields, etc.) and / or an API provided by the software client. Using the actions available in a current state can reduce the action space of a software policy network in various ways, including by masking actions that are not currently available, by only providing available actions to software policy network models that can utilize variable length inputs and variable length outputs (e.g., pointer networks), etc. This reduction in action space allows the policy neural network model to be trained more efficiently and / or to operate more efficiently, which further conserves client device resources, such as battery power, processor cycles, memory usage, and / or additional client device resources.

[0015] Other implementations can include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Yet other implementations can include a system of one or more computers that include one or more processors, which are operable to execute stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0016] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein (generally described above and / or elsewhere herein) are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIGURE illustrates a block diagram of an example environment in which various implementations disclosed herein can be implemented.

[0018] Figure 2 is a flow diagram illustrating an example process of selecting a software policy neural network using an overall policy neural network and generating a sequence of actions using the software policy neural network.

[0019] Figure 3 is a flow diagram illustrating an example process of training an overall reinforcement learning framework in accordance with various implementations disclosed herein.

[0020] Figure 4 FIGURE illustrates a block diagram of an example environment in which various implementations disclosed herein can be implemented.

[0021] Figure 5 FIGURE illustrates an example architecture of a computing device. DETAILED DESCRIPTION

[0022] Turning to Figure 1 FIGURE illustrates an example environment in which various implementations disclosed herein can be implemented. Figure 1This includes client device 102, overall agent engine 104, software agent engine 106, overall training engine 110, and software training engine 114. For example, client device 102 can be a standalone voice-activated speaker device, desktop computing device, laptop computing device, smartphone computing device, computing device in a user's vehicle (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), and / or a user's wearable device including a computing device (e.g., a watch with a computing device, glasses with a computing device, virtual or augmented reality computing device, etc.). Additional and / or alternative client devices may be provided.

[0023] The overall agent engine 104, software agent engine 106, overall training engine 110, and software training engine 114 are example components in which the technologies and / or systems, components, and technologies described herein can be implemented and interfaced with the example components. Operations performed by one or more engines 104, 106, 110, and / or 114 can be distributed across multiple computing systems. In some embodiments, one or more aspects of engines 104, 106, 110, and / or 114 can be combined in a single system and / or one or more aspects can be implemented on client device 102. For example, in some embodiments of those embodiments, aspects of the overall agent engine 104 can be combined with aspects of the overall training engine 110. Engines according to many embodiments can be implemented separately in one or more computing devices that communicate, for example, via a communication network. The communication network can include wide area networks, such as the Internet, one or more local area networks (LANs) (such as Wi-Fi LANs, mesh networks, etc.), and / or one or more bus subsystems. Communication networks may optionally utilize one or more standard communication technologies, protocols, and / or inter-process communication technologies.

[0024] In many implementations, the client device 102 can train the overall policy model 108 using the overall training engine 110. For example, the overall training engine 110 can train one or more parameters of the overall policy model 108 based on a sequence of user actions, such as a user selecting a software client on the client device and then performing an action using the software client. Additionally or alternatively, the overall training engine 110 can update one or more parameters of the overall policy model 108 using a reward signal. The reward signal can be generated using the overall training engine 110, the overall agent engine 104, one or more additional engines of the client device 102, and the like. For example, the reward signal can be based on content generated by a particular software client associated with a software policy model selected using the overall policy model. Additionally or alternatively, the reward signal can be based on one or more further actions performed by the user after the output is rendered by the client device. A negative reward signal can be associated with the user repeating the same user interface input after the output is rendered by the client device. For example, the user providing a user input of “turn off the light” after the output is rendered by the client device based on the user input of “turn off the light” can indicate that the client device did not turn on the light. Similarly, a positive reward signal can be associated with the user providing a different user interface input in response to the output being rendered by the client device. For example, the user can provide an interface input of “play rock music” after the output is rendered by the client device based on the user interface input of “turn off the light” can indicate that the client device did not turn on the light.

[0025] Additionally or alternatively, the client device 102 can train the software policy model 112 using the software training engine 114. For example, the software training engine 114 can train one or more parameters of one or more software policy models 112 based on a sequence of user actions, such as a user selecting a software client on the client device and then performing an action using the software client. Additionally or alternatively, the software training engine 114 can update one or more parameters of the software policy model 112 using a reward signal. In some implementations, the reward signal used to update the software policy model 112 can be the same reward signal used to update the overall policy model 108 used to select the particular software policy model 112. In some other implementations, the reward signal used to update the software policy model 112 is a different reward signal than the reward signal used to update the overall policy model 108. The reward signal can be generated using the software training engine 114, the software agent engine 106, one or more additional engines of the client device 102, and the like.

[0026] Using the overall policy model 108, the overall agent engine 104 can process an intent (associated with the received user interface input) to select a particular software policy model 112. The intent can be generated, for example, by an NLU engine (e.g., the natural language processor of Figure 4 In some implementations, the overall agent engine 104 can additionally process one or more values of one or more slots associated with the intent and also associated with the user input. For example, for the user input“dim the kitchen light,” the overall agent engine 104 can process the intent associated with“dimming” and / or the value corresponding to“kitchen lights” using the overall policy model 108. Thus, in selecting the particular software policy model 112, the overall agent engine 104 can process the intent and / or values associated with the intent using the overall policy model 108. Further, using the particular software policy model 112, the software agent engine 106 can generate a sequence of actions using the intent, the sequence of client device state data, and / or additional input. In some implementations, in generating the sequence of actions, the software agent engine 106 can additionally process values of slots associated with the intent using the particular software policy model 112.

[0027] Figure 2 is a flowchart illustrating an example process 200 of generating a sequence of actions using a software policy network that is selected using an overall policy network. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. Such a system can include one or more components of a client device, such as a processor. Moreover, while the operations of process 200 are illustrated in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted and / or added.

[0028] In block 202, the system receives a user interface input provided by a user of a client device. For example, the system can receive a spoken input captured by one or more microphones of the client device, a gesture input captured by one or more cameras of the client device, a typed input provided through one or more virtual keyboards of the client device, and / or other user interface input.

[0029] In block 204, the system selects a particular software policy neural network by processing the intent corresponding to the user interface input using the overall policy neural network. As described herein, the intent can be generated by a separate component, such as a natural language processor that processes text included in or generated based on the user interface input (e.g., using a speech-to-text processor). In many implementations, selecting the particular software policy network can cause a corresponding software client to open. In various implementations, the particular software policy neural network is selected from a set of software policy neural networks, each of which corresponds to a different software client of the client device. For example, the client device can include a ‘hypothetical music client’, a ‘hypothetical pizza client’, and a “smart light client”. A first software policy neural network can be associated with the ‘hypothetical music client’, a second software policy neural network can be associated with the ‘hypothetical pizza client’, and a third software policy neural network can be associated with the ‘smart light client’. In response to the client device receiving the user interface input “order a large cheese pizza”, the overall software policy neural network can select the second software policy neural network associated with the ‘hypothetical pizza client’ corresponding to the intent to order a pizza.

[0030] In many implementations, the client device can include several software clients for satisfying particular user intents. For example, the client device can include a ‘hypothetical music client’ and an ‘additional music client’, each with a corresponding software policy neural network. For example, one or more of the software clients can be a corresponding application installed on the client device. In various implementations, the system can randomly (or pseudo-randomly) select a software policy network when it determines that multiple software policy networks can be selected using the overall policy network. For example, the system can select the software policy network corresponding to the ‘hypothetical music client’ after receiving a first intent to play music, and can select the software policy network corresponding to the ‘additional music client’ after receiving a second intent to play music.

[0031] Further, in many implementations, the user interface input can include multiple intents. For example, the user interface input “order a large cheese pizza and play party music” includes a first intent associated with ordering a pizza and a second intent to play music. The overall policy neural network can process the first intent (i.e., the intent associated with ordering a large cheese pizza) to select a first software policy neural network, and the overall policy neural network can process the second intent (i.e., the intent associated with playing party music) to select a second software policy neural network.

[0032] If, in block 204, the system is unable to select a particular software policy network using the holistic policy network, the system can prompt the user to perform a sequence of actions on the client device and can use the sequence of actions as a training instance to update one or more parameters of the holistic policy network and / or the software policy network corresponding to the user-selected software client. For example, the user can provide the user interface input “increase the temperature by five degrees.” If the system is unable to select a software policy network by processing the intent to adjust the thermostat using the holistic policy network, the system can provide the user with the response content “unable to process your request, please manually increase the temperature so we can learn how to in the future” to prompt the user to provide a training instance related to adjusting the temperature.

[0033] In block 206, the system generates a particular sequence of actions using the selected particular software policy neural network by processing the sequence of state data and the intent using the particular software policy neural network. The sequence of state data is based on the state of the particular software client during control of the particular software client based on the particular sequence of actions. For example, a sequence of actions to turn on a light can include: opening the software client to turn on the light; selecting the room corresponding to the desired light; selecting the desired light; and turning on the desired light. After each action, the state of the software client changes. For example, the software client can start on a ‘home’ screen. After performing the action on the home screen, the state of the software client can change to an alternative screen.

[0034] In many implementations, the current state of the software client and the intent can be processed using the selected software policy network to generate a first action. After the client device performs the first action, the state of the software client becomes a second state of the software client and the second state of the client device and the intent are processed using the selected software policy network to generate a second action. This can continue until a sequence of actions is generated (and performed) to complete the intent.

[0035] In various implementations, the software policy neural network can be associated with an action space of the software client. For example, the action space can include all possible actions that the software client can perform (and / or all possible actions that can be performed by the client device), and the action sequence is generated based on actions in the action space. Some actions can be available in any state of the software client. For example, an action to close the software client can be available in all states of the software client. However, many actions will not be available in each state of the client device. For example, the software client can be used to order food from a hypothetical cafe and can include an action to populate payment information for a selected order. This action is not available until food to order has been selected (i.e., is not available in each state of the client device). In various implementations, the system can determine one or more actions that are available in a state, and can process the available actions when generating a particular action sequence. For example, the accessibility API can be used to determine actions that are available in the current state. Additionally or alternatively, the available actions can be determined based on analysis of rendered content of the software client, such as graphical buttons, text entry fields, etc. Further, the available actions can include simulations of human touches for interacting with the software client, such as selecting a particular button, scrolling up, scrolling down, swiping left, swiping right, entering text into a text field, etc. For example, the software client can include four selectable buttons, where action 1 corresponds to selecting button 1, action 2 corresponds to selecting button 2, action 3 corresponds to selecting button 3, and action 4 corresponds to selecting button 4.

[0036] In many implementations, portions of the action space that do not correspond to available actions can be masked out. For example, the action space can include: action 1, action 2, action 3, and action 4. The system can determine that action 1 and action 3 are available for the current state, and can mask out action 2 and action 4. For example, the probabilities of action 2 and action 4 can be set to zero, ensuring that generated actions in an action sequence generated using the software policy network will not include action 2 and / or action 4.

[0037] In many implementations, the software policy neural network can process variable length inputs to generate variable length outputs. For example, a pointer network can process variable length inputs and generate a variable length output that "points to" (e.g., using a pointer data structure) the input. In various implementations, the available actions (as well as state data and intents) can be processed using a pointer network to determine a variable length output for generating an action. By not processing unavailable actions, the pointer network can improve computational efficiency during processing using the software policy neural network. Moreover, during training, the pointer network can be more efficiently trained due to not processing unavailable actions, enabling the policy network to more quickly converge to an optimal policy. As a further example, in a first state, the action space can be reduced to action 1 and action 2. In a second state, the action space can be reduced to action 2, action 3, and action 4; in a third state, the action space can again be reduced to action 1 and action 2. In other words, the action space can dynamically change within a single software agent as the state of the software client changes.

[0038] If, in block 206, the system is unable to generate the sequence of actions using the selected software policy network, the system can prompt the user to perform the request and use the sequence of actions generated by the user as an additional training instance for the overall policy network and / or the selected policy network. For example, the system can have selected an incorrect software policy network using the overall policy network, and the training instance can be used to update one or more parameters of the overall policy network. Additionally or alternatively, the overall policy network can select the correct particular software policy network, but the system is unable to generate the sequence of actions using the correct software policy network. In this case, the training instance can be used to update one or more parameters in the correct software policy network.

[0039] In block 208, the system causes the client device to render output based on the content generated by the particular software client corresponding to the particular software policy neural network. For example, the user can provide the input "turn off the kitchen light," and in addition to turning off the light, the system can provide a spoken response "ok, turning off the kitchen lights." Similarly, the user can provide the input "order my morning coffee," and the system can provide a visual response (i.e., text) "OK, your coffee has been ordered."

[0040] Figure 3is a flow diagram illustrating an example process 300 of training an overall reinforcement learning framework. For convenience, the operations of flow diagram are described with reference to a system that performs the operations. Such a system can include one or more components of a client device, such as a processor. Moreover, while the operations of process 300 are illustrated in a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted and / or added.

[0041] In block 302, the system receives a round of user interface input provided by a user of a client device. In many implementations, a round is a sequence of actions with corresponding states that start with an initial state and end with a final state. For example, a round can include an initial state of a client device with no software clients open, an action of selecting a software client, and a corresponding state of the client device once the action is complete, and a second action within the software client and a corresponding state of the client device once the second action is complete, where the state of the client device once the second action is complete is the final state (i.e., the sequence of actions has reached a state of the user’s intent).

[0042] In block 304, the system generates updated overall policy parameters for the overall policy model using the round of user interface input. For example, the system can determine the software selected in the round, and can update one or more parameters (such as weights) in the overall policy model associated with the software policy network corresponding to the selection.

[0043] In block 306, the system updates the overall policy model using the updated overall policy parameters.

[0044] In block 308, the system generates updated software policy parameters for the software policy model using the round of user interface input. For example, in each action of the sequence of actions, the system can determine an intermediate reward for the corresponding action, and update one or more parameters of the software policy network based on each individual action. As a further example, the system can generate updated software policy parameters using the entire round of user interface input.

[0045] In block 310, the system updates the software policy model using the updated software policy parameters.

[0046] Turning now to Figure 4 , an example environment is illustrated in which various implementations can be performed. Figure 4The client computing device 402 includes an instance of an automated assistant client 404. One or more cloud-based automated assistant components 410 can be implemented on one or more computing systems (collectively referred to as a cloud computing system) that are communicatively coupled to the client device 402 over one or more local and / or wide area networks (e.g., the Internet), generally indicated as 408.

[0047] The instance of the automated assistant client 404, through its interaction with the one or more cloud-based automated assistant components 410, can form what appears to be a logical instance of the automated assistant 400 from the perspective of a user with whom the user can have a human-to-computer dialog. It will be appreciated that in some implementations, a user with whom the automated assistant client 404 executing on the client device 402 interfaces can in fact interface with his or her own logical instance of the automated assistant 400. For brevity and simplicity, the term "automated assistant" used herein as a "service" to a particular user will generally refer to the combination of the automated assistant client 404 executing on the client device 402 operated by the user and the one or more cloud-based automated assistant components 410 (which can be shared among multiple automated assistant clients of multiple client computing devices).

[0048] For example, the client device 402 can be a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communication system and an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television, and / or a wearable of the user that includes a computing device (e.g., a watch of the user that has a computing device, glasses of the user that have a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices can be provided. In various implementations, the client computing device 402 can optionally operate one or more other applications in addition to the automated assistant client 404, such as a message exchange client (e.g., SMS, MMS, online chat), a browser, etc. In some of those various implementations, one or more of the other applications can optionally interface with the automated assistant 400 (e.g., through an application program interface), or include their own instance of an automated assistant application that also interfaces with the cloud-based automated components 410.

[0049] The automated assistant 400 engages in human-to-computer dialog with a user through user interface input and output devices of the client device 402. To protect user privacy and / or conserve resources, in many cases, the user must explicitly invoke the automated assistant 400 before the automated assistant fully processes speech. The explicit invocation of the automated assistant 400 can occur in response to certain user interface inputs received at the client device 402. For example, user interface inputs that can invoke the automated assistant 400 through the client device 402 can optionally include actuation of hardware and / or virtual buttons of the client device 402. In addition, the automated assistant client can include one or more local engines 406, such as an invocation engine operable to detect the presence of one or more spoken invocation phrases. The invocation engine can invoke the automated assistant 400 in response to detection of a spoken invocation phrase. For example, the invocation engine can invoke the automated assistant 400 in response to detecting a spoken invocation phrase, such as "Hey Assistant," "OK Assistant," and / or "Assistant." The invocation engine can continuously process (e.g., if not in an inactive mode) a stream of audio data frames based on output from one or more microphones of the client device 402 to monitor for the occurrence of a spoken invocation phrase. When monitoring for the occurrence of a spoken invocation phrase, the invocation engine discards (e.g., after temporary storage in a buffer) any audio data frames that do not include a spoken invocation phrase. However, when the invocation engine detects the occurrence of a spoken invocation phrase in a processed audio data frame, the invocation engine can invoke the automated assistant 400. As used herein, "invoking" the automated assistant 400 can include causing one or more previously inactive functionalities of the automated assistant 400 to activate. For example, invoking the automated assistant 400 can include causing one or more local engines 406 and / or cloud-based automated assistant components 410 to further process the audio data frame based on which the invocation phrase was detected and / or one or more following audio data frames (however, prior to invocation, no further processing of the audio data frames occurs). For example, the local and / or cloud-based components can generate a refined version of the audio data and / or perform other processing in response to invocation of the automated assistant 400. In some implementations, the spoken invocation phrase can be processed to generate a speaker embedding that is used to generate a refined version of the audio data that follows the spoken invocation phrase. In some implementations, the spoken invocation phrase can be processed to identify an account associated with a speaker of the spoken invocation phrase and a stored speaker embedding associated with the account to use to generate a refined version of the audio data that follows the spoken invocation phrase.

[0050] One or more local engines 406 of the automated assistant 400 are optional and may, for example, include the invocation engine described above, a local speech-to-text (“STT”) engine (which converts captured audio to text), a local text-to-speech (“TTS”) engine (which converts text to speech), a local natural language processor (which determines semantic meaning of audio and / or text converted from audio), an overall agent engine (as described above), a software agent engine (as described above), and / or other local components. Because the client device 402 is relatively limited in computing resources (e.g., processor cycles, memory, battery, etc.), the local engines 406 can have limited functionality relative to any counterparts included in the cloud-based automated assistant components 410.

[0051] The cloud-based automated assistant components 410 leverage the virtually unlimited resources of the cloud to perform more robust and / or accurate processing of audio data and / or other user interface inputs relative to any counterparts of the local engines 406. Again, in various implementations, the client device 402 can provide audio data and / or other data to the cloud-based automated assistant components 410 in response to the invocation engine detecting a spoken invocation phrase or detecting some other explicit invocation of the automated assistant 400.

[0052] The illustrated cloud-based automated assistant components 410 include a cloud-based TTS module 412, a cloud-based STT module 414, and a natural language processor 416. In certain implementations, one or more of the engines and / or modules of the automated assistant 400 can be omitted, combined, and / or implemented in components separate from the automated assistant 400. Further, in some implementations, the automated assistant 400 can include additional and / or alternative engines and / or modules.

[0053] The cloud-based STT module 414 can convert audio data to text, which can then be provided to the natural language processor 416. The cloud-based TTS module 412 can convert text data (e.g., natural language responses formulated by the automated assistant 400) to computer-generated speech output. In some implementations, the TTS module 412 can provide the computer-generated speech output to the client device 402 to be output directly, e.g., using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the cloud-based automated assistant components 410 can be provided to one or more local engines 406, which can then convert the text data to computer-generated speech that is output locally.

[0054] The natural language processor 416 of the automated assistant 400 processes free-form natural language input and generates annotated output based on the natural language input for use by one or more other components of the automated assistant 400. For example, the natural language processor 426 can process natural language free-form input that is input as text that is converted by the STT module 414 from audio data provided by a user through the client device 402. The generated annotated output can include one or more annotations of the natural language input and optionally one or more (e.g., all) of the terms of the natural language input.

[0055] In some implementations, the natural language processor 416 is configured to identify and annotate various types of grammatical information in the natural language input. In some implementations, the natural language processor 416 can additionally and / or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), and the like. In some implementations, the natural language processor 416 can additionally and / or alternatively include a coreference resolver (not depicted) configured to group or cluster references to the same entity based on one or more contextual cues. For example, the coreference resolver can be used to resolve the term "there" to "Hypothetical Café" in the natural language input "I liked Hypothetical Café last time we ate there." In some implementations, one or more components of the natural language processor 416 can rely on annotations from one or more other components of the natural language processor 416. In some implementations, one or more components of the natural language processor 416 can use relevant prior input and / or other relevant data other than a particular natural language input when processing the particular natural language input to determine one or more annotations.

[0056] In some implementations, the cloud-based automated assistant component 410 can include a dialog state tracker (not depicted) that can be configured to keep track of a "dialog state," e.g., that includes a belief state of one or more user's goals (or "intentions") during the course of a human-to-computer dialog session and / or across multiple dialog sessions. In determining the dialog state, some dialog state trackers can attempt to determine the most likely values for slots instantiated in the dialog based on user and system utterances in the dialog session. Some techniques utilize a fixed ontology that defines a set of slots and a set of values associated with those slots. Additionally or alternatively, some techniques can be customized for individual slots and / or domains. For example, some techniques can require training a model for each slot type in each domain.

[0057] The cloud-based automated assistant component 410 can include a dialog manager (not depicted) that can be configured to map a current dialog state (e.g., provided by the dialog state tracker) to one or more "response actions" from a plurality of candidate response actions that are then performed by the automated assistant 400. Response actions can take a variety of forms depending on the current dialog state. For example, initial and midstream dialog states corresponding to turns of a dialog session that occur prior to the last turn (e.g., when a final user intended task is performed) can map to various response actions that include the automated assistant 100 outputting additional natural language dialog. For example, such a responsive dialog can include a user's request to provide parameters (i.e., fill slots) for some action that the dialog state tracker believes the user intends to perform. In some implementations, response actions can include actions such as "request" (e.g., seek a parameter for slot filling), "offer" (e.g., suggest an action or course of action to the user), "select," "inform" (e.g., provide requested information to the user), "no match" (e.g., inform the user that the user's last input was not understood), a command to a peripheral device (e.g., turn off a light bulb), and the like.

[0058] In various implementations, the automated assistant 400 can leverage the overall multi-agent reinforcement learning techniques disclosed herein to act on various user inputs. For example, a received input can be processed by the STT module 414, and the resulting text analyzed by the natural language processor 416 to determine one or more intents specified by the user input, and optionally one or more values for slots of the intents. Further, the automated assistant 400 can include or interface with an overall agent engine and a software agent engine. With the overall policy model, the overall agent engine can process the intents and optionally the values to select one or more software policy networks to satisfy the intents. Further, the software agent engine can leverage the software policy networks to generate actions to control a corresponding software client based on the intents (and optionally the values) and state data of the software client during the control period. The assistant 400 can render (e.g., visually and / or aurally) output that is based on content generated by the software client in response to being controlled in satisfying the intents.

[0059] Figure 5 FIG. 5 is a block diagram of an example computing device 510 that can optionally be used for implementing one or more aspects of the technology described herein. In some implementations, one or more of the client computing devices and / or other components can include one or more components of the example computing device 510.

[0060] The computing device 510 typically includes at least one processor 514 which communicates with a number of peripheral devices via bus subsystem 512. These peripheral devices can include a storage subsystem 524, including, for example, a memory subsystem 525 and a file storage subsystem 526, user interface output devices 520, user interface input devices 522, and a network interface subsystem 516. The input and output devices allow user interaction with the computing device 510. Network interface subsystem 516 provides an interface to outside networks and devices, and is coupled to corresponding interface devices in other computing devices.

[0061] The user interface input devices 522 can include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information to the computing device 510 or to a communication network.

[0062] User interface input devices 520 can include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen, or other types of input devices. User interface input devices 520 can also include a microphone, a camera or a video camera, for example. User interface input devices 520 can also include an audio input device such as a microphone for inputting voice commands. User interface input devices 520 can also include an audio or video recording device, such as a digital audio recorder or a digital video camera. User interface input devices 520 can also include a television tuner or broadcast receiving component, a satellite dish, a cable modem, a satellite modem, a telephone receiver, or other types of components for receiving television or radio broadcasts. User interface input devices 520 can also include an Internet or other network access component such as a modem or other components for establishing communications over the Internet or other networks. User interface input devices 520 can also include a non- visual display such as a audio display device, for example. Generally, use of the term "input device" is intended to include all possible types of devices and ways to input information to computing device 510 or to another machine or computing device.

[0063] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 can include logic to perform selected aspects of one or more processes of Figure 2 and / or Figure 3 various components depicted in Figure 1 and / or Figure 4 .

[0064] These software modules are generally executed by processor 514 alone or in combination with other processors. Memory 525 used in the storage subsystem 524 can include a number of memories including a main random access memory ("RAM") 530 for storage of instructions and data during program execution and a read only memory ("ROM") 532 in which fixed instructions are stored. A file storage subsystem 526 can provide persistent (nonvolatile) storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations can be stored by file storage subsystem 526 in the storage subsystem 524, or in another machine accessible by the processor(s) 514.

[0065] Bus subsystem 512 provides a mechanism for letting the various components and subsystems of computing device 510 communicate with each other as intended. Although bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem can use multiple busses.

[0066] Computing device 510 can have various types of configurations, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the variety of configurations that can be used to implement some implementations, the description of computing device 510 in Figure 5 is merely intended as a specific example. Many other configurations of computing device 510 can have more or fewer components than the computing device depicted in Figure 5 .

[0067] In situations in which the systems described herein collect personal information about users (or in which personal information can be used), users can be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and / or how to receive content from a content server. Users can also have the opportunity to take one or more steps to remove personally identifiable information from existing data the system has collected, or to limit the contact for the user with previously stored data.

[0068] In some implementations, a method implemented by one or more processors is provided, the method including receiving, by one or more user interface input devices of a client device, user interface input provided by a user of the client device. The method also includes selecting, from a plurality of candidate software policy neural networks, a particular software policy neural network, wherein selecting the particular policy neural network includes selecting the particular policy neural network based on output generated by processing one or more features of the user interface input using an overall policy neural network, wherein the overall policy neural network represents an overall reinforcement learning policy, wherein the one or more features of the user interface input include an intent identified based on analyzing the user interface input, and wherein the particular software policy neural network represents a reinforcement learning policy for a particular software client. The method further includes, in response to selecting the particular software policy neural network: generating, using the particular software policy neural network, a particular sequence of actions and controlling the particular software client based on the particular sequence of actions. In some implementations, generating the particular sequence of actions includes processing, using the particular software policy neural network, a sequence of state data for the particular software client and the intent, wherein the sequence of state data is based on a state of the particular software client during controlling the particular client based on the particular sequence of actions. The method further includes causing the client device to render output based on content generated by the particular software client in response to controlling the particular software client based on the particular sequence of actions.

[0069] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0070] In some implementations, the particular software policy neural network is used to generate output in an action space, where generating the particular sequence of actions includes identifying available actions for a given state data of the sequence of state data. The method further includes generating the given action in the particular sequence of actions based on processing the given state data and based on the identified available actions for the given state data. In some versions of those implementations, generating the particular sequence of actions further includes using an accessibility application programming interface to identify the available actions. In some versions of those implementations, the method further includes identifying a portion of the action space that does not correspond to the available actions for the given state data. In some versions of those implementations, generating the given action based on processing the given state data and based on the identified available actions includes, in the action space, masking any portion of the action space that does not correspond to the available actions for the given state data, and selecting the given action from the portion of the action space that was not masked by the masking. In some versions of those implementations, generating the given action based on processing the given data and based on processing the identified available actions further includes processing the available actions and the given state data and intent using the particular software policy neural network to restrict the action space to the available actions. In some versions of those implementations, the particular software policy neural network includes a pointer network configured to determine a variable length output for generating the given action.

[0071] In some implementations, the user interface input provided by the user of the client device includes natural language input spoken by the user and directed to an automated assistant application.

[0072] In some implementations, the method further includes analyzing the user interface input to identify the intent and an additional intent, wherein the one or more features of the user interface input that are used to select the particular software policy neural network include the intent but not the additional intent. The method further includes selecting, from the plurality of candidate software policy neural networks, an additional software policy neural network by processing the additional intent using the overall policy neural network, wherein the additional software policy neural network represents a reinforcement learning policy of an additional software client. The method further includes generating an additional particular sequence of actions using the additional software policy neural network, and controlling the additional software client based on the additional particular sequence of actions. Generating the additional particular sequence of actions includes processing an additional sequence of state data of the additional software client and the additional intent using the additional software policy neural network, wherein the additional sequence of state data is based on an additional state of the additional software client during controlling the additional software client based on the additional particular sequence of actions, and causing the client device to render additional content that is based on additional content generated by the additional software client in response to controlling the additional software client based on the additional sequence of actions. In some versions of those implementations, the additional software policy neural network is configured to generate an output in a further action space, wherein generating the additional sequence of actions includes identifying available actions for a given state data of the additional sequence of state data. In some versions of those implementations, the method further includes generating a given action in the additional sequence of actions based on processing the given state data of the additional sequence of state data and based on the identified available actions for the given state data of the additional sequence of state data. In some versions of those implementations, generating the additional sequence of actions includes identifying a portion of the further action space that does not correspond to the available actions for the given state data of the additional sequence of state data. In some versions of those implementations, generating the given action in the additional sequence of actions based on processing the given state data of the additional sequence of state data and based on the identified available actions for the given state data of the additional sequence of state data includes masking any portion of the further action space that does not correspond to the available actions for the given state data of the additional sequence of state data in the further action space. In some versions of those implementations, the method further includes selecting the given action in the additional sequence of actions from the portion of the further action space that is not masked by the masking. In some versions of those implementations, generating the given action in the additional sequence of actions based on processing the given state data of the additional sequence of state data and based on the identified available actions for the given state data of the additional sequence of state data includes processing, using the additional software policy neural network, the available actions, the given state data of the additional sequence of state data, and the additional intent to restrict the further action space to the available actions. In some versions of those implementations, the additional software policy neural network includes a pointer network configured to determine a variable-length output used to generate the given action in the additional sequence of actions.

[0073] In some implementations, the client device includes an automated assistant client, and wherein the automated assistant client uses the software policy neural network to generate the particular sequence of actions and controls the particular software client based on the particular sequence of actions. In some versions of those implementations, the received user interface input is spoken utterance input of the user and includes a invocation phrase portion and an additional portion. In some versions of those implementations, the method further includes determining the additional portion of the user interface input. In some versions of those implementations, the method further includes analyzing the additional portion to identify the intent.

[0074] In some implementations, in response to causing the client device to render the output, the output is based on content generated by the particular software client in response to being controlled based on the sequence of actions: the method further includes determining an overall reward signal that provides an indication of success in using the overall policy neural network to select the particular software policy neural network corresponding to the intent. In some versions of those implementations, the method further includes determining updated overall policy parameters for the overall policy neural network using the overall reward signal. In some versions of those implementations, the method further includes updating the overall policy neural network using the updated overall policy parameters. In some versions of those implementations, the overall reward signal is based on the content generated by the particular software client and / or based on one or more further actions performed by the user after causing the client device to render the output based on the content generated by the particular software client.

[0075] In some implementations, in response to causing the client device to render the output, the output is based on content generated by the particular software client in response to being controlled based on the sequence of actions: the method further includes determining a software reward signal that provides an indication of success in using the particular software policy neural network to generate the particular sequence of actions. In some versions of those implementations, the method further includes determining updated software client policy parameters for the software policy neural network using the software reward signal. In some versions of those implementations, the method further includes updating the software policy neural network using the updated software client policy parameters. In some versions of those implementations, the software reward signal is based on the content generated by the particular software client and / or based on one or more further actions performed by the user after causing the client device to render the output based on the content generated by the particular software client.

[0076] In some implementations, processing the sequence of state data and the intent for the particular software client using the particular software policy neural network includes: determining the sequence of state data for the particular software client using an accessibility application programming interface.

[0077] In some implementations, the overall policy neural network is a deep neural network, and wherein the software policy neural network is an additional deep neural network.

[0078] In some implementations, the overall policy neural network is at least partially included in an operating system of the client device.

[0079] In some implementations, a method implemented by one or more processors is provided that includes receiving, by one or more user interface input devices of a client device, a round of user interface input provided by a user of the client device, wherein the round of user interface input includes a sequence of actions for a particular software client. The method further includes generating, using the round of user interface input, updated overall policy parameters for an overall policy neural network, wherein the overall policy neural network represents a reinforcement learning policy, wherein the updated overall policy neural network is used to select a particular software policy neural network corresponding to the particular software client. The method further includes updating the overall policy neural network using the updated overall policy parameters. The method further includes generating, using the round of user interface input, updated software policy parameters for the software policy neural network. The method further includes updating the software policy neural network using the updated software policy parameters.

[0080] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0081] In some implementations, receiving the round of user interface input provided by the user of the client device is in response to an automated assistant client of the client device determining that the particular software policy neural network cannot be selected using the overall policy neural network.

[0082] In some implementations, receiving the round of user interface input provided by the user of the client device is in response to an automated assistant client of the client device determining that the sequence of actions cannot be generated using the software policy neural network.

[0083] In some implementations, the round of user interface input provided by the user of the client device includes a plurality of instances of user interface input, wherein each instance of user interface input includes a starting client device state, a subsequent client device state that transitions from the starting client device state, and a user action that transitions from the starting client device state to the subsequent client device state.

Claims

1. A reinforcement learning method implemented by one or more processors, comprising: Receive user interface input provided by a user of the client device via one or more user interface input devices of the client device; A software policy neural network is selected from multiple candidate software policy neural networks based on the output generated by processing one or more features of the user interface input using a holistic policy neural network. The holistic policy neural network is used to select an appropriate software policy network to parse the intent specified by the user interface input and to control the corresponding software client used to parse the intent. Wherein, the overall policy neural network represents the overall reinforcement learning policy. Wherein, the one or more features of the user interface input include an intent identified based on analysis of the user interface input, and The software policy neural network represents a reinforcement learning policy of the software client. The software policy neural network is trained based on multiple previous user action sequences, the previous user action sequences including the user selecting the software client on the client device and then using the software client to perform the user action. In response to the selection of the software policy neural network: The software policy neural network is used to generate action sequences and to control the software client based on these action sequences. Generating the action sequence includes: The software policy neural network is used to process the state data sequence of the software client and the intent. The state data sequence is based on the state of the software client during the control of the software client based on the action sequence; and The client device renders an output based on content generated by the software client in response to control of the software client based on the action sequence; Analyze the user interface input to identify the intent and additional intents. Wherein, the one or more features of the user interface input used to select the software policy neural network include the intent, but do not include the additional intent; By processing the additional intent using the overall policy neural network, an additional software policy neural network is selected from the plurality of candidate software policy neural networks. Wherein, the additional software policy neural network represents the reinforcement learning policy of the additional software client, and The additional software policy neural network is used to generate additional action sequences, and the additional software client is controlled based on these additional action sequences. Generating the additional action sequence includes: The attached software policy neural network is used to process the attached state data sequence of the attached software client and the attached intent. The additional state data sequence is based on the additional state of the additional software client during the control of the additional software client based on the additional action sequence; and The client device renders additional output based on additional content generated by the additional software client in response to controlling the additional software client based on the additional action sequence.

2. The method according to claim 1, wherein, The software policy neural network is used to generate outputs in the action space, wherein generating the action sequence includes: Identify available actions for a given state data in the state data sequence; A given action in the action sequence is generated based on processing the given state data and the identified available actions for the given state data.

3. The method according to claim 2, wherein, Generating the action sequence includes: Identify the portion of the action space that does not correspond to the available actions for the given state data.

4. The method according to claim 2, wherein, Generating the given action based on the processing of the given state data and the identified available actions includes: In the action space, mask any portion of the action space that does not correspond to the available actions for the given state data; and The given action is selected from the portion of the action space that is not covered by the mask.

5. The method according to claim 2, wherein, Generating the given action based on the processing of the given state data and the identified available actions includes: The software policy neural network is used to process the available actions, the given state data, and the intent to restrict the action space to the available actions.

6. The method according to claim 5, wherein, The software policy neural network includes a pointer network configured to determine a variable-length output for generating the given action.

7. The method according to claim 1, wherein, The user interface input provided by the user of the client device includes natural language input spoken by the user and refers to an automation assistant application.

8. The method according to claim 1, wherein, The additional software policy neural network is used to generate outputs in a further action space, wherein generating the additional action sequence includes: Identify available actions for a given state data in the additional state data sequence; A given action in the additional action sequence is generated based on the given state data of the additional state data sequence and the identified available actions based on the given state data of the additional state data sequence.

9. The method according to claim 8, wherein, Generating the additional action sequence includes: Identify the portion of the further action space that does not correspond to the available actions for the given state data in the additional state data sequence.

10. The method according to claim 8, wherein, Generating the given action in the additional action sequence based on processing the given state data in the additional state data sequence and based on the identified available actions for the given state data in the additional state data sequence includes: In the further action space, mask any portion of the further action space that does not correspond to the available actions for the given state data in the additional state data sequence; and The given action is selected from the portion of the additional action sequence that is not masked by the mask in the further action space.

11. The method according to claim 8, wherein, Generating the given action in the additional action sequence based on the given state data from the processing of the additional state data sequence and the available actions identified for the given state data in the additional state data sequence includes: The additional software policy neural network is used to process the available actions, the given state data of the additional state data sequence, and the additional intent, in order to limit the further action space to the available actions.

12. The method according to claim 11, wherein, The additional software policy neural network includes a pointer network configured to determine a variable-length output for generating the given action in the additional action sequence.

13. The method according to claim 1, wherein, The client device includes an automation assistant client, wherein the automation assistant client uses the software policy neural network to generate the action sequence and controls the software client based on the action sequence.

14. The method according to claim 13, wherein, The received user interface input is the user's spoken input and includes a calling phrase portion and an additional portion, and further includes: Determine the additional portion of the user interface input; and Analyze the additional portion to identify the intent.

15. The method of claim 1, further comprising: In response to the client device rendering output, the output being based on content generated by the software client in response to control based on the action sequence: A global reward signal is determined, which provides an indication of successfully using the global policy neural network to select the software policy neural network corresponding to the intent.

16. The method of claim 15, further comprising: The overall reward signal is used to determine updated overall policy parameters for the overall policy neural network; as well as The updated global policy parameters are used to update the global policy neural network.

17. The method according to claim 15, wherein, The overall reward signal is based on the content generated by the software client and / or on one or more further actions performed by the user after the client device renders the output based on the content generated by the software client.

18. The method of claim 1, further comprising: In response to the client device rendering output, the output is based on content generated by the software client in response to control based on the action sequence. A software reward signal is determined, which provides an indication of the successful generation of the action sequence using the software policy neural network.

19. The method of claim 18, further comprising: The software reward signal is used to determine updated software client policy parameters for the software policy neural network; as well as The software policy neural network is updated using the updated software client policy parameters.

20. The method according to claim 19, wherein, The software reward signal is based on the content generated by the software client and / or on one or more further actions performed by the user after the client device renders the output based on the content generated by the software client.

21. The method according to claim 1, wherein, Processing the state data sequence and intent for the software client using the software policy neural network includes: The accessibility application programming interface is used to determine the state data sequence for the software client.

22. The method according to claim 1, wherein, The overall policy neural network is a deep neural network, and the software policy neural network is an additional deep neural network.

23. The method according to any one of claims 1-22, wherein, The overall policy neural network is at least partially included in the operating system of the client device.

24. A reinforcement learning system, the system comprising one or more processors for performing the method according to any one of claims 1-23.

25. A non-volatile computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 23.

Images