Reinforcement learning for active sequence processing

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 62 / 936,315, filed on November 15, 2019 and to U.S. Provisional Application No 62 / 971,877 filed February 7, 2020.BACKGROUND

[0002] This specification relates to a neural network system that processes a sequence of task inputs to generate a task output in order to perform a machine learning task.

[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. WO 2018 / 224471 A1 discloses a system for selecting actions to be performed by an agent interacting with an environment. WO 2019 / 155065 A1 discloses a network system to generate output data values from input data values according to one or more learned data.SUMMARY

[0004] The invention is defined by the independent claims. Additional embodiments are defined by the dependent claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1 shows an example neural network system for active sequence processing. FIG. 2 shows further details of an example neural network system. FIG. 3 is a flow diagram of an example process for training a machine learning system for active sequence processing. FIG. 4 is a flow diagram of an example process for generating a machine learning task output. Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0006] FIG.1 shows an example neural network system 100 for active sequence processing. The system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques below can be implemented.

[0007] The neural network system 100 may be configured to receive a sequence of task inputs 104 and to perform a machine learning task. The sequence of tasks may be processed more efficiently by exploiting temporal redundancy in the sequence. The sequence of task inputs 104 may each be real-world data collected by at least one sensor, such as a sequence of video frames (a "video") captured by one or more video cameras, and the machine learning task is in this case to recognize one or more actions performed in the sequence of video frames. For example, the sequence of video frames may be a sequence of video frames of a human performing an activity and the task may be to identify the activity. The cost of perception may be explicitly modelled to process the sequence of task inputs 104 more efficiently. For example a reinforcement learning environment may be created from a sequence of video frames while minimizing perception cost to process as few frames as possible.

[0008] An example where the sequence of tasks inputs 104 is a sequence of video frames will be used to exemplify the operation of the neural network system described herein. However, the machine learning task may alternatively, be a different kind of machine learning task that operates on a sequence of inputs i.e. a speech recognition task (in this case, the task inputs are derived from successive audio samples of a speech waveform collected by a microphone at respective times; optionally, the audio samples may be data in the time domain and / or in the frequency domain, such as Short-time Fourier transforms (STFT) of successive time windows of the speech waveform).

[0009] The system 100 may process multiple sequences of video frames (i.e. multiple videos) successively. The system 100 attempts to classify each video with a respective class label y while consuming as few frames as possible. For example, since the task is action recognition, the class labels may include, for example, labels for actions from a set of actions, e.g. dancing, swimming, playing guitar etc. In this case, the class label assigned to a given video to indicate the action shown in the video is termed an "activity class label". At the end of each video the system 100 can use an accumulated representation to predict the activity class label y. The task may be modelled as a Markov Decision Process (MDP).

[0010] The system 100 includes a reinforcement learning (RL) neural network 102. The RL neural network 102 is configured to generate for each task input 104 a respective decision about whether to encode the task input or skip the task input. The RL neural network 102 provides the respective decision for each task input to a task neural network 106.

[0011] The task neural network 106 is configured to receive from the RL neural network 102, for each task input in the sequence of task inputs 104, a respective decision that determines whether to encode the task input or to skip the task input. The task neural network 106 is further configured to process each of the un-skipped task inputs, i.e., each of the task inputs for which the decision is to encode the task input in the sequence of task inputs to generate a respective accumulated feature for the un-skipped task input. In some implementations generating a respective accumulated feature includes computing a hidden representation of each un-skipped task. The hidden representation can be integrated over time. The respective accumulated feature for a given un-skipped task input characterizes features of the un-skipped task input and of any previous un-skipped task inputs in the sequence.

[0012] In a first implementation the task neural network 106 may be further arranged to generate a machine learning task output 108 for the machine learning task based on the last accumulated feature generated for the last un-skipped task input in the sequence. In a different implementation, the task neural network 106 may be arranged to generate a temporary machine learning output based for each un-skipped task input on the respective accumulated feature and generate a machine learning task output 108 for the machine learning task by averaging the temporary machine learning outputs generated for the un-skipped task inputs in the sequence. That is, each temporary machine learning output is an output that is of the type that is required for the machine learning task and is referred to as "temporary" because the output is generated before the entire sequence has been processed.

[0013] In some implementations, to generate a machine learning task output 108 for the machine learning task, the task neural network 106 generates an output that specifies one or more labels that identify an activity.

[0014] FIG. 2 shows a further details of an example neural network system 200 for active sequence processing.

[0015] FIG. 2 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques below can be implemented. The system 200 includes the RL neural network 102 and the task neural network 106

[0016] In some implementations the task neural network 106 may further include an encoder 202, an accumulated feature neural network 204, and a final neural network 206.

[0017] The encoder 202 is configured to process the current un-skipped task input to generate an encoded feature representation for the current un-skipped task input. In some implementations the encoder 202 may be a convolutional neural network (CNN). Generally, the encoder 202 can have any neural network architecture that allows the encoder 202 to process a task input, e.g., a video frame, an embedding of a word or other piece of text (e.g. the output of a further neural network (not shown) which receives the word or other piece of text), or features of an audio sample, to generate a lower-dimensional representation of the task input.

[0018] The encoder 202 provides the encoded feature representation to an accumulated feature neural network 204. In some implementations the accumulated feature neural network may be a recurrent neural network (RNN), for example, a long-term short-term memory (LSTM). The accumulated feature neural network 204 is configured to, for each of the un-skipped task inputs, process the encoded feature representation of the current un-skipped task input and a previous accumulated feature associated with the previous un-skipped task input to generate a respective accumulated feature for the current un-skipped task input.

[0019] In some implementations, the accumulated feature neural network 204 is configured to generate a temporary machine learning output based on the respective accumulated feature and, once the entire sequence has been processed, generate a machine learning task output 108 for the machine learning task by combining the respective temporary machine learning outputs generated for the un-skipped task inputs in the sequence, such as by averaging the temporary machine learning outputs generated for the un-skipped task inputs in the sequence.

[0020] The accumulated feature neural network 204 also generates for each un-skipped task input a corresponding feature output, which is also a vector having one or more components (typically multiple components). The accumulated feature neural network 204 may generate this based on the respective accumulated feature. The encoder 202 can further use the feature output from the respective accumulated feature of the current un-skipped task input and provide the feature output to the RL neural network 102. For example, a residual skip connection over the accumulated feature neural network can be applied, in which case the feature output is a concatenation of the encoder output with the accumulated feature of the current un-skipped task input.

[0021] The RL neural network can then use the feature output to generate a next decision for the next un-skipped task input in the sequence.

[0022] The encoder 202 and accumulated feature neural network 204 may run only when needed to encode information. For example, the encoder 202 and accumulated feature neural network 204 may not run for the task inputs which the RL neural network 102 indicates should be skipped, resulting in a saving of resources. In this case, the feature output supplied to the RL neural network 102 may remain the same until the RL neural network indicates that one of the sequence of task inputs should not be skipped. Note that when the RL neural network is generating a decision for all task inputs up to and including the first task input for which the RL neural network decides that the task input should be encoded, the RL neural network may receive a default feature output, which may be the same for all those task inputs.

[0023] The RL neural network 102 maintains an action space comprising an encode action and a skip action.

[0024] The RL neural network 102 is configured to, for each task input in the sequence process, use the received feature output to generate a corresponding distribution of actions in the action space, and generate the next decision for the next task input in the sequence by sampling an action from the action space according to the corresponding distribution of actions.

[0025] For each of the un-skipped task inputs in the sequence, the RL neural network 102 may be configured to receive, from the task neural network 104, the feature output 208 generated by the accumulated feature neural network, and to generate a next decision for the next task input in the sequence. For each of the skipped task inputs in the sequence, the RL neural network may be configured to receive, from the task neural network, the feature output generated by the accumulated feature neural network for the most recent un-skipped task input (e.g. a frame of the video sequence), and produce a probability distribution over possible next actions and by sampling from the distribution generate an encode or skip action generates a next decision for the next task input after the skipped task input in the sequence. That is, the RL neural network 102 can generate a first probability for the encode action and a second probability for the skip action and then sample either the encode action or the skip action based on the first or second probabilities.

[0026] In some implementations, to determine whether to encode or skip a frame may comprise the RL neural network 102 learns a policy π that takes as input the feature output 208 h of the accumulated feature neural network and produces a probability distribution over possible next actions. In some implementations sampling this distribution can be used to generate the next encode or skip action. The RL neural network 102 may run at the frame rate, i.e. may run for each task input, taking h as input. When frames are skipped h does not change and the policy neural network sees the same input. An embedding of the current time step can be added, so the RL network can keep track of the passage of time even when h does not change.

[0027] In some implementations the RL neural network 102 may be a Recurrent Neural Network (RNN), for example, a long short-term memory (LSTM).

[0028] During a training phase of the neural network system 200 (e.g. as described below with reference to Fig. 3), the RL neural network 102 may receive a reward for making the next decision for the task input. For example, the RL neural network 102 receives a reward of 0 for skipping the frame and -1 for encoding the frame. The RL neural network 102 receives a reward based on the quality of the task output relative to the ground truth task output, for example if the task output is correctly labelled. In some implementations there is no (zero) net cost for skipping a frame. For example the RL neural network 102 may receive a negative reward for deciding to encode a frame. The reward for the skip action may typically be greater than the reward for the encode action.

[0029] The final neural network 206 may further be configured to process the last accumulated feature to generate the machine learning task output for the machine learning task. Outputs of the accumulated feature neural network 204 can be concatenated with the corresponding input through a skip connection. The concatenated inputs and outputs can be projected into the activity class space, obtaining class logits. The class logits can be averaged over time obtaining a distribution over the possible class labels y = ^ ∑ i = 1 k class MLP h i . h i represents the concatenation between the encoder neural network output and the accumulated feature neural network output, at each un-skipped time step i. When using temporary outputs, these h i are passed through the class_MLP to obtain temporary logits, which are then averaged. When temporary outputs are not used, only h i corresponding to the last un-skipped step is passed through the class_MLP to obtain the logits. k denotes the number of encoded (i.e. un-skipped) task inputs and i represents the indices of the frames that were encoded. The indices may be sequential or non-sequential.

[0030] For example, the machine learning task output may be a class label or other classification. The final neural network 206 may be a multilayer perceptron (MLP) neural network.

[0031] FIG. 3 is a flow diagram of an example process 300 for training a machine learning system for active sequence processing. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system including one or more neural networks for active sequence processing, appropriately programmed in accordance with this specification, can perform the process 300. For example, the system can be the neural network system for active sequence processing 100 of FIG. 1 or 200 of FIG. 2. The system jointly trains RL neural network and the task neural network to minimize a loss objective function using a stochastic gradient descent method in which gradients from the RL neural network 102 are not back-propagated into the task neural network. The system trains 302 the task neural network using a supervised signal (minimizing the cross entropy loss between the true class y and the predicted class ŷ). The system trains 304 the RL neural network 102 through reinforcement learning. For example, a Markov Decision Process defined by the tuple (S, A, R, T) where S represents the set of states, A the set of possible actions, R the reward function, and T the transition model may be defined. The representation components may be considered as part of an environment ε w , i.e. a parametric environment parametrized by w where w are the weights of the encoder and of the recurrent neural network accumulating the features, which get updated at each training iteration, changing the environment that the policy sees, and the policy as part the RL neural network. The set of states may be defined as S ∈ R d< , where d is the size of the accumulated feature neural network output (i.e. the feature output). The action space may be defined as A = {encode, skip}. The policy π θ may be given by the mapping π θ : R d< → {encode, skip} and is parameterized by the parameters θ of the RL neural network. In some implementations the transition model from s t to s t+1 is deterministic and is defined as: h t encode → h ′ , t + 1 , h t skip → h , t + 1 , h N ⋅ → e .

[0032] At the end of a video sequence (t = N), the system may transition automatically to an end-of-episode (e) state, irrespective of the action taken. The rewards are R = {r encode , r skip , r class }. r class is a reward the RL neural network receives a reward dependent on the quality of the task output relative to the ground truth output. In some implementations it may by a hard reward (e.g. 1 if the classification is correct, 0 otherwise). In other implementations it may be a soft reward, equal to the normalized amplitude of the logit of the correct class.

[0033] During training, the RL neural network 102 learns a distribution of tasks using a training dataset of video sequences (e.g. for certain video sequences the task may be to classify the video sequence as a first action, and for other video sequences the task may be to classify the video sequence as a second, different action), wherein each task is determined by (v i , w j ) where v i is a video sequence from the dataset and w j are environment weights at the current training iteration, i.e. the parameters of the task neural network. At different training iterations the environment (e.g. the output of the task neural network) may look different for the same sequence sampled from the dataset because the parameters of the task neural network change during training. The parametric nature of the environment's dependence on the weights w differs from a classical RL network.

[0034] In some implementations the system trains 304 the RL neural network 102 based on an advantage-actor critic (A2C) approach. In an A2C approach the stochastic gradient decent method may be REINFORCE. The expected return of a policy E π [R t ], may be maximized by computing policy gradients using the REINFORCE algorithm to update the parameters θ. The empirical return R t = ∑ k = 0 N − t γ k r t + k is the total discounted return accumulated by the RL neural network from time step t until the end of the episode (which has total length N), with discount factor γ ∈ (0,1]. The value of state s under policy π is defined as V π< (s) = E π [R t |s t = s] and represents the expected return for following policy π from state s. The expected return from each state s t can be maximized by the RL neural network. A REINFORCE likelihood-ratio trick may be used to update the parameters θin the direction ∇ θ log log π(s t ; θ)R t , which is an unbiased estimate of ∇ θ E π [R t ].

[0035] In some implementations the training of the RL neural network may be unstable. This may be negated by learning a function of the state. This function of the state may be known as a baseline b t (s t ) which is subtracted from the return to reduce bias. Using a learnt estimate of the baseline (also known as a value function) the estimate of the gradient remains unbiased even after baseline subtraction. The new gradients are given by ∇ θ log log π(a t |s t ; θ)(R t - V π< (s t ) ) where At = R t - V π< (s t ) may be considered as the advantage in terms of taking action a t as compared to the average return.

[0036] Training 304 the neural network may include exploration of the state space. An entropy bonus can be added to the actions of the RL neural network during training. The entropy bonus may be given by: G = − ∑ i p a i log log p a i . An entropy bonus may be used to increase the spread of action distribution, preventing the RL neural network from being too sure of taking any one action and helping exploration during learning. In some implementations the loss objective function may comprise: a first component that is a cross entropy loss of a ground-truth machine learning task output and a predicted machine learning task output and that measures how well the task neural network is predicting the ground-truth machine learning task output, a second component that is the expected sum of rewards collected by the RL neural network 102, and a third component that is an action exploration entropy. For example the loss objective function may be given by: L = H y y ^ − ∑ t log π s _ t ; θ A _ t + R t − V π s t 2 2 − βG t Where β is a fixed weight applied to the per-step action entropy cost term and H(y, ŷ) denotes the cross-entropy loss for the supervised classification.

[0037] At every training iteration a batch of sequences from the video dataset may be sampled. The sampled batches may be used to unroll the RL neural network 102 and collect trajectories. The action logits, action samples and value predictions given by the RL neural network together with the rewards at each step (including class predictions) may be used to compute the empirical returns R t as discounted cumulative sums of the rewards, for example with γ = 0.99 and advantages. The gradients can further be computed and the policy updated.

[0038] In some implementations when the system trains 304 the RL neural network the number of video frames that can be consumed by the task neural network in any given sequence may be fixed in advance. The system selects which frames to consume within a given budget. Assuming a video sequence of length N, and a budget of k, there are less than nchoosek(N, k) possibilities. This may be described as a fixed, or limited, budget system. A fixed or limited budget constraint allows a user of the neural network system to strictly control resources. The number of encoded frames may be counted and the trajectories clipped when the budget is reached. The RL neural network may be constrained to reduce k below a perception cost. In other implementations clipping may be used without an encoding cost, so the RL neural network is allowed to consume k frames out of N without incurring a cost.

[0039] In other implementations the system may be allowed to consume any number of frames to achieve the task while paying a penalty for each consumed frame and aiming to achieve a reduction in the overall cost. The penalty is part of the reward structure used to train the RL network with REINFORCE. These per-step penalties are used to compute the total returns that get multiplied with the gradients of the policy. The returns become part of the loss policy. Assuming a video sequence of N frames there are 2 N< possibilities in an example that has an unlimited budget.

[0040] In other implementations, the RL neural network 102 may be configured to predict how many more task inputs the task neural network needs to encode until an accuracy of a machine learning task output generated by the task neural network satisfies a threshold. In one example a pre-trained per-frame action classifier may be used to train the RL neural network. The classifier may output a score for each frame in the sequence. For example, for a given budget of k frames the frames with the top k confidence scores for the correct class among the classifier's predictions can be selected. The indices of these frames may be used to generate a surrogate "ground truth" for the policy neural network - which can initially be trained in a supervised way to emit an encoding action (i.e. to indicate that an test input should be encoded) at the time-steps identified by the classifier as being most informative. The label produced by the classifier at time t is used by the policy neural network as ground truth for the policy at t - 1 since the action is output to be applied to the next frame. Teacher forcing may be used to ensure the actions of the RL neural network at the beginning of training do not hinder the representation learning i.e. the actions sampled from the RL neural network can be overwritten with the ground truth given by the per-frame classifier. For example, let c t denote the score of the correct class given by a per-frame classifier at time t and A k = argtop k (c t ), the set of indices of the top k scores. Then, the imitation label may be given by l t = 1 {t∈Ak} and the training seeks to minimize L = H y y ^ = 1 N ∑ t = 1 N H l t a t − 1 . After pre-training, supervisor guidance can be removed from the pre-frame classifier and the policy may be fined-tuned using the RL loss.

[0041] FIG. 4 is a flow diagram of an example process 400 for generating a machine learning task output. The system receives 402 a sequence of task inputs to perform a machine learning task. In an unclaimed embodiment, the sequence of task inputs can be a sequence of video frames and the machine learning task can be a different video processing task, e.g. to predict the topic of the sequence of video frames or to recognize an object depicted in the sequence of video frames.

[0042] As another unclaimed example, the machine learning task is a natural language processing task where the sequence of task inputs is a sequence of text (or other data representing a sample of natural language, such as a speech waveform captured by a microphone) and the task output is a prediction of a topic of the sequence of text a sentiment expressed in the sequence of text, or another output for another natural language processing task.

[0043] RL neural network 102 is used to generate 404 for each task input in the sequence of task inputs a respective decision that determines whether to encode or skip the task input. The RL neural network makes the respective decision according a policy which determines whether to encode the task input or skip the task input. For example the policy may determine a reward for the system should the encoder decide not to encode the task.

[0044] The RL neural network provides 406 the respective decision of each task input to the task neural network, that is, for each frame whether the task neural network should encode or skip the task.

[0045] The task neural network receives 408 the sequence of task inputs and a respective decision from the RL neural network. The decision determines whether to encode or skip the task input.

[0046] The task neural network processes 410 each of the un-skipped task inputs in the sequence of task inputs to generate a respective accumulated feature for the un-skipped task input, wherein the respective accumulated feature characterizes features of the un-skipped task input and of previous un-skipped task inputs in the sequence.

[0047] The task neural network generates 412 a machine learning task output for the machine learning task based on the last accumulated feature generate for the last un-skipped task input in the sequence.

[0048] Alternatively the task neural network may generate a temporary machine learning output based on the respective accumulated feature and generate 412 a machine learning task output for the machine learning task by averaging the temporary machine learning outputs generated for the un-skipped task inputs in the sequence.

[0049] This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0050] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0051] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0052] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0053] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0054] Similarly, in this specification the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0055] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0056] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0057] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0058] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0059] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0060] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

[0061] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0062] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0063] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

[0064] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0065] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented system (100) configured to receive a sequence of task inputs (104) and to perform a machine learning task, the system comprising a reinforcement learning (RL) neural network (102) and a task neural network (106), wherein the RL neural network (102) is configured to: generate, for each task input of the sequence of task inputs, a respective decision that determines whether to encode the task input or to skip the task input (404), and provide the respective decision of each task input to the task neural network (406); and wherein the task neural network (106) is configured to: receive the sequence of task inputs (408), receive, from the RL neural network, for each task input of the sequence of task inputs, a respective decision that determines whether to encode the task input or to skip the task input (408), process each of the un-skipped task inputs in the sequence of task inputs to generate a respective accumulated feature for the un-skipped task input, wherein the respective accumulated feature characterizes features of the un-skipped task input and of previous un-skipped task inputs in the sequence (410), and either: (i) generate a machine learning task output for the machine learning task based on the last accumulated feature generated for the last un-skipped task input in the sequence (412), or (ii) for each of the un-skipped task inputs, generate a temporary machine learning output based on the respective accumulated feature; and generate a machine learning task output for the machine learning task by averaging the temporary machine learning outputs generated for the un-skipped task inputs in the sequence; wherein the sequence of task inputs (104) is: (i) a sequence of video frames, and the machine learning task is to recognize one or more actions performed in the sequence of video frames, or (ii) a sequence of voice inputs, and the machine learning task is to predict the topic of the sequence of voice inputs, and wherein the RL neural network and the task neural network are jointly trained to minimize a loss objective function using a stochastic gradient descent method, wherein gradients from the RL neural network are not backpropagated into the task neural network, the RL neural network being configured to receive a reward for making a decision for a task input, the reward for a skip action which determines to skip the task input being greater than the reward for an encode action which determines to encode the task input, and the reward being also based on the quality of a task output relative to a ground truth task output.

2. The computer-implemented system (100) of claim 1, wherein the task neural network (106) is configured to generate a machine learning task output for the machine learning task based on the last accumulated feature generated for the last un-skipped task input in the sequence, and the task neural network (106) comprises an encoder (202), an accumulated feature neural network (204), and a final neural network (206), wherein, for each of the un-skipped task inputs, the encoder (202) is configured to: process the current un-skipped task input to generate an encoded feature representation for the current un-skipped task input, and provide the encoded feature representation to the accumulated feature neural network (204); wherein, for each of the un-skipped task inputs, the accumulated feature neural network (204) is configured to: process the encoded feature representation of the current un-skipped task input and a previous accumulated feature associated with the previous un-skipped task input to generate a respective accumulated feature for the current un-skipped task input, generate a feature output from the respective accumulated feature of the current un-skipped task input, and provide the feature output to the RL neural network (102), the feature output being used by the RL neural network (102) to generate a next decision for the next un-skipped task input in the sequence; and wherein the final neural network (206) is configured to process the last accumulated feature to generate the machine learning task output for the machine learning task.

3. The computer-implemented system (100) of claim 1, wherein the task neural network (106) is configured, for each of the un-skipped task inputs, to generate a temporary machine learning output based on the respective accumulated feature; and generate a machine learning task output for the machine learning task by averaging the temporary machine learning outputs generated for the un-skipped task inputs in the sequence, and the task neural network (106) comprises an encoder (202), an accumulated feature neural network (204), and a final neural network (206), wherein, for each of the un-skipped task inputs, the encoder is configured to: process the current un-skipped task input to generate an encoded feature representation for the current un-skipped task input, and provide the encoded feature representation to the accumulated feature neural network (204); wherein, for each of the un-skipped task inputs, the accumulated feature neural network (204) is configured to: process the encoded feature representation of the current un-skipped task input and a previous accumulated feature associated with the previous un-skipped task input to generate a respective accumulated feature for the current un-skipped task input, generate a feature output from the respective accumulated feature of the current un-skipped task input, and provide the feature output to the RL neural network (102), the feature output being used by the RL neural network (102) to generate a next decision for the next un-skipped task input in the sequence; and wherein the final neural network (206) is configured to compute the machine learning task output based on the temporary machine learning outputs.

4. The computer-implemented system of claim 3, wherein, for each of the un-skipped task inputs, the accumulated feature neural network (204) is configured to generate the feature output from the respective accumulated feature of the current un-skipped task input by: concatenating the encoded feature representation of the current un-skipped task input and the respective accumulated feature through a skip connection to generate an adjusted accumulated feature, and generating the feature output from the adjusted accumulated feature.

5. The computer-implemented system of any one of claims 2 to 4, wherein the encoder is a convolutional neural network, if the sequence of task inputs is a sequence of video frames.

6. The computer-implemented system of any one of claims 2-5, wherein the feature output includes a time embedding that identifies the current time step.

7. The computer-implemented system of any one of claims 2-6, wherein the final neural network is a multilayer perceptron (MLP) neural network.

8. The computer-implemented system of claim 2 or claim 3, wherein: for each of the un-skipped task inputs in the sequence, the RL neural network is (102) configured to receive, from the task neural network (106), the feature output generated by the accumulated feature neural network (204), and to generate a next decision for the next task input in the sequence, and for each of the skipped task inputs in the sequence, the RL neural network (102) is configured to receive, from the task neural network (106), the feature output generated by the accumulated feature neural network (204) for the most recent un-skipped task input, and to generate a next decision for the next task input after the skipped task input in the sequence.

9. The computer-implemented system of claim 8, wherein the RL neural network (102) maintains an action space comprising an encode action and a skip action, and wherein the RL neural network (102) is configured to, for each task input in the sequence: process, using a policy neural network of the RL neural network (102), the received feature output to generate a corresponding distribution of actions in the action space, and generate the next decision for the next task input in the sequence by sampling an action from the action space according to the corresponding distribution of actions.

10. The computer-implemented system (100) of claim 1, wherein the stochastic gradient descent method is REINFORCE.

11. The computer-implemented system (100) of claim 10, wherein the loss objective function comprises: (i) a first component that is a cross entropy loss of a ground-truth machine learning task output and a predicted machine learning task output and that measures how well the task neural network is predicting the ground-truth machine learning task output, (ii) a second component that is the expected sum of rewards collected by the RL neural network (102), and (iii) a third component that is an action exploration entropy.

12. The computer-implemented system of any one of any preceding claim, wherein the RL neural network (102) is trained by a pre-trained per-frame action classifier to predict how many more task inputs the task neural network (106) needs to encode until an accuracy of a machine learning task output generated by the task neural network (106) satisfies a threshold.

13. One or more computer readable medium storing instructions that, when executed by one or more computers, cause the one or more computers to implement the system according to any one of claims 1-12.