Deriving and utilizing fine-grained feedback from generative model streams

Abstract

Systems and methods for deriving and using fine-grained feedback to train generative neural networks. One of the methods includes obtaining a data item generated by a generative neural network in response to a user query, the data item comprising a sequence of samples; providing the generated data item for presentation to a user on a user device, wherein the respective samples are presented in an order according to the sequence; during the presentation of the generated data item to the user on the user device: obtaining an implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the plurality of samples; and using the implicit feedback signal to update the generative neural network.

Description

DERIVING AND UTILIZING FINE-GRAINED FEEDBACK FROM GENERATIVEMODEL STREAMSBACKGROUND

[0001] This specification relates to processing data using machine learning models.

[0002] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0003] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.SUMMARY

[0004] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that obtains implicit feedback signals from one or more users and uses the implicit feedback signals to update a generative neural network to align with the preferences of the one or more users.

[0005] That is, the system receives a user implicit feedback signal regarding a data item that includes a sequence of samples generated by a generative neural network and presented to the user on a user device, e.g., in response to a query from the user. Then the system transforms the feedback signal into a training signal to update the generative neural network.

[0006] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0007] Reinforcement learning from human feedback is an effective method to update a generative neural network to align with the preferences of one or more users. Tn principle, the more types of preference feedback and greater quantities of feedback that are available from the one or more users, the more aligned with the preferences of the one or more users a generative neural network that is updated using the feedback will become.

[0008] For example, a pairwise ranking of two generated data items (sequences of samples) by a user for the same user query can be used to update a generative neural network to align with the user’s preferences.

[0009] As another example, a user approving a data item as satisfactory in response to the user’s query represents a positive feedback signal that can be used to update the generative neural network to align with the user’s preferences.

[0010] In practice, however, pairwise ranking of data items and explicit approval feedback per data item by users may be insufficient to robustly update a generative neural network to align with all the preferences of users.

[0011] One reason for the insufficiency is that collecting enough data to update the generative neural network through explicit feedback is difficult. Data reflecting pairwise rankings or explicit ratings of data items may be difficult to collect because such data requires users to submit additional, explicit feedback after being presented with a generated data item.

[0012] Another reason for the insufficiency is that the feedback received from a user in this manner is coarse-grained, only providing information regarding the entirety of the data item and not information regarding the samples in the sequence that constitute the data item. With such coarse feedback, it is difficult for insight about preferences regarding particular samples included in a long data item to be gleaned and incorporated into an update of the generative neural network.

[0013] This specification, on the other hand, describes using rich, dense implicit feedback that one or more users passively provide to create training signals to update a generative neural network to align to the preferences of the one or more users.

[0014] These feedback signals are generated from the interactions of a user with a presented sequence of samples (data item) on a user device, without requiring the user to change the user’s behavior or the user to submit any additional inputs than if feedback signals were not generated. Also, the different types of user interactions can create different types of implicit feedback signals per sample, providing rich, dense feedback information to create informative training signals to update the generative neural network.

[0015] For example, implicit feedback signals can be used to select training data (query7and data item pairs) to update the generative neural network.

[0016] As another example, the implicit feedback signals can be used to train a surrogate function, e.g., a reward model, which can then subsequently be used to update the generative neural network through reinforcement learning techniques.

[0017] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0018] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1 shows an example generative model updating system.

[0020] FIG. 2 shows an example user device that presents a data item's sequence of samples to a user and generates an implicit feedback signal from the user’s interaction with the sequence of samples.

[0021] FIG. 3 is a flow diagram of an example process to update the generative neural network using the implicit feedback signal.

[0022] FIG. 4 shows an example training engine that trains a reward model.

[0023] FIG. 5 is a flow diagram of an example process to update the generative neural network using the reward model.

[0024] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0025] FIG. 1 shows an example generative model updating system 100. The generative model updating system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0026] The generative model updating system 100 is a system that receives a user implicit feedback signal 110 regarding a data item 106 that includes a sequence of samples generated by a generative neural network 104 and presented on a user device 108 in response to a user's query 102 and transforms the feedback signal into a training signal to update the generative neural network 104.

[0027] That is, a training engine 112 within the system 100 updates the generative neural network 104 to better align with the preferences of one or more users based on user queries 102, data items 106, and implicit feedback signals 110.

[0028] A data item 106 includes a sequence of samples that represents ordered, contextually relevant data with respect to the user query 102.

[0029] For example, a data item 106 can be text, e.g., a sentence, paragraph, essay, etc., and the samples can be text tokens, e.g., characters, words, sentences, and so on. The order of the text tokens within the data item 106 is fundamental to the coherent meaning of the data item106. To illustrate the order’s importance, first consider a corresponding sentence data item 106 "After a long chase, the cat caught the mouse.” for which the tokens are the words and punctuation. Now, consider an alternative token sequence for the same tokens ‘'After a long chase, the mouse caught the cat.” Both example data items 106 have the same tokens but different semantic meanings, illustrating the importance of sequence order.

[0030] As another example, a data item 106 can be a video and the samples can be video frames or groups of video frames. The order of the samples, e.g., video frames or groups of video frames, within the data item 106 is fundamental to the meaning of the data item 106. Consider a video clip of someone opening a book. Reversing the sequence of frames represents a video clip of someone closing a book, an opposite semantic meaning from the same set of video frames. Again, the order of the sequence is important.

[0031] As another example, a data item 106 can be an image and the samples can be pixels or patches of pixels. The order of the samples in the sequence can ensure adjacent dependency of the pixels to construct a coherent image and is therefore important for incremental construction of the image.

[0032] As another example of a data item 106 that is an image, the sequence of samples can each be images, beginning with a noisy image, i.e., an image with distortions, followed by progressively less noisy images, and ending with a clean image. The order of such a sequence can be important for preserving the integrity of the image throughout a sequence of incremental denoising.

[0033] As another example, a data item 106 can be an audio segment, the sequences of samples can be amplitude values for sequential time points of an audio waveform representation of the audio segment. If the order of the samples were lost, the characteristics, e.g., pitch and loudness, of the audio segment would be lost.

[0034] As another example of a data item 106 that is an audio segment, the sequences of samples can each be an audio waveform, beginning with a noisy audio waveform, i.e., an audio waveform with distortions, followed by progressively less noisy waveforms, and ending with a clean waveform. The order of such a sequence can be important for sequential incremental denoising of the audio waveform.

[0035] As another example of a data item 106 that is an audio segment, the sequences of samples can be audio tokens, i.e., compact representations of discrete units of the audio segment, e.g., encodings produced by codecs, embeddings produced by neural networks, mappings of the audio segment to high level features (such as component frequencies, musical notes, etc.), and so on.

[0036] The generative neural network 104 can be any neural network configured to process a user query 102 of a first modality and generate a data item 106. i.e., a sequence of samples, of a second modality. The first modality type of the user query 102 and second modality type of the data item’s 106 sequence of samples can be the same modality type or different modality types.

[0037] In particular, the generative neural network 104 may employ various architectures or techniques, such as transformers, attention-based mechanisms, variational auto-encoders, recurrent neural networks, convolutional neural networks, or generative adversarial networks, which are all effective for capturing complex dependencies between samples within data items 106.

[0038] For example, the generative neural network 104 can be a text-to-text generative neural network, i.e., the generative neural network 104 can be configured to process a user query 102 in text format and generate a data item 106 including a sequence of samples in text format. For a sequence of text samples, the samples may be text tokens, e.g., characters, words, sentences, and so on. Examples of such text-to-text generative neural networks are described in ArXiv: 2204.02311 and ArXiv: 1910.10683.

[0039] As another example, the generative neural network 104 can be a text-to-video generative neural network, i.e., the generative neural network 104 can be configured to process a user query 102 in text format and generate a data item 106 including a sequence of samples in video format. For a sequence of video samples, the samples may be video frames, or groups of video frames. Examples of such text-to-video generative neural networks are described in ArXiv: 2210.02303 and ArXiv: 2210.02399.

[0040] As another example, the generative neural network 104 can be a text-to-audio generative neural network, i.e., the generative neural network 104 can be configured to process a user query 102 in text format and generate a data item 106 including a sequence of samples in audio format. For a sequence of audio samples, the samples may be audio tokens, e.g., codec encodings, neural network embeddings, and so on. Examples of such text-to-audio generative neural networks are described in ArXiv: 2209.03143 and ArXiv: 2306.12925.

[0041] In certain situations, the system 100 can convert the user query 102 or data item 106 from an original modality to a new modality'.

[0042] For instance, a smartphone can perform speech-to-text conversion in a straightforward process where spoken words are transcribed into text tokens. For such an instance, a speech- to-text conversion may be used so that speech representing a user query 102 can be processed as text by a text-to-text generative neural network 104. Continuing with this example, thedata item 106 including a sequence of text samples generated by the text-to-text generative neural network 104 can be converted back to speech to be presented to the user.

[0043] To update the generative neural network 104, the system obtains a data item 106, generated by a generative neural network 104 in response to a user query 102.

[0044] Then, the system 100 presents the generated data item 106 including a sequence of samples to a user on a user device 108, wherein the respective samples are presented in an order according to the sequence. That is, samples earlier in the order according to the sequence are shown to the user before those that appear later in the order.

[0045] For example, if the data item 106 is a paragraph, such that the samples are words , then the generative model 104 producing the paragraph will present words one at a time in the precise order of the paragraph for presentation to a user.

[0046] As another example, if the data item 106 is a video scene, such that the samples are video frames, then the generative model 104 producing the video scene will present video frames one after the other in the precise order of the video scene for presentation to a user.

[0047] Then, during the presentation of the generated data item 106 to the user on the user device 108 the system 100 obtains an implicit feedback signal 110 of a particular type that indicates user feedback associated with a particular sample of the sequence of samples.

[0048] Using the implicit feedback signal 110, the system 100 updates the generative neural network 104.|00049| Generally, the user device 108 can be any device that can receive and present the data item’s 106 sequence of samples to the user and allows the user to interact with the sequence of samples to generate implicit feedback signals 110. Examples of a user device 108 include a smartphone, tablet, laptop, desktop, smartwatch, smart tv, and so on.

[0050] Generally, the data item’s 106 sequence of samples can be streamed from the generative neural network 104 to be presented to the user, or the sequence of samples can be generated fully to then be presented to the user in the form of a stream. That is, presenting a sequence of samples or groups of samples to a user on a user device 108 involves presenting samples or groups of samples incrementally rather than all at once. Each sample or group of samples are introduced sequentially, allowing the user to focus on one sample or group of samples at a time. In the case that samples are live streamed, i.e., presented to the user in realtime as they are generated, they can be presented to the user sooner than first waiting for the full generation of the stream before presenting samples.

[0051] For example, a sequence of text samples, e.g., words or sentences, can be presented on a user device display screen to a user that can then interact with the sequence of samplesthrough user device interface controls. Such device interface controls can allow the user to interactively navigate the text samples to control the pace of presentation of the samples, represent samples, skip presenting samples, and end presentation of samples.

[0052] Alternatively to having the user manually navigate the text samples, the sequence of text samples can be presented to a user in a live stream format. That is, the text samples can be displayed in real-time at a certain pace on the user device display screen to a user. The user can then interact with the sequence of text samples through playback controls such as changing the pace of presentation of samples, stopping presentation of samples, rewinding the presentation of samples, and skipping forward through the stream presentation of text samples.

[0053] As another alternative to presenting the sequence of text samples, the text samples can be presented as audio through a user device text-to-speech conversion of text tokens to speech tokens, i.e., units of sound that can be combined to form speech of words. A user device audio playback system can present the speech tokens in a live stream format and enforce the sequential order of tokens to prevent the user from presenting samples in an order other than intended. In other implementations, the user device audio playback system can provide playback controls such as stopping, skipping, rewinding, etc., to the user.

[0054] In addition to speech tokens, a sequence of audio samples, e.g., audio tokens, can be presented to a user on a user device audio playback sy stem in a live stream format. Similarly to the presentation of speech tokens, the user device audio playback system can present audio tokens and enforce the sequential order of tokens. In some implementations, the user device audio playback system can also provide playback controls for the user to control the presentation of audio tokens.

[0055] As another example, a sequence of video samples, e.g., video frames or groups of video frames, can be presented on a user device display screen to a user in a live stream format and the user can control the presentation of the samples through user device playback controls, e.g., pause, stop, play, fast-forward, rewind, etc.

[0056] The implicit feedback signal 110 can be any implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the data item's 106 sequence of samples.

[0057] One example of the particular sample can be a sample before the last sample in the sequence, and the implicit feedback signal 110 can indicate that the user terminated the presentation after presentation of the particular sample and before the presentation of samples after the particular sample, i.e., an early termination signal. In addition to the particularsample, the early termination implicit feedback signal 110 can be associated with a portion of the samples in the data item 106 that includes the particular sample and any samples preceding the particular sample in the sequence and does not include any samples following the particular sample in the sequence.

[0058] For example, the early termination signal may correspond to a user viewing a sequence of text samples on a user device text playback system and terminating the presentation before all samples belonging to the data item have been presented. The particular sample before the last sample can be the most recently focused word or sentence viewed before ending the presentation and all samples before the particular sample can be associated with early termination signal.

[0059] As another example, the early termination signal may correspond to a user viewing a sequence of video samples on a user device video playback system and terminating the presentation before all samples have been played back. The particular sample before the last sample can be the last displayed sample, e.g., video frame or group of video frames, before ending the presentation and all samples before the particular sample can also be associated with the early termination signal.

[0060] As another example, the early termination signal may correspond to a user listening to a sequence of audio samples on a user device audio playback system and terminating the presentation before all samples have been played back. The particular sample before the last sample can be the last played back sample before ending the playback and all samples before the particular sample can be associated with the early termination signal.

[0061] Another example of the particular sample can be a sample before a last sample in the sequence, and the implicit feedback signal 110 can indicate that the user replayed a segment of the data item 106 corresponding to the particular sample, i.e., a replay signal. That is, previously presented samples of the sequence are re-presented to the user.

[0062] For example, the replay signal may correspond to a user re-viewing a previously presented text sample from the latest presented text sample position using user device text playback controls. The particular sample before the last sample can be the focused word or sentence re-presented.

[0063] As another example, the replay signal may correspond to a user viewing a sequence of video samples on a user device video playback system and rewinding the video playback to a previously presented sample. The particular sample before the last sample can be the video frame or group of video frames re-presented after rewinding the video.

[0064] As another example, the replay signal may correspond to a user listening to a sequence of audio samples on a user device audio playback system and rewinding the audio playback to a previously presented sample. The particular sample before the last sample can be the audio sample re-presented after rewinding the audio playback.

[0065] Another example of the particular sample can be a sample before a last sample in the sequence, and the implicit feedback signal 110 can indicate that the user skipped a segment of the data item corresponding to the particular sample, i.e., a skipped signal.

[0066] For example, the skipped signal may correspond to a user using a user device text playback system control during presentation of text to skip the presentation of intermediate text samples that would have been presented incrementally otherwise. The particular sample can be the last sample presented before the user used the text playback system control.

[0067] As another example, the skipped signal may correspond to a user using a '‘seek to” command on a user device video playback system to continue video playback at a specific time point of the video, i.e., continue presenting to the user on a user device 108 a future video sample in the sequence without presenting intermediate samples relative to the current sample of the sequence. The particular sample can be the last sample presented before “skipping” using the “seek to” command.

[0068] As another example, the skipped signal may correspond to a user using a “seek to” command on a user device audio playback system to continue audio playback at a specific time point of the audio segment. The particular sample can be the last played back sample presented before “skipping” using the “seek to” command.

[0069] Another example of the particular sample can be the last sample in the data item’s 106 sequence of samples, and the implicit feedback signal 110 can indicate that the user completed the presentation of the data item, i.e.. a completion signal.

[0070] For example, the completion signal may correspond to a user viewing all text samples from the sequence of text samples. The particular sample is the last sample and can correspond to the last word or sentence in the sequence.

[0071] As another example, the completion signal may correspond to a user viewing a sequence of video samples on a user device video playback system to completion. The particular sample is the last sample and can correspond to the last video frame or group of video frames in the sequence.

[0072] As another example, the completion signal may correspond to a user listening to a sequence of audio samples on a user device audio playback system to completion. Theparticular sample is the last sample and can correspond to the last audio sample in the sequence.

[0073] Further details of an example user device 108 presenting a data item’s 106 sequence of samples to a user and generating an implicit feedback signal 110 are described below with reference to FIG. 2.

[0074] After obtaining the feedback signal 110, the system uses a training engine 112 and at least the feedback signal 110 to update the generative neural network 104.

[0075] In some implementations, the training engine 112 determines whether to include a training example that includes the user query 102 and the generated data item 106 in training data for the generative neural network 104 based on the implicit feedback signal 110 and updates the generative neural network 104 on the training data through supervised learning.

[0076] For example, a positive implicit feedback signal 110, such as a completion signal, considered such due to the implication that a user finishes a presentation of the sequence of samples if the user enjoys the presentation, can be used to select a user query 102 and data item 106 pair to update the generative neural network 104.

[0077] In other implementations, the training engine 112 not only determines whether to include a training example that includes the user query 102 and the data item 106 in training data for the generative neural network 104 based on the implicit feedback signal 110, but also determines a weight to be assigned to the training example when training the generative neural network 104 on the training data based on the implicit feedback signal 110.

[0078] For example, the replay implicit feedback signal 1 10 can be used to weight training data in proportion to a metric that measures the amount of replay. That is, the weighting of a training data item 106 can be proportional to the number of replay signals associated with the data item 106, or, particular samples of the data item 106 can be weighted in proportion to if they are associated with a replay signal.

[0079] Further details of updating the generative neural network 104 using the implicit feedback signal 110 are described below with reference to FIG. 3.

[0080] In some implementations, using the implicit feedback signal 110 to update the generative neural network 104 includes training, using the implicit feedback signal 1 10, at least the particular sample, and the user query 102, a reward model that predicts, given at least an input portion of an input data item 106 and a corresponding user query 102, a likelihood that a user would submit an implicit feedback of the particular type in association with the input portion of the input data item 106 when presented with the input data item 106.

[0081] In certain instances, the reward model can be a neural network with an architecture that is similar to the generative neural network’s 104 architecture. Both the reward model and the generative neural network can process a portion of an input data item 106 and the user query 102, however, while the generative neural network features a neural network head to predict probabilities associated with a finite tokens of a vocabulary, e.g., text tokens, pixels, audio tokens, etc., the reward model instead features a regression head, i.e., the final layer of the reward model neural network outputs continuous values to generate the likelihood of an implicit feedback signal associated with the portion of the input data item.

[0082] In other instances, the reward model can be a neural network with an architecture that is dissimilar to the generative neural network's 104 architecture. In such cases, the reward model neural network can have any appropriate architecture that processes at least an input portion of an input data item 106 and a corresponding user query 102 to generate a likelihood that a user would submit an implicit feedback of the particular type in association with the input portion of the input data item 106 when presented with the input data item 106.

[0083] Further details of training the reward model using a training engine 112 are described below with reference to FIG. 4.

[0084] After training the reward model, the training engine 112 can update the generative neural network 104 using the reward model.

[0085] For example, the training engine 112 can use reinforcement learning methods, such as policy gradient methods, i.e., methods that directly optimize the generative neural network 1 4 using the gradient of scores determined through a reward model. Examples of such policy gradient methods include Proximal Policy Optimization described inArXiv: 1707.06347, and Trust Region Policy Optimization described in ArXiv: 1502.05477.

[0086] Further details of updating the generative neural network 104 using the reward model are described below with reference to FIG. 5.

[0087] The training engine 112 can also use direct policy optimization to update the generative neural network 104. By sampling multiple possible data items of the generative neural network 104 for a given new user query’, the training engine 112 can use the reward model to rank the possible data items and then perform direct policy optimization to update the generative neural network 104 to more likely produce data items 106 that would yield high reward scores.

[0088] In another implementation of the training engine 112 using direct policy optimization, instead of using the reward model to score multiple possible data items, the multiple possible data items can be presented to a user to receive implicit feedback signals directly. Thetraining engine 112 can then use the implicit feedback signals to rank the possible data items and then perform direct policy optimization to update the generative neural network 104, avoiding the use of a reward model.

[0089] For example, if two data items corresponding to video segments are presented to a user, and the first data item is presented to completion while the second data item is only partially presented, i.e., the training engine 112 receives a completion signal for the first data item and an early termination signal for the second data item, then the training engine 112 can rank the data items and update the generative neural network 104 using direct policy optimization.

[0090] FIG. 2 shows an example user device 200 that presents a data item’s 106 sequence of samples to a user and generates an implicit feedback signal 110 of a particular type from the user’s interaction with the sequence of samples.

[0091] The example user device 200 is a laptop computer that processes a sequence of samples in video format with each sample corresponding to a video frame, presents the sequence of video frames 202 to a user 206, receives user feedback associated with a particular sample through the user interaction with a user device interface 204. and generates the corresponding implicit feedback signal 110.

[0092] The user device interface 204 includes video playback symbols, e.g., rewind, stop, play, fast forward, and skip forward, that determine how the video samples are presented. During presentation of the sequence of video frames 202. when the user selects the stop symbol during video presentation, as indicated by a dashed line connecting the user to the stop symbol, it signifies the user’s choice to end the video presentation, i.e., an early termination feedback. The particular video sample associated with this feedback is indicated via the dashed line connecting the stop symbol to a particular video frame.

[0093] A user feedback, e.g., the early termination feedback, does not necessarily have to originate directly from the user device interface 204. In general, any interaction of the user with the user device 200 during presentation of samples can be utilized. For example, the user closing a video playback application, or turning off the user device 200, are also interactions that would indicate early termination feedback from the user.

[0094] The implicit feedback signal 110 resulting from the user interaction with the user device 200 encapsulates at least the type of feedback signal, and the particular sample, i.e., the video frame the user stopped video presentation on.

[0095] FIG. 3 is a flow diagram of an example process 300 to update the generative neural network using the implicit feedback signal. For convenience, the process 300 will bedescribed as being performed by a system of one or more computers located in one or more locations. For example, a generative model updating system, e.g.. the generative model updating system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0096] To begin example process 300, the system obtains a data item generated by a generative neural network in response to a user query, the data item including a sequence of samples (step 302). As described above, the generative neural network can process a query and generate a sequence of samples of any pair of modality types.

[0097] Next, the system provides the generated data item for presentation to a user on a user device, wherein the respective samples are presented in an order according to the sequence (step 304). The user device can be any device that can appropriately present the sequence of samples, and the device may also present the sequence of samples using a modality type that can represented by the modality type of the sequence of samples but is not the same modality type, e.g., a text-to-speech presentation of a sequence of text samples as described above.

[0098] During the presentation of the generated data item to the user on the user device, the system obtains an implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the plurality of samples (step 306). The user device presenting the sequence of samples receives the user feedback associated with a particular sample and generates the implicit feedback signal that includes at least the feedback signal type and the particular sample. In some implementations, multiple feedback signals for a given past period of time are collected before sending any feedback signal to the system. For example, the system can process a usage report produced by the user device regarding details of one or more user inputs during multiple past presentations for the system to generate feedback signals.

[0099] The particular sample and feedback signal type can be determined by the user’s interaction with the samples through a user device interface, e.g., as described with reference to FIG. 2.[000100] In addition to implicit feedback signals related to sample presentation control, the system can also receive implicit feedback signals from user expressions, such as emoji reactions. That is, a user may express feelings such as joy, surprise, confusion, and so on using a user device interface in response to the presentation of samples and these expressions can define implicit feedback signals.[000101] For example, a user watching a comedy video being presented as a sequence of groups of video frames may select a "laughing face” emoji in response to an enjoyablegroup of samples, which can be used to define a “funny” implicit feedback signal ty pe. The “funny” signal can then be associated with the group of video frames coinciding with the user’s laughing face emoji selection to define the particular sample.[000102] As another example, a user reading a story7presented as a sequence of text samples may select an “astonished face” emoji as a response to reading captivating text samples, which can be used to define a “captivating” implicit feedback signal type. The “captivating” signal can then be associated with the text samples coinciding with the user’s astonished face emoji selection to define the particular sample.[000103] In some implementations, the system can process past sensor data during past presentations to process indirect interactions of the one or more users with samples into feedback signals.[000104] The system uses the implicit feedback signal to update the generative neural network (step 308).[000105] The system can update the generative neural network using the implicit feedback signal using any of a variety of techniques.[000106] As one example, the system selects query -data-item pairs that the implicit feedback signal indicates the user has preference for, i.e., positive training examples, to include in a training dataset to update the generative neural network through supervised learning.|000107| As another example, the system first trains a reward model using multiple training examples, each of which includes a corresponding user query7, one or more particular samples, and a corresponding feedback signal, through supervised learning, then uses the reward model to update the generative neural network through reinforcement learning. Training a reward model and updating the generative neural network using a reward model will be described in more detail below with reference to FIG. 4 and FIG. 5 respectively.[000108] FIG. 4 shows an example training engine 400 that trains a reward model.[000109] The reward model 404 can be any machine learning model that, after training, can process a new user query7406 and a portion of new sequence of samples 408. i.e., any contiguous subset of the sequence, to generate a likelihood 410, i.e., a likelihood of an implicit feedback signal of a particular type associated with the portion, e.g., a likelihood of an early termination signal at a particular video frame during presentation of a sequence of video frames.[000110] For example, the reward model 404 can be a neural network that includes any of fully connected layers, convolutional layers, recurrent layers, attention-based layers, andso on, as is appropriate for the modality types included in the data item’s 106 sequence of samples.[000111] A reward model training engine 402 trains the reward model 404 using multiple different examples of user queries 102, data items’ 106 sequences of samples, and feedback signals 110 with the same particular type of feedback signal. That is, the reward model training engine 402 receives training examples generated by the system, where each example includes a respective implicit feedback signal 110. at least a respective particular sample of the sequence of samples, and a respective query 102. Then the rew ard model training engine 402 uses supervised learning to train the rew ard model 404 with the training examples.[000112] For example, the reward model training engine 402 uses the reward model’s 404 estimates of likelihoods of a specific feedback signal type for particular samples for a training example, compares these predictions to actual feedback signals for the training example, and then optimizes the reward model’s 404 parameters to minimize prediction errors using gradient-based methods. Updates to the reward model’s 404 parameters can occur after evaluating many training examples or only a single training example.[000113] The particular training method the rew ard model training engine 402 employs to minimize the error of predictions can be any of various algorithms and optimization techniques.|000114| For example, the parameters may be updated via a gradient descent method, e.g., stochastic gradient descent, that minimizes an objective function, e.g., the cross-entropy loss function and a L-2 norm regularization term.[000115] Once trained, the reward model 404 can be used to update the generative neural network 104. After updating the generative neural network 104, as additional new training examples become available, the reward model 404 can be updated with new training examples and cyclical iterative refinement of both the generative neural netw ork 104 and the reward model 404 can follow . Specifically, the reward model is updated with new7training examples, while the generative neural network 104 is refined based on the updated reward model.[000116] FIG. 5 is a flow diagram of an example process 500 for training the generative neural netw ork using the rew ard model. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a generative model updating system, e.g., the generative model updating system100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.[000117] Specifically, example process 500 trains the generative neural network using the reward model, which is continually updated with new training examples, even after the initial training of the reward model, e.g., as described with reference to FIG. 4. The reward model is configured to process a user query and a portion of the data item to generate a respective likelihood of a particular implicit feedback signal type associated with the portion of the data item.[000118] To begin example process 500, the system obtains a new user query' (step 502).[000119] Next, the system generates, using the generative neural network, a new data item in response to the new user query' (step 504)[000120] For each of one or more portions of the new data item, the system processes the new user query and the portion of the new data item using the reward model to generate a respective likelihood for the portion (step 506), and then the system determines a reward score for the portion from the respective likelihood (step 508). Each portion includes a particular sample and all preceding samples in the sequence, i.e., the prefix for the particular sample when including the query', but excludes any sample that follows. These portions can encompass all possible prefixes of the data item. Alternatively, these portions can include only a subset of all possible prefixes.[000121] For example, the one or more portions of the new data item can be selected at random so that all portions of the data item have equal chance of being selected. Randomly selecting portions in this manner can help reduce computational cost of training the generative neural network in a future step.[000122] As another example, the one or more portions of the new data item can be selected according to a set of rules, such as requiring portions to include samples with specific content. Selecting portions according to specific content can help update the generative neural network to align to preferences regarding the specific content.[000123] After step 508, the system can reduce the number of portions of the new data item by sampling the portions proportionally to the rewards they receive, i.e., importance sampling. Those portions with higher rewards are more likely to be selected and are the portions most likely to contribute to updating the generative neural network to more align with preferences.[000124] The reward score can be any score derived from at least the respective likelihood of implicit feedback signal for the respective portion of the data item.[000125] For example, the reward score may be a log transform of an estimated likelihood for an implicit feedback signal, e.g., a log transform of the early termination likelihood of the data item portion.[000126] Different types of implicit feedback signal can have different transformations, i.e.. the system can use different transformation to map likelihoods to reward scores for different types of implicit feedback signals. For example, for some types of implicit feedback signals, e.g., replaying a particular portion, the transformation can be one that yields a positive correlation between the likelihood and the reward score, so that higher likelihoods result in higher reward scores. For example, for some types of implicit feedback signals, e.g., replaying a particular portion, the transformation can be one that yields a positive correlation between the likelihood and the reward score, so that higher likelihoods result in higher reward scores. Examples of such transformations are the log transformation, the identity transformation, a multiplicative transformation that multiplies the likelihood by a positive scaling factor, and so on. As another example, for other types of implicit feedback signals, e.g., early termination, the transformation can be one that yields a negative correlation between the likelihood and the reward score, so that higher likelihoods result in lower reward scores. Examples of such transformations are the negative of the log transformation, the negative of the identity transformation, a multiplicative transformation that multiplies the likelihood by a negative scaling factor, and so on.[000127] In some implementations, the reward score may be a weighted sum of multiple reward scores. That is, in some implementations, multiple reward models are trained for multiple types of feedback signals and their reward scores can be combined as a weighted linear combination.[000128] For example, a weighted sum of reward scores, e.g., reward scores for early termination, replay, skipped and completion signals, can be combined using a single fully connected layer neural network, and the weights of the fully connected layer can be fixed or trained.[000129] In some cases, in order to combine the reward scores of opposing characteristics, i.e., “good rewards” and “bad rewards”, e.g., a replay signal associated with a first sample represents a “good reward” while an early termination signal associated with a different later second sample represents a “bad reward”, reward score transforms for each feedback signal type are those that result in sign agreement, i.e., good scores positivelycorrelate with an increasing combined score and bad rewards negatively correlate with an increasing combined score.[000130] For example, in the case of combining the reward scores associated with a replay signal and an early termination signal, the replay signal reward score transform can be the logit of the estimated likelihood of a replay signal, while the early termination signal reward score transform can be the negative logit of the estimated likelihood of an early termination signal. For such transforms, an increasing likelihood of a replay signal for the first sample increases the combined reward score, and a decreasing likelihood of an early termination signal of the second sample also increases the combined reward score.[000131] In some implementations, how to combine the multiple reward scores into one reward score can be a learned function.[000132] For example, if high-quality feedback is associated with a data item, e.g., conducting a user study to generate explicit expert user feedback of data item samples, another training engine can be used to train a function to map the implicit feedback signals to the high-quality feedback.[000133] The system trains the generative neural network using the reward scores for the one or more portions using reinforcement learning (step 510).[000134] In order to train the generative neural network, the parameters of the generative neural network are adjusted through any of variety of optimization techniques that leverage the reward scores.[000135] For example, training the generative neural network using the reward scores may involve using a policy gradient method. That is, the trainable parameters associated with the generative neural network are optimized to maximize expected cumulative reward scores associated with the portions of the data item over many queries. The optimization can proceed through a gradient ascent procedure that iteratively updates the trainable parameters of the generative neural network as are described in ArXiv: 1707.06347, and ArXiv: 1502.05477.[000136] When the training engine trains a reward model to update the generative neural network through a policy gradient technique that maximizes the expected cumulative combined reward score associated with the sequence of samples over many queries, the reward model offers the advantage of being dense, i.e., the expected cumulative reward score over many queries is less noisy than alternatives and therefore stabilizes gradient ascent optimization of the generative neural network.[000137] The density of scores, i.e., reward scores for all samples in a sequence, of the reward model is advantageous when optimizing a generative neural network to maximize the expected cumulative reward score associated with the sequence of samples. Generating reward scores for all samples in a sequence results in less variance of the cumulative reward score than an alternative reward model, e.g., a reward model trained only using a single feedback signal associated with an entire data item, would produce. That is, the alternative reward model only assigns a value to the entire data item, effectively the same as assigning the same reward score to all samples of a data item, resulting in more noisy estimates of the cumulative reward score, and therefore, also resulting in more difficult optimization.[000138] When the training engine trains a combined reward model to update the generative neural network through a policy gradient technique that maximizes the expected cumulative combined reward score associated with the sequence of samples over many queries, the combined reward model offers the advantage of being rich, i.e., the combined reward score provides information regarding many implicit feedback types on a per sample basis.[000139] The combined reward model contains rich information due to its many reward models estimating likelihoods of different particular types of implicit feedback signals in concert on each sample of a sequence of samples. Consequently, any given combined reward score reflects multiple user feedbacks simultaneously, and updates to the generative neural network reflect these multiple feedbacks. In contrast, an alternative reward model trained only using a single feedback signal associated with an entire data item, e.g., approval of a data item by a user, lacks the informative fine-grain information that this combined reward model provides.[000140] This specification uses the term ■■configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.[000141] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.[000142] The term '‘data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.[000143] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.[000144] In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structuredat all. and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.[000145] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.[000146] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [000147] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory' or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry'. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [000148] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including byway of example semiconductor memory^ devices, e g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.[000149] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory7feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.[000150] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardw are accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.[000151] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow7framework or a Jax framework.[000152] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area netw ork (LAN) and a wide area network (WAN), e.g., the Internet.[000153] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g.. for purposes of displaying data to and receiving user input from a user interacting with thedevice, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device.[000154] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.[000155] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.[000156] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.[000157] What is claimed is:

Claims

CLAIMS1. A method performed by one or computers, the method comprising: obtaining a data item generated by a generative neural network in response to a user query, the data item comprising a sequence of samples; providing the generated data item for presentation to a user on a user device, wherein the respective samples are presented in an order according to the sequence; during the presentation of the generated data item to the user on the user device: obtaining an implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the plurality of samples; and using the implicit feedback signal to update the generative neural network.

2. The method of claim 1, wherein the particular sample is before a last sample in the sequence, and wherein obtaining an implicit feedback signal that indicates user feedback associated with a particular sample of the plurality of samples comprises: determining that the user terminated the presentation after the particular sample was presented.

3. The method of claim 2, wherein using the implicit feedback signal to update the generative neural network comprises: associating the implicit feedback signal with a portion of the samples in the data item that includes the particular sample and any samples preceding the particular sample in the sequence and does not include any samples following the particular sample in the sequence.

4. The method of claim 1, wherein the particular sample is a last sample in the sequence, and wherein obtaining an implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the plurality of samples comprises: determining that the user completed the presentation of the data item.

5. The method of claim 1, wherein the particular sample is before a last sample in the sequence, and wherein obtaining an implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the plurality of samples comprises: determining that the user replayed a segment of the data item corresponding to theparticular sample.

6. The method of claim 1, wherein the particular sample is before a last sample in the sequence, and wherein obtaining an implicit feedback signal of a particular type that indicates user feedback associated with a particular sample of the plurality of samples comprises: determining that, during the presentation, the user skipped a segment of the data item corresponding to the particular sample.

7. The method of any preceding claim, wherein using the implicit feedback signal to update the generative neural network comprises: determining whether to include a training example that comprises the user query and the data item in training data for the generative neural network based on the implicit feedback signal; and training the generative neural network on the training data through supervised learning.

8. The method of claim 7, wherein using the implicit feedback signal to update the generative neural network further comprises: in response to determining to include the training example in the training data, determining a weight to be assigned to the training example when training the generative neural network on the training data based on the implicit feedback signal.

9. The method of any preceding claim, wherein using the implicit feedback signal to update the generative neural network comprises: training, using the implicit feedback signal, at least the particular sample, and the user query, a reward model that predicts, given an input portion of an input data item and a corresponding user query, a likelihood that a user would submit an implicit feedback of the particular type in association with the input portion of the input data item when presented with the input data item.

10. The method of claim 9, wherein using the implicit feedback signal to update the generative neural network further comprises: training the generative neural network using the reward model.

11. The method of claim 10, wherein training the generative neural network using the reward model comprises: obtaining a new user query; generating, using the generative neural network, a new data item in response to the new user query; for each of one or more portions of the new data item: processing the new user query and the portion of the new data item using the reward model to generate a respective likelihood for the portion; and determining a reward score for the portion from the respective likelihood; and training the generative neural network using the reward scores for the one or more portions using reinforcement learning.

12. The method of any preceding claim, wherein the data item is text and each sample is a text token.

13. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-12.

14. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-12.

Images