Dynamic entity representations for sequence generation

Abstract

The present disclosure relates to dynamic entity representations for sequence generation. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output sequences using entity memory data. In particular, a neural network is used to generate an output sequence conditioned on an input sequence and on entity memory data.

Description

Technical Field

[0001] This specification relates to using neural networks to process inputs to generate output sequences. Background Technology

[0002] A neural network is a machine learning model that uses one or more non-linear units to predict the output from a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., another hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of a set of corresponding parameters. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which generates an output sequence conditioned on an input sequence and data identifying one or more cue entities.

[0004] The subject matter described herein can be implemented in specific embodiments to achieve one or more of the following advantages.

[0005] The system described in this specification uses a neural network conditioned on an input sequence including one or more input tokens and entity memory data to regressively generate an output sequence including a corresponding output token at each of one or more output positions in the output sequence. The system receives data identifying one or more cue entities and maintains entity memory data to include a corresponding representation of each of the one or more cue entities. The system initializes the entity memory data for each cue entity using one or more corresponding tokens from the data identifying the cue entities.

[0006] Maintaining memory data for each memory entity, as described in this specification, enables the neural network to incorporate entities more accurately into the output sequence. That is, maintaining entity memory data for each cue entity allows the neural network to incorporate a more consistent set of entities throughout the output sequence, where each entity in the set is associated with a more consistent set of attributes throughout the output sequence. In contrast, more conventional systems without entity memory data generate output sequences with less consistent sets of entities, where entities tend to fall outside the output sequence on long output sequences (e.g., during autoregressive output generation, a sufficiently long output sequence will begin to discard the beginning of the output sequence). Additionally, more conventional systems tend to generate output sequences with less consistent sets of attributes for each entity in the entity set.

[0007] The system described in this specification is capable of initializing entity memory data for each cue entity in memory data by processing data that identifies the cue entities. Using the data that identifies the cue entities allows the user to specify a custom set of important entities, each with custom associated attributes for use when generating the output sequence. In contrast, other output sequence generation techniques can process only the input sequence without specifically specifying important entities for the generation of the output sequence.

[0008] Therefore, by using the described technique, the "first neural network block" constituting the pre-trained neural network does not need to be able to effectively contextualize every possible entity and incorporate it into a large number of possible entities. Thus, by augmenting the first block with a "second neural network block," the described method allows the training of the neural network to consume fewer computational resources compared to training a model from scratch using only the "first neural network block," as attempted with conventional techniques. Furthermore, the overall neural network can achieve comparable or better performance using fewer "first neural network blocks" by effectively incorporating the "second neural network block," reducing the number of required parameters, and reducing the memory footprint of the neural network both during inference and training.

[0009] Details of one or more embodiments of the subject matter of this specification are set forth in the following figures and description.

[0010] Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. Attached Figure Description

[0011] Figure 1 This is a diagram of an example neural network system.

[0012] Figure 2 This is a flowchart of an example process for generating the output sequence.

[0013] Figure 3 This is a flowchart illustrating an example process for using two neural network layers to process layer inputs.

[0014] Figure 4 This is a flowchart of an example process for initializing scene memory data.

[0015] Figure 5 An example of how the system operates is shown.

[0016] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0017] Figure 1This is a diagram of an example neural network system 100. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the following systems, components, and techniques are capable of being implemented.

[0018] The neural network system 100 is a system that generates an output sequence 150 conditioned on an input sequence 102 and data that identifies one or more cue entities 104.

[0019] In some cases, system 100 obtains data that identifies each of one or more prompting entities 104 and an input sequence 102 that includes one or more input flags.

[0020] In some cases, input sequence 102 is received from a user of the system. For example, the user may be able to provide input sequence 102, and the user may be able to identify, or the system may be able to determine, which markers in input sequence 102 refer to entities.

[0021] In some other cases, input sequence 102 is a system-generated placeholder input sequence, for example, a sequence that includes only a predefined "start" flag. In this example, the user is able to provide data that only identifies entities that the user deems relevant.

[0022] As another example, the system can receive data from another system to identify the cue entity 104. For example, the other system can provide an entity related to the current context in which system 100 needs to generate the output sequence 150.

[0023] System 100 maintains entity memory data 120, which includes corresponding entity data for each of one or more prompt entities. That is, system 100 initializes entity memory data 120 after receiving data identifying a prompt entity. As will be described in more detail below, the entity data for each entity characterizes the entity and the context in which the entity appears in the received data.

[0024] See below for reference Figure 2 and Figure 5 A more detailed description of the initialization of physical memory data.

[0025] System 100 uses neural network 110 to process input sequence 102 and physical memory data 120 to generate output sequence 150, which includes a corresponding output flag for each of one or more output positions.

[0026] Typically, system 100 is able to autoregressively generate each output sign of output sequence 150 by using neural network 110 to process the combined sequence, the combined sequence comprising at least the input sequence and a cascade of any output signs preceding the output signs in the output sequence.

[0027] The neural network 110 includes one or more two-layer 130.

[0028] For example, a neural network can include a layer stack comprising (i) one or more bilayer layers and (ii) an output layer.

[0029] Each layer in the stack can receive a layer input, which includes a corresponding flag for each flag in the combined sequence. For the first layer in the stack, the input is the flags in the combined sequence. For each layer after the first layer in the stack, the input is the output of the previous layer.

[0030] As a specific example, a layer stack can include an embedding layer, followed by multiple double layers, and finally an output layer. As another specific example, a layer stack can include an embedding layer, followed by a layer stack comprising two regular attention layers, and finally an output layer.

[0031] When generating any given output label, the output layer can process the layer output from the output position of the last double layer 130 in one or more double layers of the neural network 110 to generate a corresponding score distribution on the vocabulary of output labels for the output position in the output sequence, and then select the appropriate output label from the vocabulary of output labels for the output position based on the corresponding score distribution of the output position. For example, the layer can sample labels or greedily select the label with the highest score.

[0032] Each double layer 130 includes a corresponding first neural network block 136 and a corresponding second neural network block 138.

[0033] The first neural network block 136 is a self-attention block that updates the flags in the layer inputs of the two-layer 130 by performing self-attention to generate a corresponding hidden representation for each input flag in the layer inputs.

[0034] The second neural network block 138 is a block that uses entity memory data 120 to update the flags in the layer inputs for the two layers 130 to generate a corresponding entity-aware representation for each layer input flag in the layer inputs.

[0035] The two layers of 130 are then combined with the hidden representation and the entity-aware representation to generate the two-layer 130 layer output.

[0036] Therefore, the dual layer 130 uses both the output generated so far as part of the output sequence 150 and the entity memory data 120 to update the flags in the layer's input, thereby producing a neural network 110 that is able to greatly improve the way it processes entity references in the output sequence 150 generated by the neural network.

[0037] See below for reference. Figure 2-5 The operations performed by the dual-layer 130 are described in more detail.

[0038] The neural network 110 can be configured to process any suitable input sequence that includes one or more input symbols (e.g., input symbols from a vocabulary of input symbols). The vocabulary of input symbols can include input symbols representing characters (e.g., letters or pictographs), word segments, words, special separators, and punctuation marks. For example, the input symbols can represent characters, word segments, and words from human languages ​​(e.g., English, Korean, etc.). In another example, the input symbols can represent code segments from compiled languages ​​(e.g., C, C++, Python, etc.). In yet another example, the input symbols can represent other symbols that are consistently instilled with semantic meaning.

[0039] The neural network 110 can be configured to process any appropriate data that identifies each of one or more cue entities. The one or more cue entities can be, for example, significant entities to be generated in the output sequence, such as characters in a narrative or topics of discussion in a report. The data that identifies each of the one or more cue entities can include one or more flags, such as one or more flags identifying the cue entity's name (e.g., name) and / or one or more input flags from a vocabulary of input flags describing attributes associated with the cue entity.

[0040] The neural network 110 can be configured to generate any suitable output sequence 150 including one or more output flags (e.g., output flags from a vocabulary of output flags). The vocabulary of output flags can include output flags representing characters (e.g., letters or pictographs), word segments, words, special delimiters, and punctuation marks. For example, output flags can represent characters, word segments, and words from human languages ​​(e.g., English, Korean, etc.). In another example, output flags can represent code segments from compiled languages ​​(e.g., C, C++, Python, etc.). In yet another example, output flags can represent other symbols that are consistently instilled with semantic meaning.

[0041] In one example, input sequence 102 can include input prompts from a user, and one or more prompt entities can include topics important to the user. Neural network 110 can process one or more input sequences from the user to generate a corresponding output sequence that represents a response to the user's input sequence. For example, neural network 110 can be part of a chatbot, and a user may be interacting with the chatbot to receive answers to questions, such as a customer service chatbot for a company, or an interactive FAQ bot for dynamically resolving the most frequently asked questions about a company or service.

[0042] In another example, system 100 may be part of an automated medical diagnostic system, and the prompting entity may be an entity provided by the user that represents the user's health status, such as the current system, existing conditions, medications, etc. The output sequence can be generated as part of a dialogue with the user concerning the user's health status.

[0043] In another example, input sequence 102 can include a text sequence, and one or more cue entities can include topics to be summarized from the text sequence. Output sequence 150 can include a general summary of the text sequence and a corresponding sub-summary of each of the one or more cue entities.

[0044] In another example, input sequence 102 can represent the overture in a song, and output sequence can be the continuation of the song. Cue entities can be instruments to be played in the output sequence (e.g., a generic or "average" version of an instrument, or each instrument having certain desired qualities such as being constructed of certain materials, having certain shapes, representing a particular famous instrument (such as the Stradivarius), or any combination thereof). Cue entities can collectively represent a group of instruments, such as instruments played in an orchestra. In yet another example, cue entities can represent a particular style or quality of music, such as hard rock, death metal vocals, or operatic singing to be imitated in the output sequence. In yet another example, cue entities can represent the style of an individual artist or band to be imitated in the output sequence.

[0045] In another example, input sequence 102 can include a text sequence representing the beginning of the narrative, and cue entities can include important characters, places, ideas, things, or combinations thereof in the narrative. Output sequence 150 can be a continuation of the narrative.

[0046] In another example, input sequence 102 can include lines of computer code, and cue entities can include desired code segments, algorithms, methodologies, or semantic entities to be used in the code (e.g., for loops, while loops, etc.). Output sequence 150 can represent a continuation of a line of computer code, a specific use case of a cue entity, or a corresponding alternative example of a line of computer code rewritten using each cue entity. System 100 can then provide the generated computer code for execution by one or more computers to perform a computational task.

[0047] As another example, the prompting entity can recognize entities in the environment, the input sequence 102 can specify a task to be performed by an agent in the environment (e.g., a robot or other mechanical agent), and the output sequence can be instructions given to the agent to make the agent perform the task, such as natural language instructions or other instructions.

[0048] In some implementations, the corresponding first neural network block 132 in each two-layer 130 may come from a self-attention model with a modified architecture for generating or processing longer sequences, such as the transformer-XL (T-XL) machine learning model. After autoregressively generating N output labels in the output sequence, the T-XL model (or other model) is able to store representations of the N output labels in the T-XL memory. The T-XL model is able to store corresponding representations of multiple segments of the N labels in the T-XL memory. Each time after generating an additional N output labels, the T-XL is able to store a representation of the additional N output labels in the T-XL memory, where the representation was generated by the T-XL model. The T-XL model is able to regressively generate each output label in the output sequence by processing a combination sequence of corresponding representations that are at least already in the T-XL memory and any output labels that are stored in the T-XL memory as part of the corresponding representations.

[0049] Therefore, processing a combined sequence as used in this specification can include processing all individual flags in the combined sequence or processing a compressed representation of some or all flags in the combined sequence.

[0050] Before using neural network 110 to generate output sequence 150, system 100 or another training system trains neural network 110 so that neural network 110 can accurately generate output sequence.

[0051] Specifically, the training system is capable of training the neural network 110 on training data that includes multiple training examples. Each training example includes (i) a training input sequence and (ii) a training output sequence that should be generated by the system 100 by processing the training input sequence.

[0052] The training system can perform this training in any of a variety of ways. As an example, it is possible to pre-train the first network block in each two-layer structure, and then train the neural network using the included first and second network blocks to improve how the neural network processes entity mentions.

[0053] Figure 2 This is a flowchart of an example process 200 for generating an output sequence. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification (e.g., Figure 1 The neural network system 100 described in the text is capable of executing process 200.

[0054] The system receives data that identifies one or more prompt entities (step 202) and receives an input sequence that includes one or more input flags (step 204).

[0055] The system maintains physical memory data (step 206).

[0056] Specifically, the entity memory data includes corresponding entity data for each of one or more prompt entities, and the corresponding entity data includes a corresponding entity representation of the prompt entity.

[0057] In some implementations, the corresponding entity data for each entity includes the entity's static key vector.

[0058] In some other implementations, the corresponding entity data for each entity includes both a static key vector and a dynamic value vector, which can be updated by the system as generation proceeds.

[0059] In some implementations, the entity memory data further includes corresponding non-entity data for each of one or more non-entities, which represents entity-independent information. Like the entity data, the non-entity data can include static keys or both static keys and dynamic values.

[0060] The system uses a neural network with one or more two layers to process the input sequence and physical memory data to generate an output sequence, which includes a corresponding output flag at each of one or more output positions in the output sequence (step 208). Specifically, as described above, the system generates the output flags in the output sequence autoregressively by processing the combined sequence one after another for each flag.

[0061] As part of generating a marker at any given output position in the output sequence, the system generates a corresponding layer input for each of one or more bilayers and uses the bilayer to process the layer input to generate the layer output of the bilayer.

[0062] As mentioned above, layer input typically includes a corresponding flag for each flag in the combined sequence and can be generated by the layer preceding the double layer in the layer stack.

[0063] Each bilayer has at least (i) a corresponding first neural network block and (ii) a corresponding second neural network block, and uses the two network blocks to generate the corresponding layer output of the bilayer when generating a given flag.

[0064] In other words, neural networks typically consist of stacked layers (including one or more double layers), and to generate a label at any given position in the output sequence, they process a combined sequence, which includes the input sequence and any output labels already generated at positions preceding the given position. In some cases, the system processes a compressed representation of some labels in the combined sequence as described above. In other cases, the neural network 110 is able to have a fixed “context window” and, as part of processing the combined output sequence, the system is able to discard labels outside the context window.

[0065] In some implementations, the system also includes entity cues in the combined sequence. Entity cues include corresponding flags for each entity in the entity-identifying memory data, optionally separated by special delimiter flags. Including entity cues allows for two-layer attention to entity flags and improves the consistency of the generated data.

[0066] See below for reference Figure 3 A more detailed description of using two layers to process layer inputs for a given two layers.

[0067] Figure 3 This is a flowchart of an example process 300 for processing layer input using two layers. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification (e.g., Figure 1 The neural network system 100 described in the text is capable of executing process 300.

[0068] The two layers receive layer inputs for the output location (step 302), which are based on at least the input sequence and include one or more layer input flags. For example, when the neural network is configured to process combined sequences, the layer inputs include the corresponding layer input flags for each flag in the current combined sequence.

[0069] The two layers use the corresponding first neural network block to process the layer input to generate a corresponding hidden representation for each layer input flag in the layer input (step 304).

[0070] As described above, the corresponding first neural network block is typically a self-attention layer block that applies self-attention to the symbols in the layer input to generate hidden representations.

[0071] The first block can use any of the various self-attention variants to perform this processing.

[0072] In some implementations, the first block is an attention block from a self-attention model (e.g., a transformer-XL (T-XL) machine learning model) with a modified architecture to generate or process longer sequences. After autoregressively generating N output labels in the output sequence, the T-XL model (or other model) is able to store representations of the N output labels in the T-XL memory. The T-XL model is able to store corresponding representations of multiple segments of the N labels in the T-XL memory. Each time after generating an additional N output labels, T-XL is able to store a representation of the additional N output labels in the T-XL memory, where the representation was generated by the T-XL model. The T-XL model is able to regressively generate each output label in the output sequence by processing a combination sequence of corresponding representations that are at least already in the T-XL memory and any output labels that are stored in the T-XL memory as part of the corresponding representations that precede the output labels.

[0073] Therefore, in some implementations, the first block of attention is on the layer inputs in the T-XL memory and the layer inputs that have not yet been stored in the T-XL memory.

[0074] The first block can also include other components besides the self-attention layer (i.e., processing is performed before or after the self-attention layer). Examples of such components include feedforward layers, normalization layers, residual connection layers, etc.

[0075] The two layers use corresponding second neural network blocks to process the layer inputs and entity memory data to generate a corresponding entity-aware representation for each layer input sign in the layer input (step 306).

[0076] Typically, for each layer input flag in the layer input, the second neural network block uses entity memory data to update the layer input flag to generate an entity-aware representation of the layer input flag.

[0077] As a specific example, the corresponding second neural network block can include a cross-attention neural network layer that applies cross-attention to the entity memory data. Specifically, the cross-attention layer can, for each layer input flag, generate a query derived from the layer input flag and perform cross-attention on the entity memory data having keys and values ​​derived from at least the corresponding entity representations in the entity memory data to update the layer input. For example, when the entity memory data includes only static keys, both the key and value can be equal to or derived from the static key. When the entity memory data includes both static keys and dynamic values, the key can be equal to or derived from the static key, while the value can be equal to or derived from the dynamic value.

[0078] The second block can also include other components besides the cross-attention layer (i.e., processing is performed before or after the cross-attention layer). Examples of such components include feedforward layers, normalization layers, residual connection layers, etc.

[0079] The hidden representation and entity-aware representation are processed in two layers to generate a layer output of the output location (step 308), the layer output having one or more layer output flags—that is, including a corresponding layer output flag for each flag in the layer input.

[0080] Generally, two layers are combined to form a hidden representation and an entity-aware representation to generate the layer output.

[0081] For any given sign, the double layer can combine the sign's representation in any appropriate way.

[0082] As a specific example, two layers can use gated neural network blocks with multiple gating parameters to combine hidden representations and entity-aware representations to generate layer output labels in the layer output.

[0083] For example, a gated neural network block can process the hidden representation and the corresponding entity perception representation according to multiple gating parameters for each hidden representation to generate a corresponding gating vector, and then combine the hidden representation and the corresponding entity perception representation according to the corresponding gating vector to generate the corresponding layer output label in the layer output.

[0084] To generate gated vectors, gated neural network blocks can cascade hidden representations and entity-aware representations to generate combined representations and process the combined representations according to gating parameters to generate corresponding gated vectors, for example, by processing the combined representations via one or more fully connected layers.

[0085] To combine the hidden representation and the corresponding entity-aware representation according to the corresponding gating vector, the gated neural network block can process the corresponding gating vector to generate a hidden weight vector and perform element-wise multiplication of the hidden weight vector and the hidden representation to generate an intermediate hidden representation. Similarly, the block can process the corresponding gating vector to generate an entity weight vector and perform element-wise multiplication of the entity weight vector and the entity-aware representation to generate an intermediate entity-aware representation. The block can then sum the intermediate hidden representation and the intermediate entity-aware representation to generate the corresponding layer output label.

[0086] As mentioned above, in some implementations, physical memory data is static after initialization; however, in other implementations, the system can update dynamic values ​​in the physical memory data after initialization. See below for reference. Figure 4 Describes the dynamic value update.

[0087] Figure 4This is a flowchart of an example process 400 for initializing physical memory data. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification (e.g., Figure 1 The neural network system 100 described in the text is capable of executing process 400.

[0088] As described above, for each entity, the entity memory data can include (i) a static key or (ii) a static key and a dynamic value.

[0089] To initialize this data, for each entity, the system processes data that identifies that entity. In some implementations, the system is able to receive a separate text segment describing each entity. In other implementations, the system is able to receive a single text segment describing all entities. For example, each entity can be mentioned in the initial input sequence received by the system from the user.

[0090] Specifically, for each entity, the system is able to use a neural network to process each flag in the corresponding data of the entity being identified to generate a corresponding embedding for each flag (step 402). During processing, the system uses only the first block within the two layers and not the second block. That is, during this processing, for each two layers, the system receives a layer input including one or more layer input flags, where each layer input flag corresponds to a corresponding flag in the flags of the entity being identified, and uses the corresponding first neural network block within the two layers to process the layer input flags to generate a corresponding layer output flag for each layer input flag without using the corresponding second neural network block of the two layers.

[0091] The system then uses the corresponding embedding of the entity's flag (i.e., the embedding of the entity's flag within the data that identifies the entity) to initialize the corresponding entity representation of the entity (step 404).

[0092] As a specific example, the system is able to determine the average value of the corresponding embeddings of the flags of the prompt entity and use the average value of the corresponding embeddings of the flags of the prompt entity to initialize the corresponding entity representation of the prompt entity.

[0093] When the physical memory data consists only of static keys, the system can initialize the static keys to be equal to the average value. When the physical memory data consists of both static keys and dynamic values, the system can initialize both the static keys and dynamic values ​​to be equal to the average value.

[0094] When the physical memory data includes dynamic values, the system can update the dynamic values ​​at certain points while generating the output sequence.

[0095] Specifically, the system can update the dynamic value after each Nth flag is added to the combined sequence processed by the neural network. Typically, N is a fixed integer greater than one and can be a hyperparameter of the system. That is, for tasks where the system interacts with the user while generating the output sequence, the system can perform the update after N flags—a combination of flags that can be generated for the user and system-generated flags—have been added to the combined sequence. For tasks where the system generates a long output sequence without user interaction after receiving a prompt entity and an input sequence, the system can perform the update after N flags have been generated by the system.

[0096] In order to update the dynamic values, the system determines the corresponding representation of the last N combined sequence flags for each of one or more prompting entities (step 406).

[0097] For example, the system can use the corresponding first neural network block of the last double layer in one or more double layers of the neural network to determine the hidden representation of the last N combined sequence flags and use the corresponding second neural network block of the last double layer in one or more double layers of the neural network to determine the corresponding presence weights for the last N combined sequence flags of the cue entity. That is, the system can use the outputs of the first and second blocks for the last N combined sequence flags when processing the last flag in the combined sequence. The system then determines the corresponding representation of the last N combined sequence flags of the cue entity by processing the hidden representation and the presence weights.

[0098] The system then uses the representation of the prompt entity to update the dynamic value in the entity memory data of each prompt entity (step 408).

[0099] In particular, the system is able to update the dynamic value of a given entity by processing at least the corresponding representation of the entity through updating neural network blocks.

[0100] For example, the system can use an updated neural network block to determine representation weights for the corresponding representation, and then update the dynamic values ​​in the memory data of the memory entity by processing the dynamic values, representation weights, and corresponding representations. For example, the system can determine the updated dynamic values ​​as a weighted sum of dynamic values ​​and representations, where the representations are weighted by representation weights and the dynamic values ​​are weighted by subtracting the representation weights.

[0101] Figure 5 Example 500 illustrates the operation of the system.

[0102] In Example 500, the entity memory data includes the corresponding static keys and corresponding dynamic values ​​for the following three entities: “Sarakin”, “Community”, and “Animal”.

[0103] The system is able to represent the three entities in a combined sequence processed by a neural network as entity cues.

[0104] If able to Figure 5 As seen in the image, the neural network utilizes Transformer-XL to generate long output sequences in multiple large blocks. The system has already generated the first 39 large blocks of the output sequence, now represented in the "T-XL" memory, and is currently generating the 40th large block.

[0105] To generate the next output in block 40, the bilayer within the system operates on a combination sequence that includes flags and entity hints derived from the output already generated in the block ("Sarah King saved the animal"). Due to the architecture of Transformer-XL, the first block within each bilayer also operates on representations of earlier blocks stored in T-XL memory.

[0106] In particular, such as Figure 5 As shown, the two-layer structure within the neural network comprises a first block and a second block. The first block performs self-attention across the combined sequence (and optionally, the data in the Transformer-XL memory), while the second block performs cross-attention on the physical memory data for each flag in the combined sequence.

[0107] Then, a gating mechanism is used to combine the outputs of the two blocks to generate a single layer output flag for each flag in the combined sequence.

[0108] When the criteria for updating dynamic values ​​are met, the system is able to use an update neural network (“FFN”) to update the dynamic values.

[0109] As mentioned above, neural networks can be trained in any of a variety of ways. For example... Figure 5 As shown, the second neural network block can be trained through "entity supervision".

[0110] In particular, in some implementations, the corresponding first neural network block for one or more two layers may have been pre-trained as part of a different neural network that does not include the corresponding second neural network block. For example, the first neural network block may have been pre-trained as part of a different neural network performing a language modeling task. For example, different neural networks may have been trained on a large corpus of unlabeled text data through unsupervised learning.

[0111] After pre-training the corresponding first neural network block, the system is able to train the neural network on training data including the target network input and the corresponding target network output for each network input.

[0112] Specifically, the system is able to train a neural network to optimize an objective function that, for each of a plurality of training network inputs and for each output position in the target network output for the training network inputs, measures the corresponding error between: (i) the corresponding target score distribution on the vocabulary of the output labels at the position, i.e., the target distribution for identifying the corresponding labels in the target network outputs, and (ii) the score distribution generated by the neural network for the output positions by processing the training network inputs.

[0113] like Figure 5 As shown, the objective function can also include a regularization loss that measures the error between (i) the intermediate output (cross-attention score) of the corresponding second neural network block and (ii) the target intermediate output (gold mention) of the corresponding second neural network block for each of one or more bilayers.

[0114] In some implementations, the system keeps the first block fixed at pre-trained values ​​during this training period. In other implementations, the system fine-tunes the first block while training the second block.

[0115] As used in this specification, “embedded” is a vector of numeric values ​​(e.g., floating-point or other types of numeric values) having a predetermined dimension, for example, having a predetermined number of values.

[0116] As mentioned above, a self-attention block is a neural network layer that includes an attention mechanism that operates on the self-attention block input (or input derived from the layer input) to generate a self-attention block output. The self-attention mechanism can be causally masked, such that any given position in the input sequence is not concerned with any position after that given position in the input sequence (e.g., using data from them). Many different possible attention mechanisms exist.Some examples of self-attention layers, including attention mechanisms, are described in the following: Vaswani et al., “Attention is all you need,” 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards.” A human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0117] Typically, attention mechanisms map a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is calculated using a compatibility function of the query and its corresponding key (e.g., a dot product or scaled dot product).

[0118] Typically, self-attention mechanisms are configured to correlate different positions within the same sequence to determine a transformed version of the sequence as the output. For example, the input to the attention layer can include a vector of each element of the input sequence. These vectors provide input to the self-attention mechanism and are used by the mechanism to determine a new representation of the same sequence for the attention layer output, which similarly includes a vector of each element of the input sequence. The output of the self-attention mechanism can be used as the attention layer output, or it can be processed through one or more of feedforward layers, skip connections, or normalization operations to provide the attention layer output.

[0119] In some implementations, the attention mechanism is configured to apply, for example, a matrix W to the attention layer input X, which serves as the input data of the attention layer. Q Defined query transformation, for example, by matrix W K Defined key transformations and, for example, by matrix W V Each of the defined value transformations is used to derive a query matrix Q = XW, which includes the corresponding query for each vector in the input sequence. Q The key matrix K = XW includes the corresponding keys for each vector in the input sequence. K And a value matrix V = XW that includes the corresponding values ​​for each vector in the input sequence. V These are used to determine the sequence of presence for the output. For example, an attention mechanism could be a dot-product attention mechanism applied by applying each query vector to each key vector to determine a weight for each value vector, and then using those weights to combine the value vectors to determine the output of the self-attention layer for each element in the input sequence. The output of the self-attention layer can be scaled by a scaling factor, for example, by the square root of the dimensions of the queries and keys, to achieve scaled dot-product attention. Therefore, for example, the output of the attention mechanism can be determined as... Where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism includes an "additive attention" mechanism that uses a feedforward network with hidden layers to compute the compatibility function. The output of the attention mechanism can be further processed by one or more fully connected feedforward neural network layers.

[0120] Attention mechanisms can achieve multi-head attention, meaning that multiple different attention mechanisms can be applied in parallel. The outputs of these can then be combined with learned linear transformations, applied as needed to reduce the original dimensionality, such as through cascading.

[0121] This specification uses the term "configured" with respect to system and computer program components. For one or more computer systems to be configured to perform a particular operation or action, it means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when run by a data processing device, cause the device to perform those operations or actions.

[0122] The embodiments of the subject matter and functional operation described in this specification can be implemented using digital electronic circuit systems, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their equivalents), or a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for use by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for operation by a data processing apparatus.

[0123] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an operating environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0124] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to run on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0125] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and that can be stored on storage devices in one or more locations. Thus, for example, an indexed database can comprise multiple collections of data, each of which can be organized and accessed differently.

[0126] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0127] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0128] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The essential components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0129] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0130] To provide interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of information to a personal device, such as a smartphone running a messaging application, and receiving response information from the user in return.

[0131] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production, such as inference workloads.

[0132] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow.

[0133] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can use to interact with an implementation of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0134] A computing system can include clients and servers. Clients and servers are typically geographically separated and interact via communication networks. The client-server relationship occurs through computer programs running on their respective computers and having a client-server relationship with each other. In some implementations, the server sends data, such as HTML pages, to the user device, for example, to display data to a user interacting with the device as a client, and to receive user input from the user interacting with the device as a client. It is possible to receive data generated on the user device, such as the result of user interaction, from the device at the server.

[0135] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in this specification in the context of a single embodiment can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as acting in certain combinations and even initially claimed in this way, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0136] Similarly, although operations are depicted in the accompanying drawings and described in the claims in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring the performance of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0137] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for generating a text output sequence using dynamic entity representations, the method being performed by a system including a neural network and comprising: Receive text data that identifies one or more prompt entities; Receive a text input sequence including one or more input flags; Maintain text entity memory data including the corresponding text entity data of each of the one or more prompt entities, wherein the corresponding text entity data of each prompt entity includes the corresponding entity representation of the prompt entity; and The text input sequence and the text entity memory data are processed using a neural network having one or more dual layers, wherein each dual layer includes at least (i) a corresponding first neural network block and (ii) a corresponding second neural network block to generate a text output sequence, the text output sequence including a corresponding output flag for each of one or more output positions in the text output sequence, for each output position including: For each of the one or more double layers: Receive layer input at the output location, the layer input being based on at least the text input sequence and including one or more layer input flags; The corresponding first neural network block is used to process the layer input to generate a corresponding hidden representation for each layer input flag in the layer input; The corresponding second neural network block is used to process the layer input and the text entity memory data to generate a corresponding entity-aware representation for each layer input flag in the layer input; and The hidden representation and the entity-aware representation are processed to generate a layer output with one or more layer output flags at the output location.

2. The method according to claim 1, wherein, The neural network generates each output flag of the text output sequence autoregressively by processing a combination sequence for each output position, the combination sequence comprising a concatenation of at least the text input sequence and any output flags in the text output sequence preceding the output position, wherein the layer input for each output position is derived from the combination sequence.

3. The method according to claim 2, wherein, Each of the one or more cue entities is identified by one or more flags, and the combined sequence further includes, for each cue entity, the one or more flags that identify the cue entity.

4. The method according to claim 1, wherein, For each double layer, using the corresponding second neural network block to process the layer input and the text entity memory data to generate the corresponding entity-aware representation of each layer input flag in the layer input includes: For each layer input flag, the corresponding second neural network block is used to process the layer input flag and the text entity memory data to generate the corresponding entity-aware representation of the layer input flag.

5. The method according to claim 4, wherein, For each double layer, the corresponding second neural network block includes a cross-attention neural network layer that applies cross-attention based on the query derived from the layer input flags and the keys and values ​​derived from the corresponding entity representations in at least the text entity memory data.

6. The method according to claim 3, wherein, For each bilayer, processing the hidden representation and the entity-aware representation to generate the layer output includes: The hidden representation and the entity-aware representation are combined using a gated neural network block with multiple gating parameters to generate the layer output flag in the layer output.

7. The method according to claim 6, wherein, Combining the hidden representation and the entity-aware representation using the gated neural network block with multiple gating parameters to generate the layer output includes: For each hidden representation: The hidden representation and the corresponding entity-aware representation are processed according to the plurality of gating parameters to generate corresponding gating vectors; and The hidden representation and the corresponding entity-aware representation are combined according to the corresponding gating vector to generate the corresponding layer output flag in the layer output.

8. The method according to claim 7, wherein, Processing the hidden representation and the corresponding entity-aware representation according to the multiple gating parameters to generate the corresponding gating vector includes: Cascade the hidden representation and the entity-aware representation to generate a combined representation; and The combined representation is processed according to the gating parameters to generate the corresponding gating vector.

9. The method according to claim 7, wherein, Combining the hidden representation and the corresponding entity-aware representation according to the corresponding gating vector to generate the corresponding layer output flag includes: Process the corresponding gate vectors to generate hidden weight vectors; Perform element-wise multiplication of the hidden weight vector and the hidden representation to generate an intermediate hidden representation; Process the corresponding gate vectors to generate entity weight vectors; Perform element-wise multiplication of the entity weight vector and the entity-aware representation to generate an intermediate entity-aware representation; and The intermediate hidden representation and the intermediate entity-aware representation are summed to generate the corresponding layer output flag.

10. The method of claim 1, further comprising, before using the neural network to process the text input sequence and the text entity memory data to generate the text output sequence: The corresponding entity representation of each cue entity in the text entity memory data is initialized by processing the data that identifies the cue entity.

11. The method according to claim 10, wherein, Initializing the corresponding entity representation of each cue entity in the text entity memory data by processing the data that identifies the cue entity includes: The neural network is used to process each flag in the data that identifies the cue entity to generate a corresponding embedding of the flag, wherein processing the flag using the neural network includes, for each two-layer: Receive layer input including one or more layer input flags, wherein each layer input flag corresponds to a corresponding one of the flags identifying the prompt entity; and Without using the corresponding second neural network block of the two layers, the corresponding first neural network block is used to process the layer input flags to generate a corresponding layer output flag for each layer input flag; and The corresponding entity representation of the prompt entity is initialized using the corresponding embedding of the sign of the prompt entity.

12. The method according to claim 11, wherein, Initializing the corresponding entity representation of the prompt entity using the corresponding embedding of the flag of the prompt entity includes: Determine the average value of the corresponding embeddings of the flags of the prompting entity; and The corresponding entity representation of the prompt entity is initialized using the average value of the corresponding embedding of the sign of the prompt entity.

13. The method according to claim 12, wherein, The corresponding entity representation of each of the one or more prompt entities is a combination of a corresponding static key and a corresponding dynamic value, and wherein initializing the corresponding entity representation of each prompt entity using the average value of the corresponding embedding of the flag of the prompt entity includes: Initialize the corresponding static key of the prompt entity to the average value of the corresponding embedding of the prompt entity's flag; and The corresponding dynamic value of the prompt entity is initialized to the average value of the corresponding embedding of the prompt entity's flag.

14. The method according to claim 12, wherein, The corresponding entity representation of each of the one or more prompt entities is a corresponding static key, and wherein initializing the corresponding entity representation of each prompt entity includes: The corresponding static key of the prompt entity is initialized to the average value of the corresponding embedding of the prompt entity's flag.

15. The method according to claim 13, wherein, Maintain text entity memory data including the corresponding text entity data of each of the one or more prompt entities, wherein the corresponding text entity data of each prompt entity includes the corresponding entity representation of the prompt entity, including: After every Nth flag is added to the combined sequence, the corresponding dynamic value in the text entity memory data is updated for each of the one or more cue entities, where N is a fixed integer greater than one.

16. The method according to claim 15, wherein, For each of the one or more prompt entities, updating the corresponding dynamic value in the text entity memory data includes: For each of the one or more prompting entities, determine the corresponding representation of the last N combined sequence flags; and For each cue entity, the dynamic value in the text entity memory data is updated by processing the corresponding representation of at least the cue entity using an update neural network block.

17. The method according to claim 16, wherein, Determining the corresponding representation of the last N combined sequence flags for each of the one or more prompting entities includes: The hidden representation of the last N combined sequence symbols is determined using the corresponding first neural network block of the last double layer of the one or more double layers in the neural network; The corresponding second neural network block of the last double layer in one or more double layers of the neural network is used to determine the corresponding presence weight for the last N combined sequence flags for the cueing entity; and The corresponding representation of the last N combined sequence flags is determined for the cueing entity by processing the hidden representation and the attendance weight.

18. The method according to claim 16, wherein, Updating the dynamic value in the memory data for each cue entity by processing at least the corresponding representation using an updated neural network block includes: The updated neural network block is used to determine the representation weights for the corresponding representation; and The dynamic value in the memory data of the memory entity is updated by processing the dynamic value, the representation weight, and the corresponding representation.

19. The method according to claim 1, wherein, The text entity storage data further includes corresponding non-entity data for each of one or more non-entities, the corresponding non-entity data representing entity-independent information.

20. The method according to any one of claims 1-19, wherein, Using a neural network with one or more two layers to process the text input sequence and the text entity memory data further includes, for each of the output positions: Process the layer outputs from the output positions of the last double layer of the one or more double layers in the neural network to generate a corresponding score distribution on a vocabulary of the output labels of the output positions in the text output sequence; as well as The corresponding output flag is selected from the vocabulary of output flags at the output position based on the corresponding score distribution at the output position.

21. The method according to claim 20, wherein, The corresponding first neural network block used for the one or more two-layer structures has been pre-trained as part of a different neural network that does not include the corresponding second neural network block.

22. The method of claim 21, further comprising, after pre-training the corresponding first neural network block, training the neural network to optimize an objective function, the objective function measuring a corresponding error between (i) a corresponding target score distribution on the vocabulary of the output sign at the position, and (ii) the score distribution generated by the neural network for the output position by processing the training network input.

23. The method according to claim 22, wherein, The objective function further measures the regularization loss of each of the one or more bilayers among: (i) the intermediate output of the corresponding second neural network block and (ii) the target intermediate output of the corresponding second neural network block.

24. The method according to claim 23, wherein, For each double layer, the corresponding second neural network block includes a cross-attention neural network layer that applies cross-attention based on the query derived from the layer input flags and the keys and values ​​derived from the corresponding entity representations in at least the text entity memory data, and the intermediate output is the cross-attention weights generated by the cross-attention neural network layer and the target intermediate output is the target set of the cross-attention weights.

25. A system comprising: One or more computers; as well as One or more storage devices communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1-24.

26. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1-24.

Images