Computer-implemented method, computer program, and computer system (encoding with a transformer incorporating metadata)

Abstract

To provide a method, system and program for transformer-based encoding incorporating metadata in natural language textual or non-textual data.SOLUTION: A method comprises: constructing a relativity matrix from metadata of a corpus of natural language text documents, where a row-column intersection in the relativity matrix corresponds to a relation between two instances of a type of metadata; and training an encoder model to generate a trained encoder model so as to compute an embedding corresponding to a token of a natural language text document within the corpus and the relativity matrix, the encoder model comprising a first encoder layer, the first encoder layer comprising a token embedding portion, a relativity embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, the training comprising adjusting a set of parameters of the encoder model.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates generally to methods, systems, and computer program products for autoencoding natural language text or non-text data, and more particularly to methods, systems, and computer program products for encoding with transformers that incorporate metadata. [Background technology]

[0002] Transformer-based autoencoders, also known as transformer-based autoencoding networks, or simply transformers, have an encoder-decoder architecture. The encoder portion of the transformer contains a set of encoding layers that iteratively process the input sequence, layer by layer. Each encoder layer produces an encoding, a multidimensional number also known as a vector, that contains information about which parts of the input are related to each other. The decoder portion of the transformer contains a set of decoding layers that iteratively process the encoder's output, layer by layer. Each decoder layer uses the encoding's embedded context information to generate an output sequence that corresponds to the input sequence. To incorporate context, the encoder and decoder layers contain an attention mechanism for each input that includes other inputs in the sequence weighted by the relevance of the other inputs. The decoder layer also contains an additional attention mechanism that draws on information from the output of the previous decoder layer and from the encoding.

[0003] As used herein, a Transformer refers to an autoencoding network with an encoder-decoder architecture, an encoder with a set of encoding layers but no decoder layers, and a decoder with a set of decoder layers but no encoder layers. For example, a typical text classification implementation includes only an encoder layer followed by a classifier layer, but no decoder layer. Transformers can be used to process sequences of natural language text or non-text data.

[0004] Exemplary embodiments recognize that to encode text using context, attention mechanisms typically combine embeddings corresponding to tokens (token embeddings) with positional embeddings (encoding the position of the token within a portion or segment of text) and segmentation embeddings (encoding the segment of text in which the token is located). However, both positional and segmentation embeddings are based solely on the text within a document. As a result, metadata—information about the text within a document, such as turn and speaker information in a dialogue, thread information in a discussion forum transcript, movie subtitles, section hierarchy information, and other elements of structured text—is often not used when encoding natural language text, even when available.

[0005] The exemplary embodiments also recognize that even when metadata is used in text processing, the model architecture does not explicitly consider attention between tokens provided by the metadata or model the influence of the metadata through layers of transformers. Rather, portions of the metadata are encoded as embeddings and combined with token embeddings in existing attention mechanisms. For example, in a conversation transcript, the metadata may indicate that some portions were spoken or transcribed by participant 1 and some were spoken and transcribed by participant 2. Thus, all of the participant 1 portions may be considered one segment of text, and all of the participant 2 portions may be considered a second segment of text, and corresponding segmentation embeddings are determined and incorporated into the text encoding. Alternatively, the system may compute additional embeddings that encode which speaker is associated with a token, and the additional embeddings are combined with the token, position, and segmentation embeddings in existing attention mechanisms. However, encoding specific parts or types of metadata into specific embeddings does not easily yield models that are trainable for many types of text and metadata, perform comparably to existing models when processing text without metadata, and can be used in a variety of downstream tasks. Furthermore, model architectures for processing non-text data either do not explicitly consider attention between tokens provided by metadata, or do not model the influence of metadata through layers of transformers.

[0006] Thus, the exemplary embodiments also recognize that there is an unmet need for a transformer-based autoencoder architecture that includes parameters that are specific to a model of attention due to metadata, and that it is possible to use a metadata encoding scheme that does not require encoding specific parts or types of metadata into specific embeddings. Summary of the Invention [Problem to be solved by the invention]

[0007] Metadata, even when available, is often not used when encoding natural language text. [Means for solving the problem]

[0008] Exemplary embodiments provide methods, systems, and computer program products. The embodiments include a method for constructing a correlation matrix from metadata of a corpus of natural language text documents, the intersections of rows and columns in the correlation matrix corresponding to relationships between two instances of a type of metadata. The embodiments train an encoder model to compute embeddings corresponding to tokens and the correlation matrix for the natural language text documents in the corpus to generate a trained encoder model, the encoder model including a first encoder layer, the first encoder layer including a token embedding portion, a correlation embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, and the training includes adjusting a set of parameters of the encoder model. Thus, the embodiments provide encoding using a transformer that incorporates metadata.

[0009] In another embodiment, the token embedding portion computes a set of token embeddings, where the token embeddings in the set of token embeddings correspond to tokens of the natural language text documents in the corpus. Accordingly, an embodiment provides a method for computing a set of token embeddings.

[0010] In another embodiment, the tokens comprise parts of words of a natural language text document. Accordingly, embodiments provide a method for computing token embeddings from parts of words of a natural language text document.

[0011] In another embodiment, the token embedding comprises a multi-dimensional numerical representation of the token. Accordingly, embodiments provide a method for computing a token embedding comprising a multi-dimensional numerical representation.

[0012] In another embodiment, the token embedding comprises a combination of a multidimensional numeric representation of the token, a multidimensional numeric representation of the token's location within the natural language text document, and a multidimensional numeric representation of the segment of the natural language text document in which the token is located. Accordingly, embodiments provide a method for computing a token embedding comprising a multidimensional numeric representation.

[0013] In another embodiment, the correlation embedding portion computes a set of correlation embeddings, and the correlation embeddings in the set of correlation embeddings include multidimensional numerical representations of row and column intersections. Thus, an embodiment provides a method for computing a set of correlation embeddings.

[0014] In another embodiment, the token self-attention portion adjusts the input token embeddings according to a set of token attention weights, where the token attention weights in the set of token attention weights correspond to relationships in the natural language text document between two tokens, and the set of token attention weights are calculated during training. Thus, an embodiment provides a method for adjusting the input token embeddings according to a set of token attention weights.

[0015] In another embodiment, the metadata self-attention portion adjusts the input correlation embedding according to a set of metadata attention weights, where the set of metadata attention weights is calculated during training. Thus, an embodiment provides a method for adjusting the input correlation embedding according to a set of metadata attention weights.

[0016] In another embodiment, the fusion portion combines the outputs of the token self-attention portion and the metadata self-attention portion. Accordingly, embodiments provide further details of the fusion portion.

[0017] In another embodiment, the training comprises: initializing a set of parameters for the token embedding portion to a base set of token embedding parameters; initializing a set of parameters for the token self-attention portion to a base set of token self-attention parameters; performing a first training of the encoder model to generate a partially trained encoder model, the first training including adjusting a set of parameters for the correlation embedding portion and a set of parameters for the metadata self-attention portion, while the set of parameters for the token embedding portion is set to a base set of token embedding parameters and the set of parameters for the token self-attention portion is set to a base set of token self-attention parameters; and performing second training to generate a trained encoder model, the second training including adjusting a set of parameters of the encoding model; Thus, embodiments provide further details of the training of the encoder model.

[0018] In another embodiment, the encoder model further includes a first decoder layer, the first decoder layer including a decoder token self-attention portion, a decoder metadata self-attention portion, a decoder fusion portion, and a decoder attention portion, and the training includes adjusting a set of parameters of the first decoder layer. Accordingly, the embodiment provides further details of the first decoder layer.

[0019] In another embodiment, the decoder attention portion adjusts the output of the encoder layer according to a set of attention weights, which are calculated during training. Accordingly, the embodiment provides further details of the decoder attention portion.

[0020] Embodiments include a computer-usable program product that includes one or more computer-readable storage devices and program instructions stored on at least one of the one or more storage devices.

[0021] An embodiment includes a computer system including one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, where program instructions are stored in at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories. [Brief explanation of the drawings]

[0022] The particular novel features which are believed to be characteristic of the invention are set forth in the appended claims. However, the invention itself, together with its preferred mode of use, further objects and advantages, will best be understood by reference to the following detailed description of illustrative embodiments taken in conjunction with the accompanying drawings.

[0023] [Figure 1] 1 depicts a block diagram of a network of data processing systems in which illustrative embodiments may be implemented;

[0024] [Figure 2] 1 illustrates a block diagram of a data processing system in which illustrative embodiments may be implemented;

[0025] [Figure 3]1 illustrates a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata, according to an exemplary embodiment.

[0026] [Figure 4] 1 illustrates a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata, according to an exemplary embodiment.

[0027] [Figure 5] 1 illustrates a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata according to an exemplary embodiment.

[0028] [Figure 6] 1 illustrates a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata, according to an exemplary embodiment.

[0029] [Figure 7] 1 illustrates a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata, according to an exemplary embodiment.

[0030] [Figure 8] 10 illustrates an example of encoding by a transformer incorporating metadata, according to an exemplary embodiment.

[0031] [Figure 9] 10 illustrates a continuation of the encoding example with a transformer incorporating metadata, according to an exemplary embodiment.

[0032] [Figure 10] 10 illustrates a continuation of the encoding example with a transformer incorporating metadata, according to an exemplary embodiment.

[0033] [Figure 11]10 illustrates a continuation of the encoding example with a transformer incorporating metadata, according to an exemplary embodiment.

[0034] [Figure 12] 10 illustrates a continuation of the encoding example with a transformer incorporating metadata, according to an exemplary embodiment.

[0035] [Figure 13] 1 illustrates a flowchart of an exemplary process for encoding with a transformer incorporating metadata, according to an exemplary embodiment.

[0036] [Figure 14] 1 illustrates a cloud computing environment in accordance with an embodiment of the present invention.

[0037] [Figure 15] 1 illustrates abstraction model layers according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0038] The exemplary embodiments also recognize that there is an unmet need for a Transformer-based autoencoder architecture that includes parameters that are specific to a model of attention due to metadata, and that it is possible to use a metadata encoding scheme that does not require encoding specific portions or types of metadata into specific embeddings. The exemplary embodiments recognize that currently available tools or solutions do not address or provide an adequate solution to these needs. The exemplary embodiments used to illustrate the invention generally address and solve the problems described above, as well as other problems associated with Transformer-based encoding that incorporates metadata.

[0039] Embodiments may be implemented as a software application. An application implementing embodiments may be configured as a modification of an existing natural language text processing system or non-text data processing system, as a separate application operating in conjunction with an existing natural language text processing system or non-text data processing system, a standalone application, or some combination thereof.

[0040] Specifically, some example embodiments provide a method for constructing a correlation matrix from metadata or non-text data of a corpus of natural language text documents, and training an encoding model to compute embeddings corresponding to portions of the documents or non-text data in the corpus and the correlation matrix to generate a trained encoding model.

[0041] Embodiments construct a correlation matrix from metadata or non-text data of a corpus of natural language text documents. A correlation matrix is ​​a two-dimensional matrix that stores relationships between instances of a certain type of metadata. In particular, rows and columns of the correlation matrix represent instances of a certain type of metadata, and intersections of rows and columns store data on relationships between two instances of a certain type of metadata.

[0042] In one non-limiting example of a correlation matrix coding scheme, if there is a relationship between two instances of a certain type of metadata, an entry in the correlation matrix is ​​set to 1, and if there is no relationship, it is set to zero. For example, in a conversation transcript, the metadata may indicate which part was spoken or transcribed by which participant. Thus, the corresponding rows and columns of the correlation matrix may indicate the participants. If there is a relationship between the participant indicated by the row of the entry and the participant indicated by the column of the entry, the entry in the correlation matrix is ​​set to 1, and if there is no relationship, it is set to zero.

[0043] In another non-limiting example of a correlation matrix coding scheme, an entry in a correlation matrix is ​​set to the distance between two instances of a certain type of metadata in the metadata. For example, in a conversation transcript, the metadata may indicate which part was spoken or transcribed by which participant. Thus, the corresponding rows and columns of the correlation matrix may indicate the participants, and the entry in the correlation matrix may be set to the number of turns in the conversation between the participant indicated by the entry's row and the participant indicated by the entry's column. In another example of a conversation transcript, the metadata may record timestamps corresponding to each change or turn of a participant in the conversation. Thus, the corresponding rows and columns of the correlation matrix may indicate a particular turn, and the entry in the correlation matrix may be set to the amount of elapsed time between the turn indicated by the entry's row and the turn indicated by the entry's column. Alternatively, the corresponding rows and columns of the correlation matrix may indicate a particular turn, and the entry in the correlation matrix may be set to the number of turns between the turn indicated by the entry's row and the turn indicated by the entry's column.

[0044] In another non-limiting example, a document or conversation transcript is represented by a tree structure described by metadata. The rows and columns of a corresponding correlation matrix may indicate a particular node in the tree structure. An entry in the correlation matrix may be set to the number of tree links between the node indicated by the entry's row and the node indicated by the entry's column. In a variant, an entry in the correlation matrix may be set to the positive distance between node i and node j if node i is a predecessor of node j in the tree, the negative distance between node i and node j if node j is a predecessor of node i in the tree, or the sum of the tree depth and the distance between node i and node j otherwise.

[0045] Other non-limiting examples of encoding metadata relationships using correlation matrices in the manner described herein to indicate relative distances between instances of metadata include relationships between people, scene locations, or other interactions within text, audio, or video content, such as transcripts, scripts, or movies; relationships between entities represented in the content; and relationships between elements of sparse trees, knowledge graphs, hyperlinks, or other structured information relationships.

[0046] An embodiment includes a Transformer model. The model receives as input a sequence of tokens and a correlation matrix and generates embeddings, also called encodings, corresponding to the input tokens, adjusted according to a context including the correlation matrix. When processing natural language text, a token is a word, part of a word, or other portion of a natural language text sequence, such as a sentence in a document. The input sequence is typically a tokenized sentence or other tokenized portion of text, but need not be a grammatically correct sentence. When processing non-text data, a token is a portion of non-text data, such as a single frame of video or a predetermined length of video data. The model includes one or more encoder layers connected in series. In one embodiment, the last encoder layer is followed by an optional task-specific layer that outputs predictions, such as text classification. The model also includes an embedding portion that generates embeddings from the model inputs and passes the generated embeddings to the first encoder layer for adjustment.

[0047] In particular, the embedding portion receives as input a sequence of tokens. The embedding portion is configured in currently available neural network configurations to generate token embeddings that encode the tokens. The embedding portion is also optionally configured in currently available neural network configurations to generate positional embeddings that encode the location of the token within a portion or segment of text, and segmentation embeddings that encode which segment of text the token is located in. If positional and segmentation embeddings are generated, the embedding portion is configured in currently available neural network configurations to combine the token, position, and segmentation embeddings into a token embedding. In one embodiment, the token, position, and segmentation embeddings are combined by adding them together. In another embodiment, the token, position, and segmentation embeddings are combined using a weighted sum, where the weights are model parameters determined during model training.

[0048] The embedding portion also receives a correlation matrix as input. The embedding portion generates a set of correlation embeddings that each encode an entry in the correlation matrix. One embodiment uses a projection matrix initialized to a starting configuration (e.g., a set of pseudo-randomly selected values) and tuned model training to generate the set of correlation embeddings. Another embodiment uses a set of heuristics to generate the set of correlation embeddings. Other techniques for generating the set of correlation embeddings are possible and are contemplated within the scope of the exemplary embodiments.

[0049] In one embodiment, each encoder layer is identically configured and includes a token self-attention portion, one or more metadata self-attention portions, and fusion, feedforward, and optional summation and normalization portions. In another embodiment, each encoder layer is not identically configured. Instead, some encoder layers do not include one or more metadata self-attention portions or fusion portions. In one embodiment, only the last six encoder layers include metadata self-attention portions and fusion portions, speeding up model training time compared to an embodiment in which each encoding layer is identically configured.

[0050] In one currently known technique, the token self-attention part is composed of three weight matrices: the query weight W Q , key weight W K , and value weight W V In particular, for each token i, the input embedding x i is multiplied by each of the weight matrices to obtain the query vector q i =x i W Q , key vector k i =x i W K , and the value vector v i =x i W V Then, A i,j =q i k j , q i and k j is the dot product between A i,j is the attention weight from token i to token j. The attention weight is divided by the square root of the dimension of the key vector (to stabilize the gradient between matrix values ​​during training) and passed to softmax, which normalizes the weights to sum to 1. Thus, the token self-attention part is the attention weight A i,j Obtain context attention from token i to token j according to

[0051] The metadata self-attention part is the metadata attention weight A meta It computes the set of correlation embeddings S meta If there is more than one correlation embedding input set, each feeds into a separate metadata self-attention portion. meta ij =(q i +B)S metaT ij , where B is a learnable bias term and S metaT ij is S meta In another embodiment, A meta ij =(q i +B)S metaT ij +S meta ij (k j +B') T , where B and B' are learnable bias terms, and (k j +B') T is (k j +B'). Another embodiment uses a neural network to generate the correlation embedding S meta The set of,K,is combined with,K,, a matrix of the set of key vectors.,Then, a dot product is calculated between the combined result,and,Q,, a matrix of the set of query vectors, and the result,is,BS,. metaT ij The correlation embedding S meta Metadata attention weight A meta Other techniques for calculating Ĥ are possible and are contemplated within the scope of the exemplary embodiments.

[0052] The fusion portion combines the results of the token self-attention and one or more metadata self-attention portions. One embodiment uses a neural network to combine the results of the token self-attention and the metadata self-attention portion. Another embodiment combines A and A metaOther techniques for combining the token self-attention results and the metadata self-attention portion are possible and are contemplated within the scope of the exemplary embodiments.

[0053] In one embodiment, the summing and normalizing section sums the output from the fusion section and the input to the token self-attention section, then normalizes the result and passes the normalized result to the feedforward section. In another embodiment, there is no summing and normalizing section, and the output of the fusion section is passed directly to the feedforward section. In one embodiment, the feedforward section is implemented using a linear layer, with weights described by a weight matrix. The weights are adjusted during model training. In another embodiment, each metadata self-attention section is followed by a summing and normalizing section that combines the input and output of that particular metadata self-attention section, and there is no summing and normalizing section that combines the output from the fusion section and the input to the token self-attention section.

[0054] In one embodiment, the output of the feedforward section and the input to the feedforward section are added together and normalized, and the result is passed to the input section of the second encoder layer for further processing. In another embodiment, the output of the feedforward section is passed to the input section of the second encoder layer for further processing.

[0055] In an embodiment, the Transformer model includes one or more decoder layers connected in series, each receiving as input a sequence of token embeddings and a set of correlation embeddings generated by the embedding portion, and each decoder layer also receiving as input the output from the last encoder layer in the series of encoder layers.

[0056] In one embodiment, each decoder layer is identically configured and includes a token self-attention portion, one or more metadata self-attention portions, and fusion, attention, feedforward, and optional summation and normalization portions. In another embodiment, each decoder layer is not identically configured. Instead, some decoder layers do not include one or more metadata self-attention portions or fusion portions.

[0057] The token self-attention section, metadata self-attention section, and fusion section of the decoder layer operate similarly to those of the encoder layer. In one embodiment, the summing and normalizing section sums the output from the fusion section and the input to the token self-attention section, then normalizes the result and passes the normalized result to the attention section. In another embodiment, the summing and normalizing section does not exist, and the output of the fusion section is passed directly to the attention section.

[0058] The attention section combines the output from the fusion section or the summation and normalization section with the output from the last encoder layer in the set of encoder layers. In one embodiment, the output from the last encoder layer is in the form of a key vector and a value vector, and the output from the fusion section or the summation and normalization section is in the form of a query vector. The attention section combines the inputs using currently known techniques.

[0059] In one embodiment, the summing and normalizing section sums the output from the attention section and the input to the attention section, then normalizes the result, and passes the normalized result to the feedforward section. In another embodiment, the summing and normalizing section is not present, and the output of the attention section is passed directly to the feedforward section. In one embodiment, the feedforward section is implemented using a linear layer, with weights described by a weight matrix. The weights are adjusted during model training. In one embodiment, the output of the feedforward section and the input to the feedforward section are summed together and normalized, and the result is passed to the input section of the second decoder layer for further processing. In another embodiment, the output of the feedforward section is passed to the input section of the second decoder layer for further processing.

[0060] An embodiment trains a Transformer model. During training, the embodiment learns to set model parameters, including query weights, key weights, value weights, attention and metadata attention weights, one or more learnable bias terms, and parameters of the embedding portion, to values ​​that produce output training data labeled as accurate. One embodiment trains the entire model at once by initializing the model parameters to a base set of parameters (e.g., a set of pseudo-randomly selected parameters), and trains the model by tuning the set of parameters using a set of training data.

[0061] Another embodiment trains the model in multiple stages. The embodiment sets the parameters of the token embedding and token self-attention portions to a base set of parameters. As a non-limiting example, the base set of parameters is a pseudo-randomly selected set of parameters. As another non-limiting example, the base set of parameters is from a trained token embedding and token self-attention portion that has already been trained using currently available techniques. Using the trained token embedding and token self-attention portion as the base set of parameters saves training time by starting from a partially trained state. In a first training stage, the embodiment trains the model by holding the parameters of the token embedding and token self-attention portions constant and using a set of training data to adjust the parameters of the correlation embedding, one or more metadata self-attention portions, and the fusion portion. Then, in a second training stage, the embodiment trains the entire model by adjusting one or more of the entire set of parameters.

[0062] The model training data can be adjusted based on the final use of the model's output. Some non-limiting examples of final uses of the model's output are masked language modeling (attempting to predict what word should replace the mask token using context words around the mask token or whitespace to be filled), masked context regression (masking randomly selected utterances and predicting encoding vectors for the masked utterances), distributed utterance order ranking (organizing randomly shuffled utterances of a conversation into a coherent dialog context), and utterance or sentence order selection. For example, for a general-purpose model used on both plain text (no metadata) and natural language text with accompanying metadata, the training data should include both plain text and text of different genres with accompanying metadata. However, for a model used on data incorporating text of one genre with accompanying metadata, e.g., a model used only for dialog-related tasks, the training data only needs to include dialog-specific text with accompanying metadata, such as conversational turn, speaker, or timestamp data.

[0063] Once a model has been trained, either generally or for a particular genre, embodiments optionally further train the model to perform specific downstream tasks. Some non-limiting examples of downstream tasks for documents are masked sentence selection in structured documents, question answering and answer retrieval from documents, and structured document summarization. Some non-limiting examples of downstream tasks for dialogs are next utterance selection, utterance order selection (ordering utterances or other parts of a dialog), masked utterance selection, next utterance generation, and traditional question answering.

[0064] The metadata-incorporating transformer encoding scheme described herein is not available in currently available methods in the art of natural language text processing-related work. The method of the embodiments described herein, when implemented to run on a device or data processing system, comprises a significant advancement in the capabilities of that device or data processing system in constructing a correlation matrix from metadata or non-text data of a corpus of natural language text documents, and training an encoding model to compute embeddings corresponding to the tokens and correlation matrices of the natural language text documents in the corpus to generate a trained encoding model.

[0065] Exemplary embodiments are described with respect to particular types of tokens, embeddings, correlations, matrices, encoding portions, attention portions, fusion portions, neural networks, adjustments, devices, data processing systems, environments, components, and applications, by way of example only. Any particular manifestations of these and other similar artifacts are not intended as limitations of the invention. Any suitable manifestations of these and other similar artifacts may be selected within the scope of exemplary embodiments.

[0066] Furthermore, exemplary embodiments may be implemented with respect to any type of data, data source, or access to a data source via a data network. Within the scope of the present invention, any type of data storage device may provide data to an embodiment of the present invention, either locally at a data processing system or via a data network. Where an embodiment is described using a mobile device, within the scope of exemplary embodiments, any type of data storage device suitable for use with a mobile device may provide data to such an embodiment, either locally at the mobile device or via a data network.

[0067] The exemplary embodiments are described using specific code, designs, architectures, protocols, layouts, diagrams, and tools, by way of example only, and are not limited to the exemplary embodiments. Furthermore, the exemplary embodiments are described in some instances using specific software, tools, and data processing environments, by way of example only, for clarity of explanation. The exemplary embodiments may be used with other equivalent or similar purpose structures, systems, applications, or architectures. For example, other equivalent mobile devices, structures, systems, applications, or architectures therefor may be used with such embodiments of the present invention within the scope of the present invention. The exemplary embodiments may be implemented in hardware, software, or a combination thereof.

[0068] Examples in this disclosure are used for clarity of explanation only and are not intended to limit the exemplary embodiments. Additional data, operations, actions, tasks, activities, and operations are contemplated by this disclosure and are contemplated within the scope of the exemplary embodiments.

[0069] Any advantages listed herein are examples only and are not intended to limit the exemplary embodiments. Additional or different advantages may be realized depending on the particular exemplary embodiment. Furthermore, a particular exemplary embodiment may have some, all, or none of the advantages listed above.

[0070] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0071] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processes, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0072] The characteristics are as follows:

[0073] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the service provider.

[0074] Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (eg, cell phones, laptops, and PDAs).

[0075] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence exists in that consumers generally have no control or knowledge over the exact location of the provided resources, although they may be able to specify location at a higher level of abstraction (e.g., country, state, or data center).

[0076] Rapid Elasticity: This capacity can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any time.

[0077] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0078] The service model is as follows:

[0079] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0080] Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, on a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration.

[0081] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources, upon which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0082] The deployment model is as follows:

[0083] Private Cloud: Cloud infrastructure operates solely for an organization. It may be managed by that organization or a third party and may exist on-premise or off-premise.

[0084] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy and compliance considerations). It may be managed by those organizations or a third party and may exist on-premises or off-premises.

[0085] Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0086] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standard or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0087] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0088] Referring now to the figures, and in particular to Figures 1 and 2, these figures are exemplary diagrams of data processing environments in which illustrative embodiments may be implemented. Figures 1 and 2 are examples only and are not intended to assert or imply any limitation with respect to the environments in which different embodiments may be implemented. Particular implementations may make many modifications to the depicted environments based on the following description.

[0089] 1 illustrates a block diagram of a network of data processing systems in which exemplary embodiments may be implemented. Data processing environment 100 is a network of computers in which exemplary embodiments may be implemented. Data processing environment 100 includes network 102. Network 102 is the medium used to provide communications links between various devices and computers connected together within data processing environment 100. Network 102 may include connections such as wired or wireless communications links, or fiber optic cables.

[0090] Client or server are only exemplary roles for particular data processing systems connected to network 102 and are not intended to exclude other configurations or roles of these data processing systems. Server 104 and server 106 are coupled to storage unit 108 and network 102. Software applications may execute on any computer within data processing environment 100. Clients 110, 112, and 114 are also coupled to network 102. A data processing system, such as server 104 or 106 or client 110, 112, or 114, may contain data and may have software applications or software tools executing thereon.

[0091] 1 illustrates certain components that may be used in an exemplary implementation of one embodiment. For example, servers 104 and 106 and clients 110, 112, and 114 are illustrated as servers and clients by way of example only and without implying any limitation to a client-server architecture. As another example, one embodiment may be distributed across several data processing systems and data networks as shown, while another embodiment may be implemented on a single data processing system within the scope of an exemplary embodiment. Data processing systems 104, 106, 110, 112, and 114 also represent exemplary nodes in clusters, partitions, and other configurations suitable for implementing an embodiment.

[0092] Device 132 is one example of a device described herein. For example, device 132 may take the form of a smartphone, a tablet computer, a laptop computer, a client 110 in stationary or portable form, a wearable computing device, or any other suitable device. Any software application described in FIG. 1 as executing on another data processing system may be configured to execute on device 132 in a similar manner. Any data or information stored or generated in another data processing system in FIG. 1 may be configured to be stored or generated on device 132 in a similar manner.

[0093] The application 105 implements one embodiment described herein. The application 105 executes on any of the servers 104 and 106, the clients 110, 112 and 114, and the device 132.

[0094] Servers 104 and 106, storage unit 108, clients 110, 112 and 114, and device 132 may be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connections. Clients 110, 112 and 114 may be, for example, personal computers or network computers.

[0095] In the shown example, server 104 may provide data such as boot files, operating system images, and applications to clients 110, 112, and 114. Clients 110, 112, and 114 may be clients of server 104 in this example. Clients 110, 112, 114, or some combination thereof, may include their own data, boot files, operating system images, and applications. Data processing environment 100 may include additional servers, clients, and other devices not shown.

[0096] In the depicted example, data processing environment 100 may be the Internet. Network 102 may represent a collection of networks and gateways that communicate with each other using Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols. At the core of the Internet is a backbone of data communication links between major nodes or host computers, including thousands of commercial, government, educational, and other computer systems, that route data and messages. Of course, data processing environment 100 may also be implemented as numerous different types of networks, such as, for example, an intranet, a local area network (LAN), or a wide area network (WAN). Figure 1 is intended as an example, and not as an architectural limitation for different illustrative embodiments.

[0097] Among other uses, data processing environment 100 may be used to implement a client-server environment in which exemplary embodiments may be implemented. A client-server environment allows software applications and data to be distributed across a network such that applications function using interactivity between client and server data processing systems. Data processing environment 100 may also employ a service-oriented architecture in which interoperable software components distributed across a network are packaged as a coherent business application. Data processing environment 100 may take the form of a cloud, using a cloud computing model of service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the provider of the service.

[0098] Referring to Figure 2, this figure shows a block diagram of a data processing system in which exemplary embodiments may be implemented. Data processing system 200 is an example of a computer, such as servers 104 and 106 or clients 110, 112, and 114 in Figure 1, or another type of device in which computer-usable program code or instructions implementing processes may be located for exemplary embodiments.

[0099] Data processing system 200 also represents a data processing system or configuration therein, such as data processing system 132 of FIG. 1 , in which computer-usable program code or instructions implementing the processes of the exemplary embodiments may be located. Data processing system 200 is described as a computer by way of example only, and not limitation. Implementations in the form of other devices, such as device 132 of FIG. 1 , may modify data processing system 200, for example, by adding a touch interface, or even omit certain illustrated components from data processing system 200, without departing from the general description of the operation and functionality of data processing system 200 described herein.

[0100] In the illustrated example, data processing system 200 uses a hub architecture including a northbridge and memory controller hub (NB / MCH) 202 and a southbridge and input / output (I / O) controller hub (SB / ICH) 204. A processing unit 206, a main memory 208, and a graphics processor 210 are coupled to northbridge and memory controller hub (NB / MCH) 202. Processing unit 206 may include one or more processors and may be implemented using one or more heterogeneous processor systems. Processing unit 206 may be a multi-core processor. Graphics processor 210 may be coupled to NB / MCH 202 through an accelerated graphics port (AGP) in certain implementations.

[0101] In the illustrated example, a local area network (LAN) adapter 212 is coupled to a southbridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a universal serial bus (USB) and other ports 232, and PCI / PCIe devices 234 are coupled to the southbridge and I / O controller hub 204 through a bus 238. A hard disk drive (HDD) or solid state drive (SSD) 226 and a CD-ROM 230 are coupled to the southbridge and I / O controller hub 204 through a bus 240. The PCI / PCIe devices 234 may include, for example, an Ethernet adapter, an add-in card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. The ROM 224 may be, for example, a flash binary input / output system (BIOS). The hard disk drive 226 and CD-ROM 230 may use, for example, an integrated drive electronics (IDE), a serial advanced technology attachment (SATA) interface, or variations thereof, such as external SATA (eSATA) and micro SATA (mSATA). A super I / O (SIO) device 236 may be coupled to the southbridge and I / O controller hub (SB / ICH) 204 through a bus 238.

[0102] Memories such as main memory 208, ROM 224, or flash memory (not shown) are some examples of computer-usable storage devices. Hard disk drives or solid state drives 226, CD-ROM 230, and other similarly usable devices are some examples of computer-usable storage devices that include computer-usable storage media.

[0103] An operating system executes on processing unit 206. The operating system coordinates and provides control of various components within data processing system 200 of FIG. 2. The operating system may be a commercially available operating system for any type of computing platform, including, but not limited to, server systems, personal computers, and mobile devices. An object-oriented or other type of programming system may operate in conjunction with the operating system and may provide calls to the operating system from programs or applications executing on data processing system 200.

[0104] 1 , instructions for the operating system, the object-oriented programming system, and applications or programs, such as application 105 of FIG. 1 , may be located on storage devices, for example in the form of code 226A on hard disk drive 226, and loaded into at least one of one or more memories, for example, main memory 208, for execution by processing unit 206. The processes of the exemplary embodiments may be performed by processing unit 206 using computer-implemented instructions, which may be located in a memory, such as main memory 208, read-only memory 224, or one or more peripheral devices.

[0105] Furthermore, in one case, code 226A may be downloaded from remote system 201B over network 201A, while similar code 201C is stored in storage device 201D. In another case, code 226A may be downloaded to remote system 201B over network 201A, while the downloaded code 201C is stored in storage device 201D.

[0106] The hardware in Figures 1-2 may vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives, may be used in addition to or in place of the hardware depicted in Figures 1-2. The processes of the illustrative embodiments may also be applied to multiprocessor data processing systems.

[0107] In some illustrative examples, data processing system 200 may be a personal digital assistant (PDA) typically configured with flash memory to provide non-volatile memory for storing operating system files or user-generated data, or a combination thereof. The bus system may include one or more buses, e.g., a system bus, an I / O bus, and a PCI bus. Of course, the bus system may be implemented using any type of communications fabric or architecture that provides for a transfer of data between different components or devices coupled to the fabric or architecture.

[0108] The communications unit may include one or more devices, such as a modem or network adapter, used to transmit and receive data. The memory may be, for example, main memory 208 or a cache, such as found in the northbridge and memory controller hub 202. The processing unit may include one or more processors or CPUs.

[0109] 1-2 and above-described examples are not meant to imply architectural limitations. For example, data processing system 200 may also be a tablet computer, a laptop computer, or a telephone device, in addition to taking the form of a mobile or wearable device.

[0110] When a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates in the manner of data processing system 200 using virtualized artifacts of some or all of the components shown in data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 appears as a virtualized instance of all or some of the hardware processing unit 206 available in the host data processing system, main memory 208 appears as a virtualized instance of all or some of the main memory 208 that may be available in the host data processing system, and disk 226 appears as a virtualized instance of all or some of the disk 226 that may be available in the host data processing system. The host data processing system in such a case is represented by data processing system 200.

[0111] Referring to Figure 3, this figure shows a block diagram of an exemplary configuration for encoding with a transformer that incorporates metadata, according to an exemplary embodiment. Application 300 is an example of application 105 of Figure 1, and runs on any of servers 104 and 106, clients 110, 112 and 114, and device 132 of Figure 1.

[0112] The metadata correlation module 310 constructs a correlation matrix from the metadata of a corpus of natural language text documents. The rows and columns of the correlation matrix represent instances of a certain type of metadata, and the intersections of the rows and columns store data on the relationship between two instances of the certain type of metadata. In one exemplary correlation matrix coding scheme, an entry in the correlation matrix is ​​set to 1 if there is a relationship between two instances of a certain type of metadata, and set to 0 if there is no relationship. In another exemplary correlation matrix coding scheme, an entry in the correlation matrix is ​​set to the distance between two instances of a certain type of metadata within the metadata. In another exemplary correlation matrix coding scheme, a document or conversation transcript is represented by a tree structure described by the metadata, and an entry in the correlation matrix may be set based on the distance between the node indicated by the entry's row and the node indicated by the entry's column. Other examples of encoding metadata relationships using correlation matrices in the manner described herein to indicate relative distances between instances of metadata include relationships between people, scene locations, or other interactions within text, audio, or video content, such as transcripts, scripts, or movies; relationships between entities represented in the content; and relationships between elements of sparse trees, knowledge graphs, hyperlinks, or other structured information relationships.

[0113] The encoder module 320 implements the encoder portion of the Transformer model. The module 320 receives as input a sequence of tokens and a correlation matrix and generates embeddings corresponding to the input tokens adjusted according to a context that includes the correlation matrix. Further details of the module 320 are provided with reference to Figures 4 and 5.

[0114] Decoder module 325 implements the optional decoder portion of the Transformer model, which includes one or more decoder layers connected in series. Each decoder layer receives as input a sequence of token embeddings and a set of correlation embeddings generated by encoder module 320. Each decoder layer also receives as input the output from the last encoder layer in the series of encoder layers in module 320. Further details of module 325 are provided with reference to Figures 6 and 7.

[0115] The training module 330 trains the Transformer model. During training, the module 330 sets model parameters, including query weights, key weights, value weights, attention and metadata attention weights in the encoder and decoder portions, one or more learnable bias terms, and parameters in the embedding portion, to values ​​that produce output training data labeled as accurate. One implementation of the module 330 trains the entire model at once by initializing the model parameters to a base set of parameters (e.g., a pseudo-randomly selected set of parameters), and then trains the model by adjusting the set of parameters using a set of training data.

[0116] Another implementation of module 330 trains the model in multiple stages. Module 330 sets parameters for the token embedding and token self-attention portions to a base set of parameters. As a non-limiting example, the base set of parameters is a pseudo-randomly selected set of parameters. As another non-limiting example, the base set of parameters is that of a trained token embedding and token self-attention portion that has already been trained using currently available techniques. In a first training stage, module 330 trains the model by holding constant the parameters for the token embedding and token self-attention portions and adjusting the parameters for the correlation embedding, one or more metadata self-attention portions, the attention portion, and the fusion portion using a set of training data. Then, in a second training stage, module 330 trains the entire model by adjusting one or more of the entire set of parameters.

[0117] 4, a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata is shown, according to an exemplary embodiment. The encoder module 320 is identical to the encoder module 320 in FIG.

[0118] The encoder module 320 includes a set of encoder layers including an encoder layer 410, an encoder layer 420, and an encoder layer 430. The encoder layers are connected sequentially, and there may be additional encoder layers between encoder layers 420 and 430.

[0119] 5, a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata is shown, according to an exemplary embodiment. The encoder layer 410 is identical to the encoder layer 410 in FIG.

[0120] Token embedding module 510, configured in a currently available neural network configuration, generates token embeddings that encode the tokens. Module 510 is also optionally configured in a currently available neural network configuration to generate positional embeddings that encode the location of the token within a portion or segment of text, and segmentation embeddings that encode which segment of text the token is located in. If positional and segmentation embeddings are generated, module 510 is configured in a currently available neural network configuration to combine the token, position, and segmentation embeddings into the token embedding. In one implementation of module 510, the token, position, and segmentation embeddings are combined by adding them together.

[0121] The metadata embedding module 520, configured in a currently available neural network configuration, generates a set of correlation embeddings that each encode an entry in the correlation matrix.

[0122] The token self-attention module 530 calculates three weight matrices: the query weight W Q , key weight W K , and value weight W V In particular, for each token i, the input embedding x i is multiplied by each of the weight matrices to obtain the query vector q i =x i W Q , key vector k i =x i W K , and the value vector v i =x i W V Then, A i,j =q i k j , q i and k j is the dot product between A i,jis the attention weight from token i to token j. The attention weight is divided by the square root of the dimension of the key vector (to stabilize the gradient between matrix values ​​during training) and passed to a softmax which normalizes the weights to sum to 1.

[0123] The metadata self-attention module 540 calculates the metadata attention weight A meta It computes the set of correlation embeddings S meta If there is more than one correlation embedding input set, each feeds into a separate metadata self-attention portion. meta ij =(q i +B)S metaT ij , where B is a learnable bias term and S metaT ij is S meta In another embodiment, A meta ij =(q i +B)S metaT ij +S meta ij (k j +B') T , where B and B' are learnable bias terms, and (k j +B') T is (k j +B'). Another embodiment uses a neural network to generate the correlation embedding S meta The set of,K,is combined with,K,, a matrix of the set of key vectors.,Then, a dot product is calculated between the combined result,and,Q,, a matrix of the set of query vectors, and the result,is,BS,. metaT ij is added together.

[0124] Fusion module 550 combines the results of modules 530 and 540. One implementation of module 550 uses a neural network to combine the results of the token self-attention and metadata self-attention portions. Another implementation of module 550 combines the results of A and A meta Compute the element-wise product of .

[0125] 6, which shows a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata, according to an exemplary embodiment. The decoder module 325 is identical to the encoder module 325 in FIG.

[0126] Decoder module 325 includes a set of encoder layers, including decoder layer 610, decoder layer 620, and decoder layer 630. The decoder layers are connected sequentially, and there may be additional decoder layers between encoder layers 620 and 630. Each decoder layer receives as input the sequence of token embeddings and the set of correlation embeddings generated by modules 510 and 520. Each decoder layer also receives as input the output from encoder layer 430, which is the last encoder layer in the series of encoder layers.

[0127] 7, a block diagram of an exemplary configuration for encoding with a transformer incorporating metadata is shown, according to an exemplary embodiment. The decoder layer 610 is identical to the decoder layer 610 in FIG.

[0128] As shown, the decoder layer 610 includes a token self-attention module 710, one or more metadata self-attention modules 720, a fusion module 730, and an attention module 740. The token self-attention module 710, the metadata self-attention module 720, and the fusion module 730 operate similarly to the encoder layer 410. In one implementation of layer 610, an add and normalize section adds the output from the fusion module 730 and the input to the token self-attention module 710, then normalizes the result and passes the normalized result to the attention module 740. In another implementation of layer 610, the add and normalize section is not present, and the output of the fusion module 730 is passed directly to the attention module 740.

[0129] The attention module 740 combines the output from the fusion module 730 or summation and normalization portion with the output from the last encoder layer in the set of encoder layers. In one implementation of layer 610, the output from the last encoder layer is in the form of a key vector and a query vector, and the output from the fusion portion 740 or summation and normalization portion is in the form of a query vector. The attention module 740 combines the inputs using currently known techniques.

[0130] In one implementation of layer 610, the second summing and normalizing section sums the output from and the input to the attention module 740, then normalizes the result, and passes the normalized result to the feedforward section. In another embodiment, the summing and normalizing section is not present, and the output of the attention module 740 is passed directly to the feedforward section. In one implementation of layer 610, the output of the feedforward section and the input to the feedforward section are added together and normalized, and the result is passed to the input section of the second decoder layer for further processing. In another implementation, the output of the feedforward section is passed to the input section of the second decoder layer for further processing.

[0131] Referring to Figure 8, this figure shows an example of encoding by a transformer incorporating metadata according to an exemplary embodiment. The example can be performed using application 300 in Figure 3. Metadata correlation module 310 is identical to metadata correlation module 310 in Figure 3.

[0132] In particular, Figure 8 illustrates the construction of a correlation matrix 812 from the metadata of a conversation 810. In matrix 812, an entry in the correlation matrix is ​​set to one if there is a relationship between two instances of a certain type of metadata, and to zero if there is no relationship. Thus, since the metadata of a conversation 810 indicates which parts were spoken or transcribed by which participants, the rows and columns of matrix 810 indicate the participants. An entry in correlation matrix 810 is set to one if there is a relationship between the participant indicated by the entry's row and the participant indicated by the entry's column, and to zero if there is no relationship.

[0133] 8 also illustrates the construction of a correlation matrix 822 from the time-stamped metadata of a conversation 820. In the matrix 822, entries in the correlation matrix 822 are set to the distance between two instances of a certain type of metadata in the metadata. Thus, because the metadata of a conversation 820 records timestamps corresponding to each change or turn of the participants in the conversation, the rows and columns of the correlation matrix 822 indicate particular turns, and entries in the matrix 822 are set to the amount of time elapsed between the turn indicated by the entry's row and the turn indicated by the entry's column.

[0134] Referring to Figure 9, this figure shows a continuation of the example of encoding by a transformer incorporating metadata, according to an exemplary embodiment. Token embedding module 510 and metadata embedding module 520 are identical to token embedding module 510 and metadata embedding module 520 in Figure 5. Matrices 812 and 822 are identical to matrices 812 and 822 in Figure 8.

[0135] The token embedding module 510 receives a sequence of tokens as input. The module 510 generates token embeddings that encode the tokens. The module 510 also generates position embeddings and segmentation or segment embeddings, and combines the token, position, and segmentation embeddings into the token embedding 902.

[0136] The metadata embedding module 520 receives as input the correlation matrices 812 and 822 and generates correlation embeddings 912 and 922, respectively.

[0137] Referring to Figure 10, the figure shows a continuation of the example of encoding by a transformer incorporating metadata, according to an exemplary embodiment. Encoder module 320 is identical to encoder module 320 in Figure 3. Encoder layers 410, 420, and 430 are identical to encoder layers 410, 420, and 430 in Figure 4. Token self-attention module 530, metadata self-attention module 540, and fusion module 550 are identical to token self-attention module 530, metadata self-attention module 540, and fusion module 550 in Figure 5. Token embedding 902 and correlation embedding 912 and 922 are identical to token embedding 902 and correlation embedding 912 and 922 in Figure 9. Note that Figure 10 shows details of only encoder layer 610, not encoder layers 620 and 630.

[0138] The token self-attention module 530 modifies the token embeddings 902 according to a set of attention weights. Each of the metadata self-attention modules 540 modifies the correlation embeddings 912 and 922, respectively, according to a corresponding set of metadata attention weights. The fusion module 550 combines the results of modules 530 and 540. The optional summing and normalizing module 1010 sums the outputs from the fusion module 550 and the embeddings 902, then normalizes the result and passes the normalized result to the feedforward 1030. The optional summing and normalizing module 1040 sums together and normalizes the output of the feedforward 1030 and the input to the feedforward 1030 and passes the result to the encoder layer 420. The final encoder layer in the series, the encoder layer 430, produces the encoder output 1050.

[0139] Referring to Figure 11, this figure shows a continuation of the example of encoding by a transformer incorporating metadata, according to an exemplary embodiment. The decoder module 325 is identical to the decoder module 325 in Figure 3. The decoder layers 610, 620, and 630 are identical to the encoder layers 610, 620, and 630 in Figure 6. The token self-attention module 710, the metadata self-attention module 720, the fusion module 730, and the attention module 740 are identical to the token self-attention module 710, one or more metadata self-attention modules 720, the fusion module 730, and the attention module 740 in Figure 7. The token embedding 902 and the correlation embedding 912 and 922 are identical to the token embedding 902 and the correlation embedding 912 and 922 in Figure 9. The encoder output 1050 is identical to the encoder output 1050 in Figure 10. Note that FIG. 11 shows details of only decoder layer 610, and does not show details of decoder layers 620 and 630.

[0140] The token self-attention module 710 modifies the token embeddings 902 according to a set of attention weights. Each of the metadata self-attention modules 720 modifies the correlation embeddings 912 and 922, respectively, according to a corresponding set of metadata attention weights. The fusion module 730 combines the results of modules 710 and 720. An optional summing and normalization module 1110 sums the outputs from the fusion module 730 and the embeddings 902, then normalizes the result, and passes the normalized result to the attention module 740. The attention module 740 combines the output from the fusion module 740 or the summing and normalization module 1110 with the encoder output 1050. An optional summing and normalization module 1120 sums the output from the attention module 740 and the input to the attention module 740, then normalizes the result, and passes the normalized result to the feedforward 1030. An optional summing and normalizing module 1140 sums and normalizes the output of the feedforward 1130 and the input to the feedforward 1130 together and passes the result to a decoder layer token 1150 .

[0141] 12, this figure shows a continuation of the example of encoding by a transformer incorporating metadata, according to an exemplary embodiment. The training module 330 is identical to the training module 330 in FIG.

[0142] 12 illustrates model training for a particular downstream task. Thus, the training module 330 trains an untrained encoder model 1220 using a dialog training corpus 1210 to generate a trained encoder model 1230, and then further trains the model 1230 using dialog task-specific training data 1240, resulting in a dialog task-trained dialog model 1250. Similarly, the training module 330 trains the untrained encoder model 1220 using a document training corpus 1212 to generate a trained encoder model 1232, and then further trains the model 1232 using document task-specific training data 1242, resulting in a document task-trained dialog model 1252.

[0143] 13, which illustrates a flowchart of an exemplary process for encoding with a transformer incorporating metadata, according to an exemplary embodiment. The process 1300 may be implemented in the application 300 in FIG.

[0144] At block 1302, the application constructs a correlation matrix from metadata or non-text data of a corpus of natural language text documents. At block 1304, the application initializes a set of parameters for the token embedding portion of the encoder model to a base set of token embedding parameters. At block 1306, the application initializes a set of parameters for the token self-attention portion of the model to a base set of token self-attention parameters. At block 1308, the application partially trains the model by adjusting the set of parameters for the correlation embedding portion of the model and the set of parameters for the metadata self-attention portion of the model, while the set of parameters for the token embedding portion is set to the base set of token embedding parameters and the set of parameters for the token self-attention portion is set to the base set of token self-attention parameters. At block 1310, the application trains the partially trained model by adjusting the set of parameters for the model. The application then terminates.

[0145] Referring now to FIG. 14 , an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. They may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described above. This enables the cloud computing environment 50 to provide infrastructure-as-a-service, platform-as-a-service, or software-as-a-service, or combinations thereof, without the need for cloud consumers to maintain resources on local computing devices. It should be understood that the types of computing devices 54A-N shown are for illustrative purposes only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device through any type of network or network-addressable connection (e.g., using a web browser), or both.

[0146] Referring now to Figure 15, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 14) is shown. It should be understood upfront that the components, layers, and functions shown are intended to be illustrative only, and that embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0147] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, reduced instruction set computer (RISC) architecture-based server 62, server 63, blade server 64, storage device 65, and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0148] The virtualization layer 70 provides an abstraction layer from which examples of virtual entities such as virtual servers 71, virtual storage devices 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 can be sourced.

[0149] In one example, management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance arrangements and procurement for cloud computing resources where future requirements are anticipated according to SLAs.

[0150] The workload tier 90 provides examples of functions for which a cloud computing environment may be utilized. Non-limiting examples of workloads and functions that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and application selection based on synthetic system fault generation software 96.

[0151] Thus, computer-implemented methods, systems, or apparatus, and computer program products are provided in exemplary embodiments for encoding by a transformer incorporating metadata, and other related features, functions, or operations. When an embodiment, or portion thereof, is described with respect to a certain type of device, computer-implemented method, system, or apparatus, the computer program product, or portion thereof, is adapted or configured for use with a suitable and equivalent artifact for that type of device.

[0152] When an embodiment is described as being implemented in an application, provision of the application in a software-as-a-service (SaaS) model is contemplated within the scope of the exemplary embodiments. In the SaaS model, the capabilities of an application implementing an embodiment are provided to a user by running the application in a cloud infrastructure. Users can access the application through a thin-client interface, such as a web browser (e.g., web-based email) or other lightweight client application, using a variety of client devices. Users do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage of the cloud infrastructure. In some cases, users may not even manage or control the capabilities of the SaaS application. In some other cases, a SaaS implementation of an application may allow for possible exceptions to limited user-specific application configuration settings.

[0153] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0154] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0155] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0156] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present invention.

[0157] Aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0158] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, whereby the instructions can instruct a computer, programmable data processing apparatus or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0159] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowchart or block diagram, or a combination thereof.

[0160] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions, that implement the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may actually be implemented as a single step, or may be executed simultaneously, substantially simultaneously, partially, or fully overlapping in time, or the blocks may even be executed in the reverse order depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by special-purpose hardware-based systems that perform the specified functions or operations or execute a combination of special-purpose hardware and computer instructions.

Claims

1. 1. A computer-implemented method comprising: constructing a correlation matrix from metadata of a corpus of natural language text documents, wherein row and column intersections in the correlation matrix correspond to relationships between two instances of a certain type of metadata; training an encoder model to compute embeddings corresponding to tokens of natural language text documents in the corpus and the correlation matrix to generate a trained encoder model, the encoder model including a first encoder layer, the first encoder layer including a token embedding portion, a correlation embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, and the training includes adjusting a set of parameters of the encoder model; 1. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the token embedding portion computes a set of token embeddings, and token embeddings in the set of token embeddings correspond to tokens of natural language text documents in the corpus.

3. The computer-implemented method of claim 2 , wherein the tokens comprise portions of words of the natural language text document.

4. The computer-implemented method of claim 2 , wherein the token embedding comprises a multi-dimensional numerical representation of the token.

5. 3. The computer-implemented method of claim 2, wherein the token embedding comprises a combination of a multidimensional numerical representation of the token, a multidimensional numerical representation of the location of the token within the natural language text document, and a multidimensional numerical representation of the segment of the natural language text document in which the token is located.

6. 6. The computer-implemented method of claim 1, wherein the correlation embedding portion computes a set of correlation embeddings, and wherein correlation embeddings in the set of correlation embeddings comprise multidimensional numerical representations of intersections of the rows and columns.

7. 6. The computer-implemented method of claim 1, wherein the token self-attention portion adjusts input token embeddings according to a set of token attention weights, where a token attention weight in the set of token attention weights corresponds to a relationship in the natural language text document between two tokens, and the set of token attention weights is calculated during the training.

8. 6. The computer-implemented method of claim 1, wherein the metadata self-attention portion adjusts input correlation embeddings according to a set of metadata attention weights, the set of metadata attention weights being calculated during the training.

9. The computer-implemented method of claim 1 , wherein the fusion portion combines outputs of the token self-attention portion and the metadata self-attention portion.

10. The training includes: initializing a set of parameters for the token embedding portion to a base set of token embedding parameters; initializing a set of parameters for the token self-attention portion to a base set of token self-attention parameters; performing a first training of the encoder model to generate a partially trained encoder model, the first training including adjusting a set of parameters for the correlation embedding portion and a set of parameters for the metadata self-attention portion, while the set of parameters for the token embedding portion is set to the base set of token embedding parameters and the set of parameters for the token self-attention portion is set to the base set of token self-attention parameters; and performing second training to generate the trained encoder model, the second training including adjusting the set of parameters of the encoder model; The computer-implemented method of claim 1 , comprising:

11. 6. The computer-implemented method of claim 1, wherein the encoder model further includes a first decoder layer, the first decoder layer including a decoder token self-attention portion, a decoder metadata self-attention portion, a decoder fusion portion, and a decoder attention portion, and the training includes adjusting a set of parameters of the first decoder layer.

12. The computer-implemented method of claim 11 , wherein the decoder attention portion adjusts an output of an encoder layer according to a set of attention weights, the set of attention weights being calculated during the training.

13. 1. A computer program for natural language text auto-encoding using a transformer, comprising: constructing a correlation matrix from metadata of a corpus of natural language text documents, wherein row and column intersections in the correlation matrix correspond to relationships between two instances of a certain type of metadata; training an encoder model to compute embeddings corresponding to tokens of natural language text documents in the corpus and the correlation matrix to generate a trained encoder model, the encoder model including a first encoder layer, the first encoder layer including a token embedding portion, a correlation embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, and the training includes adjusting a set of parameters of the encoder model; A computer program that executes

14. 14. The computer program product of claim 13, wherein the token embedding portion computes a set of token embeddings, token embeddings in the set of token embeddings corresponding to tokens of natural language text documents in the corpus.

15. The computer program product of claim 14 , wherein the tokens comprise portions of words of the natural language text document.

16. The computer program product of claim 14 , wherein the token embedding comprises a multi-dimensional numerical representation of the token.

17. 17. The computer program of claim 13, wherein the computer program is stored on at least one of one or more storage media of a local data processing system, and the stored computer program is transferred from a remote data processing system over a network.

18. 17. A computer program according to any one of claims 13 to 16, wherein the computer program is stored on at least one of one or more storage media of a server data processing system, and the stored computer program is downloaded over a network to a remote data processing system for use in a computer-readable storage device associated with the remote data processing system.

19. 17. The computer program of claim 13, wherein the computer program is provided as a service in a cloud environment.

20. 1. A computer system comprising one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, wherein program instructions are stored in at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions comprising: program instructions for constructing a correlation matrix from metadata of a corpus of natural language text documents, wherein row and column intersections in the correlation matrix correspond to relationships between two instances of a type of metadata; program instructions for training an encoder model to compute embeddings corresponding to tokens of natural language text documents in the corpus and the correlation matrix to generate a trained encoder model, the encoder model including a first encoder layer, the first encoder layer including a token embedding portion, a correlation embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, and the training including adjusting a set of parameters of the encoder model; and 2. A computer system comprising:

Images