Method and system for transformer-based encoding with metadata

Abstract

The present disclosure relates to transformer-based encoding with metadata. A relevance matrix is constructed from metadata of a corpus of natural language text documents, a row-column intersection in the relevance matrix corresponding to a relationship between two instances of a type of metadata. An encoder model is trained, generating a trained encoder model to compute an embedding corresponding to a token of a natural language text document within the corpus and within the relevance matrix, the encoder model including a first encoder layer including a token embedding portion, a relevance embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, the training including adjusting a set of parameters of the encoder model.

Description

Technical Field

[0001] This invention generally relates to methods, systems, and computer program products for automatically encoding natural language text or non-text data. More specifically, this invention relates to methods, systems, and computer program products for transformer-based encoding incorporating metadata. Background Technology

[0002] Transformer-based autoencoders, also known as transformer-based autoencoder networks, or simply transformers, have an encoder-decoder architecture. The encoder part of the transformer consists of a set of encoding layers that iteratively process the input sequence, one after another. Each encoder layer generates an encoding (a multidimensional number, also called a vector) containing information about which parts of the input are related to each other. The decoder part of the transformer consists of a set of decoding layers that iteratively process the output of the encoder, one after another. Each decoder layer uses contextual information combined with the encodings to generate an output sequence corresponding to the input sequence. To incorporate context, the encoder and decoder layers include attention mechanisms that, for each input, include other inputs in the sequence weighted by their relevance to other inputs. The decoder layers also have additional attention mechanisms that draw information from the output of previous decoder layers as well as from the encodings.

[0003] As used herein, a transformer refers to a transformer-based autoencoder network with an encoder-decoder architecture, an encoder that includes the set of encoder layers but not the set of decoder layers, and a decoder that includes the set of decoder layers but not the set of encoder layers. For example, a typical text classification embodiment includes only an encoder layer, followed by a classifier layer, and does not include a decoder layer. Transformers can be used to process natural language text or non-text data sequences.

[0004] The illustrative embodiments recognize that, in order to encode text using context, attention mechanisms typically combine token-corresponding embeddings (token embeddings) with positional embeddings (encoding the position of a token within a portion or segment of text) and segmentation embeddings (encoding which segment of text the token is located in). However, both positional and segmentation embeddings are based solely on the text within the document. As a result, metadata, information about the text within the document (such as turn and speaker information in a dialogue), thread information in transcripts of discussion forums, movie subtitles, segmentation hierarchy information, and other elements of structured text, even if available, are often not used when encoding natural language text.

[0005] The illustrative embodiments also recognize that even when using metadata in text processing, the model architecture does not explicitly consider the attention between tokens supplied by metadata or model the effect of metadata through layers of transformers. Instead, portions of metadata are encoded as embeddings and combined with token embeddings in existing attention mechanisms. For example, in a conversational transcript, metadata may indicate that some portions were dictated or texted by participant 1, while some portions were dictated or texted by participant 2. Thus, all portions by participant 1 can be considered as one text segment, and all portions by participant 2 can be considered as a second text segment, and the corresponding segment embeddings are determined and incorporated into the text encoding. Alternatively, the system may compute additional embeddings encoding which speaker is associated with the token, and these additional embeddings are combined with token, position, and segment embeddings in existing attention mechanisms. However, encoding specific portions or types of metadata into specific embeddings is not readily generalizable to models that can be trained on many types of text and metadata, function equally well as existing models when processing text without metadata, and are applicable to a variety of downstream tasks. Furthermore, the model architecture used to process non-textual data does not explicitly explain the attention between tokens supplied by metadata or model the effect of metadata through layers of transformers.

[0006] Therefore, the illustrative embodiments also recognize an unmet need for a transformer-based autoencoder architecture that includes parameters specifically designed to model concerns attributable to metadata and is capable of using a metadata encoding scheme that does not require encoding specific parts or types of metadata into specific embeddings. Summary of the Invention

[0007] Illustrative embodiments provide a method, system, and computer program product. One embodiment includes a method for constructing a relevance matrix from metadata of a corpus of natural language text documents, where row and column intersections correspond to relationships between two instances of a type of metadata. One embodiment trains and generates a trained encoder model for computing embeddings corresponding to tokens of natural language text documents within the corpus and the relevance matrix. The encoder model includes a first encoder layer comprising a token embedding component, a relevance embedding component, a token self-focusing component, a metadata self-focusing component, and a fusion component. The training includes adjusting the parameter set of the encoder model. Therefore, this embodiment provides transformer-based encoding incorporating metadata.

[0008] In another embodiment, the token embedding portion computes a token embedding set, wherein the token embeddings in the token embedding set correspond to tokens of natural language text documents within the corpus. Therefore, this embodiment provides a method for computing a token embedding set.

[0009] In another embodiment, the token comprises a portion of the words in the natural language text document. Therefore, this embodiment provides a method for calculating a token embedding from a portion of the words in a natural language text document.

[0010] In another embodiment, the token embedding includes a multidimensional digital representation of the token. Therefore, this embodiment provides a method for calculating a token embedding that includes a multidimensional digital representation.

[0011] In another embodiment, the token embedding includes a combination of a multidimensional numerical representation of the token, a multidimensional numerical representation of the token's position within the natural language text document, and a multidimensional numerical representation of the segment in the natural language text document in which the token is located. Therefore, this embodiment provides a method for calculating a token embedding that includes a multidimensional numerical representation.

[0012] In another embodiment, the relevance embedding portion computes a relevance embedding set, the relevance embeddings in the set including multidimensional numerical representations of the row-column intersections. Therefore, this embodiment provides a method for computing a relevance embedding set.

[0013] In another embodiment, the token self-attention portion adjusts the input token embedding based on a token attention weight set, where the token attention weights in the set correspond to the relationships between two tokens within the natural language text document, and this token attention weight set is computed during training. Therefore, this embodiment provides a method for adjusting the input token embedding based on a token attention weight set.

[0014] In another embodiment, the metadata self-focus portion adjusts the input relevance embedding based on a metadata focus weight set, which is computed during the training period. Therefore, this embodiment provides a method for adjusting the input relevance embedding based on a metadata focus weight set.

[0015] In another embodiment, the fusion portion combines the outputs of the token self-following portion and the metadata self-following portion. Therefore, this embodiment also provides details of the fusion portion.

[0016] In another embodiment, the training includes: initializing the parameter set of the token embedding portion to a basic set of token embedding parameters; initializing the parameter set of the token self-focused portion to a basic set of token self-focused parameters; a first training to generate a partially trained encoder model, the first training including adjusting the parameter set of the relevance embedding portion and the parameter set of the metadata self-focused portion when the parameter set of the token embedding portion is set to the basic set of token embedding parameters and the parameter set of the token self-focused portion is set to the basic set of token self-focused parameters; and a second training to generate a trained encoder model, the second training including adjusting the parameter set of the encoder model. Therefore, this embodiment provides further details on training the encoder model.

[0017] In another embodiment, the encoder model further includes a first decoder layer comprising a decoder token self-focusing component, a decoder metadata self-focusing component, a decoder fusion component, and a decoder focus component, wherein the training includes adjusting the parameter set of the first decoder layer. Therefore, this embodiment provides further details about the first decoder layer.

[0018] In another embodiment, the decoder attention component adjusts the output of the encoder layer based on a set of attention weights computed during training. Therefore, this embodiment provides further details regarding the decoder attention component.

[0019] The embodiments include a computer-usable program product. The computer-usable program product includes one or more computer-readable storage devices, and program instructions stored on at least one of the one or more computer-readable storage devices.

[0020] The embodiment includes a computer system. The computer system includes one or more processors, one or more computer-readable memories and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices, the program instructions being executed by at least one of the one or more processors via at least one of the one or more memories. Attached Figure Description

[0021] Certain novel features considered characteristic of the invention are set forth in the appended claims. However, the invention itself, as well as its preferred modes of use, further objects and advantages, will be best understood by reading in conjunction with the accompanying drawings and by referring to the following detailed description of illustrative embodiments, wherein:

[0022] Figure 1 A block diagram of a network that can implement the illustrative embodiments of the data processing system is depicted;

[0023] Figure 2 A block diagram of a data processing system that can implement illustrative embodiments is depicted;

[0024] Figure 3 A block diagram depicts an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment;

[0025] Figure 4 A block diagram depicts an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment;

[0026] Figure 5 A block diagram depicts an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment;

[0027] Figure 6 A block diagram depicts an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment;

[0028] Figure 7 A block diagram depicts an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment;

[0029] Figure 8 An example of transformer-based encoding incorporating metadata, according to an illustrative embodiment, is described;

[0030] Figure 9 A further example of transformer-based encoding incorporating metadata, according to an illustrative embodiment, is described;

[0031] Figure 10 A further example of transformer-based encoding incorporating metadata, according to an illustrative embodiment, is described;

[0032] Figure 11 A further example of transformer-based encoding incorporating metadata, according to an illustrative embodiment, is described;

[0033] Figure 12 A further example of transformer-based encoding incorporating metadata, according to an illustrative embodiment, is described;

[0034] Figure 13 A flowchart depicts an example process of transformer-based encoding incorporating metadata according to an illustrative embodiment;

[0035] Figure 14 A cloud computing environment according to an embodiment of the present invention is described; and

[0036] Figure 15 An abstract model layer according to an embodiment of the present invention is described. Detailed Implementation

[0037] The illustrative embodiments also recognize an unmet need for a transformer-based autoencoder architecture that includes parameters specifically designed for modeling concerns attributable to metadata and is capable of using metadata encoding schemes that do not require encoding specific portions or types of metadata into specific embeddings. The illustrative embodiments recognize that currently available tools or solutions do not address these needs or provide sufficient solutions for them. The illustrative embodiments used to describe the invention generally address and resolve the aforementioned problems and other issues related to transformer-based encoding incorporating metadata.

[0038] The embodiments can be implemented as software applications. An application implementing an embodiment can be configured as a modification of an existing natural language text processing system or a system for processing non-text data, and can be configured as a standalone application, a standalone application, or some combination thereof that operates in conjunction with an existing natural language text processing system or a system for processing non-text data.

[0039] Specifically, some illustrative embodiments provide a method for constructing a relativity matrix from metadata of a corpus of non-textual data or natural language text documents, and training and generating a trained encoding model for calculating the embedding of a portion of the documents or non-textual data within the corpus into the relativity matrix.

[0040] The example constructs a relevance matrix from metadata in a corpus of non-text data or natural language text documents. A relevance matrix is ​​a two-dimensional matrix that stores the relationships between instances of a type of metadata. Specifically, the rows and columns of the relevance matrix represent instances of a type of metadata, and the row-column intersections store data about the relationships between two instances of a type of metadata.

[0041] In a non-restrictive example of a relevance matrix encoding scheme, an entry in the relevance matrix is ​​set to 1 if a relationship exists between two instances of a type of metadata, and set to 0 if no relationship exists. For example, in a session transcript, metadata might indicate which parts were spoken or written by which participant. Therefore, the rows and columns of the corresponding relevance matrix could represent participants. If a relationship exists between the participant represented by the row of an entry and the participant represented by the column of an entry, the entry in the relevance matrix is ​​set to 1; otherwise, it is set to 0.

[0042] In another non-limiting example of a relevance matrix encoding scheme, entries in the relevance matrix are set as the distance between two instances of a type of metadata within the metadata. For example, in a session transcript, the metadata might indicate which parts were spoken or written by which participant. Thus, the rows and columns of the corresponding relevance matrix could represent participants, and entries in the relevance matrix could be set as the number of session rounds between the participant represented by the row of that entry and the participant represented by the column of that entry. In another example of a session transcript, the metadata might record timestamps corresponding to each change in a participant or round in the session. Therefore, the rows and columns of the corresponding relevance matrix could represent specific rounds, and entries in the relevance matrix could be set as the amount of time elapsed between the round represented by the row of that entry and the round represented by the column of that entry. Alternatively, the rows and columns of the corresponding relevance matrix could represent specific rounds, and entries in the relevance matrix could be set as the number of rounds between the round represented by the row of that entry and the round represented by the column of that entry.

[0043] In another non-limiting example, a document or dialogue transcript is represented by a tree structure described by metadata. The rows and columns of the corresponding relevance matrix can represent specific nodes in the tree structure. An entry in the relevance matrix can be set to the number of tree links between the node represented by the row of that entry and the node represented by the column of that entry. In a variation, if node i is an ancestor of node j in the tree, then the entry in the relevance matrix can be set to the positive distance between node i and node j; if node j is an ancestor of node i in the tree, then the entry in the relevance matrix can be set to the negative distance between node i and node j; otherwise, the entry in the relevance matrix can be set to the sum of the depth of the tree and the distance between node i and node j.

[0044] Other non-limiting examples of encoding metadata relationships using a relevance matrix to represent the relative distance between instances of metadata in the manner described herein include relationships between characters, scene locations or text, other interactions within audio or video content (such as transcripts, scripts, or movies), relationships between entities expressed in the content, and relationships between elements of parse trees, knowledge graphs, hyperlinks, or other structured information relationships.

[0045] The embodiment includes a transformer model. This model receives a sequence of tokens and a relevance matrix as input and generates embeddings, also known as encodings, that correspond to the input tokens and are adjusted according to the context including the relevance matrix. When processing natural language text, tokens are words, parts of words, or other portions of a natural language text sequence (such as sentences within a document), and the input sequence is typically a tokenized sentence or other tokenized portion of text, but not necessarily a grammatically correct sentence. When processing non-textual data, tokens are portions of non-textual data, such as a single frame of a video or a predetermined duration of video data. The model includes one or more encoder layers connected in series. In one embodiment, the last encoder layer is followed by an optional task-specific layer for output prediction (such as text classification). The model also includes an embedding section that generates embeddings from the model input and passes the generated embeddings to the first encoder layer for adjustment of the generated embeddings.

[0046] Specifically, the embedding part receives a sequence of tokens as input. The embedding part is configured in a currently available neural network configuration to generate a token embedding that encodes the tokens. Optionally, the embedding part may also be configured in a currently available neural network configuration to generate a positional embedding that encodes the position of the token within a portion or segment of text, and a segmental embedding that encodes which text segment the token is located in. If positional and segmental embeddings are generated, the embedding part is configured in a currently available neural network configuration to combine the token embedding, positional embedding, and segmental embedding into a single token embedding. In one implementation, the token, positional, and segmental embeddings are combined by summing them together. In another implementation, a weighted sum is used to combine the token embedding, positional embedding, and segmental embedding, where the weights are model parameters determined during model training.

[0047] The embedding part also receives a correlation matrix as input. The embedding part generates a set of correlation embeddings, each encoding an entry in the correlation matrix. One implementation uses a projection matrix initialized to an initial configuration (e.g., a set of pseudo-randomly selected values) and trained on an adjusted model to generate this set of correlation embeddings. Another embodiment uses a set of heuristics to generate this set of correlation embeddings. Other techniques for generating this set of correlation embeddings are also possible and are conceived within the scope of the illustrative embodiments.

[0048] In one embodiment, each encoder layer is configured identically and includes a token self-focusing component, one or more metadata self-focusing components, a fusion component, a feedforward component, and optional summation and normalization components. In another embodiment, each encoder layer is not configured identically. Instead, some encoder layers do not include one or more metadata self-focusing components or fusion components. In one embodiment, only the last six encoder layers include metadata self-focusing and fusion components, which speeds up model training time compared to embodiments where each encoder layer is configured identically.

[0049] In one currently known technique, the token self-focus component is described by three weight matrices: query weight W Q Key weight W K Sum weight W V Specifically, for each token i, the input embedding x i Multiply by each weight matrix to produce the query vector q i =x i W Q Key vector k i =x i W K Sum vector v i =x i W V So, A i,j =q i k j q i and k j The dot product between A and A, where A i,j This is the attention weight from token i to token j. The attention weight is divided by the square root of the dimension of the key vector (to stabilize the gradient between matrix values ​​during training), and then passed through softmax, which normalizes the weights to a sum of 1. Thus, the token self-attention component is determined by the attention weight A. i,j Obtain contextual interest in token j from token i.

[0050] The metadata self-focusing part calculates a set of metadata focus weights A. meta This set of metadata focuses on weights using a set of relevance embeddings of the input S. meta To calculate. If there are more than one set of relevance embeddings for the input, each set is fed into a separate metadata self-focus section. In one embodiment, A meta ij =(qi+B)S meta T ij Where B is the learnable bias term, and S meta T ij It is S meta A transposed version. In another embodiment, A meta ij=(qi+B)S meta T ij +S meta ij (k j +B') T , where B and B' are learnable bias terms, and (k j +B') T It is (k) j A transpose of +B'). Another embodiment uses a neural network to embed this set of correlations into S. meta Combine it with K (the matrix of the key vectors). Then, calculate the sum of the combination result with Q, the matrix of the query vectors, and BS. meta T ij The dot product of the summation results. Used to extract the correlation embedding S. meta Calculate metadata focus weight A meta Other techniques are also possible and are conceived within the scope of the illustrative embodiments.

[0051] The fusion component combines the results of the token self-focusing component and one or more metadata self-focusing components. One embodiment uses a neural network to combine the results of the token self-focusing component and the metadata self-focusing component. Another embodiment calculates A and A... meta The element-wise product. Other techniques for combining the results of the token self-focus portion and the metadata self-focus portion are also possible and are conceived within the scope of the illustrative embodiments.

[0052] In one embodiment, the summation and normalization section adds the output from the fusion section to the input to the token self-focus section, then normalizes the result and passes the normalized result to the feedforward section. In another embodiment, the summation and normalization section does not exist, and the output of the fusion section is passed directly to the feedforward section. In one embodiment, the feedforward section is implemented using a linear layer with weights described by a weight matrix. The weights are adjusted during model training. In another embodiment, each metadata self-focus section is followed by a summation and normalization section that combines the input and output of that particular metadata self-focus section, and there is no summation and normalization section that combines the output from the fusion section and the input to the token self-focus section.

[0053] In one embodiment, the output of the feedforward portion and the input to the feedforward portion are summed and normalized, and the result is passed to the input portion of the second encoder layer for additional processing. In another embodiment, the output of the feedforward portion is passed to the input portion of the second encoder layer for additional processing.

[0054] In one embodiment, the transformer model includes one or more decoder layers connected in series. Each decoder layer receives a sequence of token embeddings and the set of related embeddings generated by the embedding portion as input. Each decoder layer also receives the output of the last encoder layer in the series as input.

[0055] In one embodiment, each decoder layer is configured identically and includes a token self-focusing component, one or more metadata self-focusing components, a fusion component, a focus component, a feedforward component, and optional summation and normalization components. In another embodiment, each decoder layer is not configured identically. Instead, some decoder layers do not include one or more metadata self-focusing components or fusion components.

[0056] The token self-focus, metadata self-focus, and fusion parts of the decoder layer operate similarly to those of the encoder layer. In one embodiment, the addition and normalization parts add the output from the fusion part to the input to the token self-focus part, then normalize the result and pass the normalized result to the focus part. In another embodiment, there is no addition and normalization part, and the output of the fusion part is passed directly to the focus part.

[0057] The focus section combines the output from the fusion section or the summation and normalization section with the output from the last encoder layer in the group of encoder layers. In one embodiment, the output from the last encoder layer is in the form of a key vector and a value vector, and the output from the fusion section or the summation and normalization section is in the form of a query vector. The focus section combines the inputs using currently known techniques.

[0058] In one embodiment, the summation and normalization section adds the output from the focus region and the input to the focus region, then normalizes the result and passes the normalized result to the feedforward section. In another embodiment, the summation and normalization section does not exist, and the output of the focus region is passed directly to the feedforward section. In one embodiment, the feedforward section is implemented using a linear layer with weights described by a weight matrix. The weights are adjusted during model training. In one embodiment, the output of the feedforward section and the input to the feedforward section are summed and normalized, and the result is passed to the input of a second decoder layer for additional processing. In another embodiment, the output of the feedforward section is passed to the input of a second decoder layer for additional processing.

[0059] The embodiment trains a transformer model. During training, this embodiment learns to set the model parameters (including query weights, key weights, value weights, attention and metadata attention weights, one or more learnable bias terms, and parameters of the embedding part) to produce values ​​from the correctly labeled output training data. One embodiment trains the entire model at once by initializing the model parameters to a basic parameter set (e.g., a pseudo-randomly selected parameter set) and tuning that set of parameters using a set of training data.

[0060] Another embodiment trains the model in stages. This embodiment sets the parameters of the token embedding and token self-focusing parts as a basic parameter set. As a non-limiting example, the basic parameter set is a pseudo-randomly selected set of parameters. As another non-limiting example, the basic parameter set is a set of parameters of the already trained token embedding and token self-focusing parts, trained using currently available techniques. Using those parameters of the already trained token embedding and token self-focusing parts as the basic set of parameters saves training time by starting from a partially trained state. In the first training phase, the embodiment keeps the parameters of the token embedding and token self-focusing parts constant and trains the model by adjusting the parameters of the relevance embedding, one or more metadata self-focusing parts, and the fusion part using a set of training data. Then, in the second training phase, the embodiment trains the entire model by adjusting one or more parameters across the entire parameter set.

[0061] The training data for a model can be tailored based on the final use of the model's output. Some non-restrictive examples of the final use of the model's output include performing masked language modeling (using contextual words or blanks to be filled around a mask token to try to predict what words should replace that mask token), masked context regression (masking randomly selected utterances and predicting the encoded vector of the utterance being masked), distributed utterance ordering (organizing randomly shuffled utterances of a dialogue into a consistent dialogue context), and utterance or sentence order selection. For example, for a general model to be used on both plain text (without metadata) and natural language text with accompanying metadata, the training data should include both plain text and different text genres with accompanying metadata. However, for a model to be used on data combining text with one genre of accompanying metadata, such as a model used only for dialogue-related tasks, the training data only needs to include dialogue-specific text with accompanying metadata such as dialogue turns, speakers, or timestamp data.

[0062] Once the model has been trained (generally or for a specific genre), one embodiment optionally further trains the model to perform specific downstream tasks. Some non-limiting examples of downstream tasks for documents are masked sentence selection in structured documents, question-and-answer retrieval from documents, and structured document summarization. Some non-limiting examples of downstream tasks for dialogues are next utterance selection, utterance order selection (placing utterances or other parts of a dialogue in sequence), masked utterance selection, next utterance generation, and conversational question-and-answer.

[0063] The transformer-based encoding method incorporating metadata described herein is not available in currently available methods in the art of efforts related to natural language text processing. The methods of the embodiments described herein, when implemented on a device or data processing system, include substantial improvements to the functionality of that device or data processing system by constructing a relevance matrix from metadata of a corpus of non-textual data or natural language text documents, and training and generating a trained encoding model for calculating embeddings corresponding to tokens of natural language text documents within the corpus and the relevance matrix.

[0064] The illustrative embodiments are described by way of example only, relating to certain types of tokens, embeddings, correlations, matrices, encoding portions, attention portions, fusion portions, neural networks, tuning, devices, data processing systems, environments, components, and applications. Any specific manifestations of these and other similar products are not intended to limit the invention. Any suitable manifestations of these and other similar products may be chosen within the scope of the illustrative embodiments.

[0065] Furthermore, illustrative embodiments can be implemented with respect to any type of data, data source, or access to a data source via a data network. Within the scope of this invention, any type of data storage device can provide data to embodiments of the invention locally within a data processing system or via a data network. Within the scope of the illustrative embodiments, where embodiments are described using mobile devices, any type of data storage device suitable for use with mobile devices can provide data to this embodiment locally within the mobile device or via a data network.

[0066] The illustrative embodiments are described using specific code, designs, architectures, protocols, layouts, diagrams, and tools as examples only, and are not limited to these illustrative embodiments. Furthermore, for clarity, specific software, tools, and data processing environments are used in some instances as examples only to describe the illustrative embodiments. The illustrative embodiments can be used in conjunction with other comparable or similar structures, systems, applications, or architectures. For example, within the scope of this invention, other similar mobile devices, structures, systems, applications, or architectures can be used in conjunction with such embodiments of the invention. The illustrative embodiments can be implemented in hardware, software, or a combination thereof.

[0067] The examples in this disclosure are for clarity of description only and are not intended to limit the scope of the illustrative embodiments. Other data, operations, actions, tasks, activities, and manipulations will arise from this disclosure, and the same data, operations, actions, tasks, activities, and manipulations are contemplated within the scope of the illustrative embodiments.

[0068] Any advantages listed herein are merely examples and are not intended to limit these illustrative embodiments. Additional or different advantages may be achieved through specific illustrative embodiments. Furthermore, certain illustrative embodiments may have some, all, or none of the advantages listed above.

[0069] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0070] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0071] The features are as follows:

[0072] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0073] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0074] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0075] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0076] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0077] The service model is as follows:

[0078] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0079] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0080] Infrastructure as a Service (IaaS): The capabilities offered to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0081] The deployment model is as follows:

[0082] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0083] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0084] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.

[0085] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0086] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0087] Refer to the attached diagram and for details. Figure 1 and Figure 2 These figures are example diagrams of a data processing environment that can implement illustrative embodiments. Figure 1 and Figure 2 This is merely an example and is not intended to assert or imply any limitation regarding the environments in which different embodiments may be implemented. Specific implementations may be modified in many ways based on the environment depicted in the following description.

[0088] Figure 1 A block diagram of a network in which an illustrative embodiment of a data processing system may be implemented is shown. Data processing environment 100 is a computer network in which the illustrative embodiment may be implemented. Data processing environment 100 includes network 102. Network 102 is a medium for providing communication links between various devices and computers connected together within data processing environment 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0089] The client or server are merely example roles of certain data processing systems connected to network 102 and are not intended to exclude other configurations or roles of these data processing systems. Servers 104 and 106 are coupled to network 102 along with storage unit 108. Software applications can execute on any computer in data processing environment 100. Clients 110, 112, and 114 are also coupled to network 102. Data processing systems (such as servers 104 or 106 or clients 110, 112, or 114) can contain data and can have software applications or software tools executing on them.

[0090] This is merely an example and does not imply any limitations on this architecture. Figure 1Certain components that may be used in exemplary implementations of the embodiments are described. For example, servers 104 and 106 and clients 110, 112, and 114 are depicted as servers and clients only by way of example and are not intended to imply a limitation on the client-server architecture. As another example, embodiments may be distributed across several data processing systems and data networks shown, while another embodiment may be implemented on a single data processing system within the scope of the illustrative embodiments. Data processing systems 104, 106, 110, 112, and 114 also represent example nodes in clusters, partitions, and other configurations suitable for implementing the embodiments.

[0091] Device 132 is an example of the device described herein. For example, device 132 may take the form of a smartphone, tablet computer, laptop computer, client 110 in fixed or portable form, wearable computing device, or any other suitable device. Figure 1 Any software application running in another data processing system within the device can be configured to run in a similar manner in device 132. Figure 1 Any data or information stored or generated in another data processing system can be configured to be stored or generated in device 132 in a similar manner.

[0092] Application 105 implements the embodiments described herein. Application 105 is executed in any of servers 104 and 106, clients 110, 112 and 114, and device 132.

[0093] Servers 104 and 106, storage unit 108, clients 110, 112 and 114, and device 132 can be coupled to network 102 using wired connections, wireless communication protocols, or other suitable data connectivity. Clients 110, 112, and 114 can be, for example, personal computers or network computers.

[0094] In the depicted example, server 104 can provide clients 110, 112, and 114 with data such as boot files, operating system images, and applications. In this example, clients 110, 112, and 114 can be clients of server 104. Clients 110, 112, 114, or some combination thereof, can include their own data, boot files, operating system images, and applications. Data processing environment 100 can include additional servers, clients, and other devices not shown.

[0095] In the depicted example, data processing environment 100 can be the Internet. Network 102 can represent a collection of networks and gateways that communicate with each other using Transmission Control Protocol / Internet Protocol (TCP / IP) and other protocols. The core of the Internet is the skeleton of data communication links between master nodes or host computers (including thousands of commercial, government, educational, and other computer systems that route data and messages). Of course, data processing environment 100 can also be implemented as many different types of networks, such as, for example, intranets, local area networks (LANs), or wide area networks (WANs). Figure 1 This is intended as an example, not as an architectural limitation for different illustrative embodiments.

[0096] Among other uses, the data processing environment 100 can be used to implement a client-server environment in which illustrative embodiments can be implemented. The client-server environment enables software applications and data to be distributed across a network, allowing applications to function through interaction between client data processing systems and server data processing systems. The data processing environment 100 can also employ a service-oriented architecture, where interoperable software components distributed across a network can be encapsulated together as a consistent business application. The data processing environment 100 can also take the form of a cloud and employ a service-delivered cloud computing model to enable convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service providers.

[0097] refer to Figure 2 The figure depicts a block diagram of a data processing system that can implement illustrative embodiments. Data processing system 200 is an example of a computer, such as... Figure 1 The servers 104 and 106 or clients 110, 112 and 114, or another type of device in which computer-usable program code or instructions for implementing the illustrative embodiments may reside.

[0098] Data processing system 200 also represents a data processing system or a configuration thereof, such as Figure 1 The data processing system 132 may contain computer-usable program code or instructions for implementing the processing of the illustrative embodiments. The data processing system 200 is described as a computer by way of example only and is not limited thereto. Other devices (such as...) Figure 1 The implementation of the data processing system 200 in the form of device 132 can modify the data processing system 200, such as by adding a touch interface, and even by removing certain depicted components from the data processing system 200, without departing from the general description of the operation and function of the data processing system 200 described herein.

[0099] In the depicted example, the data processing system 200 employs a central architecture including a Northbridge and Memory Controller Hub (NB / MCH) 202 and a Southbridge and Input / Output (I / O) Controller Hub (SB / ICH) 204. A processing unit 206, main memory 208, and a graphics processor 210 are coupled to the Northbridge and Memory Controller Hub (NB / MCH) 202. The processing unit 206 may contain one or more processors and may be implemented using one or more heterogeneous processor systems. The processing unit 206 may be a multi-core processor. In some implementations, the graphics processor 210 may be coupled to the NB / MCH 202 via an Accelerated Graphics Port (AGP).

[0100] In the depicted example, a local area network (LAN) adapter 212 is coupled to the Southbridge and I / O controller hub (SB / ICH) 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a universal serial bus (USB) and other ports 232, and a PCI / PCIe device 234 are coupled to the Southbridge and I / O controller hub 204 via bus 238. A hard disk drive (HDD) or solid-state drive (SSD) 226 and a CD-ROM 230 are coupled to the Southbridge and I / O controller hub 204 via bus 240. The PCI / PCIe device 234 may include, for example, an Ethernet adapter, an insert card, and a PC card for a notebook computer. PCI uses a card bus controller, while PCIe does not. The ROM 224 may be, for example, a flash binary input / output system (BIOS). Hard disk drive 226 and CD-ROM 230 can use, for example, integrated drive electronics (IDE), serial advanced technology accessory (SATA) interface, or variants such as external SATA (eSATA) and micro SATA (mSATA). Super I / O (SIO) device 236 can be coupled to the southbridge and I / O controller hub (SB / ICH) 204 via bus 238.

[0101] Memory such as main memory 208, ROM 224, or flash memory (not shown) are some examples of computer-available storage devices. Hard disk drives or solid-state drives 226, CD-ROMs 230, and other similar available devices are some examples of computer-available storage devices that include computer-available storage media.

[0102] The operating system runs on processing unit 206. The operating system coordinates and provides... Figure 2The data processing system 200 controls various components within it. The operating system can be a commercially available operating system for any type of computing platform, including but not limited to server systems, personal computers, and mobile devices. Object-oriented or other types of programming systems can operate in conjunction with the operating system and provide calls to the operating system from programs or applications executing on the data processing system 200.

[0103] Operating systems, object-oriented programming systems, and applications or programs (such as...) Figure 1 The instructions for application 105 are located on a storage device (such as in the form of code 226A on hard disk drive 226) and can be loaded into at least one of one or more memories (such as main memory 208) for execution by processing unit 206. Processing in an exemplary embodiment can be executed by processing unit 206 using computer-implemented instructions that may reside in memory, such as, for example, main memory 208, read-only memory 224, or one or more peripheral devices.

[0104] Furthermore, in one scenario, code 226A can be downloaded from a remote system 201B via network 201A, where similar code 201C is stored on storage device 201D. In another scenario, code 226A can be downloaded to a remote system 201B via network 201A, where the downloaded code 201C is stored on storage device 201D.

[0105] Figure 1-2 The hardware in the implementation may vary depending on the specific implementation. (Except for or instead of) Figure 1-2 The hardware described herein can be replaced with other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disc drives. Furthermore, the processes of the illustrative embodiments can be applied to multiprocessor data processing systems.

[0106] In some illustrative examples, the data processing system 200 may be a personal digital assistant (PDA), which is typically configured with flash memory to provide non-volatile memory for storing operating system files and / or user-generated data. The bus system may include one or more buses, such as a system bus, I / O bus, and PCI bus. Of course, the bus system can be implemented using any type of communication structure or architecture that provides data transfer between different components or devices attached to the structure or architecture.

[0107] The communication unit may include one or more devices for sending and receiving data, such as a modem or network adapter. Memory may be, for example, main memory 208 or a cache, such as the cache found in the northbridge and memory controller hub 202. The processing unit may contain one or more processors or CPUs.

[0108] Figure 1-2 The examples depicted and those described above are not intended to imply architectural limitations. For example, the data processing system 200 could take the form of a tablet computer, laptop computer, or telephone device, in addition to being a mobile or wearable device.

[0109] When a computer or data processing system is described as a virtual machine, virtual device, or virtual component, the virtual machine, virtual device, or virtual component operates in a manner similar to data processing system 200, using virtualized representations of some or all of the components depicted in data processing system 200. For example, in a virtual machine, virtual device, or virtual component, processing unit 206 is represented as a virtualized instance of all or some of the hardware processing units 206 available in the host data processing system, main memory 208 is represented as a virtualized instance of all or some of the main memory 208 available in the host data processing system, and disk 226 is represented as a virtualized instance of all or some of the disk 226 available in the host data processing system. In this case, the host data processing system is represented by data processing system 200.

[0110] See Figure 3 This figure depicts a block diagram of an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment. Application 300 is... Figure 1 Example of application 105, and in Figure 1 It can be executed in any of the servers 104 and 106, clients 110, 112 and 114, and device 132.

[0111] Metadata relevance module 310 constructs a relevance matrix from metadata in a corpus of natural language text documents. The rows and columns of the relevance matrix represent instances of a type of metadata, and the row-column intersections store data on the relationships between two instances of a type of metadata. In one example relevance matrix encoding scheme, an entry in the relevance matrix is ​​set to 1 if a relationship exists between two instances of a type of metadata, and set to 0 if no relationship exists. In another example relevance matrix encoding scheme, entries in the relevance matrix are set to the distance between two instances of a type of metadata within the metadata. In yet another example relevance matrix encoding scheme, a document or dialogue transcript is represented using a tree structure described by the metadata, and entries in the relevance matrix can be set based on the distance between nodes represented by rows of entries and nodes represented by columns of entries. Other examples of using relevance matrices to represent relative distances between instances of metadata to encode metadata relationships in the manner described herein include relationships between characters, scene locations, or other interactions within text, audio, or video content (such as transcripts, scripts, or movies), relationships between entities expressed in the content, and relationships between elements of parse trees, knowledge graphs, hyperlinks, or other structured information relationships.

[0112] Encoder module 320 implements the encoder portion of the transformer model. Module 320 receives a token sequence and a correlation matrix as input and generates an embedding corresponding to the input tokens and adjusted according to the context including the correlation matrix. See also Figure 4 and 5 Presenting more details of module 320.

[0113] Decoder module 325 implements an optional decoder portion of the transformer model, comprising one or more decoder layers connected in series. Each decoder layer receives a sequence of token embeddings and a set of correlation embeddings as input, generated by encoder module 320. Each decoder layer also receives the output as input from the last encoder layer in a series of encoder layers within module 320. See also Figure 6 and 7 More details about module 325 are presented.

[0114] Training module 330 trains the transformer model. During training, module 330 sets the model parameters (including query weights, key weights, value weights, attention and metadata attention weights within the encoder and decoder parts, one or more learnable bias terms, and parameters in the embedding part) to produce the values ​​that have been correctly labeled as output training data. One implementation of module 330 trains the entire model at once by initializing the model parameters to a basic parameter set (e.g., a pseudo-randomly selected parameter set) and training the model using the training dataset.

[0115] Another implementation of Module 330 trains the model in stages. Module 330 sets the parameters of the token embedding and token self-focus parts into a basic parameter set. As a non-restrictive example, the basic parameter set is a pseudo-randomly selected parameter set. As another non-restrictive example, the basic parameter set is a set of parameters of the already trained token embedding and token self-focus parts, trained using currently available techniques. In the first training phase, Module 330 keeps the parameters of the token embedding and token self-focus parts constant and trains the model by adjusting the parameters of the relevance embedding, one or more metadata self-focus parts, the focus part, and the fusion part using the training dataset. Then, in the second training phase, Module 330 trains the entire model by adjusting one or more parameters in the entire parameter set.

[0116] See Figure 4 This figure depicts a block diagram of an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment. Encoder module 320 and Figure 3 The encoder module 320 is the same.

[0117] Encoder module 320 includes a set of encoder layers, including encoder layer 410, encoder layer 420 and encoder layer 430. The encoder layers are connected in series, and an additional encoder layer may exist between encoder layers 420 and 430.

[0118] See Figure 5 This figure depicts a block diagram of an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment. Encoder layer 410 and Figure 4 The encoder layer 410 is the same.

[0119] The token embedding module 510, configured in the currently available neural network configuration, generates a token embedding that encodes the token. Module 510 may also be optionally configured in the currently available neural network configuration to generate a positional embedding that encodes the position of the token within a portion or segment of text, and a segmental embedding that encodes which text segment the token is located in. If positional and segmental embeddings are generated, module 510 is configured in the currently available neural network configuration to combine the token embedding, positional embedding, and segmental embedding into a single token embedding. In one implementation of module 510, the token embedding, positional embedding, and segmental embedding are combined by adding them together.

[0120] The metadata embedding module 520, configured in the currently available neural network configuration, generates a set of relevance embeddings, each embedding encoding an entry in the relevance matrix.

[0121] The token self-following module 530 is described by three weight matrices: query weight WQ Key weight W K Sum weights

[0122] W V Specifically, for each token i, the input embedding x i Multiply by each weight matrix to produce the query vector q i =x i W Q Key vector k i =x i W K Sum vector v i =x i W V So, A i,j =q i k j q i and k j The dot product between A and A, where A i,j These are the attention weights from token i to token j. The attention weights are divided by the square root of the dimension of the key vector (to stabilize the gradient between matrix values ​​during training) and then normalized to a softmax that sums to 1.

[0123] The metadata self-focusing module 540 calculates a set of metadata focus weights A. meta This set of metadata focuses on weights using a set of relevance embeddings of the input S. meta To calculate. If there are more than one set of relevance embeddings for the input, each set of relevance embeddings is fed into a separate metadata self-focus section. In one embodiment, A meta ij =(q i +B)S meta T ij Where B is the learnable bias term, and S meta T ij It is S meta A transposed version. In another embodiment, A meta ij =(q i +B)

[0124] S meta T ij +S meta ij (k j +B') T , where B and B' are learnable bias terms, and (k j +B') T It is (k) j A transpose of +B'). Another embodiment uses a neural network to embed this set of correlations into S. metaCombine it with K (the matrix of the key vectors). Then, calculate the sum of the combination result with Q, the matrix of the query vectors, and BS. meta T ij The dot product of the summation results.

[0125] Fusion module 550 combines the results of modules 530 and 540. One implementation of module 550 uses a neural network to combine the results of the token self-focused portion and the metadata self-focused portion. Another implementation of module 550 computes A and A... meta The element product.

[0126] refer to Figure 6 This figure depicts a block diagram of an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment. Decoder module 325 and... Figure 3 The encoder module 325 is the same.

[0127] Decoder module 325 includes a set of decoder layers, including decoder layer 610, decoder layer 620, and decoder layer 630. The decoder layers are connected in series, and additional decoder layers may exist between encoder layers 620 and 630. Each decoder layer receives a sequence of token embeddings generated by modules 510 and 520 and a set of related embeddings as input. Each decoder layer also receives the last encoder layer in a series of encoder layers as input from encoder layer 430.

[0128] refer to Figure 7 This figure depicts a block diagram of an example configuration of transformer-based encoding incorporating metadata according to an illustrative embodiment. Decoder layer 610 and... Figure 6 The decoder layer 610 is the same.

[0129] As depicted, decoder layer 610 includes a token self-focus module 710, one or more metadata self-focus modules 720, a fusion module 730, and a focus module 740. The token self-focus module 710, metadata self-focus module 720, and fusion module 730 operate similarly to those of encoder layer 410. In one implementation of layer 610, the addition and normalization section adds the output from fusion module 730 to the input to token self-focus module 710, then normalizes the result and passes the normalized result to focus module 740. In another implementation of layer 610, the addition and normalization section is absent, and the output of fusion module 730 is directly passed to focus module 740.

[0130] The attention module 740 combines the output from the fusion module 730 or the summation and normalization section with the output from the last encoder layer in the group of encoder layers. In one implementation of layer 610, the output from the last encoder layer is in the form of a key vector and a query vector, and the output from the fusion section 740 or the summation and normalization section is in the form of a query vector. The attention module 740 uses currently known techniques to combine the inputs.

[0131] In one implementation of layer 610, the second addition and normalization section adds the output from the interest module 740 and the input to the interest module 740, then normalizes the result and passes the normalized result to the feedforward section. In another embodiment, the addition and normalization section does not exist, and the output of the interest module 740 is directly passed to the feedforward section. In one embodiment of layer 610, the output of the feedforward section and the input to the feedforward section are added together and normalized, and the result is passed to the input section of the second decoder layer for additional processing. In another embodiment, the output of the feedforward section is passed to the input section of the second decoder layer for additional processing.

[0132] refer to Figure 8 This figure depicts an example of transformer-based encoding incorporating metadata according to an illustrative embodiment. This example can be used... Figure 3 Application 300 is used to execute this. Metadata relevance module 310 and... Figure 3 The metadata relevance module 310 is the same.

[0133] Specifically, Figure 8 The description describes the construction of a relevance matrix 812 from the metadata of session 810. In matrix 812, an entry in the relevance matrix is ​​set to 1 if a relationship exists between two instances of a certain type of metadata, and set to 0 if no relationship exists. Thus, because the metadata of session 810 indicates which parts were spoken or written by which participant, the rows and columns of matrix 810 represent participants. An entry in the relevance matrix 810 is set to 1 if a relationship exists between the participant represented by the row of the entry and the participant represented by the column of the entry, and set to 0 if no relationship exists.

[0134] Figure 8The construction of a relevance matrix 822 from the timestamped metadata of session 820 is also described. In matrix 822, entries are set as the distance between two instances of one type of metadata within the metadata. Thus, because the metadata of session 820 records timestamps corresponding to each change in participants or rounds within the session, the rows and columns of the relevance matrix 822 represent specific rounds, and entries in matrix 822 are set as the amount of time elapsed between the round represented by the row of that entry and the round represented by the column of that entry.

[0135] See Figure 9 The figure depicts a further example of transformer-based encoding incorporating metadata according to an illustrative embodiment. Token embedding module 510 and metadata embedding module 520 are connected with... Figure 5 The token embedding module 510 and metadata embedding module 520 are the same. Matrix 812 and 822 are... Figure 8 The matrices 812 and 822 in the matrix are the same.

[0136] The token embedding module 510 receives a token sequence as input. Module 510 generates a token embedding that encodes the tokens. Module 510 also generates a position embedding and a segment or segment embedding, and combines the token embedding, position embedding, and segment embedding into a token embedding 902.

[0137] The metadata embedding module 520 receives correlation matrices 812 and 822 as input and generates correlation embeddings 912 and 922 respectively.

[0138] See Figure 10 This figure depicts a further example of transformer-based encoding incorporating metadata, according to an illustrative embodiment. Encoder module 320 and Figure 3 The encoder module 320 is the same as that in the previous model. Encoder layers 410, 420, and 430 are the same as those in the previous model. Figure 4 The encoder layers 410, 420, and 430 are identical. The token self-focus module 530, metadata self-focus module 540, and fusion module 550 are also identical. Figure 5 The token self-focusing module 530, metadata self-focusing module 540, and fusion module 550 are identical. Token embedding 902 and relevance embedding 912 and 922 are the same as those in the original text. Figure 9 The token embedding 902 and the related embeddings 912 and 922 are the same. It is important to note that... Figure 10 Only the details of encoder layer 610 are described; the details of encoder layers 620 and 630 are not described.

[0139] The token self-focus module 530 modifies the token embedding 902 according to a set of focus weights. Each of the metadata self-focus modules 540 modifies the relevance embeddings 912 and 922 respectively according to the corresponding metadata focus weight set. The fusion module 550 combines the results of modules 530 and 540. An optional addition and normalization module 1010 adds the output from the fusion module 550 to the embedding 902, then normalizes the result and passes the normalized result to the feedforward 1030. An optional addition and normalization module 1040 adds the output of the feedforward 1030 to the input of the feedforward 1030 and normalizes the result, then passes the result to the encoder layer 420. The encoder layer 430 (the last encoder layer in the series) generates the encoder output 1050.

[0140] See Figure 11 This figure depicts a further example of transformer-based encoding incorporating metadata according to an illustrative embodiment. Decoder module 325 and... Figure 3 The decoder module 325 is the same. Decoder layers 610, 620, and 630 are the same as... Figure 6 The encoder layers 610, 620, and 630 are identical. The token self-focus module 710, metadata self-focus module 720, fusion module 730, and focus module 740 are the same. Figure 7 The token self-focusing module 710, one or more metadata self-focusing modules 720, fusion module 730, and focus module 740 are identical. Token embedding 902 and relevance embeddings 912 and 922 are... Figure 9 The token embedding 902 and the related embeddings 912 and 922 are the same. Encoder output 1050 is... Figure 10 The encoder output is the same as 1050. It is important to note that... Figure 11 Only the details of decoder layer 610 are described; the details of decoder layers 620 and 630 are not described.

[0141] The token self-focus module 710 modifies the token embedding 902 according to a set of focus weights. Each of the metadata self-focus modules 720 modifies the relevance embeddings 912 and 922 according to its corresponding metadata focus weight set. The fusion module 730 combines the results of modules 710 and 720. An optional addition and normalization module 1110 adds the outputs from the fusion module 730 and embedding 902, then normalizes the result and passes the normalized result to the focus module 740. The focus module 740 combines the output from the fusion module 740 or the addition and normalization module 1110 with the encoder output 1050. An optional addition and normalization module 1120 adds the output from the focus module 740 and the input to the focus module 740, then normalizes the result and passes the normalized result to the feedforward module 1030. An optional addition and normalization module 1140 adds the output of the feedforward module 1130 and the input to the feedforward module 1130, normalizes the result, and passes the result to the decoder layer token 1150.

[0142] See Figure 12 This figure depicts a further example of transformer-based encoding incorporating metadata, according to an illustrative embodiment. Training module 330 and... Figure 3 The training module 330 is the same.

[0143] Specifically, Figure 12 The training of models for specific downstream tasks is described. Thus, training module 330 trains an untrained encoder model 1220 using a dialogue training corpus 1210 to produce a trained encoder model 1230, and then further trains model 1230 using dialogue task-specific training data 1240 to produce a dialogue task-trained encoder model 1250. Similarly, training module 330 trains an untrained encoder model 1220 using a document training corpus 1212 to produce a trained encoder model 1232, and then further trains model 1232 using document task-specific training data 1242 to obtain a document task-trained dialogue model 1252.

[0144] refer to Figure 13 The figure depicts a flowchart of an example process for transformer-based encoding incorporating metadata according to an illustrative embodiment. Process 1300 can be performed in... Figure 3 The application is implemented in 300.

[0145] In box 1302, the application constructs a relevance matrix from metadata of non-textual data or a corpus of natural language text documents. In box 1304, the application initializes the parameter set of the token embedding part of the encoder model to the basic set of token embedding parameters. In box 1306, the application initializes the parameter set of the token self-focused part of the model to the basic set of token self-focused parameters. In box 1308, the application partially trains the model by adjusting the parameter sets of the relevance embedding part and the metadata self-focused part, while the parameter set of the token embedding part is set to the basic set of token embedding parameters, and the parameter set of the token self-focused part is set to the basic set of token self-focused parameters. In box 1310, the application trains the partially trained model by adjusting the model's parameter set. The application then terminates.

[0146] See now Figure 14 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. Nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services for which cloud consumers do not need to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N depicted are intended to be illustrative only, and the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0147] See now Figure 15 This demonstrates a cloud computing environment of 50 ( Figure 14 This provides a set of functional abstraction layers. It should be understood in advance that the depicted components, layers, and functions are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0148] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframe 61; based on...

[0149] A RISC (Reduced Instruction Set Computer) architecture server 62; a server 63; a blade server 64; a storage device 65; and a network and networking component 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0150] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0151] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources based on anticipated future needs according to the SLA.

[0152] Workload tier 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this tier include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and application selection based on cumulative vulnerability risk assessment 96.

[0153] Therefore, computer-implemented methods, systems, or devices, and computer program products, based on transformer-based coding, are provided in the illustrative embodiments for incorporating metadata and other relevant features, functions, or operations. When embodiments or portions thereof are described with respect to the type of device, the computer-implemented methods, systems, or devices, computer program products, or portions thereof are adapted or configured for use with suitable and equivalent performance to that type of device.

[0154] Where the embodiments are described as being implemented within an application, the delivery concept of the application in the Software as a Service (SaaS) model is within the scope of the illustrative embodiments. In the SaaS model, the capabilities of the application implementing the embodiments are provided to the user by executing the application within a cloud infrastructure. Users can access the application using various client devices through thin client interfaces such as web browsers (e.g., web-based email) or other lightweight client applications. Users do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, or storage of the cloud infrastructure. In some cases, users may not even manage or control the capabilities of the SaaS application. In some other cases, the SaaS implementation of the application may allow for limited user-specific application configuration settings that may be anomalous.

[0155] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0156] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0157] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0158] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0159] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0160] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other apparatus to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0161] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other apparatus to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other apparatus to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other apparatus perform the functions / actions specified in one or more boxes in a flowchart and / or block diagram.

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A computer-implemented method for transformer-based encoding, comprising: constructing a relevance matrix from metadata of a corpus of natural language text documents, a row-column intersection in the relevance matrix corresponding to a relationship between two instances of turn-based metadata of a conversation; and training an encoder model for computing embeddings corresponding to tokens of natural language text documents within the corpus and the relevance matrix, the encoder model comprising a first encoder layer comprising a token embedding portion, a relevance embedding portion, a token self-attention portion, a metadata self-attention portion, and a fusion portion, the relevance embedding portion generating input relevance embeddings encoding entries in the relevance matrix, the metadata self-attention portion adjusting the input relevance embeddings according to a set of metadata attention weights, the fusion portion combining outputs of the token self-attention portion and the metadata self-attention portion, the training comprising adjusting a set of parameters of the encoder model, the training generating a trained encoder model, wherein the training comprises a training phase in which parameters of the token embedding portion are held constant with parameters of the token self-attention portion, and wherein the training phase further adjusts parameters of the relevance embedding portion and at least one parameter selected from a set of parameters comprising the metadata self-attention portion, another attention portion, and the fusion portion.

2. The computer-implemented method of claim 1, wherein, the token embedding portion computes a set of token embeddings, a token embedding in the set of token embeddings corresponding to a token of a natural language text document within the corpus.

3. The computer-implemented method of claim 2, wherein, the token comprises a portion of a word of the natural language text document.

4. The computer-implemented method of claim 2, wherein, the token embedding comprises a multi-dimensional numerical representation of the token.

5. The computer-implemented method of claim 2, wherein, the token embedding comprises a combination of a multi-dimensional numerical representation of the token, a multi-dimensional numerical representation of a position of the token within the natural language text document, and a multi-dimensional numerical representation of a segment of the natural language text document in which the token is located.

6. The computer-implemented method of claim 1, wherein, the relevance embedding portion computes a set of relevance embeddings, a relevance embedding in the set of relevance embeddings comprising a multi-dimensional numerical representation of the row-column intersection.

7. The computer-implemented method of claim 1, wherein, the token self-attention portion adjusts an input token embedding according to a set of token attention weights, a token attention weight in the set of token attention weights corresponding to a relationship within the natural language text document between two tokens, the set of token attention weights computed during the training.

8. The computer-implemented method of claim 1, wherein, the set of metadata attention weights are computed during the training.

9. The computer-implemented method of claim 1, wherein, the training comprises: initializing a set of parameters of the token embedding portion to a token embedding parameter base set; initializing a set of parameters of the token self-attention portion to a token self-attention parameter base set; a first training generating a partially trained encoder model, the first training comprising adjusting a set of parameters of the relevance embedding portion and a set of parameters of the metadata self-attention portion when the set of parameters of the token embedding portion is set to the token embedding parameter base set and the set of parameters of the token self-attention portion is set to the token self-attention parameter base set; and a second training to generate a trained encoder model, the second training comprising adjusting a parameter set of the partially trained encoder model.

10. The computer-implemented method of claim 1, wherein, the encoder model further comprises a first decoder layer comprising a decoder token self-attention portion, a decoder metadata self-attention portion, a decoder fusion portion, and a decoder attention portion, the training comprising adjusting a parameter set of the first decoder layer.

11. The computer-implemented method of claim 10, wherein, the decoder attention portion adjusts an output of an encoder layer according to an attention weight set, the attention weight set being computed during the training.

12. A computer program product for transformer-based coding, the computer program product comprising: program instructions comprising program instructions for performing the method according to any one of claims 1 to 11.

13. A computer system comprising one or more processors, one or more computer- readable memories, and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more computer-readable storage devices for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, the stored program instructions comprising program instructions for performing the method according to any one of claims 1 to 11.

Images