Systems and methods for semantic-based pre-training for dialogue understanding

Abstract

A system and method for pre-training a dialog model using semantic information, the system and method including: generating a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker; learning core semantic units of the input dialog based on nodes of the dialog-level AMR graph; learning semantic relationships between words of sentences of the input dialog based on edges of the dialog-level AMR graph; learning an overall agreement between the input dialog and the dialog-level AMR graph; and training a dialog model based on the learned core semantic units, the semantic relationships between words, and the overall agreement.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to U.S. patent application Ser. No. 17 / 887,134, filed August 12, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] Apparatus and methods consistent with example embodiments of the present disclosure relate to a semantic-based pre-training framework that leverages deep semantic representations for dialog pre-training.

[0003] 2. Description of Related Art Semantic knowledge is used in both social chat systems and task-oriented dialog systems. For example, PEGASUS is a spoken language interface for online air travel planning that converts sentences into semantic frames that are then used to plan the trip. As another example, a semantic dialog model can be used to perform database operations based on semantic features. As yet another example, conversational semantic parsing can be used to combine intents and slots into a semantic tree and solve intent classification and slot-filling tasks as semantic parsing. Conversational semantic parsing can also be used to represent task-oriented dialogs as semantic graphs and perform dialog state tracking.

[0004] Although incorporating semantic information into dialogue systems has been shown to be useful for many dialogue tasks, existing models are typically trained on superficial dialogue text and have proven weak in understanding the primary semantic meaning of the dialogue context. Furthermore, such methods focus only on domain-specific benchmark data, leaving the full potential of semantic structure untapped. Furthermore, such methods require either human annotation or external parsers to capture semantic structure, which increases the cost of real-world applications and / or introduces error propagation. Summary of the Invention

[0005] According to various embodiments, systems and methods are provided for performing semantic-based pre-training for dialog understanding.

[0006] According to aspects of one or more exemplary embodiments, a method performed by at least one processor for pre-training a dialog model using semantic information includes generating a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker; learning core semantic units of the input dialog based on nodes of the dialog-level AMR graph; learning semantic relationships between words of sentences of the input dialog based on edges of the dialog-level AMR graph; learning an overall agreement between the input dialog and the dialog-level AMR graph; and training a dialog model based on the learned core semantic units, the semantic relationships between words, and the overall agreement.

[0007] The method includes constructing an utterance-level AMR graph by converting utterances in an input dialog into AMR individually using a pre-trained AMR parser, and connecting the utterance-level AMR graph with a root node such that edges of the AMR graph are labeled with associated speakers.

[0008] The method includes identifying one or more semantic perceptual units of an input dialog based on nodes of a dialog-level AMR graph, and increasing attention given by a dialog model to tokens of the input dialog corresponding to the one or more semantic perceptual units.

[0009] The method includes identifying tokens of the input dialogue that are aligned with nodes of the dialogue-level AMR graph as semantic perceptual units of the input dialogue.

[0010] The method includes assigning a masking probability to each token of the input dialog such that the masking probability assigned to a token corresponding to one or more semantic perceptual units is higher than the masking probabilities assigned to other tokens of the input dialog.

[0011] The method includes projecting edges of a dialogue-level AMR graph to corresponding sentences of an input dialogue according to node-to-word alignments, and training a predictor to generate the projected edges.

[0012] The method includes generating hidden states for contextualized words of an input dialogue by using a Transformer encoder, and predicting relationships between words based on the hidden states by using a deep biaffine neural parser.

[0013] The method includes linearizing a dialog-level AMR graph, converting the linearized AMR to a set of hidden states using a pre-trained encoder, and maximizing a similarity score between the set of hidden states and the dialog-level AMR graph.

[0014] The method includes using cosine similarity as a distance scoring operation and employing a contrastive learning framework to train a dialogue model.

[0015] According to an aspect of one or more exemplary embodiments, an apparatus for pre-training a dialog model using semantic information includes: a memory having computer programming code stored thereon; and at least one processor configured to operate as instructed by the computer programming code, the computer programming code including: generation code configured to cause the at least one processor to generate a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker; training code configured to cause the at least one processor to: learn core semantic units of the input dialog based on nodes of the dialog-level AMR graph, learn semantic relationships between words of sentences of the input dialog based on edges of the dialog-level AMR graph, and learn overall agreement between the input dialog and the dialog-level AMR graph; and training code configured to cause the at least one processor to train a dialog model based on the learned core semantic units, the semantic relationships between words, and the overall agreement.

[0016] The apparatus includes construction code configured to cause at least one processor to construct an utterance-level AMR graph by converting utterances in an input dialog individually into AMR using a pre-trained AMR parser; and connection code configured to cause the at least one processor to connect the utterance-level AMR graph with a root node, where edges of the AMR graph are labeled with associated speakers.

[0017] The apparatus includes an identification code configured to cause at least one processor to identify one or more semantic perceptual units of an input dialog based on nodes of a dialog-level AMR graph, and an attention increasing code configured to cause the at least one processor to increase attention given by a dialog model to tokens of the input dialog that correspond to the one or more semantic perceptual units.

[0018] The apparatus is configured to cause at least one processor to identify tokens of the input dialog that are aligned with nodes of the dialog-level AMR graph as semantic perceptual units of the input dialog.

[0019] The apparatus includes attention augmentation code configured to cause at least one processor to assign a masking probability to each token of an input dialog such that the masking probability assigned to a token corresponding to one or more semantic perceptual units is higher than the masking probabilities assigned to other tokens of the dialog.

[0020] The apparatus includes training code configured to cause at least one processor to project edges of a dialogue-level AMR graph onto corresponding sentences of an input dialogue according to a node-to-word alignment, and train a predictor to generate the projected edges.

[0021] The apparatus includes training code configured to cause at least one processor to generate hidden states for contextualized words of an input dialogue by using a transformer encoder, and predict relationships between the words based on the hidden states by using a deep bi-fin neural parser.

[0022] The apparatus includes training code configured to cause at least one processor to linearize a dialogue-level AMR graph, convert the linearized AMR to a set of hidden states using a pre-trained encoder, and maximize a similarity score between the set of hidden states and the dialogue-level AMR graph.

[0023] The apparatus includes training code configured to cause at least one processor to use cosine similarity as a distance scoring operation and employ a contrastive learning framework to train a dialogue model.

[0024] In accordance with aspects of one or more exemplary embodiments, a non-transitory computer-readable medium has stored thereon a computer program that, when executed by a processor, causes the processor to: generate a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker; learn core semantic units of the input dialog based on nodes of the dialog-level AMR graph; learn semantic relationships between words of sentences of the input dialog based on edges of the dialog-level AMR graph; learn overall agreement between the input dialog and the dialog-level AMR graph; and train a dialog model based on the learned core semantic units, the semantic relationships between words, and the overall agreement.

[0025] Additional aspects will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practicing the embodiments set forth in the present disclosure.

[0026] Features, aspects, and advantages of certain exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements and in which: [Brief explanation of the drawings]

[0027] [Figure 1] 1 illustrates an AMR graph in accordance with one or more exemplary embodiments. [Figure 2a] 10 illustrates another AMR graph in accordance with one or more exemplary embodiments. [Figure 2b] 1 illustrates a semantically guided masking strategy according to one or more exemplary embodiments. [Figure 2c] 1 illustrates semantic relationship prediction in accordance with one or more exemplary embodiments. [Figure 2d] 1 illustrates a semantic match in accordance with one or more exemplary embodiments. [Figure 3] FIG. 1 illustrates a flow diagram of a method for pre-training a dialogue model using semantic information, according to one or more exemplary embodiments. [Figure 4] 1 depicts an illustration of components of one or more devices in accordance with one or more exemplary embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0028] The following detailed description of exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0029] The foregoing disclosure has been presented for purposes of illustration and description and is not intended to be exhaustive or to limit the scope of the present invention to the precise implementations disclosed. Modifications and variations are possible in light of the above disclosure or may be realized by practicing the implementations. Furthermore, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Additionally, in the process flow diagrams or descriptions presented below, it is understood that one or more steps may be omitted, one or more steps may be added, one or more steps may be performed (at least in part) simultaneously, and one or more steps may be reordered.

[0030] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement such systems and / or methods does not limit the manner in which this document is implemented. Thus, this document describes the steps and behavior of the systems and / or methods without reference to specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0031] Although particular combinations of features are claimed herein and / or disclosed in the specification, such combinations are not intended to limit the disclosure of possible implementations. Indeed, many such features may be combined in ways not specifically claimed herein and / or disclosed herein. Although each dependent claim set forth below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claims herein.

[0032] No element, act, or instruction used herein should be construed as critical or essential unless expressly identified as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with the word "one or more." Where only one item is intended, the word "one" or similar words are used. Also, as used herein, the words "has / have / having," "include / including," or similar terms are intended to be open-ended. Furthermore, the phrase "based on" is intended to mean "based, at least in part, on," unless expressly specified otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include A only, B only, or both A and B.

[0033] As mentioned above, related techniques train models on superficial dialogue text, focus only on domain-specific benchmark data, and require either human annotation or external parsers. Therefore, related techniques do not exploit the full potential of semantic structures, increase costs, may cause error propagation, and are known to be weak in understanding the key semantic meanings of dialogue contexts.

[0034] Exemplary embodiments provide systems and methods for performing semantic pre-training for dialog understanding. According to one aspect of the present disclosure, a semantic pre-training framework that leverages deep semantic representations for dialog pre-training is provided. For example, a semantic graph-based pre-training for dialogs (SARA) framework may extend standard pre-training frameworks with three tasks: learning core semantic units, semantic relations, and overall semantic representations according to an abstract semantic representation (AMR) graph. The SARA framework may enhance pre-trained dialog models with semantic information during pre-training. This can be achieved by using AMR as explicit semantic knowledge / structure during pre-training to capture core semantic information in dialogs for more granular supervision when pre-training the model. This improves the model's ability to infer semantic structure from conversations, eliminating the need for an external AMR parser in downstream applications.

[0035] 1 illustrates an AMR graph in accordance with one or more example embodiments. As shown in FIG. 1: "The police hummed to the boy as he walked to town." Regarding the sentence, AMR highlights the core semantic units in this sentence (e.g., "police," "hum," and "boy") and uses a rooted directed graph to connect them with semantic relations (e.g., ":arg0," ":time").

[0036] According to one aspect of the present disclosure, AMR graphs can be explicitly leveraged for pre-training dialog models. For example, the SARA framework supports three pre-training subtasks: 1) semantic-based masked language modeling (MLM), 2) semantic relation prediction, and 3) semantic matching. The semantic-based masked language modeling subtask may extend the standard masked language modeling task by increasing attention to core semantic units in the dialogue to learn core semantic units. The semantic relation prediction subtask may learn semantic relations between words. The semantic matching subtask may learn global matches between the dialogue and its corresponding AMR graph. In this way, the SARA framework combines the advantages of contextualized representations of pre-trained models and explicit semantic knowledge, while eliminating the need for external semantic parsers in downstream applications.

[0037] According to one aspect of the present disclosure, a pre-trained dialogue model (e.g., a pre-trained Transformer Encoder) may be used to continue pre-training the model on a dialogue(s) in a multi-task setting, using the AMR(s) of the dialogue(s) as explicit semantic knowledge for continued pre-training. This multi-task setting may include three pre-training sub-tasks (e.g., semantic-based masking, semantic relation prediction, and semantic matching).

[0038] According to one aspect of the present disclosure, one dialog-level AMR graph may be created for one input dialog. For example, multiple dialog-level AMR graphs may be created for each of multiple input dialogs. Each input dialog may include multiple sentences attributed to one or more speakers. The dialog-level AMR graph may be created by constructing one or more utterance-level AMR graphs and connecting these utterance-level AMR graphs to a root node, where edges are labeled with corresponding speakers. The utterance-level AMR graph may be generated by converting utterances individually into AMR using a pre-trained AMR parser.

[0039] For example, the input dialogue sequence is TIFF0007819359000001.tif5170, wherein: TIFF0007819359000002.tif3170 is the number of tokens in the dialog. The corresponding AMR is a directed acyclic graph. TIFF0007819359000003.tif4170, wherein TIFF0007819359000004.tif4170 represents a set of nodes (i.e., AMR concepts), TIFF0007819359000005.tif4170 represents a set of labeled edges (i.e., AMR relationships). An edge is It may also be expressed as a triplet of TIFF0007819359000006.tif5170, where Node from TIFF0007819359000007.tif4170 Edge label to TIFF0007819359000008.tif4170 It has TIFF0007819359000009.tif4170.

[0040] According to one aspect of the present disclosure, a semantic-guided masking strategy may be implemented by the semantic-based MLM subtask. The semantic-guided masking strategy may increase attention given by the model to tokens containing important semantic information, instead of processing all tokens equally and potentially wasting resources on tokens that provide little signal (such as punctuation or stop words). Tokens containing important semantic information may be referred to as semantic perceptual units (i.e., core semantic units). A token may be identified as a semantic perceptual unit once it is aligned with an AMR node according to AMR-to-text alignment.

[0041] 2a illustrates another AMR graph in accordance with one or more exemplary embodiments. The AMR graph of FIG. “The police could help the housewife.” This corresponds to the sentence, where the tokens "The police," "could," and "help" can be aligned with nodes in the AMR graph and identified as semantic perceptual units.

[0042] 2b illustrates a semantically guided masking strategy in accordance with one or more exemplary embodiments. As shown in FIG. 2b, the semantically guided masking strategy assigns a higher masking probability to tokens that contain important semantic information (e.g., "police," "could," "help") compared to other tokens (e.g., "The" and "the").

[0043] Pre-trained models typically use vocabularies with subword units, so alignment pairs For TIFF0007819359000010.tif5170, the alignment is TIFF0007819359000011.tif6170 and may be extended, in which the AMR node TIFF0007819359000012.tif4170 is The set of all tokens that are subwords of the word TIFF0007819359000013.tif4170 For example, as shown in Figure 2a, the AMR node "housewife" is aligned with the subtokens "house" and "##wife."

[0044] The index of tokens identified as semantic perceptual units by the semantic-guided masking strategy is The semantic-based MLM subtask may be represented as TIFF0007819359000015.tif5170, and the semantic-based MLM subtask is the optimization of the training goal. It can be defined as TIFF0007819359000016.tif11170.

[0045] According to one aspect of the present disclosure, the subtask of semantic relation prediction may be designed to learn semantic relations between words. To this end, before training the predictor to generate projected edges, each edge of the input AMR graph (corresponding to an input dialogue) may be mapped to a corresponding sentence according to node-to-word alignment. For example, because AMR relations are defined with respect to AMR nodes rather than words in the dialogue text, the node-to-word alignment may be used. AMR edge using TIFF0007819359000017.tif4170 TIFF0007819359000018.tif4170 is It can be projected onto text using the rule TIFF0007819359000019.tif11170.

[0046] To train a predictor to generate projected edges, we use a transform encoder to generate hidden states of contextualized words. TIFF0007819359000020.tif5170 can be generated first. Based on this hidden state, relationships between words can be predicted using a deep bi-fin neural parser. For example, From TIFF0007819359000021.tif4170 To determine whether a directed edge (or arc) exists, the BiAfin parser uses two separate multilayer perceptrons (MLPs) ( TIFF0007819359000023.tif5170 and TIFF0007819359000024.tif5170) to obtain two low-dimensional representation vectors for each position, and then perform the bi-fin operation. The score may be calculated according to TIFF0007819359000025.tif26170, where: TIFF0007819359000026.tif6170 is the acronym is the representation vector of TIFF0007819359000027.tif4170, TIFF0007819359000028.tif6170 is a subordinate word TIFF0007819359000029.tif4170 represents the vector, TIFF0007819359000030.tif4170 is the parameter matrix, and these Probability of label assignment to TIFF0007819359000031.tif5170 ( TIFF0007819359000032.tif6170) is calculated. Therefore, the subtask of semantic relation prediction is optimized for the training objective. TIFF0007819359000033.tif12170, where: TIFF0007819359000034.tif4170 represents the projected AMR edges.

[0047] Figure 2c illustrates semantic relation prediction according to one or more exemplary embodiments. As shown in Figure 2c, edges in the AMR graph of Figure 2a are mapped to tokens in the corresponding sentence according to node-to-word alignment. For example, the directed edge ":arg0" from the node "help-01" to the node "police" shown in Figure 2a maps from the token "help" to the token "police" in the corresponding sentence shown in Figure 2c. As another example, because the word "housewife" in Figure 2a corresponds to the first token "house" and the second token "##wife," the edge ":arg1" from the node "help-01" to the node "housewife" maps from the token "help" to the tokens "house" and "##wife" in the corresponding sentence shown in Figure 2c.

[0048] According to one aspect of the present disclosure, the semantic matching subtask can be designed to encourage the model to learn a global match between a dialogue and its corresponding AMR graph. For example, an auxiliary network can be used to encode the AMR and maximize the similarity score between the hidden states of the text and the AMR. To achieve this goal, the AMR graph can be linearized into a sequence, and a pre-trained encoder can be used to convert this AMR graph into a set of hidden states. The linearized AMR graph can be TIFF0007819359000035.tif5170, and the vector representation of the text and the corresponding AMR may be calculated as follows: TIFF0007819359000036.tif12170In formula, TIFF0007819359000037.tif5170 is a text encoder, TIFF0007819359000038.tif5170 is an AMR encoder.

[0049] 2d illustrates semantic matching according to one or more exemplary embodiments. As shown in FIG. 2d, the linearized AMR graph is processed by an AMR encoder (e.g., TIFF0007819359000039.tif5170) and the corresponding text can be generated from a text encoder (e.g. TIFF0007819359000040.tif5170) can be input to the encoder TIFF0007819359000041.tif5170 and TIFF0007819359000042.tif5170 may be initialized with the same weights but updated separately during training, and a pooling operation may be used to reduce the sequence of vectors to one vector. For example, the hidden state of the first input token may be input to an MLP layer to obtain a "pooled" vector.

[0050] According to one aspect of the present disclosure, cosine similarity may be used as the distance scoring operation, and a contrastive learning framework may be employed to train a model with the goal of grouping semantically close text-AMR pairs together and separating unpaired examples. For TIFF0007819359000043.tif3170, the positive examples are the corresponding AMR graphs. TIFF0007819359000044.tif4170, and the negative examples are AMR graphs of nearby dialogues in that corpus. TIFF0007819359000045.tif6170 and TIFF0007819359000046.tif5170 in the dataset By representing the representation of the 4170th example pair, the semantic matching subtask is optimized for the training objective. TIFF0007819359000048.tif12170, wherein: TIFF0007819359000049.tif5170 represents cosine similarity, TIFF0007819359000050.tif5170 Collect the neighborhood index of the 4170th example in TIFF0007819359000051.tif, TIFF0007819359000052.tif4170 represents the temperature hyperparameter.

[0051] According to one aspect of the present disclosure, the model calculates the total loss of the above three subtasks. It can be trained by optimizing TIFF0007819359000053.tif5170. During the ceremony, TIFF0007819359000054.tif3170 and TIFF0007819359000055.tif5170 is TIFF0007819359000056.tif5170 and TIFF0007819359000057.tif5170. To reduce the amount of computation required, training can continue using a pre-trained model on text input.

[0052] FIG. 3 illustrates a flow diagram of a method 300 for pre-training a dialogue model using semantic information, according to one or more example embodiments.

[0053] Method 300 includes generating a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker, at 302. For example, method 300 may include constructing an utterance-level AMR graph by converting utterances in the dialog individually into AMR using a pre-trained AMR parser, and connecting the utterance-level AMR graph with a root node, where edges of the AMR graph are labeled with associated speakers.

[0054] At 304, the method 300 includes learning core semantic units of the input dialog based on the nodes of the dialog-level AMR graph. For example, the method 300 may include identifying one or more semantic perceptual units of the input dialog based on the nodes of the dialog-level AMR graph and increasing attention given by a dialog model to tokens of the input dialog corresponding to the one or more semantic perceptual units. The method 300 may further include identifying tokens of the input dialog aligned with nodes of the dialog-level AMR graph as semantic perceptual units of the input dialog. The method 300 may further include assigning a masking probability to each token of the input dialog such that the masking probability assigned to the token corresponding to the one or more semantic perceptual units is higher than the masking probability assigned to other tokens of the input dialog.

[0055] At 306, the method 300 includes learning semantic relationships between words of sentences of the input dialog based on edges of the dialog-level AMR graph. For example, the method 300 may include projecting edges of the dialog-level AMR graph to corresponding sentences of the input dialog according to node-to-word alignments and training a predictor to generate the projected edges. The method 300 may further include generating hidden states for contextualized words of the input dialog by using a transformer encoder and predicting relationships between words based on the hidden states by using a deep bi-fin neural parser.

[0056] The method 300 includes learning a global match between the input dialog and the dialog-level AMR graph, at 308. For example, the method 300 may include linearizing the dialog-level AMR graph and converting the linearized AMR to a set of hidden states using a pre-trained encoder, and maximizing a similarity score between the set of hidden states and the dialog-level AMR graph. The method 300 may further include using cosine similarity as a distance scoring operation and employing a contrastive learning framework to train the dialog model.

[0057] The method 300 includes, at 310, training a dialogue model based on the learned core semantic units, the semantic relationships between words, and the overall agreement. For example, the method 300 can be implemented using the formula This may include optimizing the total loss according to TIFF0007819359000058.tif5170.

[0058] 4 shows an illustration of components of one or more devices in accordance with one or more example embodiments. With reference to FIG. 4, device 400 may include a bus 410, a processor 420, a memory 430, a storage component 440, and a communication interface 450. It will be understood that one or more of the components may be omitted and / or one or more additional components may be included.

[0059] Bus 410 includes components that enable communication between components of device 400. Processor 420 may be implemented in hardware, firmware, or a combination of hardware and software. Processor 420 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. Processor 420 may include one or more processors that can be programmed to perform operations.

[0060] Memory 430 includes random access memory (RAM), read-only memory (ROM), and / or other types of dynamic or static storage devices (such as flash memory, magnetic memory, and / or optical memory) that store information and / or instructions used by processor 420.

[0061] Storage component 440 stores information and / or software related to the operation and use of device 400. For example, storage component 440 may include a hard disk (such as a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium and corresponding drives.

[0062] Communications interface 450 includes transceiver-like components (e.g., a transceiver and / or separate receivers and transmitters) that enable device 900 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 450 may enable device 400 to receive information from and / or provide information to another device. For example, communications interface 450 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0063] Device 400 may perform one or more processes or steps described herein. Device 400 may perform steps based on processor 420 executing software instructions stored by a non-transitory computer-readable medium (e.g., memory 430 and / or storage component 440). In this document, a computer-readable medium is defined as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0064] Software instructions may be read into memory 430 and / or storage component 440 from another computer-readable medium or another device via communication interface 450. The software instructions stored in memory 430 and / or storage component 440, when executed, may cause processor 420 to perform one or more of the processes described herein.

[0065] Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more of the processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardwired circuitry and software.

[0066] The number and arrangement of components shown in Figure 4 are provided as an example. In practice, device 400 may include additional, fewer, different, or differently arranged components compared to the components shown in Figure 4. Additionally or alternatively, one or more steps described as being performed by another set of components of device 400 may be performed by one or more components (e.g., one or more components) of device 400.

[0067] In various embodiments of the present disclosure, any of the steps or processes of FIGS. 1-3 may be implemented by or using any of the elements shown in FIG.

[0068] According to an exemplary embodiment, the device 400 may perform pre-training of a dialog model using semantic information. For example, the device 400 may perform the following operations: generate a dialog-level AMR graph for an input dialog associated with a speaker, learn core semantic units of the input dialog based on nodes in the dialog-level AMR graph, learn semantic relationships between words in sentences of the input dialog based on edges in the dialog-level AMR graph, learn overall agreement between the input dialog and the dialog-level AMR graph, and train a dialog model based on the learned core semantic units, semantic relationships between words, and overall agreement.

[0069] The foregoing disclosure provides illustration and description, and is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations may be possible in light of the above description or may be realized by practicing the implementation.

[0070] Some embodiments may relate to systems, methods, and / or computer-readable media at any feasible level of technical detail. Furthermore, one or more of the above-described components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include one or more computer-readable non-transitory storage media having computer-readable program instructions for causing the processor to perform operations.

[0071] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device (such as a punch card or ridge-in-groove structure having instructions recorded thereon), and any suitable combination thereof. Computer-readable storage medium, as used herein, should not be construed as being itself a transitory signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (such as a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0072] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0073] The computer-readable program code / instructions for carrying out the steps may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state configuration data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk or C++) and procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuit to perform an aspect or step.

[0074] The computer-readable program instructions may be provided to a general-purpose or special-purpose computer processor or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the computer processor or other programmable data processing apparatus, create means for implementing the steps / acts specified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium capable of instructing a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, whereby the computer-readable storage medium on which the instructions are stored comprises an article of manufacture, the article comprising instructions for implementing aspects of the steps / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0075] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to execute a series of process steps to create a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the steps / actions identified in one or more blocks of the flow diagrams and / or block diagrams.

[0076] The flow diagrams and block diagrams in the figures illustrate the architecture, functionality, and steps of possible implementations of systems, methods, and computer-readable media according to various embodiments. As such, each block in a flow diagram or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing particular logical step(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to the blocks depicted in the figures. In some alternative implementations, the steps noted in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may actually be executed concurrently or substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented by special-purpose hardware-based systems that perform particular steps or functions, or may execute a combination of special-purpose hardware and computer instructions.

[0077] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual specialized control hardware or software code used to implement such systems and / or methods does not limit the manner in which this document is implemented. Thus, the steps and behavior of the systems and / or methods are described herein without reference to specific software code—it will be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

Claims

1. 1. A method for pre-training a dialogue model using semantic information, executed by at least one processor, comprising: generating a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker; learning core semantic units of the input dialog based on the nodes of the dialog-level AMR graph, identifying one or more semantic perceptual units of the input dialog based on the nodes of the dialog-level AMR graph; increasing the attention given by the dialogue model to tokens of the input dialogue corresponding to the one or more semantic perceptual units; learning semantic relationships between words of sentences of the input dialogue based on edges of the dialogue-level AMR graph, projecting the edges of the dialogue-level AMR graph onto corresponding sentences of the input dialogue according to node-to-word alignment; training a predictor to generate projected edges; learning a global match between the input dialog and the dialog-level AMR graph; training the dialogue model based on the learned core semantic units, the semantic relationships between words, and the overall agreement; A method comprising:

2. generating the dialog-level AMR graph for the input dialog associated with the speaker, constructing an utterance-level AMR graph by converting utterances in the input dialog into AMR individually using a pre-trained AMR parser; and connecting the utterance-level AMR graph with a root node, wherein edges of the utterance-level AMR graph are labeled with the associated speaker.

3. identifying the one or more semantic perceptual units of the input dialog based on the nodes of the dialog-level AMR graph, The method of claim 1 , comprising identifying tokens of the input dialog that are aligned with nodes of the dialog-level AMR graph as semantic perceptual units of the input dialog.

4. the step of increasing the attention given by the dialogue model to tokens of the input dialogue corresponding to the one or more semantic perceptual units, 2. The method of claim 1, further comprising the step of assigning a masking probability to each token of the input dialog, wherein the masking probability assigned to a token corresponding to one or more semantic perceptual units is higher than the masking probability assigned to other tokens of the input dialog.

5. training the predictor to generate the projected edges, generating hidden states for contextualized words of the input dialogue by using a transformer encoder; and predicting relationships between words based on the hidden states by using a deep bi-fin neural parser.

6. learning the overall match between the input dialog and the dialog-level AMR graph, linearizing the dialogue-level AMR graph and converting the linearized dialogue-level AMR graph into a set of hidden states using a pre-trained encoder; and maximizing a similarity score between the set of hidden states and the dialog-level AMR graph.

7. maximizing the similarity score between the set of hidden states and the dialog-level AMR graph; using cosine similarity as the distance scoring operation; and employing a contrastive learning framework to train the dialogue model.

8. 1. An apparatus for pre-training a dialogue model using semantic information, comprising: a memory storing computer programming code; at least one processor configured to operate as instructed by said computer programming code, said computer programming code comprising: generation code configured to cause the at least one processor to generate a dialogue-level Abstract Semantic Representation (AMR) graph for an input dialogue associated with a speaker; and causing the at least one processor to generate a dialog-level AMR graph based on the nodes of the dialog-level AMR graph. learning core semantic units of the input dialogue; learning semantic relationships between words in sentences of the input dialogue based on edges of the dialogue-level AMR graph; and learning a global match between the input dialog and the dialog-level AMR graph; and training code configured to cause the at least one processor to perform the step of training the dialog model based on the learned core semantic units, the semantic relationships between the words, and the overall agreement; The learning code is an identification code configured to cause the at least one processor to perform the step of identifying one or more semantic perceptual units of the input dialog based on the nodes of the dialog-level AMR graph; and attention increasing code configured to cause the at least one processor to perform the step of increasing attention given by the dialogue model to tokens of the input dialogue corresponding to the one or more semantic perceptual units; The learning code may be transmitted to the at least one processor. projecting the edges of the dialogue-level AMR graph onto corresponding sentences of the input dialogue according to node-to-word alignment; training a predictor to generate projected edges; It is configured to run Device.

9. The generated code is: construction code configured to cause the at least one processor to construct an utterance-level AMR graph by converting utterances in the input dialog individually into AMR using a pre-trained AMR parser; and and a connection code configured to cause the at least one processor to perform the step of connecting the utterance-level AMR graph with a root node, wherein edges of the utterance-level AMR graph are labeled with the associated speaker.

10. 9. The apparatus of claim 8, wherein the identification code is configured to cause the at least one processor to perform the step of identifying tokens of the input dialog that are aligned with nodes of the dialog-level AMR graph as semantic perceptual units of the input dialog.

11. 9. The apparatus of claim 8, wherein the attention increasing code is configured to cause the at least one processor to perform a step of assigning a masking probability to each token of the input dialog, wherein the masking probability assigned to a token corresponding to one or more semantic perceptual units is higher than the masking probability assigned to other tokens of the input dialog.

12. The training code causes the at least one processor to: generating hidden states for contextualized words of the input dialogue by using a transformer encoder; predicting relationships between words based on the hidden states by using a deep bi-fin neural parser; 9. The apparatus of claim 8, configured to cause the

13. The learning code may be transmitted to the at least one processor. linearizing the dialogue-level AMR graph and converting the linearized dialogue-level AMR graph into a set of hidden states using a pre-trained encoder; maximizing a similarity score between the set of hidden states and the dialog-level AMR graph; 9. The apparatus of claim 8, configured to cause the

14. The learning code may be transmitted to the at least one processor. using cosine similarity as the distance scoring operation; employing a contrastive learning framework to train the dialogue model; 14. The apparatus of claim 13, configured to:

15. A computer program that, when executed by a processor, causes the processor to: generating a dialog-level Abstract Semantic Representation (AMR) graph for an input dialog associated with a speaker; learning core semantic units of the input dialog based on the nodes of the dialog-level AMR graph, identifying one or more semantic perceptual units of the input dialog based on the nodes of the dialog-level AMR graph; increasing attention given by a dialogue model to tokens of the input dialogue corresponding to the one or more semantic perceptual units; learning semantic relationships between words of sentences of the input dialogue based on edges of the dialogue-level AMR graph, projecting the edges of the dialogue-level AMR graph onto corresponding sentences of the input dialogue according to node-to-word alignment; training a predictor to generate projected edges; learning a global match between the input dialog and the dialog-level AMR graph; training the dialogue model based on the learned core semantic units, the semantic relationships between words, and the overall agreement; Computer program.

Images