A method and device for constructing an incomplete speech rewriting model

Abstract

The present application relates to a kind of incomplete speech rewriting model construction method and device, method includes: based on span dependency and insertion dependency dependency modeling and using node link resolution mode obtains the dependency graph of incomplete speech rewriting text editing operation;Using GPT model, the context similarity feature and / or rewriting consistency characteristic of current incomplete speech sentence are calculated, the context similarity feature and / or rewriting consistency characteristic are used to enhance the interactive inference of incomplete speech rewriting;The dependency graph score feature is fused with the context similarity feature and / or rewriting consistency characteristic, and the final feature after the feature fusion is pushed to rewrite result based on. More abundant semantic features can be provided for parsing model, and the speech rewriting effect is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-turn dialogue modeling, and in particular to an incomplete utterance rewriting model construction method and device. BACKGROUND

[0002] Due to the existence of ellipsis and coreference in more than 70% of utterances, incomplete utterance rewriting (IUR) has become an important task in multi-turn dialogue modeling and understanding. It rewrites ellipsis and coreference from dialogue context, aiming to reconstruct an incomplete utterance into an utterance with the same semantics but sufficient independence without context.

[0003] Related technologies using sequence labeling patterns for ellipsis identification cannot model the interaction between missing concepts and their contextual references, which leads to the inability of the model to make fully informed and explicit decisions. In addition to sequence labeling patterns, other technologies also rely on structures such as syntax and semantics to directly encode the interaction between surface tokens, thereby implicitly broadening the locality of tokens.

[0004] Research has found that current incomplete utterance rewriting models based on pre-trained language models mainly face two major problems: first, in pre-trained models, the resolution of inter-sentence coreference in incomplete utterance rewriting is still inevitably affected by the flexible correlation between utterances and long-term dependencies, and its dependence on complex knowledge sources and reasoning mechanisms. Second, in the dialogue context of any length, identify joint ellipsis or coreference between sentences. This difficulty not only makes it difficult to predict ellipsis or coreference, but also makes utterance rewriting confusing.

[0005] Therefore, although having performance advantages, the rewriting performance of current text editing models based on sequence labeling patterns in low-resource environments still has a large gap compared to humans. How to provide more semantic features for parsing models to provide clearer clues and help complex incomplete utterance rewriting reasoning is a problem that needs to be solved. SUMMARY

[0006] The embodiments of the present application provide an incomplete utterance rewriting model construction method and device, which can provide more semantic features for parsing models to improve utterance rewriting effect.

[0007] In a first aspect, the embodiments of the present application provide an incomplete utterance rewriting model construction method, characterized in that the method comprises the steps of:

[0008] dependency modeling based on span dependency relations and insertion dependency relations, the span dependency relations being used to determine a range of a phrase to be inserted in a context, the insertion dependency relations being used to determine an insertion position in an incomplete utterance, and obtaining a dependency graph of text editing operations for rewriting the incomplete utterance using a node-link parsing approach;

[0009] computing context similarity features and / or rewrite consistency features for the current incomplete utterance using a GPT model, the context similarity features and / or rewrite consistency features being used to enhance interactive inference for rewriting the incomplete utterance;

[0010] performing feature fusion on the dependency graph scoring features and the context similarity features and / or rewrite consistency features, and predicting a rewriting result based on final features after the feature fusion.

[0011] In some embodiments, the dependency modeling based on span dependency relations and insertion dependency relations and the node-link parsing approach for obtaining the dependency graph of text editing operations for rewriting the incomplete utterance include the steps of:

[0012] concatenating the context of length n and the incomplete utterance as an input sequence X, and X = w1, w2, …, wn, where w1, w2, …, wn are tokens in the input sequence X; n ;

[0013] inserting a special token BOS at the beginning of the input sequence X, and converting the input sequence X into a context-dependent vector representation H using a sequence encoder, and H = SELFATTN(Emb(X)) / BILSTM(Emb(X)), where Emb(·) represents an embedding layer, SELFATTN(·) is a Transformer encoder, and BILSTM(·) is a traditional bidirectional LSTM encoder;

[0014] introducing two independent feedforward neural layers to project the representations of the sequence encoder into different spaces, respectively;

[0015] introducing a double-affine attention scorer to score the relationship between the head and tail node pairs, and applying a multi-class cross-entropy loss to the scoring results to guide the model to learn the node-link parsing.

[0016] In some embodiments, the sequence encoder is stacked by a plurality of identical layers;

[0017] In the BiLSTM encoder setting, each token is first connected by a character-level convolutional neural network output and a randomly initialized word embedding;

[0018] In the Transformer encoder setting, only layers and embeddings defined by a pre-trained language model BERT are included.

[0019] In some embodiments, the introducing two independent feedforward neural layers to project the representations of the sequence encoder to different spaces respectively, comprises steps of:

[0020] Let the head node marked as link in the sequence be H (head) , and H (head) =FFN head (H);

[0021] Let the tail node marked as link in the sequence be H (tail) , and H (tail) =FFN tail (H), wherein FFN is a feedforward neural layer.

[0022] In some embodiments, the introducing a bi-affine attention scorer to score the relationship between the head-tail node pair and applying a multi-class cross-entropy loss to the scoring result to guide the model to learn the node link resolution, further comprises steps of:

[0023] Scoring based on a first formula, the first formula comprising:

[0024]

[0025] wherein i and j represent the indices of the head and tail nodes in the sequence respectively, W and b are learned parameters, and l is the relationship label of the link i→j, represents the scoring result of the link i→j with the label l;

[0026] Guiding the model to learn the node link resolution based on a second formula, the second formula comprising:

[0027]

[0028] wherein r(i→j) represents the relationship label between node i and node j, NONE is used to represent no relationship between the nodes, and ζ represents the cross-entropy loss between the model prediction and the standard answer.

[0029] In some embodiments, the calculating the context similarity feature, comprises steps of:

[0030] Suppose that the incomplete utterance rewriting depends on the phrase wherein u is the complement of the incomplete utterance in the context, and p is the insertion position of the complement;

[0031] For generating the context and iu <p adopting the GPT model to output the next token distribution respectively;

[0032] the generated context the token distribution of

[0033] the generated context iu <p the token distribution of

[0034] The context similarity feature is represented based on a third formula, and the third formula includes:

[0035]

[0036] wherein CSF is a context similarity feature, D KL (·) is a KL divergence.

[0037] In some embodiments, the calculation of the rewrite consistency feature includes the steps of:

[0038] inserting the phrase into the position p of the incomplete utterance, starting from the position k in the context;

[0039] The rewrite consistency feature is calculated based on a fourth formula, and the fourth formula includes:

[0040]

[0041] wherein, represents a list operation from 1 to m elements, and RCF represents a rewrite consistency feature.

[0042] In some embodiments, the feature fusion includes the steps of:

[0043] Assuming that the context length is m and the incomplete utterance length is n;

[0044] The context and the incomplete utterance are concatenated into a sequence, and the two features are extended to (m+n)×(m+n) by zero padding;

[0045] The feature fusion is achieved by adding the padded features to the bi-affine scores.

[0046] In some embodiments, the feature fusion by adding the padded features to the bi-affine scores includes the steps of:

[0047] The feature fusion is performed based on a fifth formula, and the fifth formula includes:

[0048]

[0049] wherein, represents a score result after the feature fusion, represents a link head node representation, a representation of a link tail node, CSF i,j a representation of a context consistency feature, RCF i,j a representation of a rewrite consistency feature, W and b are learned parameters.

[0050] In a second aspect, an embodiment of the present application provides an incomplete utterance rewriting model construction device, characterized in that it comprises:

[0051] a dependency modeling module, configured to perform dependency modeling according to a span dependency relationship and an insertion dependency relationship, and obtain a dependency graph of an incomplete utterance rewriting text editing operation in a node link resolution manner, the span dependency relationship being used to determine a range of a phrase to be inserted in a context, and the insertion dependency relationship being used to determine an insertion position in an incomplete utterance;

[0052] an interactive enhancement feature acquisition module, configured to calculate a context similarity feature and / or a rewrite consistency feature of a current incomplete utterance using a GPT model, the context similarity feature and / or the rewrite consistency feature being used to enhance interactive inference of incomplete utterance rewriting;

[0053] a feature fusion module, configured to perform feature fusion on the dependency graph scoring feature and the context similarity feature and / or the rewrite consistency feature, and push out a rewriting result based on a final feature after the feature fusion.

[0054] An embodiment of the present application provides an incomplete utterance rewriting model construction method and device, which regards incomplete utterance rewriting as a series of link decisions on input sequences and incremental construction of rewriting outputs, and adopts a resolution method based on dependency modeling to obtain a dependency graph of an incomplete utterance rewriting text editing operation. Considering that, in related technologies, inter-sentence co-reference resolution in incomplete utterance rewriting in a pre-training model is affected by flexible correlation between utterances and long-term dependency relationships, and is dependent on complex knowledge sources and reasoning mechanisms, and that it is difficult to predict co-reference or ellipsis between sentences in an arbitrary length of dialogue context, which leads to confusion in utterance rewriting, two new features of context similarity and / or rewrite consistency for incomplete utterance reasoning on a generative pre-training language model are proposed to extract rich semantic features to assist iterative decision-making of the pre-training language model at each step, which can help to better provide clues for syntactic structures to improve utterance rewriting effect. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description only some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0056] Figure 1 The flowchart of the incomplete utterance rewriting model construction method provided by the embodiments of the present application is shown in the figure.

[0057] Figure 2 The structure diagram of the incomplete utterance rewriting model construction device provided by the embodiments of the present application is shown in the figure.

[0058] Figure 3 The flowchart of the specific embodiment of the incomplete utterance rewriting model construction method provided by the embodiments of the present application is shown in the figure.

[0059] Figure 4 The schematic diagram of the incomplete utterance rewriting model provided by the embodiments of the present application is shown in the figure.

[0060] Figure 5 The schematic diagram of the context similarity feature and the rewriting consistency feature provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0062] As shown in the figure, Figure 1 The incomplete utterance rewriting model construction method provided by the embodiments of the present application comprises the following steps:

[0063] S100: dependency modeling based on span dependency and insertion dependency and obtaining the dependency graph of the incomplete utterance rewriting text editing operation by using node link parsing method;

[0064] S200: calculating the context similarity feature and / or the rewriting consistency feature, the context similarity feature and / or the rewriting consistency feature are used to enhance the interactive inference of the incomplete utterance rewriting;

[0065] S300: Feature fusion is performed on the dependency graph scoring feature, the context similarity feature and / or the rewrite consistency feature, and a rewrite result is output based on the final feature after the feature fusion.

[0066] It should be noted that the span dependency relationship is used to determine the range of the phrase to be inserted in the context, and the insertion dependency relationship is used to determine the insertion position in the incomplete utterance.

[0067] It can be understood that the embodiment of the present application regards incomplete utterance rewriting as a series of linked decisions on the input sequence and incremental construction of the rewriting output, and adopts a dependency modeling-based parsing method to obtain a dependency graph of the incomplete utterance rewriting text editing operation. In view of the problems that the inter-sentence co-reference resolution in the incomplete utterance rewriting in the pre-trained model in the related art is affected by the flexible correlation between the utterance and the long-term dependency relationship and its dependence on complex knowledge sources and reasoning mechanisms, and the difficulty in predicting the inter-sentence joint ellipsis or co-reference caused by the identification of the joint ellipsis or co-reference in the arbitrary length of the dialogue context, which leads to the confusion of the utterance rewriting, two new features of the incomplete utterance reasoning on the generative pre-trained language model are proposed: context similarity and / or rewrite consistency, to extract rich semantic features to assist the iterative decision of each step of the pre-trained language model, which can help to better provide clues for the syntactic structure, so as to improve the utterance rewriting effect. The experimental results show that the performance and efficiency are better than those of the strong text editing model and the seq2seq model.

[0068] In some embodiments, S100 comprises the steps of:

[0069] S110: Concatenate the context of length n and the incomplete utterance as an input sequence X, and X = w1, w2, …, wn, where w1, w2, …, wn are the tokens in the input sequence X. n ;

[0070] S120: Insert a special token BOS at the beginning of the input sequence X, and convert the input sequence X into a context-dependent vector representation H using a sequence encoder, and H = SELFATTN(Emb(X)) / BILSTM(Emb(X)), where Emb(·) represents an embedding layer, SELFATTN(·) is a Transformer encoder, and BILSTM(·) is a traditional bidirectional LSTM encoder.

[0071] S130: Introduce two independent feedforward neural layers to project the representations of the sequence encoder into different spaces, respectively.

[0072] S140: Introduce a double-affine attention scorer to score the relationship between the head and tail node pairs, and apply a multi-class cross-entropy loss to the scoring results to guide the model to learn the node link parsing.

[0073] Preferably, the sequence encoder is stacked by multiple identical layers; in the BiLSTM encoder setting, each token is concatenated by a character-level convolutional neural network output and a randomly initialized word embedding first; in the Transformer encoder setting, only layers and embeddings defined by a pre-trained language model BERT are included.

[0074] Further, two independent feed-forward neural layers are introduced in S130 to project the representations of the sequence encoder into different spaces, including steps of:

[0075] S131: Represent the head node linked by tokens in the sequence as H (head) , and H (head) = FFN head (H);

[0076] S132: Represent the tail node linked by tokens in the sequence as H (tail) , and H (tail) = FFN tail (H), where FFN is a feed-forward neural layer.

[0077] Further, a bi-affine attention scorer is introduced in S140 to score the relationship between the head-tail node pair and apply a multi-class cross-entropy loss to the scoring result to guide the model to learn node link resolution, further including steps of:

[0078] S141: Score based on a first formula, which includes:

[0079]

[0080] where i and j represent the indices of the head and tail nodes in the sequence, respectively, W and b are learned parameters, and l is the relationship label of the link i→j, represents the scoring result of the link i→j with label l;

[0081] S142: Guide the model to learn node link resolution based on a second formula, which includes:

[0082]

[0083] where r(i→j) represents the relationship label between node i and node j, NONE is used to represent no relationship between nodes, and ζ represents the cross-entropy loss between the model prediction and the standard answer.

[0084] In some embodiments, when calculating the context similarity feature in S200, the steps include:

[0085] S210: Assume that the incomplete utterance rewriting depends on the phrase wherein, in the context, the utterance u is a complement to the incomplete utterance, and the complement insertion position is p;

[0086] S211: For generating the context and iu <p The token distribution of the generated context iu is The token distribution of the generated context iu <p is

[0087] S212: Based on a third formula, the context similarity feature is represented, and the third formula includes:

[0088]

[0089] wherein, CSF is the context similarity feature, D KL (·) is the KL divergence.

[0090] As shown in Figure 5 , it can be understood that the context similarity feature can be reflected by the difference between the distribution of the next word predicted by the incomplete utterance and the dialogue context. The two generated texts aim to generate the same phrase u k:k+lambda , and both have similarity in the distribution of the next token prediction, so it can be understood as the context similarity hypothesis. According to the context similarity hypothesis, if the token from the kth position of the utterance u can be inserted into the position p of the incomplete utterance iu, the divergence between the two distributions should be small; that is, the distribution difference measures the extent to which a token at a specific position in the utterance can be inserted into the incomplete utterance in the incomplete utterance rewriting.

[0091] In some embodiments, when calculating the rewriting consistency feature in S200, the following steps are included:

[0092] S220: Insert the phrase in the context starting from the kth position into the pth position of the incomplete utterance;

[0093] S221: Based on a fourth formula, the rewriting consistency feature is calculated, and the fourth formula includes:

[0094]

[0095] wherein, represents a list operation from 1 to m elements, and RCF represents the rewriting consistency feature.

[0096] As Figure 5 shown, it can be understood that the rewriting consistency feature can be reflected by the consistency of the distribution probability of the same word predicted by the incomplete utterance and the dialogue context. For a typical incomplete utterance rewriting scenario, the phrase starting from the position k in the context, is inserted into the position p of the incomplete utterance, and the new utterance should be a reasonable sentence; that is, the partial sequence iu ≤p and the inserted word u k will form a new valid sentence fragment, and this phenomenon can be rewriting consistency. The rewriting consistency feature is used in the embodiments of the present application to enhance the determination of the completed word and the insertion position in the IUR.

[0097] In some embodiments, feature fusion is performed in S300, including the steps of:

[0098] S310: Assuming that the context length is m and the incomplete utterance length is n;

[0099] S320: Connect the context and the incomplete utterance into a sequence, and extend the two features to (m+n) x (m+n) by zero padding;

[0100] S330: Perform feature fusion by adding the padded features to the bilinear score.

[0101] Further, when performing feature fusion by adding the padded features to the bilinear score in S330, the feature fusion can be performed based on a fifth formula, and the fifth formula includes:

[0102]

[0103] wherein, denotes the link tail node representation, CSF i,j denotes the context consistency feature, RCF i,j denotes the rewriting consistency feature, and W and b are learned parameters.

[0104] The parameter-free probing method based on the generative pre-training language model provided in the embodiments of the present application provides rich semantic features for the analytical model, thereby providing clearer clues to help complex IUR reasoning.

[0105] As Figure 3 and Figure 4As shown, in one specific embodiment, the deep learning framework adopted is Pytorch, version 1.9.0, and CUDA version is 11.3. The hardware environment of the experiment is NVIDIA GeForce RTX 3090 graphics card, and the processor is Intel(R) Xeon(R) Gold 6240C. A flowchart of a method for constructing an incomplete utterance rewriting model includes the following steps:

[0106] S01: The context is spliced with the incomplete utterance sentence using the “[SEP]” symbol to input into the incomplete utterance rewriting model, that is, the dependency model constructed in the embodiment.

[0107] S02: The inputted concatenated sequence is vectorized using an embedding layer to obtain a context-independent vector.

[0108] S03: The context-independent vector is encoded using a self-attention layer or a recurrent neural network layer to obtain a contextualized vector.

[0109] S04: The preliminary score of the completion action of the context and the incomplete utterance is calculated using a Biaffine scorer.

[0110] S05: The consistency feature and the context similarity feature of the current incomplete utterance sentence are calculated using the GPT model.

[0111] S06, the final action inference is completed by combining the preliminary score, the consistency feature and the context similarity feature to realize the incomplete utterance rewriting.

[0112] It should be noted that the present application uses MULTI, REWRITE, TASK and CANARD datasets. The MULTI dataset and the REWRITE dataset are Chinese open-domain dialogue datasets, containing 194K and 18K data respectively. The TASK dataset is a task-oriented English dialogue dataset, containing 2.2K data. The CANARD dataset is an English question answering dataset in context, containing 32K data. In addition to language and domain, the sizes of the four training sets also differ in different scales, and this feature makes it a suitable test platform for evaluating the sensitivity of the model to data size.

[0113] Further, the embodiments of the present application use randomly initialized character embeddings of dimension 100, character embeddings of dimension 50 in the BiLSTM sequence encoder setting, and set the character convolution hidden size to 100. The embedding dropout is set to 0.2. Three BiLSTM layers are stacked with hidden size 600 and dropout 0.33. The head and tail node projection feed-forward neural layer hidden size is set to 600. In the Transformer encoder setting, the parameters of the bottom Transformer layer are fixed, while only the first 4 layers are fine-tuned. The mean-pool strategy is used to handle the inconsistency between word-level parse and subword-level representation. The feed-forward neural hidden size is kept the same as in the BiLSTM encoder setting. The model is trained for 200 epochs using the Adam optimizer, and whether to stop early is determined in view of the performance of the development set. The learning rate of the BiLSTM encoder and the Transformer encoder setting is set to 1e -3 and 5e -5 . For English, the model uses BERT-large-uncased, and for Chinese, the model uses BERT-base-chinese. For feature extraction of context similarity and rewrite consistency, the model uses GPT2-base for English data and uer / gpt2-chinese-cluecorpussmall for Chinese data.

[0114] Further, the embodiments of the present application use BLEU, ROUGE, EM automatic indicators to evaluate the incomplete utterance rewriting performance. The BLEU indicator evaluates the n-gram matching between the rewritten utterance and the optimal utterance by the cumulative n-gram BLEU score, which is denoted as BLEUn(Bn) in the present application. The ROUGE indicator measures the n-gram overlap (denoted as ROUGE n , R n ) and the longest matching (denoted as ROUGE L , R L ) between the rewritten utterance and the optimal utterance. The EM indicator is a more stringent measure that seeks exact matching between the rewritten utterance and the optimal utterance. In addition, the present application also uses rewrite precision, recall and f-score metrics for evaluation, which are calculated according to the n-gram set containing at least one word, to emphasize more words that are considered more difficult to copy, and the three are denoted as P n , R n , F n . Since utterance rewriting is also a language generation problem, the present application pays more attention to BLEU, ROUGE and EM indicators, and only uses precision, recall and f-score three indicators for small-scale comparison.

[0115] The present application was tested on MULTI dataset, REWRITE dataset, TASK dataset and CANARD dataset respectively, and compared with the existing incomplete utterance rewriting method, and the results showed that:

[0116] The test was performed on the MULTI dataset, and the results are shown in Table 1,

[0117] Table 1

[0118] Model [P2] [R1] F1 [P2] [R2] F2 DuS 67.4 37.2 47.9 53.9 30.3 38.8 L-Gen 65.5 40.8 50.3 52.2 32.6 40.1 L-Ptr-Gen 66.6 40.4 50.3 54.0 33.1 41.1 RUN 66.9 54.9 60.3 53.0 43.4 47.7 The invention 67.3 58.7 62.7 53.7 46.1 49.6 PAC 70.5 58.1 63.7 55.4 45.1 49.7 RUN+BERT 73.2 64.6 68.6 59.5 53.0 56.0 The invention+BERT 74.5 67.4 70.7 60.8 55.4 58.0

[0119] Table 1 (continued)

[0120]

[0121]

[0122] The test was performed on the REWRITE dataset, and the results are shown in Table 2:

[0123] Table 2

[0124] Model EM [B2] [B4] [R2] [R L ]]> L-Gen 47.3 81.2 73.6 80.9 86.3 L-Ptr-Gen 50.5 82.9 75.4 83.8 87.8 L-Ptr 51.5 82.7 75.5 84.0 88.2 L-Ptr-λ 42.3 82.9 73.8 81.1 84.1 T-Gen 35.4 72.7 62.5 74.5 82.9 T-Ptr-Gen 53.1 84.4 77.6 85.0 89.1 T-Ptr 53.0 83.9 77.1 85.1 88.7 T-Ptr-λ 52.6 85.6 78.1 85.0 89.0 RUN 53.8 86.1 79.4 85.1 89.5 The invention 57.6 86.7 79.8 85.3 90.8 T-Ptr-λ+BERT 57.5 86.5 79.9 86.9 90.5 RUN+BERT 66.4 91.4 86.2 90.4 93.5 The invention+BERT 69.4 90.8 86.5 90.9 94.3

[0125] The test was performed on the TASK dataset, and the results are shown in Table 3:

[0126] Table 3

[0127] Model EM B4 F1 L-Ptr-Gen 50.4 74.1 44.1 GECOR 1 68.5 83.9 66.1 GECOR 2 66.2 83.0 66.2 RUN 69.2 85.6 70.6 The invention 71.2 86.7 72.8

[0128] The test was performed on the CANARD dataset, and the results are shown in Table 4:

[0129] Table 4

[0130] Model

[001] B1 [B2] [B4] [R1] [R2] [R L ]] L-Ptr 52.4 46.7 37.8 72.7 54.9 68.5 Pronoun Sub 60.4 55.3 47.4 73.1 63.7 73.9 L-Ptr-Gen 67.2 60.3 50.2 78.9 62.9 74.9 RUN 70.5 61.2 49.1 79.1 61.2 74.7 The invention 71.5 62.8 51.3 81.0 63.5 76.0

[0131] It should be noted that the above existing incomplete utterance rewriting method mainly includes:

[0132] (1) PAC: Zhufeng Pan, Kun Bai, Yan Wang, Lianqiang Zhou, and Xiaojiang Liu. 2019. Improving Open-Domain Dialogue Systems via Multi-Turn Incomplete Utterance Restoration. In EMNLP. 1824-1833.

[0133] (2) GECOR: Jun Quan, Deyi Xiong, Bonnie Webber, and Changjian Hu. 2019. GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue. In EMNLP. 4546-4556.

[0134] (3) RUN: Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020. Incomplete Utterance Rewriting as Semantic Segmentation. In EMNLP. 2846-2857.

[0135] (4) DuS: Vineet Kumar, and Sachindra Joshi. 2016. Non-sentential Question Resolution using Sequence to Sequence Learning. In COLING. 2022-2031.

[0136] From the experimental results, Table 1 shows the performance of the incomplete utterance rewriting on the MULTI dataset. The method of the present application takes incomplete utterance rewriting as a dependency parsing problem, and proposes context similarity and rewriting consistency characteristics to strengthen the parsing process, thereby achieving better incomplete utterance rewriting performance than the previous best system, with an improvement of 2.4, 1.9, and 2.0 on F1, F2, and F3, respectively. For the finally generated rewritten utterances, improvements are obtained on the BLEU and ROUGE indicators, with an improvement of 0.8, 0.8, 0.5, and 1.3 on B1, B2, R1, and R2, respectively. On the one hand, these improvements are due to dependency modeling and node linking parser, which can find insertion spans and insertion positions more easily than UNet, as only single-step dependency derivation is required, rather than enumerating all words in the span as in RUN. On the other hand, the features of the generative pre-training language model can effectively remove unreasonable dependency arcs and reduce the search space of insertion decisions. Compared with RUN, the method proposed by the present application also achieves good improvements, with an improvement of 2.1, 2.0, 1.3, 0.8, 1.2, 0.6, and 1.2 on F1, F2, F3, B1, B2, R1, and R2, respectively. This further demonstrates the advantages of our dependency modeling compared to the formalization as semantic segmentation. In addition, the improvements on B1, B2, R1, and R2 indicators show that the method proposed by the present application makes the determination of context spans and insertion points more accurate.

[0137] The performance of the model and the baseline on the REWRITE dataset is shown in Table 2. Comparing the results of the LSTM-based baseline and the Transformer-based baseline, it can be found that the Transformer has an advantage in most systems, but the T-Gen effect is poor compared with L-Gen and T-Gen, which may be due to the lack of optimization of the Transformer. This shows that simply changing LSTM to Transformer is not necessarily the best choice, and when there is no BERT, the present method will continue to use BiLSTM as the encoder. On the REWRITE dataset, the complete model of the present application shows performance advantages with or without BERT. Due to the small size of the REWRITE dataset, in order to increase the stability of the results, the experiment uses more stringent EM, B2, B4, R2, and R L indicators, and the improvements on these more stringent indicators demonstrate the generalization ability of the method of the present application.

[0138] Tables 3 and 4 present the experimental results on the TASK and CANARD datasets. The results show that our proposed model achieves the best results on the EM, BLEU, and ROUGE metrics. In summary, the improvements achieved by our model on four datasets with different languages ​​and domains demonstrate that dependency modeling, contextual similarity features, and rewriting consistency properties are generally effective for incomplete discourse rewriting.

[0139] like Figure 2 As shown, based on the same technical concept, embodiments of the present invention also provide an apparatus for constructing an incomplete discourse rewriting model, which includes:

[0140] The dependency modeling module is used to perform dependency modeling based on span dependencies and insertion dependencies and to obtain the dependency graph of the incomplete discourse rewriting text editing operation using node link resolution. The span dependencies are used to determine the range of phrases to be inserted in the context, and the insertion dependencies are used to determine the insertion position in the incomplete discourse.

[0141] An interactive enhancement feature acquisition module is used to calculate context similarity features and / or rewrite consistency features for the current incomplete utterance using the GPT model. The context similarity features and / or rewrite consistency features are used to enhance the interactive reasoning of incomplete utterance rewriting.

[0142] The feature fusion module is used to fuse the dependency graph scoring features with the context similarity features and / or rewrite consistency features, and to derive the rewrite result based on the final features after feature fusion.

[0143] In some embodiments, the dependency modeling module is also used for:

[0144] The input sequence X is a concatenation of context and incomplete utterances of length n, where X = w1, w2, ..., w n ;

[0145] A special token BOS is inserted at the beginning of the input sequence X, and a sequence encoder is used to convert the input sequence X into a context-dependent vector representation H, where H = SELFATTN(Emb(X)) / BILSTM(Emb(X)), where Emb(·) represents the embedding layer, SELFATTN(·) is a Transformer encoder, and BILSTM(·) is a traditional bidirectional LSTM encoder.

[0146] Two independent feedforward neural layers are introduced to project the representation of the sequence encoder into different spaces respectively;

[0147] A dual-affine attention scorer is introduced to score the relationship between pairs of head and tail nodes and a multi-class cross-entropy loss is applied to the scoring results to guide the model to learn node link resolution.

[0148] Preferably, the sequence encoder is stacked by multiple identical layers; in the BiLSTM encoder setting, each token is concatenated by a character-level convolutional neural network output and a randomly initialized word embedding first; in the Transformer encoder setting, only layers and embeddings defined by a pre-trained language model BERT are included.

[0149] Further, the dependency modeling module is further configured to:

[0150] denote the head node marked as a link in the sequence as H (head) , and H (head) =FFN head (H).

[0151] denote the tail node marked as a link in the sequence as H (tail) , and H (tail) =FFN tail (H), where FFN is a feed-forward neural layer.

[0152] Further, the dependency modeling module is further configured to score based on a first formula, and the first formula includes:

[0153]

[0154] where i and j represent the indices of the head and tail nodes in the sequence respectively, W and b are learned parameters, and l is the relationship label of the link i→j, denotes the scoring result of the existence of the link i→j and the label l;

[0155] The dependency modeling module is further configured to guide the model to learn node link resolution based on a second formula, and the second formula includes:

[0156]

[0157] where r(i→j) represents the relationship label between node i and node j, NONE is used to represent no relationship between nodes, and ζ represents the cross-entropy loss between the model prediction and the standard answer.

[0158] In some embodiments, when the interaction-enhanced feature acquisition module calculates the context similarity feature,

[0159] Suppose that the incomplete utterance rewriting depends on the phrase where u is the complement of the incomplete utterance in the context, and p is the insertion position of the complement;

[0160] For generating the context and iu <p The GPT model outputs the next token distribution respectively, and generates the context The token distribution of the generated context iu The token distribution of the generated context iu <p The token distribution of the generated context iu

[0161] The third formula represents the context similarity feature, and the third formula includes:

[0162]

[0163] Wherein, CSF is the context similarity feature, D KL (·) is the KL divergence.

[0164] In some embodiments, when the interaction enhancement feature acquisition module calculates the rewriting consistency feature,

[0165] The phrase Starts from the position k in the context, and is inserted into the position p of the incomplete utterance;

[0166] The fourth formula is used to calculate the rewriting consistency feature, and the fourth formula includes:

[0167]

[0168] Wherein, Indicates the list operation from 1 to m elements, and RCF represents the rewriting consistency feature.

[0169] In some embodiments, when the feature fusion module performs feature fusion, it is assumed that the context length is m, the incomplete utterance length is n, the context and the incomplete utterance are connected into a sequence, and the two features are expanded to (m+n)×(m+n) by zero padding method. Feature fusion is realized by adding the filled features to the bi-affine fraction.

[0170] Further, the feature fusion module performs feature fusion based on the fifth formula, and the fifth formula includes:

[0171]

[0172] Point represents, Link tail node representation represents, CSF i,j The context consistency feature is represented by CSF i,j The rewriting consistency feature is represented by RCF, and W and b are learning parameters.

[0173] Those skilled in the art can understand that all or some of the steps in the method disclosed above and the functional modules / units in the system and device can be implemented by software, firmware, hardware, or a combination thereof. In a hardware implementation, the division between the functional modules / units referred to in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer-readable storage media, which can include computer-readable storage media (or non-transitory media) and communication media (or transitory media).

[0174] It should be noted that the terms "first" and "second" and the like in the present disclosure are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Also, the terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0175] The above description is merely one specific implementation of the application, and thus the skilled in the art can understand or implement the application without departing from the spirit or scope of the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an incomplete discourse rewriting model, characterized in that, The method comprises the steps of: dependency modeling based on span dependency and insertion dependency, and obtaining a dependency graph of incomplete discourse rewriting text editing operation by using a node link resolution manner, the span dependency is used to determine the range of the phrase to be inserted in the context, and the insertion dependency is used to determine the insertion position in the incomplete discourse; using a GPT model to calculate context similarity features and / or rewriting consistency features of the current incomplete discourse, the context similarity features and / or rewriting consistency features are used to enhance the interactive reasoning of incomplete discourse rewriting; feature fusion of the dependency graph scoring features and the context similarity features and / or rewriting consistency features, and deriving a rewriting result based on the final features after the feature fusion; the calculation of the context similarity features comprises the steps of: Assuming that the incomplete utterance rewriting depends on the phrase wherein in the context the utterance u is a complement to the incomplete utterance, the complement insertion position being p; For generating context And Output the next token distribution with GPT model respectively; The generated context The token distribution is ; The generated context The token distribution is ; the context similarity features are represented based on a third formula, the third formula comprises: , wherein, is a contextual similarity feature, is a KL divergence; the calculation of the rewriting consistency features comprises the steps of: phrase begins at position k in the above context, is inserted into position p of the incomplete utterance; the rewriting consistency features are calculated based on a fourth formula, the fourth formula comprises: ， wherein, represents a list operation from 1 to m elements, represents the override consistency feature.

2. The method of claim 1, wherein the incomplete utterance rewrite model is constructed by: the dependency modeling based on span dependency and insertion dependency, and obtaining a dependency graph of incomplete discourse rewriting text editing operation by using a node link resolution manner, comprises the steps of: concatenation of a context of length n and an incomplete utterance as input sequence , and ; In the input sequence A special token BOS is inserted at the beginning of the input sequence, and a sequence encoder is used to process the input sequence. Convert to context-sensitive vector representation ,and Where Emb(·) represents the embedding layer, SELFATTN(·) is the Transformer encoder, and BILSTM(·) is the traditional bidirectional LSTM encoder; two independent feedforward neural layers are introduced to project the representation of the sequence encoder into different spaces respectively; a double-affine attention scorer is introduced to score the relationship between head and tail node pairs, and a multi-class cross-entropy loss is applied to the scoring results to guide the model to learn node link resolution.

3. The method of claim 2, wherein the incomplete utterance rewrite model is constructed by: the sequence encoder is stacked by multiple identical layers; in the BiLSTM encoder setting, each token representation is first connected by a character-level convolutional neural network output and a randomly initialized word embedding; in the Transformer encoder setting, only layers and embeddings defined by a pre-trained language model BERT are included.

4. The method of claim 3, wherein the incomplete utterance rewrite model is constructed by: the two independent feedforward neural layers are introduced to project the representation of the sequence encoder into different spaces respectively, comprising the steps of: The head node in the sequence marked as linked is represented as , and ; The tail node in the sequence marked as a link is denoted as , and where FFN is a feed forward neural layer.

5. The method of claim 4, wherein the incomplete utterance rewrite model is constructed by: the double-affine attention scorer is introduced to score the relationship between head and tail node pairs, and a multi-class cross-entropy loss is applied to the scoring results to guide the model to learn node link resolution, further comprising the steps of: scoring based on a first formula, the first formula comprising: ， where i and j denote the indices of the head and tail nodes in the sequence, W and b are learned parameters, and I is the link The relationship label, denotes the existence of a link with score result of label I; guiding the model to learn node link resolution based on a second formula, the second formula comprising: ， wherein, represents the relationship label between node i and node j, and NONE is used to represent no relationship between nodes, represents the cross-entropy loss between the model prediction and the standard answer.

6. The method of claim 1, wherein the incomplete utterance rewrite model is constructed by: the feature fusion comprises the steps of: assuming that the context length is m and the incomplete discourse length is n; The context and the incomplete utterance are concatenated into a sequence and the two properties are extended by zero padding ; feature fusion is achieved by adding the filled features to the double-affine score.

7. The method of claim 6, wherein the incomplete utterance rewrite model is constructed by: the feature fusion is achieved by adding the filled features to the double-affine score, comprising the steps of: Based on the fifth formula to carry out feature fusion, the fifth formula includes: , wherein, denotes the score result after feature fusion, denotes a link head node representation, denotes a link tail node representation, denotes a context consistency feature, denotes a rewrite consistency feature, W and b are learned parameters.

8. An incomplete utterance rewriting model construction apparatus characterized by comprising: it comprises: a dependency modeling module for dependency modeling based on span dependency and insertion dependency, and obtaining a dependency graph of incomplete discourse rewriting text editing operation by using a node link resolution manner, the span dependency is used to determine the range of the phrase to be inserted in the context, and the insertion dependency is used to determine the insertion position in the incomplete discourse; an interactive enhancement feature acquisition module, configured to calculate a context similarity feature and / or a rewrite consistency feature for the current incomplete utterance using a GPT model, the context similarity feature and / or the rewrite consistency feature being used to enhance interactive inference of incomplete utterance rewriting; a feature fusion module, configured to fuse the dependency graph scoring feature with the context similarity feature and / or the rewrite consistency feature, and to push out a rewriting result based on a final feature after the feature fusion; when the interactive enhancement feature acquisition module calculates the context similarity feature, Assuming that the incomplete utterance rewriting depends on the phrase wherein in the context the utterance u is a complement to the incomplete utterance, the complement insertion position being p; For generating context And The GPT model outputs the next token distribution respectively, and generates the context The token distribution of the context The token distribution of the context ;​ the context similarity feature is represented based on a third formula, and the third formula includes: , wherein, is a contextual similarity feature, is a KL divergence; when the interactive enhancement feature acquisition module calculates the rewrite consistency feature, phrase begins at position k in the above context, is inserted into position p of the incomplete utterance; the rewrite consistency feature is calculated based on a fourth formula, and the fourth formula includes: ， wherein, represents a list operation from 1 to m elements, represents a rewrite consistency feature.

Images