Domain data augmentation and multi-turn dialogue method and system with multi-granularity semantic understanding

Abstract

The application relates to a multi-turn dialogue method and system for domain data enhancement and multi-granularity semantic understanding, which comprises the following steps: step A: extracting user dialogue, a reply related to the user dialogue, and labeling a label of a relevant reply speech related to the user dialogue, wherein a correct reply in the dialogue is a positive sample, and an incorrect reply is a negative sample, and a training set U is constructed; step B: using the training set U to train a deep learning network model G for domain data enhancement and multi-granularity semantic understanding, which is used for learning a time sequence semantic relationship in the user dialogue and the reply related to the user dialogue; and step C: inputting the complete user dialogue and the reply related to the user dialogue into the trained deep learning network model G to obtain a correct reply about the complete user dialogue. The method and system can effectively improve the accuracy of reply selection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a multi-turn dialogue method and system for domain data enhancement and multi-granularity semantic understanding. BACKGROUND

[0002] A dialogue system can intuitively simulate human intelligent dialogue, and has important research significance for the development of artificial intelligence. Existing work for building dialogue systems includes two types, one is generative dialogue, and the other is retrieval dialogue. The generative dialogue mainly synthesizes a reply through a natural language generation model, and has the advantages of generating various replies, but has the disadvantage of low accuracy of the generated replies, which often cannot correctly answer the user's question, and is also easy to fall into the trap of safe replies. The retrieval dialogue is to search in a corpus according to the dialogue history and the current user's question through an algorithm model, and select the reply closest to the user's question from a large number of candidate replies to return to the user. Compared with the generative dialogue, the retrieval dialogue has higher accuracy and the returned content is closer to the user's question, so the retrieval dialogue also has higher practicality.

[0003] The retrieval dialogue is divided into single-turn dialogue and multi-turn dialogue, the single-turn dialogue is mainly used for short text question and answer, and cannot process dialogue history information and has insufficient understanding ability for long text. In daily life, users often express their intentions through multiple utterances, so the single-turn dialogue has low value in actual application scenarios. The retrieval multi-turn dialogue selects from a large number of candidate replies according to the dialogue history and the current user's question, and the long-distance dependency relationship in the dialogue context, the time sequence semantic information and the topic conversion make it very difficult to select a suitable reply. In this article, the multi-turn reply selection problem in the retrieval dialogue is focused on. A context in a dialogue is given, and the context is composed of multiple sentences. The purpose of this task is to select the reply closest to the current context from a large number of candidate replies.

[0004] Before the rise of deep learning, traditional machine learning-based methods combined topic analysis algorithms such as LDA (Latent Dirichlet Allocation) and LSA (Latent Semantic Analysis) with short text matching algorithms such as cosine similarity to solve the single-turn response selection problem. However, the above models require manual complex feature selection and feature extraction work, which is time-consuming and inefficient. Due to the time series nature of the dialogue content, early algorithms using deep learning use recurrent neural networks (RNNs) to handle the time series dependencies contained in the context. With the continuous development of deep learning, researchers have applied various deep neural networks to multi-turn dialogue research. For the multi-turn dialogue response selection task, the initial work is to use a single network structure or aggregate multiple single network structures to model the context.

[0005] In 2017, Vaswani et al. proposed the Transformer model based on attention mechanism. The Transformer model has strong semantic understanding ability. The model connects all the words together through multi-head self-attention mechanism, effectively ensuring the extraction of long-distance dependency information. Zhou et al. proposed a deep attention mechanism model DAM (Deep Attention Matching Network). DAM uses the attention structure in the Transformer to perform sentence representation, and also designs cross-attention for sentence interaction. Finally, a three-dimensional convolution is used for aggregation. Yuan et al. proposed the MSN (Multi-hop Selector Network) model, which combines the advantages of the DUA model and the DAM model. MSN selects more relevant context to the question through a multi-hop selector. The above models are all based on the strong semantic understanding ability of the Transformer, which greatly improves the performance of the response selection model. However, these models still have the problem of not being able to understand the dialogue content from a global perspective. At the same time, these models mainly use word2vec to generate static word vectors, which causes the problem of polysemy to be unable to be solved.

[0006] In recent years, with the advent of pre-trained models, more and more methods using mature pre-trained models have been applied to the multi-turn dialogue field. For example, BERT, RoBERTa, ALBERT and ELECTRA. Pre-trained models can be well combined with downstream tasks. BERT-VFT first uses BERT for multi-turn reply selection tasks. It converts the original matching task into a classification task by concatenating the historical dialogue and the reply together and classifying by adding a special [CLS] token to represent the global semantic information. The experimental results show its superior performance. Subsequently, more and more methods combining pre-trained models with downstream tasks have been proposed. Gu et al. proposed the SA-BERT model. Since there are often multiple participants in a dialogue, the dialogue participant information is added during data input. The pre-trained model learns to represent the dialogue participant embedding, enhancing the model's understanding of the language style of different dialogue participants. Li et al. proposed the DCM (Deep Context Modeling) model. After encoding by BERT, the context and the reply are deeply matched to learn the semantic relevance features in the context and the reply. These models can effectively learn the relevant semantic features in the context and the reply through large-scale pre-training, further improving the model's multi-turn reply selection capability. However, the hidden features in the dialogue, such as temporal information and dialogue structure information, are still lacking sufficient mining.

[0007] In recent years, methods of adding auxiliary tasks for joint training based on pre-trained language models have gained widespread attention. Whang et al. proposed three different auxiliary tasks to optimize the pre-trained model to better adapt to the domain dataset. By proposing three auxiliary tasks of utterance deletion, utterance insertion and utterance search, the latent structural features in the dialogue are learned, and then the optimal result is selected from multiple candidate replies according to the semantic structure information. On this basis, researchers use the post-training strategy to let the pre-trained language model learn the domain data before fine-tuning to enhance the pre-trained language model's adaptation ability in the dialogue task. In addition, some research works extract the output of the pre-trained language model for further fine matching to filter out irrelevant noise in the reply. These research works add new data or modules to the pre-trained language model, or add topic embeddings based on speaker embeddings to optimize the pre-trained language model. These methods have achieved good results. However, there are many topic transitions and jumps in the context of multi-turn dialogue. Different turns of dialogue contain different amounts of information. These complex scenarios require the model to learn the internal latent information of multi-turn dialogue in more detail. SUMMARY

[0008] The application aims to provide a multi-turn dialogue method and system for domain data enhancement and multi-granularity semantic understanding, which can effectively improve the accuracy of reply selection.

[0009] To achieve the above-mentioned purpose, the technical scheme adopted by the application is as follows: a multi-turn dialogue method for domain data enhancement and multi-granularity semantic understanding, comprising the following steps:

[0010] Step A: extracting user dialogue, reply involved in user dialogue, and labeling the label of the relevant reply dialogue involved in the user dialogue, the positive sample being the correct reply in the dialogue and the negative sample being the incorrect reply, and constructing a training set U;

[0011] Step B: using the training set U to train a deep learning network model G for domain data enhancement and multi-granularity semantic understanding, which is used to learn the temporal semantic relationship in the user dialogue and the reply involved in the user dialogue;

[0012] Step C: inputting the complete user dialogue and the reply involved in the user dialogue into the trained deep learning network model G to obtain the correct reply about the complete user dialogue.

[0013] Further, the step B specifically comprises the following steps:

[0014] Step B1: encoding each training sample in the training set U to obtain the initial representation vector of the context the initial representation vector of the candidate reply and the label y;

[0015] Step B2: for each training positive sample in the training set U, the i-th sentence closest to the reply and the correct reply are intercepted and encoded to obtain the initial representation vector of the context the initial representation vector of the correct reply

[0016] Step B3: for each training sample in the training set U, a sentence in the context is randomly selected, and the word-level order in the sentence is disturbed and then encoded to obtain the initial representation vector For the remaining context, a new word [STI] is added in front of each sentence and after the last sentence, and the context is encoded to obtain the initial representation vector

[0017] Step B4: inputting the representation vectors and into a K-layer bidirectional GRU network to learn and extract local semantic information and obtain the representation vector of the user dialogue

[0018] Step B5: inputting The input is fed into the sigmoid layer, and the loss is calculated based on the target loss function. main The gradients of each parameter in the deep learning network model G are calculated using the backpropagation method, and the parameters are updated using the stochastic gradient descent method.

[0019] Step B6: Initialize the context representation vector and the initial representation vector of the correct response The process involves concatenating the two elements, masking each word in the context and response with a certain probability, and then inputting the masked context-response vector into a pre-trained language model. Will The input is fed into a linear layer, and the loss is calculated based on the target loss function. img The gradients of each parameter in the deep learning network model G are calculated using the backpropagation method, and the parameters are updated using the stochastic gradient descent method.

[0020] Step B7: Initialize the context representation vector and discourse representation vector After concatenation, the data is input into the pre-training and language model to obtain... Will The input is fed into the sigmoid layer, and the loss is calculated based on the target loss function. sti The gradients of each parameter in the deep learning network model G are calculated using the backpropagation method, and the parameters are updated using the stochastic gradient descent method.

[0021] Step B8: Based on the target loss function, calculate the gradient of each parameter in the deep learning network model G using the backpropagation method, and update the parameters using the stochastic gradient descent method; when the iterative change of the loss value generated by the deep learning network model G is less than the set threshold and no longer decreases, or when the maximum number of iterations is reached, terminate the training of the deep learning network model G.

[0022] Furthermore, step B1 specifically includes the following steps:

[0023] Step B11: Traverse the training set U, where each training sample in U is represented as D = (c, r, y). Perform word segmentation on c and r in training sample D, removing stop words and adding new words.

[0024] Where c represents the context in the dialogue, r represents the response corresponding to this context, and y represents the label of this response. The label includes {0, 1}, where 0 indicates that r is not the response corresponding to c, and 1 indicates that r is the response corresponding to c.

[0025] After word segmentation and stop word removal in context c, a new word [EOT] is added to the end of each sentence, represented as:

[0026]

[0027] wherein, is the i-th word in the remaining words of the context c after tokenization, stop word removal and new word addition, i = 1, 2,..., n, n is the number of remaining words of the context c after tokenization, stop word removal and new word addition;

[0028] After tokenization and stop word removal, the reply r is added with a new word [EOT] at the end, denoted as:

[0029]

[0030] wherein, is the i-th word in the remaining words of the reply r after tokenization, stop word removal and new word addition, i = 1, 2,..., m, m is the number of remaining words of the reply r after tokenization, stop word removal and new word addition;

[0031] For the new word [EOT] added in the context and the reply, a tensor array is used to record the position of [EOT] after concatenating c and r:

[0032] eot = {EOT1, EOT2,..., EOT q}

[0033] wherein, EOT i represents the i-th EOT in the context and the reply, i = 1, 2,..., q;

[0034] Step B12: Encoding the context c and the reply r obtained in step B11 after tokenization, stop word removal and new word addition, to obtain the initial representation vector of the context c

[0035] wherein, is denoted as:

[0036]

[0037] wherein, is the i-th word corresponding to the word vector, which is obtained by looking up in the pre-trained word vector matrix , wherein d represents the dimension of the word vector, and |V| is the number of words in the dictionary V;

[0038] Step B13: Encoding the reply r obtained in step B11 after tokenization, stop word removal and new word addition, to obtain the initial representation vector of the reply r ​​

[0039]

[0040] in, Indicates the i-th word The corresponding word vectors are obtained through a pre-trained word vector matrix. The word vector is found in the dictionary, where d represents the dimension of the word vector and |V| is the number of words in the dictionary V.

[0041] Furthermore, step B2 specifically includes the following steps:

[0042] Step B21: Transfer the context from step B11 And the reply in B11 Concatenate the utterances according to their levels to obtain c1 = {u1, u2, ..., u...} n Let r}, where n represents the number of utterances in the context; extract the i-th sentence in the context closest to the response r to obtain the interval context response c1 = {u n-i u n-i+1 , ..., u n ,r}, where n represents the number of utterances in the context, i∈[3,5];

[0043] Step B22: Reply to the interval context obtained in step B21 with c1 = {u n-i u n-i+1 , ..., u n Encoding at the word granularity level yields the initial representation vector of the interval context.

[0044]

[0045] in, express The word vector corresponding to the i-th word in the text, where i = 1, 2, ..., p, and p is... The number of word vectors is determined by the pre-trained word vector matrix. The word vector is found in the dictionary, where d represents the dimension of the word vector and |V| is the number of words in the dictionary V.

[0046] Furthermore, step B3 specifically includes the following steps:

[0047] Step B31: Transfer the context from step B11 Merging according to discourse level yields c2 = {u1, u2, ..., u...} n}, where n represents the number of utterances in the context; a sentence u is randomly selected from context c2. t And separate it from c2:

[0048]

[0049] wherein, denotes the i-th word in the utterance u t , i = 1, 2,... q, q is the number of words in the utterance u t ;

[0050] In c2 = {u1, u2,..., u n}, a new word [STI] is added in front of each sentence and behind the last sentence to get:

[0051]

[0052] wherein, denotes the i-th word in the utterance u after adding the new word [STI], i = 1, 2,... n, n is the number of words in c2;

[0053] For the new word [STI] added in c2, a tensor array is used to record the position of each [STI]:

[0054] sti = [STI1, STI2,..., STIn] n-1

[0055] wherein STI i denotes the position of the i-th [STI] in c2, i = 1, 2,..., n-1;

[0056] Step B32: Encode the context c2 obtained in step B31 which removes the target utterance u t to get the initial representation vector of c2

[0057]

[0058] wherein, denotes the i-th word vector in c2 , i = 1, 2,..., n, n is the number of word vectors in c2 is obtained by looking up in the pre-trained word vector matrix , wherein d denotes the dimension of the word vector, and |V| is the number of words in the dictionary V;

[0059] Randomly shuffle and encode the words in the target utterance u obtained in step B31 to get the initial representation vector of u t

[0060] ​

[0061] wherein, denotes u t in corresponding to the word vector, i, j = 1, 2, …, m; i≠j, n is The number of word vectors in is obtained by looking up in the pre-trained word vector matrix , wherein d represents the dimension of the word vector, and |V| is the number of words in the dictionary V.

[0062] Further, the step B4 specifically comprises the following steps:

[0063] Step B41: input the context representation vector and the reply representation vector into the pre-trained language model BERT, and output h 1 , combine the tensor array eot to obtain the position of each [EOT] word vector, and splice these [EOT] word vectors to obtain h eot ;

[0064] wherein, h 1 , h eot are respectively:

[0065]

[0066]

[0067] wherein, is the output of the i-th word vector of the pre-trained language model BERT, The calculation formula of

[0068]

[0069] wherein, x is each word vector in the context representation vector and the reply representation vector ;

[0070] Step B42: input into the 2-layer bidirectional GRU to obtain the fusion representation vector of local semantic information

[0071]

[0072] wherein is the output of the i-th word vector of the bidirectional GRU, The calculation formula of

[0073]

[0074] wherein is h eot each word vector in

[0075] Further, the step B5 specifically comprises the following steps:

[0076] Step B51: input the context vector obtained in step B42 into a fully connected layer for dimension reduction, and calculate the probability distribution through a sigmoid activation function, the calculation process being as follows:

[0077]

[0078] wherein W is a trainable parameter, b is a bias, and σ(·) is a sigmoid activation function;

[0079] Step B52: input the probability distribution g(c, r) obtained in step B51 into a loss function to calculate the loss and perform iterative updates, and use cross entropy as the loss function in the fine-tuning process of the reply selection main task:

[0080]

[0081] The gradient accumulation strategy is used, and the gradient is updated after accumulating 5 batches; the model uses the AdamW algorithm as the optimizer of gradient descent.

[0082] Further, the step B6 specifically comprises the following steps:

[0083] Step B61: mask each word vector in the interval context representation vector obtained in step B22 with a probability q to obtain a masked interval context representation vector:

[0084]

[0085] wherein, is each word vector after being masked, i = 1 x q, 2 x q,..., p x q;

[0086] Step B62: input the masked context reply vector obtained to a pre-trained language model to obtain input to a linear layer, calculate the gradient of each parameter in the deep learning network model G according to the target loss function loss img , and update the parameters using the stochastic gradient descent method;

[0087]

[0088]

[0089] wherein W1 is trainable parameter, b1 is bias, σ(·) is sigmoid activation function; the loss value is calculated by using cross-entropy as loss function, the learning rate is updated by using gradient optimization algorithm Adam, the model parameters are updated by using back propagation iteration, and the model is trained by minimizing the loss function;

[0090] wherein the calculation formula of minimizing the loss function Loss is as follows:

[0091]

[0092] wherein θ is the parameter of the encoder part in BERT, and θ1 is the parameter in the output layer connected on the encoder in the interval mask generation task;

[0093] Further, the step B7 specifically comprises the following steps:

[0094] Step B71: concatenating the context representation vector obtained in step B32 and the target speech representation vector obtained in step B33 to obtain wherein represents a cascading operation;

[0095] inputting into the pre-training model to obtain Then, the score of the position in the sti tensor array is calculated by using a full connection layer.

[0096]

[0097]

[0098] wherein W2 is trainable parameter, b2 is bias, and σ(·) is sigmoid activation function;

[0099] Step B72: the loss value is calculated by using cross-entropy as loss function, the learning rate is updated by using gradient optimization algorithm Adam, the model parameters are updated by using back propagation iteration, and the model is trained by minimizing the loss function;

[0100] wherein the calculation formula of minimizing the loss function Loss is as follows:

[0101]

[0102] wherein label S ={0, 0,..1, 0...0} represents the label of the [STI] i position in the sti array, wherein 1 represents The insertion place, 0 is contrary.

[0103] Further, in the step B8, while fine-tuning, the multi-turn dialogue model uses a multi-task joint training framework to export the parameters of the pre-trained model, learns time sequence semantic information of different granularities in the optimization process of the two auxiliary tasks and is more adaptive to the current domain data set; in the process of fine-tuning, the two auxiliary tasks and the main task are optimized at the same time, therefore, the complete loss function of the model is as follows:

[0104] Loss=Loss main +αLoss img +βLoss sti

[0105] Wherein, α, β are two hyperparameters, respectively used for controlling the influence of the two auxiliary tasks on the multi-turn dialogue model.

[0106] The application also provides a multi-turn dialogue system using the above method, comprising:

[0107] A data collection module is used for extracting user dialogue, replies involved in the user dialogue and labeling the tags of relevant reply speeches in the dialogue to construct a training set.

[0108] A preprocessing module is used for preprocessing the training samples in the training set, including word segmentation processing, removing stop words and adding new words.

[0109] An encoding module is used for finding the word vectors in the preprocessed dialogue context in the pre-trained word vector matrix to obtain the initial representation vectors of the context and the initial representation vectors of the replies.

[0110] A network training module is used for inputting the initial representation vectors of the context and the initial representation vectors of the replies obtained through different data processing into a deep learning network, sharing the parameters of the pre-trained language model in the deep learning network, obtaining the multi-granularity representation vectors of the context and the replies and training the deep learning network by using the representation vectors, using the probability that the representation vectors belong to a certain category and the labels in the training set as a loss, taking minimizing the loss as an objective to train the entire deep learning network to obtain a deep learning network model of domain data enhancement and multi-granularity semantic understanding.

[0111] A reply selection module is used for analyzing and processing the input dialogue context and candidate replies by using the trained deep learning network model of domain data enhancement and multi-granularity semantic understanding to output the reply in the candidate replies that best matches the current context content.

[0112] Compared with the prior art, the present application has the following beneficial effects: a multi-turn dialogue method and system of domain data enhancement and multi-granularity semantic understanding are provided, the method and system use a multi-task learning framework, utilize an interval mask generation task, generate random masks for the words in the intervals of fixed sizes in the context and the reply, and help the model better adapt to the current domain data set, utilize a token disorder insertion method to train BERT to learn the semantic relationship between the contexts from the multi-granularity, through the task, the model can learn the semantic relationship between the contexts from the multi-granularity, and finally, a bidirectional feature fusion mechanism is utilized to fuse different local semantic information to enhance the reply selection capability, and the accuracy of the reply selection can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0113] Figure 1 is a method flowchart of an embodiment of the present application;

[0114] Figure 2 is an architecture diagram of a deep learning network model in an embodiment of the present application;

[0115] Figure 3 is a system structure schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0116] The present application will be further described below in combination with the drawings and embodiments.

[0117] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0118] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0119] As shown in Figure 1 The present embodiment provides a multi-turn dialogue method of domain data enhancement and multi-granularity semantic understanding, which comprises the following steps:

[0120] Step A: extracting user dialogue, reply involved in the user dialogue, and labeling the label of the relevant reply speech involved in the user dialogue, the positive sample is the correct reply in the dialogue, the negative sample is the incorrect reply, and a training set U is constructed.

[0121] Step B: Using the training set U, train the deep learning network model G of field data enhancement and multi-granularity semantic understanding, which is used to learn the time semantic relationship in the user dialogue and the reply involved in the user dialogue. The architecture of the deep learning network model G is as shown in Figure 2 .

[0122] In this embodiment, the step B specifically comprises the following steps:

[0123] Step B1: Encode each training sample in the training set U to obtain the initial representation vector of the context the initial representation vector of the candidate reply and the label y.

[0124] In this embodiment, the step B1 specifically comprises the following steps:

[0125] Step B11: Traverse the training set U, and each training sample in U is represented as D=(c, r, y). Tokenize c and r in the training sample D, and remove stop words and add new words.

[0126] wherein c is the context in the dialogue, r is the reply corresponding to the context, and y is the label of the reply, and the label includes {0, 1}, 0 indicating that r is not the reply corresponding to c, and 1 indicating that r is the reply corresponding to c.

[0127] After tokenization and removal of stop words, a new word [EOT] is added at the end of each sentence in the context c, which is represented as:

[0128]

[0129] wherein, is the i-th word in the context c after tokenization, removal of stop words, and addition of new words, i=1, 2,..., n, and n is the number of words remaining in the context c after tokenization, removal of stop words, and addition of new words.

[0130] After tokenization and removal of stop words, a new word [EOT] is added at the end of each sentence in the context c, which is represented as:

[0131]

[0132] wherein, represents the i-th word in the reply r after tokenization, removal of stop words, and addition of new words, i=1, 2,..., m, and m is the number of words remaining in the reply r after tokenization, removal of stop words, and addition of new words.

[0133] For the new word [EOT] added in the context and the reply, a tensor array is used to record the position of [EOT] after concatenating c and r:

[0134] eot = {EOT1, EOT2,..., EOTq} q}

[0135] where EOTi represents the ith EOT in the context and the reply, i = 1, 2,..., q. i

[0136] Step B12: encode the context c obtained in step B11 after tokenization, stop word removal and new word addition to obtain an initial representation vector of the context c

[0137] where is represented as:

[0138]

[0139] where is the word vector corresponding to the ith word in the context c, which is obtained by looking up in the pre-trained word vector matrix , where d represents the dimension of the word vector, and |V| is the number of words in the vocabulary V.

[0140] Step B13: encode the reply r obtained in step B11 after tokenization, stop word removal and new word addition to obtain an initial representation vector of the reply r

[0141]

[0142] where is the word vector corresponding to the ith word in the reply r, which is obtained by looking up in the pre-trained word vector matrix , where d represents the dimension of the word vector, and |V| is the number of words in the vocabulary V.

[0143] Step B2: for each training positive sample in the training set U, the i sentences closest to the reply and the correct reply are intercepted for encoding to obtain an initial representation vector of the context and the initial representation vector of the correct reply

[0144] In this embodiment, the step B2 specifically comprises the following steps:

[0145] Step B21: encode the context c in step B11 and the reply r in step B11 ​​​​​According to the utterance level, splicing is performed to obtain c1={u1, u2,..., u n , r}, where n represents the number of utterances contained in the context; i sentences closest to the reply r in the context are intercepted to obtain the interval context reply c1={u n-i , u n-i+1 ,..., u n , r}, where n represents the number of utterances in the context, and i∈[3, 5].

[0146] Step B22: The interval context reply c1={u n-i , u n-i+1 ,..., u n , r} obtained in step B21 is encoded at the word granularity level to obtain the initial representation vector of the interval context

[0147]

[0148] wherein, represents the word vector corresponding to the i-th word in , i=1, 2,.., p, p is the number of word vectors in is obtained by searching in the pre-trained word vector matrix , where d represents the dimension of the word vector, and |V| is the number of words in the dictionary V.

[0149] Step B3: For each training sample in the training set U, a sentence in the context is randomly selected, and the word-level order in the sentence is randomly shuffled and then encoded to obtain the initial representation vector For the remaining context, a new word [STI] is added in front of each sentence and after the last sentence, and the context is encoded to obtain the initial representation vector

[0150] In this embodiment, step B3 specifically includes the following steps:

[0151] Step B31: The context in step B11 is merged according to the utterance level to obtain c2={u1, u2,..., u n}, where n represents the number of utterances contained in the context; a sentence u t in the context c2 is randomly selected and stripped out of c2:

[0152]

[0153] wherein, represents the i-th word in the utterance u t , i=1, 2,...q, q is the utterance ut The number of words in the text.

[0154] In c2 = {u1, u2, ..., u} n Add the new word [STI] before each sentence and after the last sentence in the code to get:

[0155] in, Let i represent the i-th word in c2 after the addition of the new word, where i = 1, 2, ..., n, and n is the number of words in c2.

[0156] For the new word [STI] added in c2, a tensor array is used to record the position of each [STI]:

[0157] sti=[STI1, STI2,...,STI n-1 ]

[0158] Among them, ST, i Let i represent the position of the i-th [STI] in c2, where i = 1, 2, ..., n-1.

[0159] Step B32: Remove the target utterance u obtained in step B31. t Encode the context c2 to obtain the initial representation vector of c2.

[0160]

[0161] in, In c2 The corresponding word vectors, i = 1, 2, ..., n, where n is The number of word vectors is determined by the pre-trained word vector matrix. The word vector is found in the dictionary, where d represents the dimension of the word vector and |V| is the number of words in the dictionary V.

[0162] The target discourse obtained in step B31 The words in the text are randomly shuffled and encoded to obtain u. t initial representation vector

[0163]

[0164] in, In ut The corresponding word vectors, i, j = 1, 2, ..., m; i ≠ j, n is The number of word vectors is determined by the pre-trained word vector matrix. where d denotes the dimension of the word vector, and |V| is the number of words in the dictionary V.

[0165] Step B4: input the representation vector and into a K-layer bidirectional GRU network, learn and extract local semantic information, and obtain the representation vector of the user dialogue

[0166] In this embodiment, step B4 specifically comprises the following steps:

[0167] Step B41: input the representation vector and the representation vector into the pre-trained language model BERT, and output h 1 Combine the tensor array eot to obtain the position of each [EOT] word vector, and concatenate these [EOT] word vectors to obtain h eot .

[0168] where h 1 and h eot are respectively:

[0169]

[0170]

[0171] where, is the output of the i-th word vector of the pre-trained language model BERT, The calculation formula of is as follows:

[0172]

[0173] where x is each word vector in the context representation vector and the reply representation vector .

[0174] Step B42: input into a 2-layer bidirectional GRU to obtain a fusion representation vector of local semantic information

[0175]

[0176] where is the output of the i-th word vector of the bidirectional GRU, The calculation formula of is as follows:

[0177]

[0178] where is h eoteach word vector in the context.

[0179] Step B5: input the context initial representation vector to a sigmoid layer, and calculate the probability distribution according to the target loss function loss main , calculate the gradient of each parameter in the deep learning network model G using the back propagation method, and update the parameters using the stochastic gradient descent method.

[0180] In this embodiment, the step B5 specifically comprises the following steps:

[0181] Step B51: input the context initial representation vector obtained in step B42 to a fully connected layer for dimension reduction, and calculate the probability distribution through a sigmoid activation function, the calculation process being as follows:

[0182]

[0183] where W is a trainable parameter, b is a bias, and σ(·) is a sigmoid activation function.

[0184] Step B52: input the probability distribution g(c, r) obtained in step B51 to a loss function to calculate the loss and perform iterative updates, and use cross entropy as the loss function in the fine-tuning process of the reply selection main task:

[0185]

[0186] Use the gradient accumulation strategy to update the gradient after accumulating 5 batches; the model uses the AdamW algorithm as the optimizer of gradient descent.

[0187] Step B6: concatenate the context initial representation vector and the initial representation vector of the correct reply , mask each word in the context and reply with a certain probability, and obtain the masked context reply vector input to the pre-trained language model to obtain input to a linear layer, and calculate the probability distribution according to the target loss function loss img , calculate the gradient of each parameter in the deep learning network model G using the back propagation method, and update the parameters using the stochastic gradient descent method.

[0188] In this embodiment, the step B6 specifically comprises the following steps:

[0189] Step B61: mask each word vector in the interval context representation vector obtained in step B22 with a probability q to obtain the masked interval context representation vector:

[0190]

[0191] wherein, is each word vector masked vector, i = 1 x q, 2 x q,..., p x q.

[0192] Step B62: input the obtained masked context reply vector into the pre-trained language model to obtain input the into the linear layer, and calculate the gradient of each parameter in the deep learning network model G according to the target loss function loss img , and update the parameters by using the stochastic gradient descent method.

[0193]

[0194]

[0195] wherein W1 is a trainable parameter, b1 is a bias, and σ(·) is a sigmoid activation function; the loss value is calculated using cross-entropy as the loss function, the learning rate is updated by the gradient optimization algorithm Adam, the model parameters are updated by iteration using back propagation, and the model is trained by minimizing the loss function.

[0196] wherein the calculation formula of the loss function Loss is as follows:

[0197]

[0198] wherein θ is the parameter of the encoder part in BERT, and θ1 is the parameter in the output layer connected on the encoder in the interval mask generation task.

[0199] Step B7: input the concatenated context initial representation vector and the utterance representation vector into the pre-trained language model to obtain input the into the sigmoid layer, and calculate the gradient of each parameter in the deep learning network model G according to the target loss function loss sti , and update the parameters by using the stochastic gradient descent method.

[0200] In this embodiment, the step B7 specifically comprises the following steps:

[0201] Step B71: concatenate the context representation vector obtained in step B32 and the target utterance representation vector obtained in step B33 to obtain wherein represents a cascaded operation.

[0202] will be input into the pre-trained model to obtain Then the score of the position in the sti tensor array is calculated through the full connection layer.

[0203]

[0204]

[0205] where W2 is a trainable parameter, b2 is a bias, and sigma(·) is a sigmoid activation function.

[0206] Step B72: Calculate the loss value using cross-entropy as the loss function, update the learning rate through the gradient optimization algorithm Adam, and iteratively update the model parameters using backpropagation to minimize the loss function to train the model.

[0207] where the calculation formula of the loss function Loss is as follows:

[0208]

[0209] where label S = {0, 0,..1, 0...0} represents the label of the [STI] i position in the sti array, where 1 indicates that it is the place to be inserted, and 0 is the opposite.

[0210] Step B8: Calculate the gradient of each parameter in the deep learning network model G using the backpropagation method according to the target loss function loss, and update the parameters using the stochastic gradient descent method; when the loss value generated by the deep learning network model G iteratively changes less than the set threshold value and no longer decreases or reaches the maximum number of iterations, terminate the training of the deep learning network model G.

[0211] Specifically, while fine-tuning, the multi-turn dialogue model uses a multi-task joint training framework to export the parameters of the pre-trained model, learn time semantic information of different granularities in the process of optimizing the two auxiliary tasks, and adapt to the current domain dataset better; In the process of fine-tuning, the two auxiliary tasks and one main task are optimized at the same time, therefore, the complete loss function of the model is as follows:

[0212] Loss = Loss main + aLoss img + bLoss sti

[0213] Where a and b are two hyperparameters, respectively used to control the influence of the two auxiliary tasks on the multi-turn dialogue model.​

[0214] Step C: inputting the complete user dialogue and the reply involved in the user dialogue into the trained deep learning network model G to obtain the correct reply about the complete user dialogue.

[0215] As shown in Figure 3 The embodiment also provides a multi-turn dialogue system using the above method, which comprises a data collection module, a preprocessing module, an encoding module, a network training module, and a reply selection module.

[0216] The data collection module is configured to extract user dialogues, replies involved in the user dialogues, and label the relevant reply utterances in the dialogues to construct a training set.

[0217] The preprocessing module is configured to preprocess the training samples in the training set, including word segmentation processing, removing stop words, and adding new words.

[0218] The encoding module is configured to find the word vectors in the preprocessed dialogue context in the pre-trained word vector matrix to obtain the initial representation vectors of the context and the initial representation vectors of the replies.

[0219] The network training module is configured to input the initial representation vectors of the context and the initial representation vectors of the replies obtained by different data processing into the deep learning network, share the parameters of the pre-trained language model in the deep learning network, obtain the multi-granularity representation vectors of the context and the replies, and train the deep learning network using the multi-granularity representation vectors, the probability that the representation vectors belong to a certain category, and the labels in the training set as a loss, train the entire deep learning network to minimize the loss, and obtain the deep learning network model of domain data enhancement and multi-granularity semantic understanding.

[0220] The reply selection module analyzes and processes the input dialogue context and candidate replies using the trained deep learning network model of domain data enhancement and multi-granularity semantic understanding, and outputs the reply in the candidate replies that best matches the current context content.

[0221] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0222] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the

[0223] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the

[0224] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the

[0225] The above description is only preferred embodiments of the present application, and is not intended to limit the present application to other forms described above. Any person skilled in the art who understands the disclosed technical content can modify or change the above embodiments into equivalent embodiments with equivalent changes. However, any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution of the present application, and according to the technical essence of the present application, still belongs to the protection scope of the present application.

Claims

1. A multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding, characterized in that, Comprising the following steps: Step A: extracting user dialogue, reply involved in user dialogue, and labeling the label of the relevant reply speech involved in the user dialogue, the positive sample is the correct reply in the dialogue, and the negative sample is the incorrect reply, and constructing a training set U; Step B: using the training set U, training the deep learning network model G of the field data enhancement and multi-granularity semantic understanding, for learning the time sequence semantic relationship in the user dialogue and the reply involved in the user dialogue; Step C: input the complete user dialogue and the reply involved in the user dialogue into the trained deep learning network model G to obtain the correct reply about the complete user dialogue; The step B specifically comprises the following steps: Step B1: encode each training sample in the training set U to obtain an initial representation vector of the context , an initial representation vector of the candidate reply , and the label y; Step B2: For each training positive sample in the training set U, the i-th sentence closest to the reply and the correct reply are intercepted to encode the initial representation vector of the context , the initial representation vector of the correct reply ; Step B3: For each training sample in the training set U, randomly select a sentence in the context and encode it after shuffling the word order in it to obtain an initial representation vector ; For the remaining context, add a new word [STI] at the beginning of each sentence and at the end of the last sentence, and encode this context to obtain an initial representation vector ; Step B4: the feature vector and is input into a K-layer bidirectional GRU network, local semantic information is learned and extracted, and the feature vector of the user dialogue is obtained. Step B5: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] The input is fed into the sigmoid layer, based on the target loss function. The gradients of each parameter in the deep learning network model G are calculated using the backpropagation method, and the parameters are updated using the stochastic gradient descent method. Step B6: the initial representation vector of the context and the initial representation vector of the correct reply are concatenated, and each word in the context and the reply is masked with a certain probability to obtain a masked context-reply vector ; the masked context-reply vector is input into a linear layer, and the gradients of the parameters in the deep learning network model G are calculated using the back propagation method according to the target loss function , and the parameters are updated using the stochastic gradient descent method; Step B7: the context initial representation vector and the utterance representation vector is input into the pre-trained language model to obtain is input into the sigmoid layer, and the gradient of each parameter in the deep learning network model G is calculated according to the target loss function using the back propagation method, and the parameters are updated using the stochastic gradient descent method;​ Step B8: according to the target loss function loss, the gradient of each parameter in the deep learning network model G is calculated by using the back propagation method, and the parameters are updated by using the stochastic gradient descent method; when the loss value generated by the deep learning network model G is less than the set threshold value and no longer decreases or reaches the maximum iteration number, the training of the deep learning network model G is terminated.

2. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 1, characterized in that, The step B1 specifically comprises the following steps: Step B11: traversing the training set U, each training sample in U is expressed as D=(c,r,y), the c and r in the training sample D are subjected to word segmentation processing, stop words are removed and new words are added; Wherein c is the context in the dialogue, r is the reply corresponding to the context, y is the label of the reply, the label includes {0, 1}, 0 represents that r is not the reply corresponding to c, and 1 represents that r is the reply corresponding to c; After the context c is subjected to word segmentation and stop word removal, a new word [EOT] is added at the end of each sentence, which is expressed as: wherein, is the i-th word in the words remaining after the context c is segmented, stop words are removed, and new words are added, n is the number of words remaining after the context c is segmented, stop words are removed, and new words are added. After the reply r is subjected to word segmentation and stop word removal, a new word [EOT] is added at the end, which is expressed as: wherein, represents the i-th word in the remaining words after word segmentation, stop word removal, and new word addition in the reply r, m is the number of remaining words after word segmentation, stop word removal, and new word addition in the reply r. For the new word [EOT] added in the context and the reply, a tensor array is used to record the position of [EOT] after c and r are spliced: wherein represents the i-th EOT in the context and the reply, ; Step B12: Context after tokenization, stop word removal and addition of new words for step B11 Encoding is performed to obtain the initial representation vector of the context c ; wherein is represented by: wherein, is the ith word corresponding word vector is obtained by looking up in a pre-trained word vector matrix where d denotes the dimension of the word vector and |V| is the number of words in the vocabulary V. Step B13: Tokenizing, removing stop words and adding new words to the reply obtained in Step B11 Encoding is performed to obtain the initial representation vector of the reply r ; wherein, represents the ith word corresponding word vector is obtained by looking up in a pre-trained word vector matrix where d represents the dimension of the word vector and |V| is the number of words in the vocabulary V.

3. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 2, characterized in that, The step B2 specifically comprises the following steps: Step B21: the context in step B11 and the reply in B11 is spliced according to the utterance level, obtaining wherein n represents the number of utterances contained in the context; the interval context reply is obtained by intercepting i utterances closest to the reply r in the context wherein n represents the number of utterances in the context, ; Step B22: Interval context reply on the interval context obtained in step B21 Encoding at the word granularity level, obtaining an initial representation vector of the interval context ; wherein, denotes the i-th word in the sentence, i = 1, 2,.., p, p is the number of words in the sentence, the number of word vectors in the sentence is obtained by looking up in the pre-trained word vector matrix where d denotes the dimension of the word vectors and |V| is the number of words in the vocabulary V.

4. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 3, characterized in that, The step B3 specifically comprises the following steps: Step B31: Transfer the context from step B11 Merging according to discourse level yields... , where n represents the number of utterances in the context; the context is randomly selected. A sentence in and from Stripped from the middle: wherein, represents the i-th word in the utterance , q is the number of words in the utterance . In the beginning of each sentence and the end of the last sentence a new word [STI] is added resulting in: wherein, represents the i-th word in after adding a new word, n is the number of words in . For New words [STI] added in the middle, a tensor array to record the position of each [STI]: wherein represents position of the i-th [STI] in the middle, ; Step B32: Encode the target utterance removed context of step B31 to obtain an initial representation vector of ; and Step B33: Encode the target utterance to obtain a final representation vector of ; and Step B34: Calculate the similarity between the initial representation vector of step B32 and the final representation vector of step B33 to obtain a similarity score. wherein, denotes in the corresponding word vector, , n is The number of word vectors in the word vector matrix is obtained by looking up in the pre-trained word vector matrix , wherein d denotes the dimension of the word vector, and |V| is the number of words in the dictionary V. The words in the target utterance obtained in step B31 are randomly shuffled and encoded to obtain an initial representation vector of . ;​ wherein, denotes in the corresponding word vector, , n is The number of word vectors in the set is obtained by looking up in the pre-trained word vector matrix where d denotes the dimension of the word vectors and |V| is the number of words in the vocabulary V.

5. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 4, characterized in that, The step B4 specifically comprises the following steps: Step B41: input the characterization vector and the characterization vector into the pre-trained language model BERT, output , combine the tensor array eot to get the position of each [EOT] word vector, and splice these [EOT] word vectors to get ; wherein , respectively. wherein, is the output of the i-th word vector of the pre-trained language model BERT, The calculation formula of is as follows: where x is the context representation vector and each word vector in the reply representation vector ​ Step B42: the following compounds were prepared according to the procedure described in Step B41: input into a bidirectional GRU of 2 layers to obtain a fusion representation vector of local semantic information ; wherein is the output of the bidirectional GRU for the i-th word vector, The calculation formula is: wherein is each word vector in 6. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 5, characterized in that, The step B5 specifically comprises the following steps: Step B51: Obtain the result from step B42 The input is passed to a fully connected layer for dimensionality reduction, and the probability distribution is calculated using the sigmoid activation function. The calculation process is as follows: where W is a trainable parameter and b is a bias, is a sigmoid activation function; Step B52: The probability distribution obtained from Step B51 is input into a loss function to calculate the loss and iteratively update, using cross-entropy as the loss function in the fine-tuning process of reply selection main task: Step B52: The probability distribution obtained from Step B51 is input into a loss function to calculate the loss and iteratively update, using cross-entropy as the loss function in the fine-tuning process of reply selection main task: Using the gradient accumulation strategy, the gradient is updated after accumulating 5 batches; the model uses the AdamW algorithm as the optimizer of gradient descent.

7. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 6, characterized in that, The step B6 specifically comprises the following steps: Step B61: Each word vector in the interval context representation vector obtained in step B22 is masked with a probability q to obtain a masked interval context representation vector: ​ wherein, is each word vector masked vector, ; Step B62: the context reply vector after masking is obtained input into the pre-trained language model ; input into the linear layer according to the target loss function , the gradients of the parameters in the deep learning network model G are calculated using the back propagation method, and the parameters are updated using the stochastic gradient descent method; wherein is a trainable parameter, is a bias, is a sigmoid activation function; the loss value is calculated using cross-entropy as the loss function, the learning rate is updated by the gradient optimization algorithm Adam, the model parameters are iteratively updated by backpropagation, and the model is trained by minimizing the loss function. Wherein, the calculation formula of the loss function Loss is as follows: wherein, are parameters of the encoder part in BERT, are parameters in the output layer of the encoder applied in the interval mask generation task; The step B7 specifically comprises the following steps: Step B71: Concatenate the context representation vector obtained from Step B32 and the target utterance representation vector obtained from Step B33 to obtain wherein denotes a concatenation operation; will be input into the pre-trained model to obtain a score for the position in the tensor array; and then calculate wherein is a trainable parameter, is a bias, is a sigmoid activation function; Step B72: using cross entropy as the loss function to calculate the loss value, updating the learning rate by using the gradient optimization algorithm Adam, updating the model parameters by using the back propagation iteration, and training the model by minimizing the loss function; Wherein, the calculation formula of the loss function Loss is as follows: in, to indicate In the array The location label, where 1 indicates yes. Where it is inserted, 0 is the opposite.

8. The multi-turn dialogue method of domain data augmentation and multi-granularity semantic understanding according to claim 7, characterized in that, In the step B8, while fine-tuning, the multi-turn dialogue model uses a multi-task joint training framework, exports the parameters of the pre-trained model, learns the time sequence semantic information of different granularities in the optimization process of the two auxiliary tasks, and is more suitable for the current field data set; in the process of fine-tuning, the two auxiliary tasks and the main task are optimized at the same time, therefore, the complete loss function of the model is as follows: wherein are two hyperparameters, respectively controlling the influence of the two auxiliary tasks on the multi-turn dialogue model.

9. A multi-turn dialogue system employing the method of any one of claims 1-8, characterized in that, Comprising: The data collection module is configured to extract user conversations, replies involved in the user conversations, and mark labels of relevant reply speeches in the conversations, and construct a training set; The preprocessing module is configured to perform preprocessing on training samples in the training set, including word segmentation processing, removal of stop words, and addition of new words; The encoding module is configured to find word vectors in the preprocessed conversation context in a pre-trained word vector matrix, to obtain an initial representation vector of the context and an initial representation vector of the reply; The network training module is configured to input the initial representation vector of the context and the initial representation vector of the reply obtained through different data processing into a deep learning network, share parameters of a pre-trained language model in the deep learning network, obtain multi-granularity representation vectors of the context and the reply, and train the deep learning network based on the multi-granularity representation vectors; the network training module is configured to use probabilities that the representation vectors belong to a certain category and the labels in the training set as losses, train the entire deep learning network with a target of minimizing the losses, and obtain a deep learning network model for domain data enhancement and multi-granularity semantic understanding; The reply selection module is configured to analyze and process the input conversation context and candidate replies by using the trained deep learning network model for domain data enhancement and multi-granularity semantic understanding, and output a reply in the candidate replies that best matches the current context content.

Images