Local information perception dialogue method and system based on pre-trained language model

Abstract

The present invention relates to a local information perception dialogue method and system based on a pre-trained language model, the method comprising the following steps: Step A: Collect multi-round dialogue texts of a specific scene, and mark the category to which each multi-round dialogue reply belongs, and construct a training set with positive and negative category labels. D ; Step B: Use the training set D Train a local information-aware deep learning network model PLIP based on a pre-trained language model to select responses corresponding to a given multi-round dialogue context; Step C: Input the multi-round dialogue context and the response set into the trained local information-aware deep learning network model PLIP to obtain the most appropriate response for the multi-round dialogue context. This method and system can effectively improve the accuracy of response selection for multi-round dialogues.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to a local information perception dialogue method and system based on a pre-trained language model. Background Art

[0002] In recent years, with the development of machine learning and deep learning networks, humans have made significant progress in intelligent dialogue with computers, and dialogue systems have gradually entered everyone's field of vision. Dialogue systems have important research value for both industry and academia, and can be widely used in many fields. The current dialogue system algorithms are mainly generative dialogue and retrieval dialogue. Among them, generative dialogue can generate an answer word by word based on a question without relying on any corpus in the reasoning stage. The generated answers have the advantage of diversity, but the answers obtained are often not logical, and sometimes they fall into the trap of safe replies. Retrieval dialogue allows the algorithm to find the most appropriate answer in the corpus based on a specific question and reply. It can extract information related to the correct reply from the question and infer the appropriate answer based on this information. The retrieval dialogue model has been widely used in multi-round dialogue systems such as Microsoft XiaoIce. Compared with the generative dialogue model, it is more reliable and has better practicality.

[0003] Lowe et al. built two benchmark models for the reply selection task in retrieval-based multi-round dialogues, based on the Recurrent Neural Networks (RNNs) algorithm and the Long Short-Term Memory (LSTM) algorithm. These two benchmark models use the RNN hidden layer unit to memorize the text features of the previous moment in the process of encoding the text, introduce time order information into the model, and overcome the shortcomings of the bag-of-words model used in the early algorithms. However, in multi-round dialogues, the dialogue history may be lengthy, and not all content is related to the reply. The above two benchmark models directly encode the entire dialogue data and cannot extract important information from the dialogue data in a targeted manner, resulting in unnecessary noise for the model. In order to extract important information from long texts, the researchers proposed to extract important information by matching the context and the reply, and decompose the reply selection task into three steps. The first step is to use the RNN-based algorithm to extract features from each utterance and reply. The second step is to match the extracted utterance features with the reply features. The third step is to use CNN and other methods to extract the information needed to calculate the score in the matching matrix. However, the semantic information that RNN can extract is limited. RNN encoding assumes that data is sequentially related, but in conversation data, topics are dynamic, and two paragraphs that are far apart may also be highly related. It is difficult for RNN encoding to accurately learn the relationship between the two paragraphs. At the same time, RNN may also experience gradient vanishing when encoding long paragraphs, and cannot obtain long-distance dependencies well. The limitations of RNN may cause the above methods to lose important information in the first step. The Transformer architecture proposed by Vaswani in 2017 can fully grasp the global dependency information with the help of a large number of self-attention and interactive attention operations, without being restricted by sequence distance. The researchers rewrote the encoder part of the Transformer and applied it to the encoding module of the model, which strengthened the model's ability to extract information. At the same time, influenced by the multi-head attention mechanism in the Transformer, these works used multi-head attention to construct semantic information of various granularities in the matching stage, enriched the feature representation of the model, and achieved significant improvement. However, the above models still have the following problems. First, insufficient consideration of global sequence information. The above model mainly uses RNN and other methods to encode all sentence representations after matching. These sentence representations may have lost important information during the encoding and matching stages. Second, the word vector representation used does not take the context into consideration. The above model mainly uses static word vectors such as Word2vec, which makes it difficult to solve the problem of polysemy and cannot accurately express semantic information according to different contexts, thus introducing noise during the encoding stage.

[0004] In 2018, Google proposed the famous BERT (Bidirectional Encoder Representations from Transformers) model. In the process of encoding conversation data, BERT will perform a deep self-attention mechanism on the word granularity in a global scope. Combined with the position embedding representation in BERT, it can effectively solve the two problems of insufficient consideration of global sequence information and word vector representation not considering the context. As a result, the focus of research on the reply selection task in multi-round conversations has shifted to methods based on pre-trained language models. The basic steps of this type of method are to first use a pre-trained language model with a multi-layer Transformer encoder to encode the entire conversation data, and then input the output representation that can represent the global information [CLS] position in the output into the classification layer for prediction. On this basis, researchers have enhanced the adaptability of pre-trained language models in conversation tasks by allowing pre-trained language models to learn domain data before fine-tuning. In addition, some research works have tried to generate additional conversation texts or generate data enhancement methods such as interlocutor embeddings, while some research works have extracted the output of pre-trained language models for further fine matching to filter out noise that is not related to the reply. These research works have added new data or modules on the basis of pre-trained language models, and have achieved good results. However, the above method simply concatenates the context and the response into the pre-trained language model, and cannot learn some potential features in the dialogue data. In addition, the addition of additional modules greatly increases the number of model parameters, but the improvement effect is very limited, and it is impossible to efficiently tap the potential of the pre-trained language model. Recently, some research works have introduced multi-task learning into the multi-round dialogue response selection task, and designed multiple sub-tasks based on the coherence and consistency of dialogue data. These sub-tasks share the parameters of the pre-trained language model with the main task, and design additional loss functions to optimize the pre-trained language model in a targeted manner. Compared with the previous architecture, the method combining the pre-trained language model with multi-task learning has shown significant results in the multi-round dialogue response selection task, further tapping the potential of the pre-trained language model, reducing the number of model parameters, and being able to more efficiently learn the potential features in the dialogue data, greatly improving the pre-trained language model's ability to understand dialogue data.

[0005] In summary, although the multi-round dialogue response selection model based on the pre-trained language model has made great progress, there are still the following problems: First, the designed subtasks are not efficient enough. Designing multiple subtasks for the pre-trained language model means that the pre-trained language model needs to be fine-tuned multiple times during training, which costs a lot of time and computing resources. The more subtasks, the greater the training cost. At the same time, the optimization goals of the subtasks are very different from those of the main task. For example, the subtask of predicting the position after inserting or deleting the speech may bring noise to the main task. Second, the semantic understanding ability of the pre-trained language model is not fully utilized. The above method only uses the [CLS] label representing global information for prediction in the main task, ignoring a large amount of information output at other positions except [CLS]. At the same time, the above model can understand the conversation semantics from different perspectives in multiple auxiliary tasks and learn valuable conversation information, but the main task only uses the output of the [CLS] position for simple classification, and does not specifically extract the information learned in the auxiliary task, resulting in the model being unable to fully utilize the important semantic information learned in the auxiliary task. Summary of the invention

[0006] The purpose of the present invention is to provide a local information perception dialogue method and system based on a pre-trained language model, which method and system are conducive to improving the accuracy of multi-round dialogue response selection.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a local information perception dialogue method based on a pre-trained language model, comprising the following steps:

[0008] Step A: Collect multi-round dialogue texts of specific scenarios, annotate the category to which each multi-round dialogue response belongs, and construct a training set D with positive and negative category labels;

[0009] Step B: Use the training set D to train the local information-aware deep learning network model PLIP based on the pre-trained language model to select the response corresponding to the given multi-round dialogue context;

[0010] Step C: Input the multi-round dialogue context and the response set into the trained local information-aware deep learning network model PLIP to obtain the most appropriate response corresponding to the multi-round dialogue context.

[0011] Furthermore, the step B specifically includes the following steps:

[0012] Step B1: Input each sample of the training set D into the deep learning network model in the form of a triple (c, r, y), where c = {u 1 ,u 2 ,...,u m} represents a conversation context containing m utterances. For the tth utterance in the context, Among them l t is the number of words in the tth utterance, r is the candidate response, l r is the number of words in the reply, y∈{0,1} is the sample label, y=1 means the candidate reply is a reasonable reply in the current context, and y=0 means it is unreasonable;

[0013] Among them, the deep learning network model PLIP outputs an evaluation score that can reflect the relevance of context and reply after encoding and calculating triples. The deep learning network model uses the multi-layer attention mechanism of the pre-trained language model to learn the contextual semantic representation combined with the context. At the same time, it adopts a multi-task learning strategy to optimize the main task, i.e., the multi-round dialogue reply selection task, promote the representation vector to understand global information, learn the relevance of context and reply, and strengthen the pre-trained language model's learning of the local context of multi-round dialogues in the auxiliary task, so as to fully tap the semantic understanding ability of the pre-trained language model.

[0014] Step B2: In the auxiliary task part, the deep learning network model PLIP uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand the local context of multi-round dialogues;

[0015] Among them, the random sliding window reply prediction task samples the conversation context data at different starting positions in the multi-round conversation context to obtain the conversation fragments, encodes the conversation fragments using the pre-trained language model, and predicts the reply of the window, so that the pre-trained language model can fully learn the semantic information of the local context;

[0016] Step B3: In the multi-round dialogue response selection task, the deep learning network model PLIP uses a local information perception module to promote the pre-trained language model to generate local semantic information, and at the same time integrates global information and local semantic information, calculates the rationality score between the multi-round dialogue context and the response, and evaluates whether the current response corresponds to the given multi-round dialogue context. Finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model, and the stochastic gradient descent method is used to update the parameters;

[0017] Step B4: When the iterative change of the loss value generated by the deep learning network model PLIP is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model is terminated.

[0018] Furthermore, the step B1 specifically includes the following steps:

[0019] Step B11: concatenate the utterances and responses in the conversation context to obtain the input x of the deep learning network model;

[0020] x={[CLS],u 1 ,[EOT],u 2 ,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0021] Among them, x is the concatenated long text, [SEP] is the delimiter, [CLS] is the tag used by the deep learning network model to learn global features, and [EOT] is a special tag used by the deep learning network model to learn local information;

[0022] Step B12: Map x into a sequence of numbers through the dictionary of the pre-trained language model, where each number is the id of a word in the vocabulary, and input the id sequence into the embedding layer inside the pre-trained language model. According to the three initialized embedding matrices, map the id sequence into word embedding representation, position embedding representation, and paragraph embedding representation;

[0023] X=Embedding word (x)+Embedding pos (x pos )+Embedding type (x type )

[0024] Among them, Embedding word Represents the mapping method of word embedding, which can map the input sequence into word vectors according to the vocabulary. pos The mapping method of representing position embedding representation can map each word position to the corresponding position embedding matrix. type The mapping method of paragraph embedding representation can map the context and reply into different vector spaces, and then add the three word vectors to get the word vector l is the number of words in x, [CLS], [SEP], and [EOT] are all considered as one word;

[0025] Step B13: Add the word embedding representation, sentence representation, and position representation of each word to obtain a fused embedding representation, and encode it using a multi-layer Transformer network to obtain a high-level semantic feature representation of the sequence;

[0026] The multi-layer Transformer network is composed of multiple stacked Transformer encoding blocks; each Transformer encoding block includes two sub-layers: a multi-head self-attention mechanism and a forward feedback layer, each of which is followed by a residual connection and a normalization layer; X is first mapped into three vectors, namely, a query vector Q, a key vector K, and a value vector V, and the calculation formula is as follows:

[0027] Q=XW Q +b Q

[0028] K=XW K +b K

[0029] V=XW V +b V

[0030] Among them, W Q , W K , W V , b Q , b K , b V represents the training parameters;

[0031] Step B14: Send the Q, K, and V vectors into the multi-head self-attention mechanism, divide their word vector dimension d into h sub-vectors, each sub-vector has a dimension of d / h, and send them to the self-attention mechanism for training. Finally, these h self-attention sub-vectors are concatenated to obtain a d-dimensional output vector C. In order to prevent overfitting, make the vector more normalized, and accelerate network convergence, add residual connections and normalization to the sub-layers of the multi-head self-attention mechanism to obtain the vector T. The calculation formula is as follows:

[0032]

[0033]

[0034] C=Concat(head 1 ,head 2 ,...,head h )W C +b C

[0035] T = LayerNorm(X+C)

[0036] Among them, head i represents the self-attention score of the i-th sub-vector, W C ,b C Represents training parameters, Concat represents concatenation operation, and LayerNorm represents layer normalization transformation;

[0037] Step B15: Send the vector T to the fully connected forward feedback sublayer, which performs two linear transformations on T to obtain the comprehensive feature FFN of the sequence. Then, T and FFN are residually connected and the final high-level feature H of the sequence is obtained after layer normalization. The computer formula is as follows:

[0038] FFN=(W F T+b F )W N +b N

[0039] H = LayerNorm (T + FFN)

[0040] Among them, W F , W N 、b F 、b N Represents training parameters.

[0041] Furthermore, the step B2 specifically includes the following steps:

[0042] Step B21: In the auxiliary task random sliding window reply prediction, the model sets the length and position of the sliding window to random, samples a large amount of local conversation context data that falls within the sliding window from the conversation context, and inserts a special label [EOT] after each utterance of the local conversation context data, as shown in the following formula:

[0043]

[0044] Among them, x' is the input of the subtask. Different from the main task, x' only retains the information inside the window, and other information is replaced by [PAD]. i is the starting position of the sliding window, w is the size of the current window, m is the number of utterances in the current context, and κ is a hyperparameter indicating the minimum window size.

[0045] Step B22: Encode the conversation window data using the pre-trained language model BERT. The formula is as follows:

[0046] E=BERT(x′)

[0047] Step B23: The vector E obtained in step B22 contains all the semantic representations of the conversation segment after being encoded by the pre-trained language model BERT. The semantic representation that best represents the current conversation segment is further selected from E to optimize the auxiliary task. In order not to destroy the [CLS] representation that can represent global information in the pre-trained language model, the model only selects the [EOT] representation E that is closest to the window response in the output of the pre-trained language model. [EOT]As the final representation vector of the random sliding window reply prediction task; the auxiliary task expresses the rationality of the window data, allowing the [EOT] label in BERT to learn the information of different segments and different times in the conversation, enriching the [EOT] representation's ability to understand local area information;

[0048] Step B24: Get the final representation vector E [EOT] After that, it is input into the classification layer to calculate the score. The calculation formula is as follows:

[0049] g(w c ,w r )=σ(W w T E [EOT] +b w )

[0050] Among them, w c 、w r represents the context and reply in the sliding window, W w is a trainable parameter in the prediction layer, and σ(·) represents the sigmoid activation function;

[0051] Step B25: The random sliding window reply prediction task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate the difference between the current score and the actual conversation window label. The specific formula is as follows:

[0052]

[0053] Where D' represents the window data set.

[0054] Furthermore, the step B3 specifically includes the following steps:

[0055] Step B31: The local information perception module embeds a special tag [EOT] after each sentence in the dialogue context, as shown in the following formula:

[0056] x={[CLS],u 1 ,[EOT],u 2 ,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0057] Under the combined effect of the deep attention mechanism and position embedding of the pre-trained language model, the special label [EOT] of each position can learn the interactive information with the surrounding text at its specific position; at the same time, in the process of optimizing the random sliding window reply prediction task, the last [EOT] label in the window is used to establish a classification task, and the ability to identify window replies is gradually learned; therefore, the representation of the [EOT] label gradually learns the correct representation of the sentence and pays more attention to the text in the local area;

[0058] Step B32: In the feature fusion stage, the local information perception module selects the n local semantic representations closest to the response from the output of the pre-trained language model as multi-granular local information, and uses splicing to aggregate the local information into a whole. The specific formula is as follows:

[0059]

[0060] Among them, l represents the entry closest to the reply, and n is a hyperparameter used to represent the number of [EOT] representations taken out;

[0061] Step B33: The local information perception module integrates the local information with the global information to obtain the final representation vector of the main task. The aggregation process is:

[0062]

[0063] Step B34: Input the aggregated representation vector into the classification layer to calculate the rationality score between the current multi-round dialogue context and the response. The formula is as follows:

[0064] g(c,r)=σ(W T E ensemble +b)

[0065] Where W is a trainable parameter, σ(·) represents the sigmoid activation function, and b is the bias term of the current classification layer;

[0066] Step B35: The deep neural network model PLIP uses gradient descent to update the parameters in the learning model. At the same time, cross entropy is used as the loss function for the multi-round dialogue response selection task. The specific formula is as follows:

[0067]

[0068] Combined with the optimization objectives of the auxiliary tasks, the final loss function of the model is:

[0069] Loss=Loss main +αLoss window

[0070] Among them, α is a hyperparameter used to control the influence of the auxiliary task random sliding window reply prediction task on the model.

[0071] The present invention also provides a local information perception dialogue system using the above method, comprising:

[0072] The data collection module collects multi-round dialogue samples in a specific field, and annotates the positive and negative labels of the answers corresponding to each question in the multi-round dialogue data, and constructs a multi-round dialogue response selection training set D with positive and negative labels;

[0073] Pre-trained language model encoding module. The pre-trained language model is mainly composed of an embedding layer and a multi-layer multi-head attention mechanism. Each training sample in the form of a triple in the training set D is sent to the pre-trained language model BERT, and the contextual semantic representation combined with the context is learned using the multi-layer attention mechanism of the pre-trained language model. At the same time, the model uses multi-task learning to fully tap the semantic understanding ability of the pre-trained language model.

[0074] In the auxiliary task module, the model exports the parameters of the pre-trained language model BERT and uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand local information in the conversation. The random sliding window reply prediction task samples window data of different positions and sizes in the context of multiple rounds of conversations, uses the exported pre-trained language model to encode the conversation window, and uses the newly added special tag [EOT] to predict the window's reply, allowing the pre-trained language model to fully learn the local language characteristics of different conversation stages and conversation lengths.

[0075] Local information perception module: In the multi-round dialogue response selection task, the model uses the local information perception module to promote the pre-trained language model BERT to generate multi-granular local semantic information, and at the same time integrates global information and local semantic information to calculate the classification score to evaluate whether the current response corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model, and the stochastic gradient descent method is used to update the parameters; and

[0076] The network training module terminates the training of the deep learning network model when the iterative change of the loss value generated by the deep learning network model is less than the set threshold and no longer decreases or reaches the maximum number of iterations.

[0077] Compared with the prior art, the present invention has the following beneficial effects: it provides a local information perception dialogue method and system based on a pre-trained language model, which adopts a multi-task learning strategy, while optimizing the main task, i.e., the multi-round dialogue reply selection task, learning the context and reply relevance, and strengthening the learning of the pre-trained language model on the local area of ​​the multi-round dialogue in the auxiliary task, fully tapping the semantic understanding ability of the pre-trained language model, thereby integrating global information and local semantic information, and obtaining the most appropriate reply corresponding to the multi-round dialogue context. Therefore, the present invention can effectively improve the accuracy of multi-round dialogue reply selection, and has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0079] Figure 2 It is a local information perception deep learning network model architecture diagram based on a pre-trained language model in an embodiment of the present invention;

[0080] Figure 3 is a schematic diagram of random sliding window reply prediction in an embodiment of the present invention;

[0081] Figure 4 is a random sliding window reply prediction structure diagram in an embodiment of the present invention;

[0082] Figure 5 It is a structural diagram of the local information perception module in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0084] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0085] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0086] like Figure 1 As shown, this embodiment provides a local information perception dialogue method based on a pre-trained language model, comprising the following steps:

[0087] Step A: Collect multi-round dialogue texts of specific scenarios, annotate the category to which each multi-round dialogue response belongs, and construct a training set D with positive and negative category labels.

[0088] Step B: Use the training set D to train the local information-aware deep learning network model PLIP based on the pre-trained language model to select the response corresponding to the given multi-round dialogue context.

[0089] The step B specifically comprises the following steps:

[0090] Step B1: Input each sample of the training set D into the deep learning network model in the form of a triple (c, r, y), where c = {u 1 ,u 2 ,...,u m} represents a conversation context containing m utterances. For the tth utterance in the context, Among them l t is the number of words in the tth utterance, r is the candidate response, l r is the number of words in the reply, y∈{0,1} is the sample label, y=1 means the candidate reply is a reasonable reply in the current context, and y=0 means it is unreasonable.

[0091] Among them, the deep learning network model PLIP outputs an evaluation score that can reflect the relevance of context and reply after encoding and calculating triples. The deep learning network model uses the multi-layer attention mechanism of the pre-trained language model to learn the contextual semantic representation combined with the context, and adopts a multi-task learning strategy. While optimizing the main task, i.e., the multi-round dialogue reply selection task, promoting the representation vector to understand global information and learn the relevance of context and reply, it strengthens the pre-trained language model's learning of the local context of multi-round dialogues in the auxiliary task, and fully taps the semantic understanding ability of the pre-trained language model. The architecture of the deep learning network model is as follows: Figure 2 shown.

[0092] The step B1 specifically comprises the following steps:

[0093] Step B11: concatenate the utterances and responses in the conversation context to obtain the input x of the deep learning network model;

[0094] x={[CLS],u 1 ,[EOT],u 2 ,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0095] Among them, x is the concatenated long text, [SEP] is the delimiter, [CLS] is the tag used by the deep learning network model to learn global features, and [EOT] is a special tag used by the deep learning network model to learn local information.

[0096] Step B12: Map x into a sequence of numbers through the dictionary of the pre-trained language model, where each number is the id of a word in the vocabulary, and input the id sequence into the embedding layer inside the pre-trained language model. According to the three initialized embedding matrices, map the id sequence into word embedding representation, position embedding representation, and paragraph embedding representation;

[0097] X=Embedding word (x)+Embedding pos (x pos )+Embedding type (x type )

[0098] Among them, Embedding word Represents the mapping method of word embedding, which can map the input sequence into word vectors according to the vocabulary. pos The mapping method of representing position embedding representation can map each word position to the corresponding position embedding matrix. type The mapping method of paragraph embedding representation can map the context and reply into different vector spaces, and then add the three word vectors to get the word vector l is the number of words in x, and [CLS], [SEP], and [EOT] are all considered as one word.

[0099] Step B13: Add the word embedding representation, sentence representation, and position representation of each word to obtain a fused embedding representation, and encode it using a multi-layer Transformer network to obtain a high-level semantic feature representation of the sequence.

[0100] The multi-layer Transformer network is composed of multiple stacked Transformer encoding blocks; each Transformer encoding block includes two sub-layers: a multi-head self-attention mechanism and a forward feedback layer, each of which is followed by a residual connection and a normalization layer; X is first mapped into three vectors, namely, a query vector Q, a key vector K, and a value vector V, and the calculation formula is as follows:

[0101] Q=XW Q +b Q

[0102] K=XW K +bK

[0103] V=XW V +b V

[0104] Among them, W Q , W K , W V 、b Q 、b K 、b V Represents training parameters.

[0105] Step B14: Send the Q, K, and V vectors into the multi-head self-attention mechanism, divide their word vector dimension d into h sub-vectors, each sub-vector has a dimension of d / h, and send them to the self-attention mechanism for training. Finally, these h self-attention sub-vectors are concatenated to obtain a d-dimensional output vector C. In order to prevent overfitting, make the vector more normalized, and accelerate network convergence, add residual connections and normalization to the sub-layers of the multi-head self-attention mechanism to obtain the vector T. The calculation formula is as follows:

[0106]

[0107]

[0108] C=Concat(head 1 ,head 2 ,...,head h )W C +b C

[0109] T = LayerNorm(X+C)

[0110] Among them, head i represents the self-attention score of the i-th sub-vector, W C ,b C Represents training parameters, Concat represents concatenation operation, and LayerNorm represents layer normalization transformation.

[0111] Step B15: Send the vector T to the fully connected forward feedback sublayer, which performs two linear transformations on T to obtain the comprehensive feature FFN of the sequence. Then, T and FFN are residually connected and the final high-level feature H of the sequence is obtained after layer normalization. The computer formula is as follows:

[0112] FFN=(W F T+b F )W N +b N

[0113] H = LayerNorm (T + FFN)

[0114] Among them, W F , W N 、b F 、b N Represents training parameters.

[0115] Step B2: In the auxiliary task part, the deep learning network model PLIP uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand the local context of multi-round dialogues.

[0116] Among them, the random sliding window reply prediction task samples the conversation context data at different starting positions in the multi-round conversation context to obtain the conversation fragment, encodes the conversation fragment using the pre-trained language model, and predicts the reply of the window, so that the pre-trained language model can fully learn the semantic information of the local context. The process and structure of random sliding window reply prediction are as follows Figure 3 , 4 shown.

[0117] The step B2 specifically comprises the following steps:

[0118] Step B21: In the auxiliary task random sliding window reply prediction, the model sets the length and position of the sliding window to random, samples a large amount of local conversation context data that falls within the sliding window from the conversation context, and inserts a special label [EOT] after each utterance of the local conversation context data, as shown in the following formula:

[0119]

[0120] Among them, x' is the input of the subtask. Different from the main task, x' only retains the information inside the window, and other information is replaced by [PAD]. i is the starting position of the sliding window, w is the size of the current window, m is the number of utterances in the current context, and κ is a hyperparameter indicating the minimum window size.

[0121] Step B22: Encode the conversation window data using the pre-trained language model BERT. The formula is as follows:

[0122] E=BERT(x′)

[0123] Step B23: The vector E obtained in step B22 contains all the semantic representations of the conversation segment after being encoded by the pre-trained language model BERT. The semantic representation that best represents the current conversation segment is further selected from E to optimize the auxiliary task. In order not to destroy the [CLS] representation that can represent global information in the pre-trained language model, the model only selects the [EOT] representation E that is closest to the window response in the output of the pre-trained language model.[EOT] As the final representation vector of the random sliding window reply prediction task; the auxiliary task expresses the rationality of the window data, allowing the [EOT] label in BERT to learn information from different fragments and moments in the conversation, enriching the [EOT] representation's ability to understand local area information.

[0124] Step B24: Get the final representation vector E [EOT] After that, it is input into the classification layer to calculate the score. The calculation formula is as follows:

[0125] g(w c ,w r )=σ(W w T E [EOT] +b w )

[0126] Among them, w c 、w r represents the context and reply in the sliding window, W w is a trainable parameter in the prediction layer, and σ(·) represents the sigmoid activation function.

[0127] Step B25: The random sliding window reply prediction task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate the difference between the current score and the actual conversation window label. The specific formula is as follows:

[0128]

[0129] Where D' represents the window data set.

[0130] Step B3: In the multi-round dialogue response selection task, the deep learning network model PLIP adopts the following Figure 5 The local information perception module shown promotes the pre-trained language model to generate local semantic information, while fusing global information with local semantic information, calculating the rationality score between the multi-round dialogue context and the reply, and evaluating whether the current reply corresponds to the given multi-round dialogue context. Finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model, and the stochastic gradient descent method is used to update the parameters.

[0131] The step B3 specifically comprises the following steps:

[0132] Step B31: The local information perception module embeds a special tag [EOT] after each sentence in the dialogue context, as shown in the following formula:

[0133] x={[CLS],u 1 ,[EOT],u 2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0134] Under the combined effect of the deep attention mechanism and position embedding of the pre-trained language model, the special label [EOT] of each position can learn the interaction information with the surrounding text at its specific position; at the same time, in the process of optimizing the random sliding window reply prediction task, the last [EOT] label in the window is used to establish a classification task, and the ability to identify window replies is gradually learned; therefore, the representation of the [EOT] label gradually learns the correct representation of the sentence and pays more attention to the text in the local area.

[0135] Step B32: In the feature fusion stage, the local information perception module selects the n local semantic representations closest to the response from the output of the pre-trained language model as multi-granular local information, and uses splicing to aggregate the local information into a whole. The specific formula is as follows:

[0136]

[0137] Among them, l represents the entry closest to the reply, and n is a hyperparameter used to indicate the number of [EOT] representations taken out.

[0138] Step B33: The local information perception module integrates the local information with the global information to obtain the final representation vector of the main task. The aggregation process is:

[0139]

[0140] Step B34: Input the aggregated representation vector into the classification layer to calculate the rationality score between the current multi-round dialogue context and the response. The formula is as follows:

[0141] g(c,r)=σ(W T E ensemble +b)

[0142] Where W is a trainable parameter, σ(·) represents the sigmoid activation function, and b is the bias term of the current classification layer.

[0143] Step B35: The deep neural network model PLIP uses gradient descent to update the parameters in the learning model. At the same time, cross entropy is used as the loss function for the multi-round dialogue response selection task. The specific formula is as follows:

[0144]

[0145] Combined with the optimization objectives of the auxiliary tasks, the final loss function of the model is:

[0146] Loss=Loss main +αLoss window

[0147] Among them, α is a hyperparameter used to control the influence of the auxiliary task random sliding window reply prediction task on the model.

[0148] Step B4: When the iterative change of the loss value generated by the deep learning network model PLIP is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model is terminated.

[0149] Step C: Input the multi-round dialogue context and the response set into the trained local information-aware deep learning network model PLIP to obtain the most appropriate response corresponding to the multi-round dialogue context.

[0150] This embodiment also provides a local information perception dialogue method and system using the above method, including a data collection module, a pre-trained language model encoding module, an auxiliary task module, a local information perception module and a network training module.

[0151] The data collection module is used to collect multi-round dialogue samples in a specific field, and mark the positive and negative labels of the answers corresponding to each question in the multi-round dialogue data, and construct a multi-round dialogue response selection training set D with positive and negative labels.

[0152] The pre-trained language model encoding module includes a pre-trained language model, which is mainly composed of an embedding layer and a multi-layer multi-head attention mechanism; each training sample in the form of a triple in the training set D is sent to the pre-trained language model BERT, and the contextual semantic representation combined with the context is learned by using the multi-layer attention mechanism of the pre-trained language model; at the same time, the model adopts a multi-task learning method to fully explore the semantic understanding ability of the pre-trained language model.

[0153] In the auxiliary task module, the model exports the parameters of the pre-trained language model BERT, and uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand local information in the conversation. The random sliding window reply prediction task samples window data of different positions and sizes in the context of multiple rounds of conversations, uses the exported pre-trained language model to encode the conversation window, and uses the newly added special tag [EOT] to predict the window's reply, allowing the pre-trained language model to fully learn the local language characteristics of different conversation stages and conversation lengths.

[0154] In the multi-round dialogue response selection task, the model uses a local information perception module to promote the pre-trained language model BERT to generate multi-granularity local semantic information, and at the same time integrates global information and local semantic information to calculate classification scores to evaluate whether the current response corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model, and the stochastic gradient descent method is used to update the parameters.

[0155] The network training module is used for training the network model. When the iterative change of the loss value generated by the deep learning network model is less than the set threshold and no longer decreases or reaches the maximum number of iterations, the training of the deep learning network model is terminated.

[0156] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A local information perception dialogue method based on a pre-trained language model, characterized in that: The following steps are involved: Step A: Collect multi-round dialogue texts of specific scenarios, annotate the category to which each multi-round dialogue response belongs, and construct a training set D with positive and negative category labels; Step B: Use the training set D to train the local information-aware deep learning network model PLIP based on the pre-trained language model to select the response corresponding to the given multi-round dialogue context; Step C: Input the multi-round dialogue context and the response set into the trained local information-aware deep learning network model PLIP to obtain the most appropriate response corresponding to the multi-round dialogue context; The step B specifically comprises the following steps: Step B1: Each sample in the training set D is grouped into a triplet (c, r， y) in the form of input deep learning network model, where c = {u1, u2, ..., u m } represents a conversation context containing m utterances. For the tth utterance in the context, Among them l t is the number of words in the tth utterance, r is the candidate response, l r is the number of words in the reply, y∈{0,1} is the sample label, y=1 means the candidate reply is a reasonable reply in the current context, and y=0 means it is unreasonable; Among them, the deep learning network model PLIP outputs an evaluation score that can reflect the relevance of context and reply after encoding and calculating triples. The deep learning network model uses the multi-layer attention mechanism of the pre-trained language model to learn the contextual semantic representation combined with the context. At the same time, it adopts a multi-task learning strategy to optimize the main task, i.e., the multi-round dialogue reply selection task, promote the representation vector to understand global information, learn the relevance of context and reply, and strengthen the pre-trained language model's learning of the local context of multi-round dialogues in the auxiliary task, so as to fully tap the semantic understanding ability of the pre-trained language model. Step B2: In the auxiliary task part, the deep learning network model PLIP uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand the local context of multi-round dialogues; Among them, the random sliding window reply prediction task samples the conversation context data at different starting positions in the multi-round conversation context to obtain the conversation fragments, encodes the conversation fragments using the pre-trained language model, and predicts the reply of the window, so that the pre-trained language model can fully learn the semantic information of the local context; Step B3: In the multi-round dialogue response selection task, the deep learning network model PLIP uses a local information perception module to promote the pre-trained language model to generate local semantic information, and at the same time integrates global information and local semantic information, calculates the rationality score between the multi-round dialogue context and the response, and evaluates whether the current response corresponds to the given multi-round dialogue context. Finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model, and the stochastic gradient descent method is used to update the parameters; Step B4: When the iterative change of the loss value generated by the deep learning network model PLIP is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model is terminated; The step B1 specifically comprises the following steps: Step B11: concatenate the utterances and responses in the conversation context to obtain the input x of the deep learning network model; x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]} Among them, x is the concatenated long text, [SEP] is the delimiter, [CLS] is the tag used by the deep learning network model to learn global features, and [EOT] is a special tag used by the deep learning network model to learn local information; Step B12: Map x into a sequence of numbers through the dictionary of the pre-trained language model, where each number is the id of a word in the vocabulary, and input the id sequence into the embedding layer inside the pre-trained language model. According to the three initialized embedding matrices, map the id sequence into word embedding representation, position embedding representation, and paragraph embedding representation; X=Embedding word (x)+Embedding pos (x pos )+Embedding type (x type ) Among them, Embedding word Represents the mapping method of word embedding, which can map the input sequence into word vectors according to the vocabulary. pos The mapping method of representing position embedding representation can map each word position to the corresponding position embedding matrix. type The mapping method of paragraph embedding representation can map the context and reply into different vector spaces, and then add the three word vectors to get the word vector l is the number of words in x, [CLS], [SEP], and [EOT] are all considered as one word; Step B13: Add the word embedding representation, sentence representation, and position representation of each word to obtain a fused embedding representation, and encode it using a multi-layer Transformer network to obtain a high-level semantic feature representation of the sequence; The multi-layer Transformer network is composed of multiple stacked Transformer encoding blocks; each Transformer encoding block includes two sub-layers: a multi-head self-attention mechanism and a forward feedback layer, each of which is followed by a residual connection and a normalization layer; X is first mapped into three vectors, namely, a query vector Q, a key vector K, and a value vector V, and the calculation formula is as follows: Q=XW Q +b Q K=XW K +b K V=XW V +b V Among them, W Q , W K , W V , b Q , b K , b V represents the training parameters; Step B14: Send the Q, K, and V vectors into the multi-head self-attention mechanism, divide their word vector dimension d into h sub-vectors, each sub-vector has a dimension of d / h, and send them to the self-attention mechanism for training. Finally, these h self-attention sub-vectors are concatenated to obtain a d-dimensional output vector C. In order to prevent overfitting, make the vector more normalized, and accelerate network convergence, add residual connections and normalization to the sub-layers of the multi-head self-attention mechanism to obtain the vector T. The calculation formula is as follows: C=Concat(head1,head2,…,head h )W C +b C T = LayerNorm(X+C) Among them, head i represents the self-attention score of the i-th sub-vector, W C , b C Represents training parameters, Concat represents concatenation operation, and LayerNorm represents layer normalization transformation; Step B15: Send the vector T to the fully connected forward feedback sublayer, which performs two linear transformations on T to obtain the comprehensive feature FFN of the sequence. Then, T and FFN are residually connected and the final high-level feature H of the sequence is obtained after layer normalization. The computer formula is as follows: FFN=(W F T+b F )W N +b N H = LayerNorm (T + FFN) Among them, W F , W N 、b F 、b N represents the training parameters; The step B2 specifically comprises the following steps: Step B21: In the auxiliary task random sliding window reply prediction, the model sets the length and position of the sliding window to random, samples a large amount of local conversation context data that falls within the sliding window from the conversation context, and inserts a special label [EOT] after each utterance of the local conversation context data, as shown in the following formula: Among them, x′ is the input of the subtask. Different from the main task, x′ only retains the information inside the window, and other information is replaced by [PAD]. i is the starting position of the sliding window, w is the size of the current window, m is the number of utterances in the current context, and κ is a hyperparameter indicating the minimum window size. Step B22: Encode the conversation window data using the pre-trained language model BERT. The formula is as follows: E=BERT(x′) Step B23: The vector E obtained in step B22 contains all the semantic representations of the conversation segment after being encoded by the pre-trained language model BERT. The semantic representation that best represents the current conversation segment is further selected from E to optimize the auxiliary task. In order not to destroy the [CLS] representation that can represent global information in the pre-trained language model, the model only selects the [EOT] representation E that is closest to the window response in the output of the pre-trained language model. [EOT] As the final representation vector of the random sliding window reply prediction task; Step B24: Get the final representation vector E [EOT] After that, it is input into the classification layer to calculate the score. The calculation formula is as follows: g(w c ,w r )=σ(W w T E [EOT] +b w ) Among them, w c 、w r represents the context and reply in the sliding window, W w is a trainable parameter in the prediction layer, and σ(·) represents the sigmoid activation function; Step B25: The random sliding window reply prediction task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate the difference between the current score and the actual conversation window label. The specific formula is as follows: Where D′ represents the window data set; The step B3 specifically comprises the following steps: Step B31: The local information perception module embeds a special tag [EOT] after each sentence in the dialogue context, as shown in the following formula: x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]} Under the combined effect of the deep attention mechanism and position embedding of the pre-trained language model, the special label [EOT] of each position can learn the interactive information with the surrounding text at its specific position; at the same time, in the process of optimizing the random sliding window reply prediction task, the last [EOT] label in the window is used to establish a classification task, and the ability to identify window replies is gradually learned; therefore, the representation of the [EOT] label gradually learns the correct representation of the sentence and pays more attention to the text in the local area; Step B32: In the feature fusion stage, the local information perception module selects the n local semantic representations closest to the response from the output of the pre-trained language model as multi-granular local information, and uses splicing to aggregate the local information into a whole. The specific formula is as follows: Among them, l represents the entry closest to the reply, and n is a hyperparameter used to represent the number of [EOT] representations taken out; Step B33: The local information perception module integrates the local information with the global information to obtain the final representation vector of the main task. The aggregation process is: Step B34: Input the aggregated representation vector into the classification layer to calculate the rationality score between the current multi-round dialogue context and the response. The formula is as follows: g(c,r)xσ(W T E ensemble +b) Where W is a trainable parameter, σ(·) represents the sigmoid activation function, and b is the bias term of the current classification layer; Step B35: The deep neural network model PLIP uses gradient descent to update the parameters in the learning model. At the same time, cross entropy is used as the loss function for the multi-round dialogue response selection task. The specific formula is as follows: Combined with the optimization objectives of the auxiliary tasks, the final loss function of the model is: Loss=Loss main +αLoss window Among them, α is a hyperparameter used to control the influence of the auxiliary task random sliding window reply prediction task on the model.

2. A local information perception dialogue system using the method as claimed in claim 1, characterized in that: include: The data collection module collects multi-round dialogue samples in a specific field, and annotates the positive and negative labels of the answers corresponding to each question in the multi-round dialogue data, and constructs a multi-round dialogue response selection training set D with positive and negative labels; Pre-trained language model encoding module. The pre-trained language model is mainly composed of an embedding layer and a multi-layer multi-head attention mechanism. Each training sample in the form of a triple in the training set D is sent to the pre-trained language model BERT, and the contextual semantic representation combined with the context is learned using the multi-layer attention mechanism of the pre-trained language model. At the same time, the model uses multi-task learning to fully tap the semantic understanding ability of the pre-trained language model. In the auxiliary task module, the model exports the parameters of the pre-trained language model BERT and uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand local information in the conversation. The random sliding window reply prediction task samples window data of different positions and sizes in the context of multiple rounds of conversations, uses the exported pre-trained language model to encode the conversation window, and uses the newly added special tag [EOT] to predict the window's reply, allowing the pre-trained language model to fully learn the local language characteristics of different conversation stages and conversation lengths. Local information perception module: In the multi-round dialogue response selection task, the model uses the local information perception module to promote the pre-trained language model BERT to generate multi-granular local semantic information, calculate the rationality score between the multi-round dialogue context and the response, and evaluate whether the current response corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model, and the stochastic gradient descent method is used to update the parameters; as well as The network training module terminates the training of the deep learning network model when the iterative change of the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations.

Images