Dialogue structure-aware dialogue method and system based on fine-grained local information enhancement

Abstract

The present invention relates to a method and system for dialog structure perception based on fine-grained local information enhancement, the method comprising the following steps: Step A: obtaining multi-round dialog texts of a specific scene, and marking the category to which each multi-round dialog reply belongs, and constructing a training set with positive and negative category labels D ; Step B: Use the training set D The fine-grained local information enhanced dialogue structure-aware deep learning network model SAFL is trained to select the corresponding response for a given multi-round dialogue context; Step C: Input the multi-round dialogue context and the response set into the trained deep learning network model SAFL to obtain the most appropriate response for the multi-round dialogue context. The method and system are conducive to improving the accuracy of multi-round dialogue response selection.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a dialogue structure perception dialogue method and system based on fine-grained local information enhancement. Background Art

[0002] In recent years, with the development of machine learning and deep learning networks, humans have made significant progress in intelligent dialogue with computers, and dialogue systems have gradually entered everyone's field of vision. Dialogue systems have important research value for both industry and academia, and can be widely used in many fields. The current dialogue system algorithms are mainly generative dialogue and retrieval dialogue. Among them, generative dialogue can generate an answer word by word based on a question without relying on any corpus in the reasoning stage. The generated answers have the advantage of diversity, but the answers obtained are often not logical, and sometimes they fall into the trap of safe replies. Retrieval dialogue allows the algorithm to find the most appropriate answer in the corpus based on a specific question and reply. It can extract information related to the correct reply from the question and infer the appropriate answer based on this information. The retrieval dialogue model has been widely used in multi-round dialogue systems such as Microsoft XiaoIce. Compared with the generative dialogue model, it is more reliable and has better practicality.

[0003] Lowe et al. built two benchmark models for the reply selection task in retrieval-based multi-round dialogues, based on the Recurrent Neural Networks (RNNs) algorithm and the Long Short-Term Memory (LSTM) algorithm. These two benchmark models use the RNN hidden layer unit to memorize the text features of the previous moment in the process of encoding the text, introduce time order information into the model, and overcome the shortcomings of the bag-of-words model used in the early algorithms. However, in multi-round dialogues, the dialogue history may be lengthy, and not all content is related to the reply. The above two benchmark models directly encode the entire dialogue data and cannot extract important information from the dialogue data in a targeted manner, resulting in unnecessary noise for the model. In order to extract important information from long texts, the researchers proposed to extract important information by matching the context and the reply, and decompose the reply selection task into three steps. The first step is to use the RNN-based algorithm to extract features from each utterance and reply. The second step is to match the extracted utterance features with the reply features. The third step is to use CNN and other methods to extract the information needed to calculate the score in the matching matrix. However, the semantic information that RNN can extract is limited. RNN encoding assumes that data is sequentially related, but in conversation data, topics are dynamic, and two paragraphs that are far apart may also be highly related. It is difficult for RNN encoding to accurately learn the relationship between the two paragraphs. At the same time, RNN may also experience gradient vanishing when encoding long paragraphs, and cannot obtain long-distance dependencies well. The limitations of RNN may cause the above methods to lose important information in the first step. The Transformer architecture proposed by Vaswani in 2017 can fully grasp the global dependency information with the help of a large number of self-attention and interactive attention operations, without being restricted by sequence distance. The researchers rewrote the encoder part of the Transformer and applied it to the encoding module of the model, which strengthened the model's ability to extract information. At the same time, influenced by the multi-head attention mechanism in the Transformer, these works used multi-head attention to construct semantic information of various granularities in the matching stage, enriched the feature representation of the model, and achieved significant improvement. However, the above models still have the following problems. First, insufficient consideration of global sequence information. The above model mainly uses RNN and other methods to encode all sentence representations after matching. These sentence representations may have lost important information during the encoding and matching stages. Second, the word vector representation used does not take the context into consideration. The above model mainly uses static word vectors such as Word2vec, which makes it difficult to solve the problem of polysemy and cannot accurately express semantic information according to different contexts, thus introducing noise during the encoding stage.

[0004] In order to solve the above problems, the research focus of the reply selection task in multi-round dialogue has gradually shifted to the method based on pre-trained language model. The basic steps of this method are to first use a pre-trained language model with a multi-layer Transformer encoder to encode the entire dialogue data, and then input the output representation that can represent the global information [CLS] position in the output into the classification layer for prediction. Recently, some latest research works have proposed effective dialogue modeling methods for multi-round dialogue tasks from the perspectives of dialogue structure and sentence structure, providing new ideas for researchers' future work. Liu et al. proposed the MDFN (Mask-based Decoupling-Fusing Network) model from the perspective of dialogue structure such as dialogue discourse and the order of dialogue speakers. Specifically, MDFN designs two types of granular attention calculation methods, namely the attention mechanism at the discourse granularity and the attention mechanism at the dialogue speaker granularity. At the same time, the attention at the discourse granularity is further subdivided into the attention operation within the discourse and the interactive attention operation between discourses, and the attention at the dialogue speaker granularity is further subdivided into the attention operation within the dialogue speaker and the interactive attention operation between dialogue speakers. With the help of the above two types of four attention calculation mechanisms, the model only needs to extract information at four specified granularities, which reduces the difficulty of the model to extract key information. However, the MDFN model focuses on modeling the relationship between sentences in each attention channel, and does not further extract according to the characteristics of the dialogue on this basis, which may cause the model to mistakenly extract irrelevant content information. At the same time, limiting the four attention operations may also limit the expressiveness of the deep learning model. Zhang et al. proposed a novel Sentence Backbone Regularization (SBR) task from the perspective of sentence structure. The SBR task uses the syntactic parsing toolkit to parse sentences into part-of-speech structures, and derives the parameters of BERT to use the subject and predicate to predict the object, combining the syntactic structure with the pre-trained language model to learn the dialogue sentence structure, and achieved good results. However, the results obtained by the syntactic parsing toolkit are not 100% accurate. Incorrect parsing results may interfere with the judgment of the model. At the same time, the optimization target with a large gap between the SBR task and the main task may also bring noise to the final judgment of the main task. Summary of the invention

[0005] The purpose of the present invention is to provide a dialogue structure-aware dialogue method and system based on fine-grained local information enhancement, which method and system are conducive to improving the accuracy of multi-round dialogue response selection.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a dialogue structure-aware dialogue method based on fine-grained local information enhancement, comprising the following steps:

[0007] Step A: Obtain multi-round dialogue texts of a specific scenario, label the category to which each multi-round dialogue response belongs, and construct a training set D with positive and negative category labels;

[0008] Step B: Use the training set D to train the fine-grained local information enhanced dialogue structure-aware deep learning network model SAFL to select the response corresponding to the given multi-round dialogue context;

[0009] Step C: Input the multi-round dialogue context and the response set into the trained deep learning network model SAFL to obtain the most appropriate response corresponding to the multi-round dialogue context.

[0010] Furthermore, the step B specifically includes the following steps:

[0011] Step B1: Take each triplet training sample in the training set D as the input of the SAFL model, where the triplet training sample includes multi-round dialogue context, reply, and label; the SAFL model uses the pre-trained language model BERT as the basis, and uses BERT's multi-layer attention mechanism to learn contextual semantic representation combined with context; at the same time, the SAFL model uses a multi-task learning method to tap into BERT's semantic understanding ability, with the main task being the multi-round dialogue reply selection task, and the auxiliary tasks being the random sliding window reply prediction task and the dialogue structure perception task;

[0012] Step B2: In the auxiliary task part, the SAFL model exports the pre-trained language model BERT, and uses the random sliding window reply prediction task and the dialogue structure perception task to optimize the pre-trained language model; the random sliding window reply prediction task samples window data of different positions and sizes in the context of multiple rounds of dialogue, and uses the pre-trained language model to encode the dialogue window, thereby enhancing the pre-trained language model's ability to understand local information of the dialogue; the dialogue structure perception task constructs a graph structure according to the order in which the dialoguers speak, and uses GCN for encoding, so as to prompt the pre-trained language model to learn the features of the correct dialogue structure, allowing the pre-trained language model to understand the dialogue data from another perspective;

[0013] Step B3: In the main task, i.e., the multi-round dialogue response selection task, the SAFL model uses the key local information distillation module to extract fine-grained local semantic information from the output of the pre-trained language model, and further fuses and filters the global information and local information through the gating mechanism. The filtered fusion features are fused with the multi-granular local information and input into the classification layer to calculate the rationality score between the multi-round dialogue context and the response, and evaluate whether the current response corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back-propagation method is used to calculate the gradient of each parameter in the deep learning network model SAFL, and the stochastic gradient descent method is used to update the parameters;

[0014] Step B4: When the iterative change of the loss value generated by the deep learning network model SAFL is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model SAFL is terminated.

[0015] Furthermore, the step B1 specifically includes the following steps:

[0016] Step B11: Let u represent the utterance in the context, r represent the reply, and m represent the number of utterances. Concatenate the SAFL input into the following long sequence:

[0017] x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0018] Among them, x is the concatenated long text; [CLS] and [SEP] are delimiters, and [EOT] is a special tag used by SAFL to learn local information;

[0019] Step B12: Map x into word embedding representation, position embedding representation and paragraph embedding representation respectively through the dictionary of the pre-trained language model BERT and the embedding layer of the model;

[0020] Step B13: Add the word embedding representation, position embedding representation and paragraph embedding representation of each word to obtain the fused embedding representation, and encode it using the BERT multi-layer Transformer network to obtain the high-level semantic feature representation E of the sequence.

[0021] Furthermore, the step B2 specifically includes the following steps:

[0022] Step B21: In the random sliding window reply prediction task, which is the first auxiliary task of the model, SAFL sets the size and position of the sliding window to random, samples the local conversation context that falls within the sliding window, and inserts a special tag [EOT] after each utterance in the local conversation context, as shown in the following formula:

[0023]

[0024] Among them, x' is the input of the subtask. Different from the main task, x' only retains the information inside the window, and other information is replaced by [PAD]. w represents the size of the current window, m represents the number of utterances in the complete dialogue context, and κ is a hyperparameter representing the minimum window size.

[0025] Step B22: The pre-trained language model BERT, i.e., the parameters of the BERT model are derived, and the parameters are used to encode the local conversation context data falling within the sliding window, so that BERT can learn conversation data of different qualities and enhance the semantic understanding ability of the pre-trained language model. The formula is as follows:

[0026] E'=BERT(x')

[0027] Step B23: SAFL returns the nearest distance window to [EOT], indicating E [EOT] The input is sent to the classification layer to calculate the rationality score between the multi-round dialogue context and the response. The formula is as follows:

[0028] g(w c ,w r )=σ(W w T E [EOT] +b w )

[0029] Among them, w c ,w r Represents the context and reply in the window data, W w is a trainable parameter in the prediction layer, and σ(·) represents the sigmoid activation function;

[0030] Step B24: The random sliding window reply prediction task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate the difference between the current score and the actual conversation window label. The specific formula is as follows:

[0031]

[0032] Where D' represents the window data set;

[0033] Step B25: The dialogue structure perception task, i.e., the second auxiliary task of the model, samples continuous dialogue segments in the context of multi-round dialogues and inputs them into BERT for encoding. The specific formula is as follows:

[0034] ”={[CLS],u1,[EOT],...,u i ,[EOT],...,u s ,[EOT]}

[0035] E″=BERT(x″)

[0036] The input data contains s utterances and their corresponding local semantic labels. These utterances are continuous dialogue segments extracted from the dialogue context, where s is a hyperparameter used to control the length of the extracted segment.

[0037] Step B26: SAFL takes each utterance in the current conversation segment as a node, builds a fully connected conversation structure graph, and assigns different relationships to each edge according to the order in which the interlocutors speak;

[0038] Step B27: The SAFL model extracts the semantic representations corresponding to s utterances from E” as the node representation in the dialogue structure graph and encodes them using GCN with relations. The specific formula is as follows:

[0039]

[0040]

[0041] Among them, R represents the set of all edge relationships, N represents the set of all nodes in the graph, and c i,r is the automatically learned edge relationship coefficient, W r , W m is the parameter that the model needs to learn; after obtaining the output of the GCN module with relation, the non-relational GCN is used to further learn the dialogue structure information. The specific formula is as follows:

[0042]

[0043] Among them, W nr , W nm It is the parameter that needs to be learned in the second layer of GCN;

[0044] Step B28: After getting the output of the dialogue structure module, the SAFL model concatenates all the outputs to get the overall dialogue structure representation and calculates the score using a linear layer:

[0045]

[0046] g(w s )=σ(WTE s-ensemble +b)

[0047] Step B29: The dialogue structure perception task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate whether the current dialogue segment structure is reasonable. The specific formula is as follows:

[0048]

[0049] Among them, w s It represents the current conversation segment data, and D” represents the conversation segment data set.

[0050] Furthermore, the step B3 specifically includes the following steps:

[0051] Step B31: The key local information distillation module embeds a special tag [EOT] after each sentence in the dialogue context, as shown in the following formula:

[0052] x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0053] Under the combined effect of the deep attention mechanism and position embedding of the pre-trained language model, the special label [EOT] of each position learns the interactive information with the surrounding text at its specific position; at the same time, in the process of optimizing the random sliding window reply prediction task, the last [EOT] label in the window is used to establish a classification task, and the ability to identify window replies is gradually learned; the embedding representation of the special label [EOT] gradually learns a representation of a correct sentence, and pays more attention to the text in the local area;

[0054] Step B32: In the feature fusion stage, the key local information distillation module selects the n local semantic representations closest to the response from the output of the pre-trained language model as multi-granular local information, and uses splicing to aggregate the local information into a whole. The specific formula is as follows:

[0055]

[0056] Among them, l represents the entry closest to the reply, and n is a hyperparameter used to represent the number of [EOT] representations taken out;

[0057] Step B33: After obtaining the multi-granularity local information representation, the key local information distillation module uses a multi-path gating mechanism to filter the noise in the global semantic representation; specifically, the module uses the local semantic representation of each granularity to make a gating selection with the global semantic representation:

[0058]

[0059]

[0060] Step B34: After the global semantic representation is selected by multi-path gating, the mixed semantic representation fusion of different granularities is obtained according to the local semantic representation of different granularities. k , affected by the gating mechanism, the mixed semantic representation of each granularity contains the most important parts of the global semantic representation and the local semantic representation of the current granularity; the key local information distillation module further concatenates the multi-granular mixed semantic representation obtained by multi-channel gating selection with the multi-granular local semantic representation as the input of the prediction layer:

[0061]

[0062] Step B35: Input the aggregated feature representation into the classification layer to calculate the rationality score of the current multi-round dialogue context and the response. The formula is as follows:

[0063] g(c,r)=σ(WTE ensemble +b)

[0064] Where W is a trainable parameter, σ(·) represents the sigmoid activation function, and b is the bias term of the current classification layer;

[0065] Step B36: SAFL uses gradient descent to update the parameters in the learning model, and uses cross entropy as the loss function for the multi-round dialogue response selection task. The specific formula is as follows:

[0066]

[0067] Combining the optimization objectives of the random sliding window reply prediction task and the dialogue structure perception task, the final loss function of SAFL is:

[0068] Loss=Loss main +αLoss window +βLoss DSA

[0069] Among them, α and β are two hyperparameters, which are used to control the influence of the two auxiliary tasks on SAFL.

[0070] The present invention also provides a dialogue structure-aware dialogue system based on fine-grained local information enhancement using the above method, comprising:

[0071] The data collection module is used to collect multi-round dialogue samples in a specific field, and annotate the positive and negative labels of the answers corresponding to each question in the multi-round dialogue data, and construct a multi-round dialogue response selection training set D with positive and negative labels;

[0072] The pre-trained language model encoding module includes a pre-trained language model, which is mainly composed of an embedding layer and a multi-layer multi-head attention mechanism. Each training sample in the form of a triple in the training set D is sent to the pre-trained language model BERT, and the contextual semantic representation combined with the context is learned by using the multi-layer attention mechanism of the pre-trained language model. At the same time, SAFL uses a multi-task learning method to fully explore the semantic understanding ability of the pre-trained language model.

[0073] In the auxiliary task module, SAFL exports the parameters of the pre-trained language model and uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand local information in the conversation. At the same time, it adopts the conversation structure perception task to establish the relationship between sentences through the graph structure, allowing the model to understand the context of multiple rounds of conversations from the perspective of the conversation structure, further enhancing the effect of the conversation system.

[0074] Key local information distillation module: In the multi-round dialogue response selection task, the SAFL model uses the key local information distillation module to extract fine-grained local semantic information from the output of the pre-trained language model, and further fuses and filters the global information and local information through the gating mechanism. The fused features obtained by the screening are fused with the multi-granularity local information and input into the classification layer to calculate the classification score to evaluate whether the current response corresponds to the given multi-round dialogue context. Finally, according to the target loss function, the back-propagation method is used to calculate the gradient of each parameter in the deep learning network model SAFL, and the stochastic gradient descent method is used to update the parameters.

[0075] The network training module terminates the training of the deep learning network model SAFL when the iterative change of the loss value generated by the deep learning network model SAFL is less than the set threshold and no longer decreases or reaches the maximum number of iterations.

[0076] Compared with the prior art, the present invention has the following beneficial effects: it provides a method and system for dialogue structure perception based on fine-grained local information enhancement, which is based on a dialogue structure perception deep learning network model enhanced with fine-grained local information, uses the multi-layer attention mechanism of the pre-trained language model to learn the contextual semantic representation combined with the context, and uses a multi-task learning method to mine the semantic understanding ability of the pre-trained language model, and also uses a key local information distillation module to extract fine-grained local semantic information from the output of the pre-trained language model, and further fuses and screens the global information and local information through a gating mechanism, and fuses the screened fusion features with multi-granularity local information, so as to obtain the most appropriate response corresponding to the multi-round dialogue context. The present invention can effectively improve the accuracy of multi-round dialogue response selection, and has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0078] Figure 2 This is a SAFL architecture diagram of a deep learning model in an embodiment of the present invention;

[0079] Figure 3 is a diagram of a dialog structure perception task structure in an embodiment of the present invention;

[0080] Figure 4 It is a structural diagram of the key local information distillation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0082] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0083] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0084] like Figure 1 As shown, this embodiment provides a dialogue structure-aware dialogue method based on fine-grained local information enhancement, comprising the following steps:

[0085] Step A: Obtain multi-round dialogue texts of specific scenarios, label the category to which each multi-round dialogue response belongs, and construct a training set D with positive and negative category labels.

[0086] Step B: Use the training set D to train the fine-grained local information enhanced dialogue structure-aware deep learning network model SAFL to select the response corresponding to a given multi-round dialogue context.

[0087] The step B specifically comprises the following steps:

[0088] Step B1: Take each triplet training sample in the training set D as the input of the SAFL model. The triplet training sample includes multi-round dialogue context, reply, and label. The SAFL model uses the pre-trained language model BERT as the basis, and uses BERT's multi-layer attention mechanism to learn contextual semantic representation combined with context. At the same time, the SAFL model uses multi-task learning to tap into BERT's semantic understanding ability. The main task is the multi-round dialogue reply selection task, and the auxiliary tasks are the random sliding window reply prediction task and the dialogue structure perception task. The deep learning model SAFL architecture is as follows: Figure 2 The step B1 specifically includes the following steps:

[0089] Step B11: Let u represent the utterance in the context, r represent the reply, and m represent the number of utterances. Concatenate the SAFL input into the following long sequence:

[0090] x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0091] Among them, x is the concatenated long text; [CLS] and [SEP] are delimiters, and [EOT] is a special tag used by SAFL to learn local information.

[0092] Step B12: Through the dictionary of the pre-trained language model BERT and the embedding layer of the model, x is mapped into word embedding representation, position embedding representation and paragraph embedding representation respectively.

[0093] Step B13: Add the word embedding representation, position embedding representation and paragraph embedding representation of each word to obtain the fused embedding representation, and encode it using the BERT multi-layer Transformer network to obtain the high-level semantic feature representation E of the sequence.

[0094] Step B2: In the auxiliary task part, the SAFL model exports the pre-trained language model BERT, and uses the random sliding window reply prediction task and the dialogue structure perception task to optimize the pre-trained language model. The random sliding window reply prediction task samples window data of different positions and sizes in the context of multiple rounds of dialogue, and uses the pre-trained language model to encode the dialogue window, thereby enhancing the pre-trained language model's ability to understand local information of the dialogue. The structure of the dialogue structure perception task is as follows: Figure 3 The dialogue structure perception task constructs a graph structure according to the order in which the dialoguers speak, and uses GCN for encoding, so as to prompt the pre-trained language model to learn the features of the correct dialogue structure and understand the dialogue data from another perspective. The step B2 specifically includes the following steps:

[0095] Step B21: In the random sliding window reply prediction task, which is the first auxiliary task of the model, SAFL sets the size and position of the sliding window to random, samples the local conversation context that falls within the sliding window, and inserts a special tag [EOT] after each utterance in the local conversation context, as shown in the following formula:

[0096]

[0097] Among them, x' is the input of the subtask. Different from the main task, x' only retains the information inside the window, and other information is replaced by [PAD]. w represents the size of the current window, m represents the number of utterances in the complete dialogue context, and κ is a hyperparameter representing the minimum window size.

[0098] Step B22: The pre-trained language model BERT, i.e., the parameters of the BERT model are derived, and the parameters are used to encode the local conversation context data falling within the sliding window, so that BERT can learn conversation data of different qualities and enhance the semantic understanding ability of the pre-trained language model. The formula is as follows:

[0099] E'=BERT(x').

[0100] Step B23: SAFL returns the nearest distance window to [EOT], indicating E [EOT] The input is sent to the classification layer to calculate the rationality score between the multi-round dialogue context and the response. The formula is as follows:

[0101] g(w c ,w r )=σ(W w T E [EOT] +b w )

[0102] Among them, w c ,w r Represents the context and reply in the window data, W w is a trainable parameter in the prediction layer, and σ(·) represents the sigmoid activation function.

[0103] Step B24: The random sliding window reply prediction task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate the difference between the current score and the actual conversation window label. The specific formula is as follows:

[0104]

[0105] Where D' represents the window dataset.

[0106] Step B25: The dialogue structure perception task, i.e., the second auxiliary task of the model, samples continuous dialogue segments in the context of multi-round dialogues and inputs them into BERT for encoding. The specific formula is as follows:

[0107] ”={[CLS],u1,[EOT],...,u i ,[EOT],...,u s ,[EOT]}

[0108] E″=BERT(x″)

[0109] The input data contains s utterances and their corresponding local semantic tags. These utterances are consecutive dialogue segments extracted from the dialogue context, where s is a hyperparameter used to control the length of the extracted segments.

[0110] Step B26: SAFL takes each utterance in the current dialogue segment as a node to build a fully connected dialogue structure graph, and assigns different relationships to each edge according to the order in which the speakers speak.

[0111] Among them, the defined relationships are shown in the following table. p(u i ) represents the speaker to whom the i-th utterance in the context belongs. p1 and p2 represent the first speaker and the second speaker respectively. i < j indicates that the position of the i-th utterance in the context is earlier than that of the j-th utterance. At the same time, since each utterance in the dialogue may have a certain correlation with other utterances, the SAFL model builds a fully connected graph and assigns different edge relationships to each edge in the graph according to the relationships in the following table, and finally obtains a graph that can express the relationships between the sentences in the dialogue text.

[0112]

[0113] Step B27: The SAFL model extracts the semantic representations corresponding to s utterances from E”, as the node representations in the dialogue structure graph, and uses the GCN with relationships for encoding. The specific formula is as follows:

[0114]

[0115]

[0116] Among them, R represents the set of all edge relationships, N represents the set of all nodes in the graph, c i,r is the edge relationship coefficient automatically learned, and W r , W m are the parameters that the model needs to learn. After obtaining the output of the GCN module with relationships, the non-relational GCN is further used to learn the dialogue structure information. The specific formula is as follows:

[0117]

[0118] Among them, W nr , W nm are the parameters that need to be learned in the second layer of GCN.

[0119] Step B28: After obtaining the output of the dialogue structure module, the SAFL model concatenates all the outputs to obtain the overall dialogue structure representation, and uses a linear layer to calculate the score:

[0120]

[0121] g(w s )=σ(WT Es-ensemble +b)

[0122] Step B29: The dialogue structure perception task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate whether the current dialogue segment structure is reasonable. The specific formula is as follows:

[0123]

[0124] Among them, w s It represents the current conversation segment data, and D” represents the conversation segment data set.

[0125] Step B3: In the main task, i.e., the multi-round dialogue response selection task, the SAFL model adopts the following Figure 4 The key local information distillation module shown extracts fine-grained local semantic information from the output of the pre-trained language model, further fuses and filters the global information and local information through the gating mechanism, and fuses the filtered fusion features with the multi-granularity local information, and inputs them into the classification layer to calculate the rationality score between the multi-round dialogue context and the reply, and evaluates whether the current reply corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back propagation method is used to calculate the gradient of each parameter in the deep learning network model SAFL, and the stochastic gradient descent method is used to update the parameters. The step B3 specifically includes the following steps:

[0126] Step B31: The key local information distillation module embeds a special tag [EOT] after each sentence in the dialogue context, as shown in the following formula:

[0127] x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]}

[0128] Under the joint effect of the deep attention mechanism and position embedding of the pre-trained language model, the special label [EOT] of each position learns the interaction information with the surrounding text at its respective specific position; at the same time, in the process of optimizing the random sliding window reply prediction task, the last [EOT] label in the window is used to establish a classification task, and the ability to identify window replies is gradually learned; the embedded representation of the special label [EOT] gradually learns a representation of a correct sentence, and pays more attention to the text in the local area.

[0129] Step B32: In the feature fusion stage, the key local information distillation module selects the n local semantic representations closest to the response from the output of the pre-trained language model as multi-granular local information, and uses splicing to aggregate the local information into a whole. The specific formula is as follows:

[0130]

[0131] Among them, l represents the entry closest to the reply, and n is a hyperparameter used to represent the number of [EOT] representations taken out.

[0132] Step B33: After obtaining the multi-granularity local information representation, the key local information distillation module uses a multi-path gating mechanism to filter the noise in the global semantic representation; specifically, the module uses the local semantic representation of each granularity to make a gating selection with the global semantic representation:

[0133]

[0134]

[0135] Step B34: After the global semantic representation is selected by multi-path gating, the mixed semantic representation fusion of different granularities is obtained according to the local semantic representation of different granularities. k , affected by the gating mechanism, the mixed semantic representation of each granularity contains the most important parts of the global semantic representation and the local semantic representation of the current granularity; the key local information distillation module further concatenates the multi-granular mixed semantic representation obtained by multi-channel gating selection with the multi-granular local semantic representation as the input of the prediction layer:

[0136]

[0137] Step B35: Input the aggregated feature representation into the classification layer to calculate the rationality score of the current multi-round dialogue context and the response. The formula is as follows:

[0138] g(c,r)=σ(W T E ensemble +b)

[0139] Among them, W is a trainable parameter, σ(·) represents the sigmoid activation function, and b is the bias term of the current classification layer.

[0140] Step B36: SAFL uses gradient descent to update the parameters in the learning model, and uses cross entropy as the loss function for the multi-round dialogue response selection task. The specific formula is as follows:

[0141]

[0142] Combining the optimization objectives of the random sliding window reply prediction task and the dialogue structure perception task, the final loss function of SAFL is:

[0143] Loss=Loss main +αLoss window +βLoss DSA

[0144] Among them, α and β are two hyperparameters, which are used to control the influence of the two auxiliary tasks on SAFL.

[0145] Step B4: When the iterative change of the loss value generated by the deep learning network model SAFL is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model SAFL is terminated.

[0146] Step C: Input the multi-round dialogue context and the response set into the trained deep learning network model SAFL to obtain the most appropriate response corresponding to the multi-round dialogue context.

[0147] This embodiment also provides a dialogue structure-aware dialogue system based on fine-grained local information enhancement using the above method, including a data collection module, a pre-trained language model encoding module, an auxiliary task module, a key local information distillation module and a network training module.

[0148] The data collection module is used to collect multi-round dialogue samples in a specific field, and mark the positive and negative labels of the answers corresponding to each question in the multi-round dialogue data, and construct a multi-round dialogue response selection training set D with positive and negative labels.

[0149] The pre-trained language model encoding module includes a pre-trained language model, which is mainly composed of an embedding layer and a multi-layer multi-head attention mechanism; each training sample in the form of a triple in the training set D is sent to the pre-trained language model BERT, and the contextual semantic representation combined with the context is learned by using the multi-layer attention mechanism of the pre-trained language model; at the same time, SAFL adopts a multi-task learning method to fully explore the semantic understanding ability of the pre-trained language model.

[0150] In the auxiliary task module, SAFL exports the parameters of the pre-trained language model and uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand local information of the conversation. At the same time, it adopts the conversation structure perception task to establish the relationship between sentences through the graph structure, allowing the model to understand the context of multiple rounds of conversations from the perspective of the conversation structure, further enhancing the effectiveness of the conversation system.

[0151] In the multi-round dialogue response selection task, the SAFL model uses a key local information distillation module to extract fine-grained local semantic information from the output of the pre-trained language model, and further fuses and filters the global information and local information through a gating mechanism. The filtered fusion features are fused with multi-granular local information and input into the classification layer to calculate the rationality score of the multi-round dialogue context and the response, and evaluate whether the current response corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back-propagation method is used to calculate the gradient of each parameter in the deep learning network model SAFL, and the stochastic gradient descent method is used to update the parameters.

[0152] The network training module is used for training the network model. When the iterative change of the loss value generated by the deep learning network model SAFL is less than the set threshold and no longer decreases or reaches the maximum number of iterations, the training of the deep learning network model SAFL is terminated.

[0153] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A dialogue structure-aware dialogue method based on fine-grained local information enhancement, characterized in that: The following steps are involved: Step A: Obtain multi-round dialogue texts of a specific scenario, label the category to which each multi-round dialogue response belongs, and construct a training set D with positive and negative category labels; Step B: Use the training set D to train the fine-grained local information enhanced dialogue structure-aware deep learning network model SAFL to select the response corresponding to the given multi-round dialogue context; Step C: Input the multi-round dialogue context and the response set into the trained deep learning network model SAFL to obtain the most appropriate response corresponding to the multi-round dialogue context; The step B specifically comprises the following steps: Step B1: Take each triplet training sample in the training set D as the input of the SAFL model, where the triplet training sample includes multi-round dialogue context, reply, and label; the SAFL model uses the pre-trained language model BERT as the basis, and uses BERT's multi-layer attention mechanism to learn contextual semantic representation combined with context; at the same time, the SAFL model uses a multi-task learning method to tap into BERT's semantic understanding ability, with the main task being the multi-round dialogue reply selection task, and the auxiliary tasks being the random sliding window reply prediction task and the dialogue structure perception task; Step B2: In the auxiliary task part, the SAFL model exports the pre-trained language model BERT, and uses the random sliding window reply prediction task and the dialogue structure perception task to optimize the pre-trained language model; the random sliding window reply prediction task samples window data of different positions and sizes in the context of multiple rounds of dialogue, and uses the pre-trained language model to encode the dialogue window, thereby enhancing the pre-trained language model's ability to understand local information of the dialogue; the dialogue structure perception task constructs a graph structure according to the order in which the dialoguers speak, and uses GCN for encoding, so as to prompt the pre-trained language model to learn the features of the correct dialogue structure, allowing the pre-trained language model to understand the dialogue data from another perspective; Step B3: In the main task, i.e., the multi-round dialogue response selection task, the SAFL model uses the key local information distillation module to extract fine-grained local semantic information from the output of the pre-trained language model, and further fuses and filters the global information and local information through the gating mechanism. The filtered fusion features are fused with the multi-granular local information and input into the classification layer to calculate the rationality score between the multi-round dialogue context and the response, and evaluate whether the current response corresponds to the given multi-round dialogue context; finally, according to the target loss function, the back-propagation method is used to calculate the gradient of each parameter in the deep learning network model SAFL, and the stochastic gradient descent method is used to update the parameters; Step B4: When the iterative change of the loss value generated by the deep learning network model SAFL is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model SAFL is terminated; The step B1 specifically comprises the following steps: Step B11: Let u represent the utterance in the context, r represent the reply, and m represent the number of utterances. Concatenate the SAFL input into the following long sequence: x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]} Among them, x is the concatenated long text; [CLS] and [SEP] are delimiters, and [EOT] is a special tag used by SAFL to learn local information; Step B12: Map x into word embedding representation, position embedding representation and paragraph embedding representation respectively through the dictionary of the pre-trained language model BERT and the embedding layer of the model; Step B13: Add the word embedding representation, position embedding representation and paragraph embedding representation of each word to obtain the fused embedding representation, and encode it using the multi-layer Transformer network of BERT to obtain the high-level semantic feature representation E of the sequence; The step B2 specifically comprises the following steps: Step B21: In the random sliding window reply prediction task, which is the first auxiliary task of the model, SAFL sets the size and position of the sliding window to random, samples the local conversation context that falls within the sliding window, and inserts a special tag [EOT] after each utterance in the local conversation context, as shown in the following formula: Among them, x' is the input of the subtask. Different from the main task, x' only retains the information inside the window, and other information is replaced by [PAD]. w represents the size of the current window, m represents the number of utterances in the complete dialogue context, and κ is a hyperparameter representing the minimum window size. Step B22: The pre-trained language model BERT is exported and used to encode the local conversation context data falling within the sliding window, so that BERT can learn conversation data of different qualities and enhance the semantic understanding ability of the pre-trained language model. The formula is as follows: E'=BERT(x') Step B23: SAFL returns the nearest distance window to [EOT], indicating E [EOT] The input is sent to the classification layer to calculate the rationality score between the multi-round dialogue context and the response. The formula is as follows: g(w c ,w r )=σ(W w T E [EOT] +b w ) Among them, w c ,w r Represents the context and reply in the window data, W w is a trainable parameter in the prediction layer, and σ(·) represents the sigmoid activation function; Step B24: The random sliding window reply prediction task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate the difference between the current score and the actual conversation window label. The specific formula is as follows: Where D' represents the window data set; Step B25: The dialogue structure perception task, i.e., the second auxiliary task of the model, samples continuous dialogue segments in the context of multi-round dialogues and inputs them into BERT for encoding. The specific formula is as follows: x″={[CLS],u1,[EOT],…,u i ,[EOT],…,u s ,[EOT]} E″=BERT(x″) The input data contains s utterances and their corresponding local semantic labels. These utterances are continuous dialogue segments extracted from the dialogue context, where s is a hyperparameter used to control the length of the extracted segment. Step B26: SAFL takes each utterance in the current conversation segment as a node, builds a fully connected conversation structure graph, and assigns different relationships to each edge according to the order in which the interlocutors speak; Step B27: The SAFL model extracts the semantic representations corresponding to s utterances from E” as the node representation in the dialogue structure graph and encodes them using GCN with relations. The specific formula is as follows: Among them, R represents the set of all edge relationships, N represents the set of all nodes in the graph, and c i,r is the automatically learned edge relationship coefficient, W r , W m is the parameter that the model needs to learn; after obtaining the output of the GCN module with relation, the non-relational GCN is used to further learn the dialogue structure information. The specific formula is as follows: Among them, W nr , W nm It is the parameter that needs to be learned in the second layer of GCN; Step B28: After getting the output of the dialogue structure module, the SAFL model concatenates all the outputs to get the overall dialogue structure representation and calculates the score using a linear layer: g(w s )6σ(W T E s-ensemble +b) Step B29: The dialogue structure perception task uses gradient descent to optimize the objective function. The objective function uses the cross entropy loss function to evaluate whether the current dialogue segment structure is reasonable. The specific formula is as follows: Among them, w s represents the current conversation segment data, and D” represents the conversation segment dataset; The step B3 specifically comprises the following steps: Step B31: The key local information distillation module embeds a special tag [EOT] after each sentence in the dialogue context, as shown in the following formula: x={[CLS],u1,[EOT],u2,[EOT],…,[EOT],u m ,[SEP],r,[SEP]} Under the combined effect of the deep attention mechanism and position embedding of the pre-trained language model, the special label [EOT] of each position learns the interactive information with the surrounding text at its specific position; at the same time, in the process of optimizing the random sliding window reply prediction task, the last [EOT] label in the window is used to establish a classification task, and the ability to identify window replies is gradually learned; the embedding representation of the special label [EOT] gradually learns a representation of a correct sentence, and pays more attention to the text in the local area; Step B32: In the feature fusion stage, the key local information distillation module selects the n local semantic representations closest to the response from the output of the pre-trained language model as multi-granular local information, and uses splicing to aggregate the local information into a whole. The specific formula is as follows: Among them, l represents the entry closest to the reply, and n is a hyperparameter used to represent the number of [EOT] representations taken out; Step B33: After obtaining the multi-granularity local information representation, the key local information distillation module uses a multi-path gating mechanism to filter the noise in the global semantic representation; specifically, the module uses the local semantic representation of each granularity to make a gating selection with the global semantic representation: Step B34: After the global semantic representation is selected by multi-path gating, the mixed semantic representation fusion of different granularities is obtained according to the local semantic representation of different granularities. k , affected by the gating mechanism, the mixed semantic representation of each granularity contains the most important parts of the global semantic representation and the local semantic representation of the current granularity; the key local information distillation module further concatenates the multi-granular mixed semantic representation obtained by multi-channel gating selection with the multi-granular local semantic representation as the input of the prediction layer: Step B35: Input the aggregated feature representation into the classification layer to calculate the rationality score of the current multi-round dialogue context and the response. The formula is as follows: g(c,r)vσ(W T E ensemble +b) Where W is a trainable parameter, σ(·) represents the sigmoid activation function, and b is the bias term of the current classification layer; Step B36: SAFL uses gradient descent to update the parameters in the learning model, and uses cross entropy as the loss function for the multi-round dialogue response selection task. The specific formula is as follows: Combining the optimization objectives of the random sliding window reply prediction task and the dialogue structure perception task, the final loss function of SAFL is: Loss=Loss main +αLoss window +βLoss DSA Among them, α and β are two hyperparameters, which are used to control the influence of the two auxiliary tasks on SAFL.

2. A dialog structure-aware dialog system based on fine-grained local information enhancement using the method of claim 1, characterized in that: include: The data collection module is used to collect multi-round dialogue samples in a specific field, and annotate the positive and negative labels of the answers corresponding to each question in the multi-round dialogue data, and construct a multi-round dialogue response selection training set D with positive and negative labels; The pre-trained language model encoding module includes a pre-trained language model, which is mainly composed of an embedding layer and a multi-layer multi-head attention mechanism. Each training sample in the form of a triple in the training set D is sent to the pre-trained language model BERT, and the contextual semantic representation combined with the context is learned by using the multi-layer attention mechanism of the pre-trained language model. At the same time, SAFL adopts a multi-task learning method to fully explore the semantic understanding ability of the pre-trained language model. In the auxiliary task module, SAFL exports the parameters of the pre-trained language model and uses the random sliding window reply prediction task to further enhance the pre-trained language model's ability to understand local information in the conversation. At the same time, it adopts the conversation structure perception task to establish the dependency relationship between sentences through the graph structure, allowing the model to understand the context of multiple rounds of conversations from the perspective of the conversation structure, further enhancing the effect of the conversation system. Key local information distillation module: In the multi-round dialogue response selection task, the SAFL model uses the key local information distillation module to extract fine-grained local semantic information from the output of the pre-trained language model, and further fuses and filters the global information and local information through the gating mechanism. The fused features obtained by the screening are fused with the multi-granularity local information and input into the classification layer to calculate the rationality score of the multi-round dialogue context and the response, and evaluate whether the current response corresponds to the given multi-round dialogue context. Finally, according to the target loss function, the back-propagation method is used to calculate the gradient of each parameter in the deep learning network model SAFL, and the stochastic gradient descent method is used to update the parameters. The network training module terminates the training of the deep learning network model SAFL when the iterative change of the loss value generated by the deep learning network model SAFL is less than the set threshold or reaches the maximum number of iterations.

Images