Context deconstruction-based dialogue state tracking method and system

Abstract

The invention relates to a dialogue state tracking method based on context deconstruction. The method comprises the following steps: constructing a data set; constructing a deep learning network model based on context deconstruction; the model inputs the dialogue into a dialogue context deconstruction coding module for coding so as to decouple the mutual influence of instant information and historical information; searching current dialogue information and historical information related to the slot through a slot utterance alignment module and a slot memory alignment module; the influence of groove utterance attention and groove memory attention on a specific groove is dynamically controlled through a meta-learning enhanced gating fusion module, and a groove specific representation vector and an optimal gating unit parameter after multi-view information fusion are obtained; capturing the dynamic correlation between slot positions in the dialogue through a slot perception dynamic fusion module; training the model by using the data set; and inputting dialogue data of the user and the system into the trained model in sequence, and outputting a current dialogue state. The method and the system are beneficial to improving the accuracy of dialogue state tracking.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a method and system for tracking a dialog state based on context deconstruction. Background Art

[0002] Dialogue State Tracking (DST) is a crucial module in task-based dialogue systems. Its goal is to monitor user goals hidden in the dialogue history and represent them as a dialogue state consisting of a series of (domain, slot, slot value) triplets. The continuous advancement of information technology has enabled people to access information, applications, and services almost instantly at any time and place with wireless connections. However, while various Internet applications and personal applications have made daily life and travel more and more convenient, more and more applications have increased the time cost and difficulty of use for users, so the demand for virtual personal intelligent assistants is increasing. Dialogue state tracking is the core technology for realizing virtual personal intelligent assistants, but human dialogue is inherently complex and ambiguous, and there is still a long way to go to create an open-domain dialogue artificial intelligence that can face any scenario.

[0003] In order to solve the drawbacks of manually crafting features to predict slot values, et al. proposed a model called Neural Belief Tracker (NBT). NBT uses convolutional neural network (CNN) instead of handcrafted features to predict slot values ​​in the word embedding processing part, which greatly improves the performance compared with the previous DST model. Inspired by this pioneering work, many neural DST methods based on long short-term memory (LSTM) networks and bidirectional gated recurrent unit (BiGRU) networks have been proposed to further improve the performance of neural DST models.

[0004] Researchers usually regard DST as a multi-classification or multi-hop classification task and propose an ontology-based dialogue state tracking method. Henderson et al. were the first to use deep learning models in DST tasks. They integrated many feature functions (such as SLU scores, Rank scores, Affirm scores, etc.) as inputs to the neural network and then predicted the probability of each slot-value pair.

[0005] In order to overcome the limitations of fixed ontology, researchers proposed a DST method based on an open vocabulary, which not only reduced the model complexity and time complexity of the DST task, but also was more conducive to the end-to-end training of task-oriented dialogue systems. Wu et al. proposed the TRADE model, which also applied a copy mechanism and used a soft-gated pointer generator to generate slot values ​​based on domain-slot pairs and dialogue context encoding. Cheng et al. proposed a tree encoder-decoder TED architecture, which uses a hierarchical tree structure to represent dialogue states and system behaviors. The TED architecture generates a tree-structured dialogue state for the current round based on the dialogue history, dialogue behavior, and the dialogue state of the previous round. Chen et al. built an interactive encoder to model the dependencies between rounds. In addition, they used the attention mechanism to construct slot-level context environments for users and systems respectively, and the state generator copied slot values ​​from the dialogue context based on this.

[0006] Recently, pre-trained language models have received widespread attention from industry and academia, and a variety of pre-trained language models have been released, such as BERT and GPT-2. Since the models are trained on large corpora, they have shown strong performance in downstream tasks. e et al. first introduced the BERT model into the DST task and proposed the SUMBT framework. SUBMT uses BERT to learn the relationship between slots and dialogue context through the slot-word attention mechanism. Inspired by the SUMBT model, Shan et al. proposed the CHAN architecture, applied BERT for multi-task learning and generated dialogue states. They first encoded the dialogue content at the word level and turn level, and then retrieved relevant information for each slot from the dialogue content by applying word-level and turn-level attention, and predicted the corresponding slot value based on the retrieved information. Heck et al. proposed a new DST method TripPy, which does not need to maintain a list of candidate values ​​and uses three copying mechanisms to extract slot values: extracting slot values ​​directly from user input, copying slot values ​​from the system notification memory that tracks system notification operations, and copying slot values ​​from different slots already included in the dialogue state. Hosseini-Asl et al. used GPT-2 as a dialogue context encoder and transformed DST into a language generation task, which also achieved good results. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for tracking a dialog state based on context deconstruction, which method and system are conducive to improving the accuracy of dialog state tracking.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is: a method for tracking a conversation state based on context deconstruction, comprising the following steps:

[0009] Step A: Collect the conversation data and conversation status data between users and the system in different scenarios and build a data set TS;

[0010] Step B: Construct a deep learning network model G based on context deconstruction; the deep learning network model G inputs the dialogue into the dialogue context deconstruction encoding module for encoding to decouple the mutual influence of instant information and historical information; retrieves the current dialogue information and historical information related to the slot through the slot discourse alignment module and the slot memory alignment module; dynamically controls the influence of slot discourse attention and slot memory attention on specific slots through the meta-learning enhanced gated fusion module, and obtains the slot-specific representation vector and the optimal gated unit parameters after multi-view information fusion; captures the dynamic correlation between slots in the dialogue through the slot-aware dynamic fusion module; and trains the deep learning network model G using the dataset TS;

[0011] Step C: Input the conversation data between the user and the system into the trained deep learning network model G in sequence, and output the current conversation status.

[0012] Furthermore, the step A specifically includes the following steps:

[0013] Step A1: Collect conversation data between users and the system in different scenarios; the conversation data in one scenario includes multiple rounds of user utterances and system responses, where user utterances represent the input text of the user in one round of conversation, and system responses represent the system's response to the above user input; each round of conversation in each scenario contains a set of conversation state data, which includes:

[0014] Slot: represents a predefined category of information in a conversation, used to identify a specific component of user intent;

[0015] Slot value: the actual information filled into a specific slot, representing the specific data extracted from the user's input corresponding to the specific slot;

[0016] Dialog state: represented as a set of key-value pairs, where the key represents the slot name and the value is the slot value;

[0017] Dialogue history: contains the sequence of all user inputs and system responses before the current turn;

[0018] Step A2: The conversation data and conversation state data between the user and the system in a scenario form a conversation sample, and multiple conversation samples together form a data set TS; a conversation sample in the data set TS includes:

[0019] T current dialogues, the current dialogue of the tth round is represented by D t =(R t , U t ), where Rt Indicates the system response, U t Represents user utterances;

[0020] The dialogue history of T rounds of dialogue, the dialogue history of the tth round is expressed as Where R t Indicates the system response, U t represents the user's utterance, D t represents the current dialogue in the tth round; (3) A dialogue sample contains the dialogue state of T rounds of dialogue, and the dialogue state of the tth round is represented as a set of key-value pairs Where S i is the slot name, is the slot candidate value corresponding to the slot in round t, and I is the number of all slots in different fields;

[0021] Step A3: Divide the dataset TS into training set D by random sampling, with 80% of the data n , 20% of the data is divided into the validation set D v The training set is used for model training and parameter optimization, and the validation set is used to evaluate the performance of the model and adjust hyperparameters.

[0022] Furthermore, the step B specifically includes the following steps:

[0023] Step B1: Input the dialogue state data contained in the t-th round of dialogue in the dataset TS into the dialogue context deconstruction encoding module for encoding to decouple the mutual influence of instant information and historical information; the dialogue context deconstruction encoding module consists of four encoders: current discourse encoder, past memory encoder, slot encoder and slot candidate value encoder; the current discourse encoder encodes the system response and user discourse combination sequence of the current dialogue to obtain the current discourse representation vector The past memory encoder encodes the combined sequence of the dialogue history and the previous round of dialogue states to obtain a memory representation vector The slot encoder and slot candidate value encoder encode all predefined slots and their corresponding slot candidate values ​​to obtain the slot representation vector h s and slot candidate value representation vector

[0024] Step B2: Represent the current speech vector and slot characterization vector h s Input to the slot utterance alignment module, retrieve the current dialogue information related to the slot, and get the slot utterance attention Representing the memory vector and slot characterization vector h sInput to the slot memory alignment module to track the historical conversation information related to the slot and obtain the slot memory attention

[0025] Step B3: Pay attention to the slot utterance obtained in step B2 Attention with slot memory The input is to the meta-learning enhanced gated fusion module; the meta-learning enhanced gated fusion module adopts a two-layer optimization strategy to determine the optimal gated unit parameters, dynamically control the influence of slot discourse attention and slot memory attention on specific slots, so as to enhance the adaptability to dialogue tasks in different fields; the meta-learning enhanced gated fusion module first uses the meta-learning loss function to perform internal optimization on the training set, and then performs external optimization on the gated unit parameters through the verification set, and finally obtains the slot-specific representation vector after multi-view information fusion and the optimal gating unit parameters

[0026] Step B4: Merge the slot representation vector obtained in step B3 Input into the slot-aware dynamic fusion module to capture the dynamic correlation between slots in the conversation and obtain the slot-aware representation vector

[0027] Step B5: The slot perception representation vector obtained in step B4 The corresponding slot candidate slot value representation vector obtained in step B1 Perform similarity matching and select the most similar slot value as the prediction result; compare the predicted value with the true value to calculate the slot prediction loss, use the back propagation algorithm to calculate the gradient of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters;

[0028] Step B6: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model G is terminated.

[0029] Furthermore, the step B1 specifically includes the following steps:

[0030] Step B11: Concatenate the system response and user utterance of the current dialogue into utterance-level representation, insert markers at the beginning and end of the sequence, and define the input representation as:

[0031]

[0032] Among them, [CLS] is a special token added before each round of input, and [SEP] is a special token used to mark the end of the dialogue round;

[0033] The dialogue history and the dialogue state of the previous round are concatenated into a memory-level dialogue context sequence. The entire memory-level dialogue context sequence of the tth round is defined as:

[0034]

[0035] Will and Input into the current speech encoder and the past memory encoder respectively to obtain the current speech representation vector and memory representation vector It is expressed as:

[0036]

[0037] Among them, BERT finetune For the BERT model using the finetune strategy for feature extraction, L is the length of the current conversation, L h is the length of the dialogue history plus the dialogue state of the previous round, and d is the dimension of the token representation vector;

[0038] Step B12: The sequence of slot names and slot values ​​in round t is represented as s and v t , replace s and v t Input into the slot encoder and slot candidate value encoder respectively to obtain the slot representation vector h s and slot value representation vector It is expressed as:

[0039] h s =BERT fixed (s)

[0040]

[0041] Among them, BERT fixed For the BERT model that uses the fixed strategy for feature extraction, L s is the length of the slot name, L v is the length of the slot value candidate, d is the dimension of the token representation vector, BERT fixed The parameter weights of are fixed during training to reduce the difficulty of slot value matching;

[0042] Since the number of slot candidate values ​​for each slot is different, Characterize the vector for each candidate value in this slot Collection It is expressed as:

[0043]

[0044] Wherein, i=1,2,...q{8≤q≤156}; Represents the i-th slot candidate value for the j-th slot.

[0045] Furthermore, the step B2 specifically includes the following steps:

[0046] Step B21: Represent the current speech vector and slot characterization vector h s Input to the slot discourse alignment module to obtain the slot discourse attention that integrates the current dialogue information It is expressed as:

[0047]

[0048]

[0049] in, is the learnable weight parameter, and the slot representation vector h s As the query matrix input, the current discourse representation vector As the key and value matrices, SelfAttention(Q, K, V) and MultiHeadAttention(Q, K, V) are the self-attention and multi-head attention mechanisms respectively. The Concat() function concatenates the outputs of h self-attention heads.

[0050] Step B22: To retain the slot name information, the slot utterances output by the multiple heads in step B21 are focused on and the slot characterization vector h s After concatenation, it is input into the feedforward neural network for linear transformation to obtain a temporary representation vector The temporary representation vector After applying the ReLU activation function, a linear transformation is performed to map it to the new space to obtain the final slot speech attention. It is expressed as:

[0051]

[0052] in, is the learnable weight parameter; is bias;

[0053] Step B23: The memory representation vector output from step B12 Slot characterization vector h s Input to the slot memory alignment module to obtain the slot memory attention of fused memory information It is expressed as:

[0054]

[0055] Attention slot memory Further transformation through the feedforward neural network to obtain the final slot memory attention It is expressed as:

[0056]

[0057] in, is the learnable weight parameter; For bias.

[0058] Furthermore, the specific implementation method of step B3 is:

[0059] The slot utterance attention obtained in step B2 and slot memory attention Input to the meta-learning enhanced gated fusion module to obtain the slot-specific representation vector It is expressed as:

[0060] f(w1)=Sigmoid(w1)

[0061]

[0062] For different slots, use meta-learning to set f(w1) to different values ​​to train a more robust model;

[0063] Among them, w1, w2 are learnable network parameters, f′(w2)=1-f(w1) means that the sum of the weights of slot speech attention and slot memory attention is 1, ⊙ means the element-by-element product operation of two vectors, Represents a vector product operation;

[0064] The goal of meta-learning is to use Find the optimal parameters w1, w2 of the gate unit and determine the allocation slot discourse attention and slot memory attention First, the meta-learning enhanced gated fusion module uses the meta-learning loss function to perform internal optimization on the training set, so that the model can minimize the loss value after training the optimal parameters based on parameters w1 and w2; the training objective of the model derived from the meta-learning loss function is:

[0065]

[0066] Among them, D n is the training set, θ represents the parameters of the main model, X t represents the conversation context, s represents the slot associated with the conversation context, represents the vector product operation, and The loss values ​​corresponding to slot discourse attention and slot memory attention are minimized by Optimize, expressed as:

[0067]

[0068] Secondly, the meta-learning enhanced gated fusion module uses the validation set D v As metadata for external optimization, the optimal parameter θ * (w1, w2) is expected to be in the validation set D v To achieve the best performance, the method of optimizing w1 and w2 is expressed as:

[0069]

[0070] in, Represents the validation set dialogue X t The corresponding loss of slot s is given by the parameter θ * The initial model prediction of (w1, w2) is calculated, and the optimal parameters of the gated unit after meta-learning are finally obtained

[0071] Furthermore, the step B4 specifically includes the following steps:

[0072] Step B41: Slot-specific characterization vector obtained from step B32 The input is sent to the slot-aware dynamic fusion module, which automatically corrects the correlation between domains and slots and provides mutual guidance between slots. Represents the matrix representation of all fusion slot information vectors, let F1 = ε t , for the lth self-attention layer, it is expressed as:

[0073]

[0074] For the lth feedforward sublayer, it is expressed as:

[0075]

[0076]

[0077] in, is the learnable parameter matrix, is the bias term, is the output of the last layer;

[0078] Step B42: In order to compare with the slot candidate value encoded by BERT, the value obtained in step B41 is Input to a linear layer LayerNorm layer to get a value close to slot S JSemantic vector representation of the truth value at round t: slot perception table

[0079] Eigenvector It is expressed as:

[0080]

[0081] in, represents the j-th slot perception representation vector, Represents the set of all slot-aware representation vectors.

[0082] Furthermore, the step B5 specifically includes the following steps:

[0083] Step B51: The slot candidate value representation vector obtained in step B12 Slot self-attention representation obtained with B42 Calculate the semantic distance as a measure of similarity; then select the slot candidate value representation vector with the smallest semantic distance to the slot self-attention representation, that is, the most similar slot candidate value as slot S j The final prediction result; use L2 norm as the distance metric; in the training phase, calculate the slot S in round t j The true value of The probability is:

[0084]

[0085] The value with the highest probability is taken as the predicted value; where exp(·) represents the exponential function, ||·||2 represents the L2 norm, S is the time slot of round t j The true value of The vector representation of ;

[0086] Step B52: Train the model to maximize the joint probability of all slots, i.e. The slot value prediction loss function for each round t is defined as the accumulation of negative log-likelihood:

[0087]

[0088] The calculated loss The learning rate is updated through the gradient optimization algorithm AdamW, and the model parameters are iteratively updated using back propagation to minimize the loss function to train the model.

[0089] The present invention also provides a conversation state tracking system based on context deconstruction, comprising a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0090] Compared with the prior art, the present invention has the following beneficial effects:

[0091] 1) The present invention splits the conversation context information into current discourse and past memory, which can decouple the mutual influence of immediate information and historical information and avoid mutual interference of information, so that the model can focus on the details contained in the current discourse while fully exploring the long-term dependencies in the conversation history.

[0092] 2) The present invention determines the ratio of allocated slots at the discourse level and slot memory level through a meta-learning gated fusion module, thereby filtering out redundant information while incorporating sufficient dialogue information, and is closer to the spatial representation of the actual slot value to a greater extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0094] Figure 2 It is an architecture diagram of the deep learning network model G based on context deconstruction in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0096] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0097] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0098] like Figure 1 As shown, this embodiment provides a method for tracking a dialog state based on context deconstruction, comprising the following steps:

[0099] Step A: Collect the conversation data and conversation status data between users and the system in different scenarios and build a data set TS;

[0100] Step B: Construct a deep learning network model G based on context deconstruction; the deep learning network model G inputs the dialogue into the dialogue context deconstruction encoding module for encoding to decouple the mutual influence of instant information and historical information; retrieves the current dialogue information and historical information related to the slot through the slot discourse alignment module and the slot memory alignment module; dynamically controls the influence of slot discourse attention and slot memory attention on specific slots through the meta-learning enhanced gated fusion module, and obtains the slot-specific representation vector and the optimal gated unit parameters after multi-view information fusion; captures the dynamic correlation between slots in the dialogue through the slot-aware dynamic fusion module; and trains the deep learning network model G using the dataset TS;

[0101] Step C: Input the conversation data between the user and the system into the trained deep learning network model G in sequence, and output the current conversation status.

[0102] In this embodiment, the step A specifically includes the following steps:

[0103] Step A1: Collect conversation data between users and the system in different scenarios. Specific scenarios include task conversation scenarios such as booking a hotel, booking a train ticket, and booking a restaurant. Conversation data in one scenario includes multiple rounds of user utterances and system responses, where user utterances represent the input text of the user in one round of conversation, and system responses represent the system's response to the user input. Each round of conversation in each scenario contains a set of conversation state data, which includes:

[0104] (1) Slot: represents a predefined information category in a conversation and is used to identify a specific component of user intent;

[0105] (2) Slot value: the actual information filled into a specific slot, representing the specific data extracted from the user’s input corresponding to the specific slot;

[0106] (3) Dialog state: represented as a set of key-value pairs, where the key represents the slot name and the value represents the slot value;

[0107] (4) Dialogue history: contains the sequence of all user inputs and system responses before the current round;

[0108] Step A2: The conversation data and conversation state data between the user and the system in a scenario form a conversation sample, and multiple conversation samples together form a data set TS; a conversation sample in the data set TS includes:

[0109] (1) There are T current dialogues. The current dialogue in the tth round is represented by D t =(R t , U t ), where R t Indicates the system response, U t Represents user utterances;

[0110] (2) The dialogue history of T rounds of dialogue, the dialogue history of the tth round is expressed as Where R t Indicates the system response, U t represents the user's utterance, D t represents the current dialogue in the tth round; (3) A dialogue sample contains the dialogue state of T rounds of dialogue, and the dialogue state of the tth round is represented as a set of key-value pairs Where S i is the slot name, is the slot candidate value corresponding to the slot in round t, and I is the number of all slots in different fields;

[0111] Step A3: Divide the dataset TS into training set D by random sampling, with 80% of the data n , 20% of the data is divided into the validation set D v The training set is used for model training and parameter optimization, and the validation set is used to evaluate the performance of the model and adjust hyperparameters.

[0112] Figure 2 It is an architecture diagram of the deep learning network model G based on context deconstruction in this embodiment. In this embodiment, the specific implementation steps of step B are as follows.

[0113] Step B1: Input the dialogue state data contained in the t-th round of dialogue in the dataset TS into the dialogue context deconstruction encoding module for encoding to decouple the mutual influence of instant information and historical information; the dialogue context deconstruction encoding module consists of four encoders: current discourse encoder, past memory encoder, slot encoder and slot candidate value encoder; the current discourse encoder encodes the system response and user discourse combination sequence of the current dialogue to obtain the current discourse representation vector The past memory encoder encodes the combined sequence of the dialogue history and the previous round of dialogue states to obtain a memory representation vector The slot encoder and slot candidate value encoder encode all predefined slots and their corresponding slot candidate values ​​to obtain the slot representation vector h s and slot candidate value representation vector

[0114] The step B1 specifically comprises the following steps:

[0115] Step B11: Concatenate the system response and user utterance of the current dialogue into utterance-level representation, insert markers at the beginning and end of the sequence, and define the input representation as:

[0116]

[0117] Among them, [CLS] is a special token added before each round of input, and [SEP] is a special token used to mark the end of the dialogue round.

[0118] The dialogue history and the dialogue state of the previous round are concatenated into a memory-level dialogue context sequence. The entire memory-level dialogue context sequence of the tth round is defined as:

[0119]

[0120] Will and Input into the current speech encoder and the past memory encoder respectively to obtain the current speech representation vector and memory representation vector It is expressed as:

[0121]

[0122] Among them, BERT finetune For the BERT model using the finetune strategy for feature extraction, L is the length of the current conversation, L h is the length of the dialogue history plus the dialogue state of the previous round, and d is the dimension of the token representation vector.

[0123] Step B12: The sequence of slot names and slot values ​​in round t is represented as s and v t , replace s and v t Input into the slot encoder and slot candidate value encoder respectively to obtain the slot representation vector h s and slot value representation vector It is expressed as:

[0124] h s =BERT fixed (s)

[0125]

[0126] Among them, BERT fixed For the BERT model that uses the fixed strategy for feature extraction, L s is the length of the slot name, L v is the length of the slot value candidate, d is the dimension of the token representation vector, BERT fixed The parameter weights of are fixed during training to reduce the difficulty of slot value matching.

[0127] Since the number of slot candidate values ​​for each slot is different, Characterize the vector for each candidate value in this slot Collection It is expressed as:

[0128]

[0129] Wherein, i=1,2,...q{8≤q≤156}; Represents the i-th slot candidate value for the j-th slot.

[0130] Step B2: Represent the current speech vector and slot characterization vector h s Input to the slot utterance alignment module, retrieve the current dialogue information related to the slot, and get the slot utterance attention Representing the memory vector and slot characterization vector h s Input to the slot memory alignment module to track the historical conversation information related to the slot and obtain the slot memory attention

[0131] The step B2 specifically comprises the following steps:

[0132] Step B21: Represent the current speech vector and slot characterization vector h s Input to the slot discourse alignment module to obtain the slot discourse attention that integrates the current dialogue information It is expressed as:

[0133]

[0134]

[0135] in, is the learnable weight parameter, and the slot representation vector h s As the query matrix input, the current discourse representation vector As the key and value matrices, SelfAttention(Q, K, V) and MultiHeadAttention(Q, K, V) are the self-attention and multi-head attention mechanisms respectively. The Concat() function concatenates the outputs of h self-attention heads.

[0136] Step B22: To retain the slot name information, the slot utterances output by the multiple heads in step B21 are focused on and the slot characterization vector h s After concatenation, it is input into the feedforward neural network for linear transformation to obtain a temporary representation vector The temporary representation vector After applying the ReLU activation function, a linear transformation is performed to map it to the new space to obtain the final slot speech attention. It is expressed as:

[0137]

[0138]

[0139] in, is the learnable weight parameter; For bias.

[0140] Step B23: The memory representation vector output from step B12 Slot characterization vector h s Input to the slot memory alignment module to obtain the slot memory attention of fused memory information It is expressed as:

[0141]

[0142] Attention slot memory Further transformation through the feedforward neural network to obtain the final slot memory attention It is expressed as:

[0143]

[0144] in, is the learnable weight parameter; For bias.

[0145] Step B3: Pay attention to the slot utterance obtained in step B2 Attention with slot memory The input is to the meta-learning enhanced gated fusion module; the meta-learning enhanced gated fusion module adopts a two-layer optimization strategy to determine the optimal gated unit parameters w1, w2, and dynamically control the influence of slot discourse attention and slot memory attention on specific slots to enhance the adaptability to dialogue tasks in different fields; the meta-learning enhanced gated fusion module first uses the meta-learning loss function to perform internal optimization on the training set, and then performs external optimization on the gated unit parameters w1, w2 through the verification set, and finally obtains the slot-specific representation vector after multi-view information fusion and the optimal gating unit parameters

[0146] The specific implementation method of step B3 is:

[0147] The slot utterance attention obtained in step B2 and slot memory attention Input to the meta-learning enhanced gated fusion module to obtain the slot-specific representation vector It is expressed as:

[0148] f(w1)=Sigmoid(w1)

[0149]

[0150] For different slots, meta-learning is used to set f(w1) to different values ​​to train a more robust model.

[0151] Among them, w1, w2 are learnable network parameters, f′(w2)=1-f(w1) means that the sum of the weights of slot speech attention and slot memory attention is 1, ⊙ means the element-by-element product operation of two vectors, Represents a vector product operation.

[0152] The goal of meta-learning is to use Find the optimal parameters w1, w2 of the gate unit and determine the allocation slot discourse attention and slot memory attention First, the meta-learning enhanced gated fusion module uses the meta-learning loss function to perform internal optimization on the training set, so that the model can minimize the loss value after training the optimal parameters based on parameters w1 and w2; the training objective of the model derived from the meta-learning loss function is:

[0153]

[0154] Among them, D n is the training set, θ represents the parameters of the main model, X t represents the conversation context, s represents the slot associated with the conversation context, represents the vector product operation, and The loss values ​​corresponding to slot discourse attention and slot memory attention are minimized by Optimize, expressed as:

[0155]

[0156] Secondly, the meta-learning enhanced gated fusion module uses the validation set D v As metadata for external optimization, the optimal parameter θ * (w1, w2) is expected to be in the validation set D v To achieve the best performance, the method of optimizing w1 and w2 is expressed as:

[0157]

[0158] in, Represents the validation set dialogue X t The corresponding loss of slot s is given by the parameter θ *The initial model prediction of (w1, w2) is calculated, and the optimal parameters of the gated unit after meta-learning are finally obtained

[0159] Step B4: Merge the slot representation vector obtained in step B3 Input into the slot-aware dynamic fusion module to capture the dynamic correlation between slots in the conversation and obtain the slot-aware representation vector

[0160] The step B4 specifically comprises the following steps:

[0161] Step B41: Slot-specific characterization vector obtained from step B32 The input is sent to the slot-aware dynamic fusion module, which automatically corrects the correlation between domains and slots and provides mutual guidance between slots. Represents the matrix representation of all fusion slot information vectors, let F1 = ε t , for the lth self-attention layer, it is expressed as:

[0162]

[0163] For the lth feedforward sublayer, it is expressed as:

[0164]

[0165] in, is the learnable parameter matrix, is the bias term, is the output of the last layer.

[0166] Step B42: In order to compare with the slot candidate value encoded by BERT, the value obtained in step B41 is Input into a linear layer LayerNorm layer to obtain a semantic vector representation close to the true value of slot SJ in round t: slot perception table

[0167] Eigenvector It is expressed as:

[0168]

[0169] in, represents the j-th slot perception representation vector, Represents the set of all slot-aware representation vectors.

[0170] Step B5: The slot perception representation vector obtained in step B4 The corresponding slot candidate slot value representation vector obtained in step B1 Perform similarity matching and select the most similar slot value as the prediction result; compare the predicted value with the true value to calculate the slot prediction loss, use the back propagation algorithm to calculate the gradient of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters.

[0171] The step B5 specifically comprises the following steps:

[0172] Step B51: The slot candidate value representation vector obtained in step B12 Slot self-attention representation obtained with B42 Calculate the semantic distance as a measure of similarity; then select the slot candidate value representation vector with the smallest semantic distance to the slot self-attention representation, that is, the most similar slot candidate value as the final prediction result of slot Sj; use the L2 norm as

[0173] is the distance metric; in the training phase, the slot S is calculated at round t j The true value of The probability is:

[0174]

[0175] The value with the highest probability is taken as the predicted value; where exp(·) represents the exponential function, ||·||2 represents the L2 norm, S is the time slot of round t j The true value of The vector representation of .

[0176] Step B52: Train the model to maximize the joint probability of all slots, i.e. The slot value prediction loss function for each round t is defined as the accumulation of negative log-likelihood:

[0177]

[0178] The calculated loss The learning rate is updated through the gradient optimization algorithm AdamW, and the model parameters are iteratively updated using back propagation to minimize the loss function to train the model.

[0179] Step B6: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model G is terminated.

[0180] This embodiment also provides a conversation state tracking system based on context deconstruction, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0182] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0183] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0184] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0185] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for tracking dialog states based on context deconstruction, characterized in that: The following steps are involved: Step A: Collect the conversation data and conversation status data between users and the system in different scenarios and build a data set TS; Step B: Construct a deep learning network model G based on context deconstruction; the deep learning network model G inputs the dialogue into the dialogue context deconstruction encoding module for encoding to decouple the mutual influence of instant information and historical information; retrieves the current dialogue information and historical information related to the slot through the slot discourse alignment module and the slot memory alignment module; dynamically controls the influence of slot discourse attention and slot memory attention on specific slots through the meta-learning enhanced gated fusion module, and obtains the slot-specific representation vector and the optimal gated unit parameters after multi-view information fusion; captures the dynamic correlation between slots in the dialogue through the slot-aware dynamic fusion module; Use the dataset TS to train the deep learning network model G; Step C: Input the conversation data between the user and the system into the trained deep learning network model G in sequence, and output the current conversation status.

2. The method for tracking dialog states based on context deconstruction according to claim 1, characterized in that: The step A specifically comprises the following steps: Step A1: Collect conversation data between users and the system in different scenarios; the conversation data in one scenario includes multiple rounds of user utterances and system responses, where user utterances represent the input text of the user in one round of conversation, and system responses represent the system's response to the above user input; each round of conversation in each scenario contains a set of conversation state data, which includes: Slot: represents a predefined category of information in a conversation, used to identify a specific component of user intent; Slot value: the actual information filled into a specific slot, representing the specific data extracted from the user's input corresponding to the specific slot; Dialog state: represented as a set of key-value pairs, where the key represents the slot name and the value is the slot value; Dialogue history: contains the sequence of all user inputs and system responses before the current turn; Step A2: The conversation data and conversation state data between the user and the system in a scenario form a conversation sample, and multiple conversation samples together form a data set TS; a conversation sample in the data set TS includes: T current dialogues, the current dialogue of the tth round is represented by D t =(R t , U t ), where R t Indicates the system response, U t Represents user utterances; The dialogue history of T rounds of dialogue, the dialogue history of the tth round is expressed as Where R t Indicates the system response, U t represents the user's utterance, D t represents the current dialogue in the tth round; (3) A dialogue sample contains the dialogue state of T rounds of dialogue, and the dialogue state of the tth round is represented as a set of key-value pairs Where S i is the slot name, is the slot candidate value corresponding to the slot in round t, and I is the number of all slots in different fields; Step A3: Divide the dataset TS into training set D by random sampling, with 80% of the data n , 20% of the data is divided into the validation set D v The training set is used for model training and parameter optimization, and the validation set is used to evaluate the performance of the model and adjust hyperparameters.

3. The method for tracking dialog states based on context deconstruction according to claim 1, characterized in that: The step B specifically comprises the following steps: Step B1: Input the dialogue state data contained in the t-th round of dialogue in the dataset TS into the dialogue context deconstruction encoding module for encoding to decouple the mutual influence of instant information and historical information; the dialogue context deconstruction encoding module consists of four encoders: current discourse encoder, past memory encoder, slot encoder and slot candidate value encoder; the current discourse encoder encodes the system response and user discourse combination sequence of the current dialogue to obtain the current discourse representation vector The past memory encoder encodes the combined sequence of the dialogue history and the previous round of dialogue states to obtain a memory representation vector The slot encoder and slot candidate value encoder encode all predefined slots and their corresponding slot candidate values ​​to obtain the slot representation vector h s and slot candidate value representation vector Step B2: Represent the current speech vector and slot characterization vector h s Input to the slot utterance alignment module, retrieve the current dialogue information related to the slot, and get the slot utterance attention Representing the memory vector and slot characterization vector h s Input to the slot memory alignment module to track the historical conversation information related to the slot and obtain the slot memory attention Step B3: Pay attention to the slot utterance obtained in step B2 Attention with slot memory The input is to the meta-learning enhanced gated fusion module; the meta-learning enhanced gated fusion module adopts a two-layer optimization strategy to determine the optimal gated unit parameters, dynamically control the influence of slot discourse attention and slot memory attention on specific slots, so as to enhance the adaptability to dialogue tasks in different fields; the meta-learning enhanced gated fusion module first uses the meta-learning loss function to perform internal optimization on the training set, and then performs external optimization on the gated unit parameters through the verification set, and finally obtains the slot-specific representation vector after multi-view information fusion and the optimal gating unit parameters Step B4: The slot-specific representation vector obtained in step B3 Input into the slot-aware dynamic fusion module to capture the dynamic correlation between slots in the conversation and obtain the slot-aware representation vector Step B5: The slot perception representation vector obtained in step B4 The corresponding slot candidate slot value representation vector obtained in step B1 Perform similarity matching and select the most similar slot value as the prediction result; compare the predicted value with the true value to calculate the slot prediction loss, use the back propagation algorithm to calculate the gradient of each parameter in the deep network, and use the stochastic gradient descent algorithm to update the parameters; Step B6: When the loss value generated by the deep learning network model is less than the set threshold or reaches the maximum number of iterations, the training of the deep learning network model G is terminated.

4. The method for tracking dialog states based on context deconstruction according to claim 3, characterized in that: The step B1 specifically comprises the following steps: Step B11: Concatenate the system response and user utterance of the current dialogue into utterance-level representation, insert markers at the beginning and end of the sequence, and define the input representation as: Among them, [CLS] is a special token added before each round of input, and [SEP] is a special token used to mark the end of the dialogue round; The dialogue history and the dialogue state of the previous round are concatenated into a memory-level dialogue context sequence. The entire memory-level dialogue context sequence of the tth round is defined as: Will and Input into the current speech encoder and the past memory encoder respectively to obtain the current speech representation vector and memory representation vector It is expressed as: Among them, BERT finetune For the BERT model using the finetune strategy for feature extraction, L is the length of the current conversation, L h is the length of the dialogue history plus the dialogue state of the previous round, and d is the dimension of the token representation vector; Step B12: The sequence of slot names and slot values ​​in round t is represented as s and v t , replace s and v t Input into the slot encoder and slot candidate value encoder respectively to obtain the slot representation vector h s and slot candidate value representation vector It is expressed as: h s =BERT fixed (s) Among them, BERT fixed For the BERT model that uses the fixed strategy for feature extraction, L s is the length of the slot name, L v is the length of the slot value candidate, d is the dimension of the token representation vector, and the parameter weights of BERTfixed are fixed during training to reduce the difficulty of slot value matching; Since the number of slot candidate values ​​for each slot is different, Characterize the vector for each candidate value in this slot Collection It is expressed as: Wherein, i=1,2,...q{8≤q≤156}; Represents the i-th slot candidate value for the j-th slot.

5. The method for tracking dialog states based on context deconstruction according to claim 4, characterized in that: The step B2 specifically comprises the following steps: Step B21: Represent the current speech vector and slot characterization vector h s Input to the slot discourse alignment module to obtain the slot discourse attention that integrates the current dialogue information It is expressed as: in, is the learnable weight parameter, and the slot representation vector h s As the query matrix input, the current discourse representation vector As the key and value matrices, SelfAttention(Q, K, V) and MultiHeadAttention(Q, K, V) are the self-attention and multi-head attention mechanisms respectively. The Concat() function concatenates the outputs of h self-attention heads. Step B22: To retain the slot name information, the slot utterances output by the multiple heads in step B21 are focused on and the slot characterization vector h s After concatenation, it is input into the feedforward neural network for linear transformation to obtain a temporary representation vector The temporary representation vector After applying the ReLU activation function, a linear transformation is performed to map it to the new space to obtain the final slot speech attention. It is expressed as: in, is the learnable weight parameter; is bias; Step B23: The memory representation vector output from step B12 Slot characterization vector h s Input to the slot memory alignment module to obtain the slot memory attention that fused the memory information It is expressed as: Attention slot memory Further transformation through the feedforward neural network to obtain the final slot memory attention It is expressed as: in, is the learnable weight parameter; For bias.

6. The method for tracking dialog states based on context deconstruction according to claim 5, characterized in that: The specific implementation method of step B3 is: The slot utterance attention obtained in step B2 and slot memory attention Input to the meta-learning enhanced gated fusion module to obtain the slot-specific representation vector It is expressed as: f(w1)=Sigmoid(w1) For different slots, use meta-learning to set f(w1) to different values ​​to train a more robust model; Among them, w1, w2 are learnable network parameters, f′(w2)=1-f(w1) means that the sum of the weights of slot speech attention and slot memory attention is 1, ⊙ means the element-by-element product operation of two vectors, Represents a vector product operation; The goal of meta-learning is to use Find the optimal parameters w1, w2 of the gate unit and determine the allocation slot discourse attention and slot memory attention First, the meta-learning enhanced gated fusion module uses the meta-learning loss function to perform internal optimization on the training set, so that the model can minimize the loss value after training the optimal parameters based on parameters w1 and w2; the meta-learning loss function derives the training objective of the model as: Among them, D n is the training set, θ represents the parameters of the main model, X t represents the conversation context, s represents the slot associated with the conversation context, represents the vector product operation, and The loss values ​​corresponding to slot discourse attention and slot memory attention are minimized by Optimize, expressed as: Secondly, the meta-learning enhanced gated fusion module uses the validation set D v As metadata for external optimization, the optimal parameter θ*(w1, w2) is expected to be in the validation set D v To achieve the best performance, the method of optimizing w1 and w2 is expressed as: in, Represents the validation set dialogue X t The corresponding loss of slot s is given by the parameter θ * The initial model prediction of (w1, w2) is calculated, and the optimal parameters of the gated unit after meta-learning are finally obtained 7. The method for tracking dialog states based on context deconstruction according to claim 6, characterized in that: The step B4 specifically comprises the following steps: Step B41: Slot-specific characterization vector obtained from step B32 The input is sent to the slot-aware dynamic fusion module, which automatically corrects the correlation between domains and slots and provides mutual guidance between slots. Represents the matrix representation of all fused slot information vectors, let F1 = ε, for the l-th self-attention layer, it is expressed as: For the lth feedforward sublayer, it is expressed as: in, is the learnable parameter matrix, is the bias term, is the output of the last layer; Step B42: In order to compare with the slot candidate value encoded by BERT, the value obtained in step B41 is Input to a linear layer LayerNorm layer to get a value close to slot S J Semantic vector representation of the truth value at round t: slot-aware representation vector It is expressed as: in, represents the j-th slot perception representation vector, Represents the set of all slot-aware representation vectors.

8. The method for tracking dialog states based on context deconstruction according to claim 7, characterized in that: The step B5 specifically comprises the following steps: Step B51: The slot candidate value representation vector obtained in step B12 Slot self-attention representation obtained with B42 Calculate the semantic distance as a measure of similarity; then select the slot candidate value representation vector with the smallest semantic distance to the slot self-attention representation, that is, the most similar slot candidate value as slot S j The final prediction result; use L2 norm as the distance metric; in the training phase, calculate the slot S in round t j The true value of The probability is: The value with the highest probability is taken as the predicted value; where exp(·) represents the exponential function, ||·||2 represents the L2 norm, S is the time slot of round t j The true value of The vector representation of ; Step B52: Train the model to maximize the joint probability of all slots, i.e. The slot value prediction loss function for each round t is defined as the accumulation of negative log-likelihood: The calculated loss The learning rate is updated through the gradient optimization algorithm AdamW, and the model parameters are iteratively updated using back propagation to minimize the loss function to train the model.

9. A conversation state tracking system based on context deconstruction, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

Images