Joint method of semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM

Abstract

This paper discloses a method for combining semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM. The method comprises the following steps: S1, constructing a model framework consisting of a shared network and two classification networks; S2, training the model described in S1, updating the parameters of the entire network model except for the word embedding layer, and obtaining a semantic slot filling and intent detection model; S3, based on the semantic slot filling and intent detection model described in S2, implementing semantic slot filling and intent detection in sentences of the dialogue text to be detected. The method automatically finds a balance between the two tasks without significantly increasing the computational load, achieving better overall performance.

Description

Technical Field

[0001] The present invention relates to the field of spoken language understanding in dialogue systems, and in particular to a joint method of semantic slot filling and intent detection based on BERT-BLSTM-RPEA, which can be applied to fields such as personal voice assistants and intelligent aviation consulting. Background Art

[0002] Intent detection and semantic slot filling are crucial components of spoken language understanding (SLU) in dialogue systems. Therefore, their performance significantly impacts the overall performance of the dialogue system. The rapid development of technologies such as artificial intelligence (AI), deep learning (DL), and big data has provided technical support for intent detection and slot filling. To improve the accuracy of slot filling and intent detection, the most widely used approach is the pipeline approach. This involves first training a SLU model to label the semantic slots, then using the semantic slot vectors predicted by the trained SLU model as input to train a model for detecting user intent. However, this serial approach cannot simultaneously capture the errors of both tasks in the intent and semantic slot spaces. Therefore, error propagation can only be used to adjust parameters based on separate tasks. To address this serial error propagation problem, existing solutions employ a joint model of semantic slot filling and intent detection.

[0003] Joint models are an important approach for combining intent detection and semantic slot filling tasks. Their goal is to train the network by propagating errors from both intent and semantic slots simultaneously. Existing joint models still face many challenges, such as suboptimal trade-off parameters, which often result in joint models performing well on only one of the tasks. Furthermore, insufficient fusion of the corresponding hidden-layer semantic slots and intent information output by the two tasks in the model can also reduce the accuracy of both tasks. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a joint semantic slot filling and intent detection method based on BERT-BLSTM-RPEA-LSTM.

[0005] A joint semantic slot filling and intent detection method based on BERT-BLSTM-RPEA-LSTM comprises the following steps: S1, constructing a model framework consisting of a shared network and two classification networks, the model comprising a word embedding layer, an encoder layer, an RPEA mechanism, and a decoding dual-task classification layer, wherein the shared network consists of "BERT-BLSTM-RPEA-LSTM", and the two classification networks correspond to feedforward neural networks for semantic slot filling and intent detection, respectively; S2, training the model described in S1, calculating the loss by the error between two prediction vectors obtained from the output of the model described in S1 and the true value, and updating the parameters of the entire network model except the word embedding layer, to obtain a semantic slot filling and intent detection model; S3, based on the semantic slot filling and intent detection model described in S2, according to the dialogue text sentences to be detected obtained from the dialogue system, realizing the filling of semantic slots and the detection of intent in the dialogue text sentences to be detected.

[0006] Optionally, the word embedding layer adds the intent classifier to the beginning of a given conversation text sentence data and inputs it into the word embedding model to obtain a contextual semantic embedding feature vector of the word. Before the text sentence is input into the word embedding model, the word '[CLS]' is added to the beginning of the sentence as an input word for intent classification, thereby unifying the two tasks of semantic slot filling and intent detection into a sequence labeling task. The sentence is then segmented to obtain the word index X' after the sentence segmentation, and finally embedded to obtain a word embedding feature vector sequence E'.

[0007] X′=Token(X)

[0008] E′=BERT(X′)

[0009] E = select(E′)

[0010] where X = {x1, x2, ..., x N} represents the input sentence word sequence with the word '[CLS]' added, N represents the maximum number of words in the input sentence, and for sentences with fewer than N words, '[PAD]' is added at the end of the sentence to fill it up, E = {e1, e2, ..., e N}; Represents the final word semantic feature sequence, where the semantic feature vector corresponding to the input word '[PAD]' is replaced by a zero vector through the select operation, so that the filler word is not used as a word for information transmission.

[0011] Furthermore, the encoder layer inputs the word embedding vector into the encoder layer to obtain the hidden layer feature vector of the word. The encoder layer selects a BLSTM network containing forward and backward LSTMs and sets the input dimension of the network to D e , the output dimension is D h=256; By inputting the obtained word semantic feature sequence E into the forward and backward LSTM networks respectively, a forward hidden layer feature vector information before t is obtained. And a method to obtain the backward hidden layer feature vector information after t Finally and Concatenate as the final hidden layer feature vector h t , which is the output of the encoder layer;

[0012]

[0013]

[0014]

[0015] in

[0016] The RPEA mechanism is to calculate a corresponding global plus local attention supplementary information vector for each word in each sentence through the RPEA mechanism, and calculate a supplementary information vector for each hidden layer feature vector obtained. When the tth hidden layer vector h t Calculate the supplementary information vector c t When , first calculate a global threshold parameter vector and a local threshold parameter vector Then c t Can be obtained by a t With r t After finding the Hadamard product, multiply it with the hidden layer feature sequence to get it; by first t And each hidden layer feature vector splicing, and then with the parameter matrix W α Perform matrix multiplication and feed it into the hyperbolic tangent threshold function, and then combine it with the parameter matrix V α Multiply to get a global attention score, and finally normalize the global score to get the global threshold parameter vector a t ; Local threshold parameter vector r t Each hidden layer feature vector is relative to h t Position and h t The calculated local attention score is obtained by the sigmoid threshold function;

[0017]

[0018]

[0019] global_s(h t , h i)=[tanh([h t ;h i ]×W α )]×V α

[0020]

[0021]

[0022] in T represents the transpose of the matrix, r t，i ∈(0,1).

[0023] Optionally, the decoding dual-task classification layer includes a decoder and a dual-task classification network, uses a forward LSTM network as a decoder, and uses FFNN as the network model for the two tasks of semantic slot filling and intent detection. The input of the decoder is a concatenated sequence of the hidden layer feature sequence and the obtained supplementary information sequence. The decoder output is the final feature vector of each word. The final feature vector corresponding to the first word of each sentence is then input into the intent detection network to obtain the intent space distribution, and the final feature vectors of the remaining words are input into the semantic slot filling network to obtain the semantic slot space distribution. Finally, the output of the dual-task classification network is normalized to obtain a prediction vector.

[0024] The step S2 specifically includes: S201, the loss of the entire network consists of semantic slot loss and intent loss, each intent or semantic slot is regarded as a different category, and the label smoothing recognition loss is used to supervise the training of the encoder, RPEA mechanism and various parameters in the decoding dual-task classification layer. Their losses are defined as:

[0025]

[0026]

[0027] L=δL I +(1-δ)L S

[0028] where q i The true label of intention i, p i is the predicted intent label; Represents the predicted label of the j-th word corresponding to the output semantic slot s, is the true semantic slot label; δ∈[0, 1] is used to adjust the degree of attention to these two losses;

[0029] S202: Feed the currently obtained loss back to the network model, and adjust the parameters in the model through the Adam optimizer for continuous iterative optimization to reduce the loss and improve the accuracy of target prediction.

[0030] The beneficial effects of the present invention are as follows: In response to the problems of unsatisfactory trade-off parameter values ​​in the joint method of intent detection and semantic slot filling and insufficient fusion of hidden layer semantic slots and intent information leading to decreased accuracy, the present invention proposes a joint method of semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM. Its main advantages are: 1) The two tasks are unified into a sequence labeling task, which facilitates the learning of ideal trade-off parameter values. 2) RPEA is able to learn more complex attention windows, selectively focusing on a subset of the source sentence, thereby providing the hidden layer feature vector with supplementary information that fully integrates the semantics of words in the sentence. 3) After using the present invention, the balance between the two tasks can be automatically found without increasing the amount of calculation, thereby obtaining better overall performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the overall structure of the BERT-BLSTM-RPEA-LSTM-based semantic slot filling and intent detection joint method provided by the present invention;

[0032] Figure 2 Schematic diagram of the functional structure of the word embedding module provided by the present invention;

[0033] Figure 3 Schematic diagram of the RPEA module structure provided by the present invention;

[0034] Figure 4 Schematic diagram of the RPEA calculation process structure provided by the present invention;

[0035] Figure 5 Schematic diagram of the decoding dual-task classification layer structure provided by the present invention. DETAILED DESCRIPTION

[0036] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, the following detailed description of specific embodiments of the present invention is provided in conjunction with the accompanying drawings, so that the above-mentioned and other objects, features, and advantages of the present invention will become more apparent. Like reference numerals denote like parts throughout the drawings. The drawings are not drawn to scale; emphasis is placed on illustrating the subject matter of the present invention.

[0037] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the invention. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the invention is provided for illustration purposes only and not for the purpose of limiting the invention as defined by the appended claims and their equivalents.

[0038] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to a "module" includes reference to one or more of such modules. The advantages and features of the present invention and methods of accomplishing the same may be more readily understood by reference to the detailed description of the embodiments below and the accompanying drawings. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the invention to those skilled in the art.

[0039] In general, this paper proposes a joint method for intent detection and semantic slot filling based on BERT, Bi-directional Long-Short Term Memory (BLSTM) and Relative Position Encoding Attention (RPEA). The main features of this method are: 1) adding the '[CLS]' marker word at the beginning of the conversation text sentence sequence to form the input sequence (x1, x2...x n ), where x1 will be used as the input feature for intent recognition, and the features other than x1 will be used as input features for semantic slot filling; 2) The pre-trained BERT semantic model performs word embedding. During the word embedding process, some coarse-grained or synthetic words are divided into fine-grained basic words and the words are embedded. Then, the fine-grained word embedding vectors of the word are summed to obtain the final word embedding vector of the word; 3) The RPEA mechanism is added between the BLSTM encoder and decoder; 4) RPEA is obtained by the joint action of relative position encoding information and the hidden information output by the encoding layer. The relative position encoding information is encoded by the position of the i-th word relative to the t-th word using a parameter matrix.

[0040] The joint semantic slot filling and intent detection method based on BERT-BLSTM-RPEA includes the following steps:

[0041] S1. Build a model framework consisting of a shared network and two classification networks, such as Figure 1 As shown in the figure, the shared network consists of BERT-BLSTM-RPEA-LSTM, and the two classification networks correspond to the feedforward neural networks for semantic slot filling and intent detection, respectively. The model is divided into four parts: word embedding layer, encoder layer, RPEA mechanism, and decoding dual-task classification layer.

[0042] S2. Train the model, calculate the loss through the error between the two prediction vectors and the true value obtained by the model output, and update the parameters of the entire network model except the word embedding layer to obtain the semantic slot filling and intent detection model.

[0043] S3. Based on the semantic slot filling and intent detection model, the semantic slot filling and intent detection in the dialogue text sentences to be detected are realized according to the dialogue text sentences to be detected obtained from the dialogue system.

[0044] The above model consists of four parts: word embedding layer, encoder layer, RPEA mechanism and decoding dual-task classification layer;

[0045] 1) The word embedding layer adds the intent classifier to the beginning of a given conversational text sentence and then inputs it into the word embedding model to obtain the word's contextual semantic embedding feature vector. Before the text sentence is input into the word embedding model, the word '[CLS]' is added to the beginning of the sentence as the input word for intent classification, thus unifying the semantic slot filling and intent detection tasks into a single sequence labeling task. The sentence is then segmented to obtain the word index X′ after segmentation, and finally embedded to obtain the word embedding feature vector sequence E′.

[0046] The word embedding model selection includes a case-insensitive dictionary, an embedding layer, 12 encoder layers, and a final hidden layer dimension of D e =768 and 12 attention heads. Due to the fine-grained nature of the word atoms in the dictionary, when using the pre-trained BERT model to extract semantic features of sentence words, the complex input word x i During the word segmentation process, it will be split into multiple basic words and embedded to obtain the rich semantic expression of the word. Finally, the multiple semantic feature vectors of the word are summed to obtain the final semantic feature vector e i The process is as follows Figure 2 As shown;

[0047] X′=Token(X)#(1)

[0048] E′=BERT(X′)#(2)

[0049] E=select(E′)#(3)

[0050] where X = {x1, x2, ..., x N} represents the input sentence word sequence with the word '[CLS]' added, N represents the maximum number of words in the input sentence, and for sentences with fewer words than N, '[PAD]' will be added at the end of the sentence to fill it. E = {e1, e2, ..., e N}; Represents the final word semantic feature sequence, where the semantic feature vector corresponding to the input word '[PAD]' is replaced by a zero vector through the select operation, so that the filler word is not used as a word for information transmission.

[0051] 2) The encoder layer inputs the word embedding vector into the encoder layer to obtain the hidden layer feature vector of the word. The encoder layer selects a BLSTM network containing forward and backward LSTMs and sets the input dimension of the network to D e , the output dimension is D h = 256. By inputting the word semantic feature sequence E obtained from claim 3 into the forward and backward LSTM networks respectively, a network that can obtain the forward hidden layer feature vector information before t is obtained. And a method to obtain the backward hidden layer feature vector information after t Finally and Concatenate as the final hidden layer feature vector h t , which is the output of the encoder layer;

[0052]

[0053]

[0054]

[0055] in

[0056] 3) The RPEA mechanism calculates a corresponding global + local attention supplementary information vector for each word in each sentence through the RPEA mechanism. The function of RPEA is as follows Figure 3 As shown, it calculates a supplementary information vector for each hidden layer feature vector obtained from claim 4. When the tth hidden layer vector h t Calculate the supplementary information vector c t When , we need to calculate a global threshold parameter vector and a local threshold parameter vector Then c t Can be obtained by a t With r t After calculating the Hadamard product, multiply it with the hidden layer feature sequence to obtain the product. The calculation process is as follows Figure 4 As shown;

[0057] By first h t And each hidden layer feature vector splicing, and then with the parameter matrix W α Perform matrix multiplication and feed it into the hyperbolic tangent threshold function, and then combine it with the parameter matrix V αMultiply to get a global attention score, and finally normalize the global score to get the global threshold parameter vector a t ; Local threshold parameter vector r t Each hidden layer feature vector relative to h t Position and h t The calculated local attention score is obtained by the sigmoid threshold function;

[0058]

[0059]

[0060] global_s(h t , h i )=[tanh([h t ;h i ]×W α )]×V α #(9)

[0061]

[0062]

[0063] in T represents the transpose of the matrix, r t，i ∈(0,1).

[0064] 4) Decoding dual-task classification layer such as Figure 5 As shown, it includes a decoder and a dual-task classification network. The present invention uses a forward LSTM network as a decoder and uses FFNN as a network model for the two tasks of semantic slot filling and intent detection. The input of the decoder is a concatenated sequence of the hidden layer feature sequence and the supplementary information sequence obtained in claim 4. The decoder output is the final feature vector of each word. The final feature vector corresponding to the first word of each sentence is then input into the intent detection network to obtain the intent space distribution. The final feature vectors of the remaining words are input into the semantic slot filling network to obtain the semantic slot space distribution. Finally, the output of the dual-task classification network is normalized to obtain the required prediction vector.

[0065] The model training steps are as follows:

[0066] The above model was trained and tested on the PyTorch framework using domain-specific dialogue system text training and test datasets. The overall network loss consists of a semantic slot loss and an intent loss. Following previous work, each intent or semantic slot is treated as a distinct category. A label smoothing recognition loss is used to supervise the training of the encoder, RPEA mechanism, and various parameters in the decoder dual-task classification layer. These losses are defined as:

[0067]

[0068]

[0069] L=δL I +(1-δ)L S #(14)

[0070] Where qi represents the true label of intent i, and pi is the predicted intent label; Represents the predicted label of the j-th word corresponding to the output semantic slot s, is the true semantic slot label; δ∈[0, 1] is used to adjust the degree of attention to these two losses.

[0071] Ultimately, the current loss is fed back into the network model, and the Adam optimizer is used to adjust the model parameters for continuous iterative optimization to reduce the loss and improve the accuracy of target predictions. The trained model is tested on the test dataset to determine the current model's detection accuracy. If the preset accuracy is not achieved, the network hyperparameters or optimizer are reset until a model that meets the preset accuracy requirements is obtained and saved as the final semantic slot filling and intent detection model.

[0072] Perform semantic slot filling and intent detection on conversational text sentences. First, simulate a specific domain environment and collect conversational text data from that domain. Then, load the semantic slot filling and intent detection model. Input the collected conversational text data to be tested into the semantic slot filling and intent detection model to obtain a semantic slot for each word in the conversational text sentence and an intent for the entire sentence.

[0073] Although the technology has been illustrated and described with respect to one or more embodiments, changes and / or modifications may be made to the illustrated examples without departing from the spirit and scope of the appended claims. In particular, with respect to the various functions performed by the components or structures (assemblies, devices, circuits, systems, etc.) described above, terms used to describe such components (including references to "means") are intended to correspond to any component or structure that performs the designated function of the described component (e.g., functionally equivalent), even if not structurally equivalent to the disclosed structure that performs the function in the example embodiments described herein, unless otherwise indicated. In addition, although a particular feature may have been disclosed with respect to one embodiment among several embodiments, such feature may be combined with one or more other features in other embodiments as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "comprising," "including," "having," "having," "containing," or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0074] In the above description, many specific details are set forth in order to fully understand the present invention. However, the above description is only a preferred embodiment of the present invention. The present invention can be implemented in many other ways different from those described herein, so the present invention is not limited to the specific implementation disclosed above. At the same time, any person skilled in the art can make many possible changes and modifications to the technical solution of the present invention using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A joint method of semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM, characterized by: The steps include: S1. Build a model framework consisting of a shared network and two classification networks. The model includes a word embedding layer, an encoder layer, a RPEA mechanism, and a decoding dual-task classification layer. The shared network consists of "BERT-BLSTM-RPEA-LSTM", and the two classification networks correspond to the feedforward neural networks for semantic slot filling and intent detection, respectively. S2. Train the model described in S1, calculate the loss based on the error between the two predicted vectors and the true value obtained by the output of the model described in S1, and update the parameters of the entire network model except the word embedding layer to obtain a semantic slot filling and intent detection model; S3. Based on the semantic slot filling and intent detection model described in S2, the semantic slot filling and intent detection in the dialogue text sentences to be detected are implemented according to the dialogue text sentences to be detected obtained from the dialogue system; The encoder layer selects a BLSTM network containing forward and backward LSTMs and sets the input dimension of the network to D e , the output dimension is D h =256; The RPEA mechanism calculates a corresponding global plus local attention supplementary information vector for each word in each sentence obtained through the RPEA mechanism; Calculate a supplementary information vector for each hidden layer feature vector obtained. When the tth hidden layer vector h t Calculate the supplementary information vector c t When , first calculate a global threshold parameter vector and a local threshold parameter vector Then c t By a t With r t After finding the Hadamard product, multiply it with the hidden layer feature sequence to get it; by first t And each hidden layer feature vector splicing, and then with the parameter matrix W α Perform matrix multiplication and feed it into the hyperbolic tangent threshold function, and then combine it with the parameter matrix V α Multiply to get a global attention score, and finally normalize the global score to get the global threshold parameter vector a t ; Local threshold parameter vector r t Each hidden layer feature vector is relative to h t Position and h t The calculated local attention score is obtained by the sigmoid threshold function; in T represents the transpose of the matrix, r t，i ∈(0, 1); N represents the maximum number of words in the input sentence.

2. A method for combining semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM according to claim 1, characterized in that: The word embedding layer adds the intent classification word at the beginning of the given conversation text sentence data and inputs it into the word embedding model to obtain the contextual semantic embedding feature vector of the word.

3. The method of combining semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM according to claim 2, characterized in that: Before the text sentence is input into the word embedding model, the word '[CLS]' is added to the beginning of the sentence as the input word for intent classification, thereby unifying the two tasks of semantic slot filling and intent detection into a sequence labeling task. The sentence is then segmented to obtain the word index X' after the sentence segmentation, and finally embedded to obtain the word embedding feature vector sequence E'. X′=Token(X) E′=BERT(X′) E = select(E′) where ={x1,x2,…,x N } represents the input sentence word sequence with the word '[CLS]' added, N represents the maximum number of words in the input sentence, and for sentences with fewer than N words, '[PAD]' is added at the end of the sentence to fill it up, E = {e1, e2, ..., e N }; Represents the semantic feature sequence of the final word, where the semantic feature vector corresponding to the input word '[PAD]' is replaced by a zero vector through the select operation, so that the filler word is not used as a word for information transmission.

4. A method for combining semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM according to claim 3, characterized in that: The encoder layer inputs the word embedding vector into the encoder layer to obtain the hidden layer feature vector of the word.

5. The method of combining semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM according to claim 1, characterized in that: By inputting the obtained word semantic feature sequence E into the forward and backward LSTM networks respectively, a network that can obtain the forward hidden layer feature vector information before t is obtained. And a method to obtain the backward hidden layer feature vector information after t Finally and Concatenate as the final hidden layer feature vector h t , which is the output of the encoder layer; in 6. A method for combining semantic slot filling and intent detection based on BERT-BLSTM-RPEA-LSTM according to claim 1, characterized in that: The decoding dual-task classification layer includes a decoder and a dual-task classification network, uses a forward LSTM network as a decoder, and uses FFNN as the network model for the two tasks of semantic slot filling and intent detection. The input of the decoder is a concatenated sequence of the hidden layer feature sequence and the obtained supplementary information sequence. The decoder output is the final feature vector of each word. The final feature vector corresponding to the first word of each sentence is then input into the intent detection network to obtain the intent space distribution, and the final feature vectors of the remaining words are input into the semantic slot filling network to obtain the semantic slot space distribution. Finally, the output of the dual-task classification network is normalized to obtain a prediction vector.

Images