Natural language understanding method and system based on zero-shot learning cross-language hierarchical architecture
By combining cross-lingual sentence embedding and word embedding, and employing a BiLSTM encoder and mBERT model, the alignment problem of intent detection and slot filling in cross-lingual dialogue systems is solved, achieving efficient cross-lingual understanding.
Patent Information
- Application Number
- CN202211190548.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing technologies in cross-language dialogue systems overlook the fact that intent detection is a sentence classification task that requires higher-level alignment between different languages, and the deep relationship between intent detection and slot filling has not been fully discovered.
We adopt a cross-lingual hierarchical architecture based on zero-shot learning, combining cross-lingual sentence embedding and word embedding. Through BiLSTM encoder and multilingual model mBERT, we handle intent detection and slot filling respectively, learn the preset intent classification in high-dimensional space, and fill semantic slots under the guidance of low-level predicted intent.
It improves slot filling performance, enabling efficient intent detection and slot filling across different languages, especially with excellent performance under zero-shot learning conditions.
Smart Images
Figure CN115688794B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to cross-lingual human-machine dialogue systems, and particularly relates to a natural language understanding method and system based on zero-shot learning cross-lingual hierarchical architecture. BACKGROUND
[0002] Natural language understanding (NLU) plays an important role in task-oriented dialogue systems, whose purpose is to parse the dialogue discourse by recognizing the user's intent and arguments of the intent. These two tasks are called intent detection and slot filling, respectively. Recently, many neural models have been proposed to jointly train these two tasks, considering intent detection and slot filling as sentence classification and sequence labeling tasks. However, most of the work has focused on monolingual datasets, which are costly to build. In addition, some dialogue systems, such as Google Home and Apple Siri, need to serve a large number of users around the world in different languages, and may face the problem of dialogue data scarcity in some languages.
[0003] Therefore, the motivation of building a cross-lingual dialogue system is to achieve zero-shot adaptation from high-resource languages to low-resource languages without any target language training data. Specifically, cross-lingual pre-trained embeddings are used to connect different languages. A small translation vocabulary is adopted to align cross-lingual embeddings through bilingual projection, which alleviates the inherent differences between different languages. Pre-trained cross-lingual models such as XLM and mBERT are used to derive the context embeddings of words in different languages. However, most existing methods consider embedding alignment between source and target languages at the word level to achieve cross-lingual adaptability, but ignore the fact that intent detection is a sentence classification task, which requires higher-level alignment between different languages. In addition, these two tasks are closely related, and slots are highly dependent on intent, but their deep relationship has not been fully discovered by existing work.
[0004] Scattered mainstream methods pay attention to learning cross-lingual embeddings. Some methods learn cross-lingual word embeddings through joint learning or post-training mapping of monolingual embeddings. Recently, cross-lingual contextual methods are built on the basis of masked language models, which encourage to narrow the representation difference in source and target language spaces. In addition to word-level alignment, some works focus on learning cross-lingual sentence embeddings to align different language expressions of sentences. SUMMARY
[0005] The purpose of this invention is to address the shortcomings of existing cross-language dialogue generation methods, which mostly consider embedding alignment between source and target languages at the word level to achieve cross-language adaptability. However, these methods neglect the fact that intent detection is a sentence classification task that requires higher-level alignment between different languages. Furthermore, these two tasks are closely related, with slots highly dependent on intent, but their deep relationship has not been fully explored in existing work. This invention combines cross-language sentence embeddings and word embeddings to handle intent detection and slot filling separately. It learns to classify predefined intents in a high-dimensional space and fills semantic slots under the guidance of low-level predicted intents. This helps to find more relevant intent arguments to improve slot filling performance.
[0006] Specifically, this invention proposes a natural language understanding method based on zero-shot learning of a cross-language hierarchical architecture, which includes:
[0007] Step 1: Obtain a corpus consisting of L words of the same language, u = [w1, w2, ... w2]. L [and its corresponding set of intent types I and slot types S;]
[0008] Step 2: Input the corpus u into the BiLSTM encoder to obtain the multilingual sentence embedding h. u embed h according to sentence u The intention distribution y of the corpus u is obtained. I Select the intended distribution y I The intent type with the highest probability in the corpus u is used as the intent prediction result o. I According to the intention distribution y I And intent type I, construct intent loss;
[0009] Step 3: Input the corpus u into the multilingual model mBERT to generate contextual word embeddings [e1, e2, ..., e] in the multilingual model. L ], embed each word into e j With intention prediction I The intention to embed h I Average as a word representation Based on each word representation Obtain the slot distribution for each word Select slot distribution The slot type with the highest probability is used as the slot prediction result [s1,s2,...,s] for corpus u. L According to the distribution of slots And slot type S, construct slot loss;
[0010] Step 4: Using the sum of the intent loss and the slot loss as the total loss, train the BiLSTM encoder and the multilingual model mBERT until the preset number of iterations is reached or the total loss converges. Save the current BiLSTM encoder and the multilingual model mBERT as a natural language understanding model. Input the natural language to be understood, which is different from the language of the corpus u, into the natural language understanding model to obtain the intent prediction result and slot prediction result of the natural language, so as to parse the dialogue of the natural language.
[0011] The natural language understanding method based on zero-shot learning cross-language hierarchical architecture, wherein step 2 obtains the intent distribution y using the following formula. I and intention prediction results o I :
[0012]
[0013] o I =Argmax(y I )
[0014] in These are the parameters obtained during training;
[0015] Step 3 yields the slot distribution y using the following formula. I :
[0016]
[0017] In the formula and These are trainable parameters.
[0018] The natural language understanding method based on zero-shot learning of a cross-language hierarchical architecture, wherein the total loss... for:
[0019]
[0020] Intended loss and This represents the loss of the tank. Let represent the intent distribution of the i-th word predicted by the model, where ^ represents the true label and S represents slot filling.
[0021] The natural language understanding method based on zero-shot learning cross-language hierarchical architecture includes a BiLSTM encoder that generates multilingual sentence embeddings, in which sentences with the same meaning but different languages are embedded and aligned in the same semantic space; and in the fine-tuning of the multilingual model mBERT, data augmentation is used to align word embeddings.
[0022] This invention also proposes a natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, which includes:
[0023] The initial module is used to obtain a corpus u = [w1, w2, ... w] consisting of L words of the same language. L [and its corresponding set of intent types I and slot types S;]
[0024] The encoding module is used to input the corpus u into the BiLSTM encoder to obtain the multilingual sentence embedding h. u embed h according to sentence u Obtain the intent distribution y of corpus u I Select the intended distribution y I The intent type with the highest probability in the corpus u is used as the intent prediction result o. I According to the intention distribution y I And intent type I, construct intent loss;
[0025] The prediction module is used to input the corpus u into the multilingual model mBERT and generate contextual word embeddings [e1, e2, ..., e] in multiple languages. L ], embed each word into e j With intention prediction I The intention to embed h I Average as a word representation Based on each word representation Obtain the slot distribution for each word Select slot distribution The slot type with the highest probability is used as the slot prediction result [s1,s2,...,s] for corpus u. L According to the distribution of slots And slot type S, construct slot loss;
[0026] The parsing module is used to train the BiLSTM encoder and the multilingual model mBERT with the sum of the intent loss and the slot loss as the total loss until a preset number of iterations is reached or the total loss converges. The current BiLSTM encoder and the multilingual model mBERT are saved as a natural language understanding model. Natural language to be understood that is different from the language of the corpus u is input into the natural language understanding model to obtain the intent prediction result and slot prediction result of the natural language, so as to parse the dialogue of the natural language.
[0027] The natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, wherein the encoding module obtains the intent distribution y using the following formula. I and intention prediction results o I :
[0028]
[0029] O I =Argmax(y I )
[0030] in These are the parameters obtained during training;
[0031] Step 3 yields the slot distribution y using the following formula. I :
[0032]
[0033] In the formula and These are trainable parameters.
[0034] The natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, wherein the total loss... for:
[0035]
[0036] Intended loss and This represents the loss of the tank. Let represent the intent distribution of the i-th word predicted by the model, where ^ represents the true label and S represents slot filling.
[0037] The natural language understanding system based on zero-shot learning cross-language hierarchical architecture includes a BiLSTM encoder that generates multilingual sentence embeddings, in which sentences with the same meaning but different languages are embedded and aligned in the same semantic space; and in the fine-tuning of the multilingual model mBERT, data augmentation is used to align word embeddings.
[0038] The present invention also proposes a storage medium for storing a program that executes any of the natural language understanding methods based on zero-shot learning cross-language hierarchical architectures.
[0039] The present invention also proposes a client for any of the aforementioned natural language understanding systems based on zero-shot learning cross-language hierarchical architectures.
[0040] like Figure 2As shown above, the advantages of this invention are as follows: This invention proposes a model called HCLD (A Hierarchical Framework for Zero-shot Cross-lingual Dialogue System) to address the shortcomings of the existing technologies. This invention is built upon a hierarchical framework to jointly complete intent detection and slot filling. In the model of this invention, it learns to classify preset intents in high-dimensional space and fills semantic slots under the guidance of low-level predicted intents, which helps to find more relevant intent arguments to improve slot filling performance. In particular, this invention also uses a pre-trained language-independent BiLSTM encoder LASER to derive sentence embeddings, which are directly used for subsequent intent detection and classification tasks, where sentence embeddings of different languages are consistent at the sentence level. Experiments were conducted on a common task-oriented dialogue dataset, and the results show that the proposed method achieves optimal performance in zero-shot adaptation. Attached Figure Description
[0041] Figure 1 This is a system block diagram of the present invention;
[0042] Figure 2 This is a chart comparing the effects of the present invention with those of existing technologies. Detailed Implementation
[0043] The primary goal of Natural Language Understanding (NLU) is to parse dialogues to obtain features of dialogue intent and slot information. These features can then be used for subsequent tasks, such as intent detection and slot filling. This invention combines cross-lingual sentence embedding and word embedding to handle intent detection and slot filling respectively. Intent detection and slot filling are key tasks in NLU for task-oriented dialogue systems. This invention first predicts the intent represented by a given multilingual sentence and directly incorporates the sentence's intent information into slot filling. Furthermore, this invention can simultaneously handle intent detection and slot filling in a cross-lingual setting. To achieve the above technical effects, this application includes the following key technical points:
[0044] Key point 1: This invention classifies the preset intent in a high-dimensional space and fills the semantic slots under the guidance of the predicted intent in a low-dimensional space. Semantic filling (Slot F1) achieves the best results on the dataset.
[0045] Key point 2: Cross-language generation based on intent recognition and semantic fill; the dialogue generation performance achieved the best results on the dataset.
[0046] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.
[0047] The zero-shot learning employed in this invention refers to supervising the model in one language and then directly transferring it to other languages for inference. Intent detection and slot filling are two key tasks of NLU. Given a sentence consisting of L words, as the corpus u = [w1, w2, ... w...],... L The corpus u contains a set of predefined intent types I and slot types S. The purpose of intent detection is to predict intent based on the corpus u. I ∈I), while slot filling is a sequence labeling problem, which involves placing the word sequence [w1, w2, ..., w...] into a sequence of words. L Mapped to semantic slots [s1, s2, ..., s L ], where s∈S.
[0048] like Figure 1 As shown, the entire architecture is divided into two layers. The higher-layer network is responsible for the intent recognition task, while the lower-layer network is responsible for the semantic slot filling task, using the intent of each sentence as guidance to direct the slot filling task within that sentence.
[0049] The hierarchical model HCLD proposed in this invention first classifies predefined intents in a high-level architecture, and then completes semantic slots under the guidance of low-level intents. In the high-dimensional space, this invention first uses LASER to derive the sensory sentence representation h. u Based on a multilingual neural media translation task, LASER can generate multilingual sentence embeddings, where sentence embeddings from different languages with similar meanings can be aligned in a similar semantic space. Such sentence-level alignment can better facilitate intent detection, a task of sentence classification. So, intent... I Prediction method
[0050]
[0051] O I =Argmax(y I )
[0052] Where y I It is an intentional distribution. These are trainable parameters. At the low level, this invention utilizes a pre-trained multilingual model mBERT to generate contextual word embeddings [e1,e2,...,e] in different languages. LIn mBERT, word embeddings are aligned within the same semantic space. However, inconsistent word context representations exist across different languages. Therefore, this invention follows CoSDA-ML, using data augmentation to better align word representations during mBERT fine-tuning. Furthermore, this invention employs intent formation to guide the slot-filling task, obtaining the intent embedding vector h from the intent labels in the original corpus through encoder transformation. I This is used to assist in the slot filling task. Each word is embedded in e. j With the corresponding intent embedded h I The average is represented by new words And help fill semantic slots [s1,s2,...,s L Therefore, the slot distribution for each word. It can be predicted in advance.
[0053]
[0054] In the formula and These are trainable parameters.
[0055] To learn both tasks together, the objective function is defined as follows:
[0056]
[0057] The formula consists of three trems, the leftmost one being... The first line represents the overall loss, the middle line represents the sum of the two losses, and the rightmost line is the formula for calculating the loss, which is based on the cross-entropy loss function. The meaning of the model prediction is the intent, where I is short for intent, i is the i-th, and ^ represents the ground truth. S is short for slot filling.
[0058] This invention was tested on cross-language task-oriented dialogue datasets, including English, Spanish, and Thai. Training and validation were performed on 30,521 and 4,181 sentences respectively in the English dialogue data, and testing was conducted on 3,043 and 1,692 sentences in Spanish and Thai, respectively, for zero-fitting.
[0059] like Figure 2As shown, this invention compares with several competitive baselines, including Zero-shot SLU, BiLSTEM with CRF (LVM), XLM and LVM, mBERT, Attention-Informed Mixed-Language Training (MLT), and CoSDA-ML. This invention uses classification accuracy (Acc.) to evaluate intent detection performance, while using F1 scores to measure slot filling performance. In the experiments of this invention, word embeddings containing 110,000 words were used as specified. The invention uses Laser to generate sentence embeddings of size 1024, while using the base case mBERT to generate cross-lingual word embeddings of size 768. When using mBERT for slot filling, the first word embedding is used as a word-level representation. This invention also sets the intent embedding size to 768 and trains the model 10 times with a batch size of 32 and a learning rate of 0.001, using AdamW to optimize the HCLD of this invention, and selecting hyperparameters through grid search. Furthermore, in the early stages of training, golden intents are used instead of pre-predicted intents. I This guides the filling of the slots to prevent errors.
[0060] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0061] This invention also proposes a natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, which includes:
[0062] The initial module is used to obtain a corpus u = [w1, w2, ... w] consisting of L words of the same language. L [and its corresponding set of intent types I and slot types S;]
[0063] The encoding module is used to input the corpus u into the BiLSTM encoder to obtain the multilingual sentence embedding h. u embed h according to sentence u Obtain the intent distribution y of corpus u I Select the intended distribution y I The intent type with the highest probability in the corpus u is used as the intent prediction result o. I According to the intention distribution y I And intent type I, construct intent loss;
[0064] The prediction module is used to input the corpus u into the multilingual model mBERT and generate contextual word embeddings [e1, e2, ..., e] in multiple languages. L ], embed each word into e j With intention prediction I The intention to embed h I Average as a word representation Based on each word representation Obtain the slot distribution for each word Select slot distribution The slot type with the highest probability is used as the slot prediction result [s1,s2,...,s] for corpus u. L According to the distribution of slots And slot type S, construct slot loss;
[0065] The parsing module is used to train the BiLSTM encoder and the multilingual model mBERT with the sum of the intent loss and the slot loss as the total loss until a preset number of iterations is reached or the total loss converges. The current BiLSTM encoder and the multilingual model mBERT are saved as a natural language understanding model. Natural language to be understood that is different from the language of the corpus u is input into the natural language understanding model to obtain the intent prediction result and slot prediction result of the natural language, so as to parse the dialogue of the natural language.
[0066] The natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, wherein the encoding module obtains the intent distribution y using the following formula. I and intention prediction results o I :
[0067]
[0068] O I =Argmax(y I )
[0069] in These are the parameters obtained during training;
[0070] Step 3 yields the slot distribution y using the following formula. I :
[0071]
[0072] In the formula and These are trainable parameters.
[0073] The natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, wherein the total loss... for:
[0074]
[0075] Intended loss and This represents the loss of the tank. Let represent the intent distribution of the i-th word predicted by the model, where ^ represents the true label and S represents slot filling.
[0076] The natural language understanding system based on zero-shot learning cross-language hierarchical architecture includes a BiLSTM encoder that generates multilingual sentence embeddings, in which sentences with the same meaning but different languages are embedded and aligned in the same semantic space; and in the fine-tuning of the multilingual model mBERT, data augmentation is used to align word embeddings.
[0077] The present invention also proposes a storage medium for storing a program that executes any of the natural language understanding methods based on zero-shot learning cross-language hierarchical architectures.
[0078] The present invention also proposes a client for any of the aforementioned natural language understanding systems based on zero-shot learning cross-language hierarchical architectures.
Claims
1. A natural language understanding method based on zero-shot learning of a cross-linguistic hierarchical architecture, characterized in that, include: Step 1: Obtain a corpus of L words of the same language, u=[w1, w2, ... w... L [and its corresponding set of intent types I and slot types S;] Step 2: Input the corpus u into the BiLSTM encoder, which can map sentences from different languages to the same semantic space, thus obtaining multilingual sentence embeddings. According to sentence embedding Obtain the intent distribution of corpus u Select intention distribution The intent type with the highest probability in the corpus u is used as the intent prediction result. According to the distribution of intent And intent type I, construct intent loss; Step 3: Input the corpus u into the multilingual model mBERT to generate contextual word embeddings [e1, e2, ej..., e...] in the multilingual model. L ], Embed each word Compared with intention prediction results Intent Embedding Average as a word representation According to the word representation of each word The slot distribution of each word is obtained. Selecting slot distribution The slot type with the highest probability in the corpus is used as the slot prediction result [s1, s2, ..., s] for corpus u. L According to the distribution of slots And slot type S, construct slot loss; Step 4: Using the sum of the intent loss and the slot loss as the total loss, train the BiLSTM encoder and the multilingual model mBERT until the preset number of iterations is reached or the total loss converges. Save the current BiLSTM encoder and the multilingual model mBERT as a natural language understanding model. Input the natural language to be understood, which is different from the language of the corpus u, into the natural language understanding model to obtain the intent prediction result and slot prediction result of the natural language, so as to parse the dialogue of the natural language.
2. The natural language understanding method based on zero-shot learning cross-language hierarchical architecture as described in claim 1, characterized in that, Step 2 yields the intention distribution using the following formula. and intention prediction results : in These are the parameters obtained during training; Step 3 yields the slot distribution using the following formula: In the formula and These are trainable parameters.
3. The natural language understanding method based on zero-shot learning cross-language hierarchical architecture as described in claim 2, characterized in that, The total loss for: Represents intentional loss, This represents the loss of the tank. Let represent the intent distribution of the i-th word predicted by the model, where ^ represents the true label and S represents the slot type.
4. The natural language understanding method based on zero-shot learning cross-language hierarchical architecture as described in claim 1, characterized in that, The BiLSTM encoder generates multilingual sentence embeddings, in which sentences with the same meaning but different languages are embedded and aligned in the same semantic space; and in the fine-tuning of the multilingual model mBERT, word embeddings of aligned words are augmented with data.
5. A natural language understanding system based on a zero-shot learning cross-language hierarchical architecture, characterized in that, include: The initial module is used to obtain a corpus of L words of the same language, u=[w1, w2, ...w...]. L [and its corresponding set of intent types I and slot types S;] The encoding module is used to input the corpus u into the BiLSTM encoder, which maps sentences from different languages to the same semantic space, resulting in multilingual sentence embeddings. According to sentence embedding Obtain the intent distribution of corpus u Select intention distribution The intent type with the highest probability in the corpus u is used as the intent prediction result. According to the distribution of intent And intent type I, construct intent loss; The prediction module is used to input the corpus u into the multilingual model mBERT and generate contextual word embeddings [e1, e2, ej, ..., ej] in multiple languages. L ], Embed each word Compared with intention prediction results Intent Embedding Average as a word representation According to the word representation of each word The slot distribution of each word is obtained. Selecting slot distribution The slot type with the highest probability in the corpus is used as the slot prediction result [s1, s2, ..., s] for corpus u. L According to the distribution of slots And slot type S, construct slot loss; The parsing module is used to train the BiLSTM encoder and the multilingual model mBERT with the sum of the intent loss and the slot loss as the total loss until a preset number of iterations is reached or the total loss converges. The current BiLSTM encoder and the multilingual model mBERT are saved as a natural language understanding model. Natural language to be understood that is different from the language of the corpus u is input into the natural language understanding model to obtain the intent prediction result and slot prediction result of the natural language, so as to parse the dialogue of the natural language.
6. The natural language understanding system based on a zero-shot learning cross-language hierarchical architecture as described in claim 5, characterized in that, The encoding module obtains the intent distribution using the following formula. and intention prediction results : in These are the parameters obtained during training; Step 3 yields the slot distribution using the following formula: In the formula and These are trainable parameters.
7. The natural language understanding system based on zero-shot learning cross-language hierarchical architecture as described in claim 6, characterized in that, The total loss for: Represents intentional loss, This represents the loss of the tank. Let represent the intent distribution of the i-th word predicted by the model, where ^ represents the true label and S represents the slot type.
8. The natural language understanding system based on zero-shot learning cross-language hierarchical architecture as described in claim 5, characterized in that, The BiLSTM encoder generates multilingual sentence embeddings, in which sentences with the same meaning but different languages are embedded and aligned in the same semantic space; and in the fine-tuning of the multilingual model mBERT, word embeddings of aligned words are augmented with data.
9. A storage medium for storing a program that executes the natural language understanding method based on a zero-shot learning cross-language hierarchical architecture as described in any one of claims 1 to 4.
10. A client for implementing the natural language understanding system based on a zero-shot learning cross-language hierarchical architecture as described in any one of claims 5 to 8.