A natural language understanding method for bidirectional interaction between intent slots and values

Abstract

The present invention discloses a natural language understanding method for bidirectional interaction of intent and slot value, adopts BERT semantic encoding module to perform text preprocessing on the utterance input by the user, and performs semantic encoding, then inputs the intent representation vector into the intent attention module, generates a semantically enhanced intent representation vector, and obtains the final intent semantic representation vector and slot semantic representation vector through the self-attention semantic interaction module, the bidirectionally interactive semantically enhanced intent semantic representation vector and slot semantic representation vector, then inputs the final intent representation vector and slot semantic representation vector into the CRF semantic decoding module, and obtains the intent and slot value of the final utterance. The present invention uses pre-training knowledge to significantly improve the recognition effect of intent recognition and slot filling, so that the system can be applied to high-precision NLU scenarios, and improves the prediction accuracy of the system.

Description

Technical Field

[0001] The present application belongs to the technical field of task-based dialogue systems, and specifically relates to a natural language understanding method for two-way interaction of intent slot values. Background Art

[0002] As dialogue systems such as customer service agents, virtual voice assistants, and voice navigation are increasingly used in people's daily lives, improving the performance and efficiency of dialogue systems has become crucial. The core function of a dialogue system is to understand the intent of user speech, record key information, and give users appropriate responses. The framework of a dialogue system can be roughly divided into five parts: speech recognition (ASR), natural language understanding (NLU), dialogue management (MD), dialogue generation (NLG), and text-to-speech synthesis (TTS).

[0003] Due to the complexity of natural language, NLU tasks have always been a key link affecting the accuracy of dialogue systems, and intent detection and slot filling are two main tasks in NLU. Intent detection refers to judging the intent of user speech, and slot filling refers to extracting semantic concepts from user speech. For example, in the sentence "find "The Sound of the Wind" within an hour", the intent of this sentence is "search for movies", "one hour" is the slot of the time category to be extracted, and "The Sound of the Wind" is the slot of the movie category to be extracted. The performance of intent detection and slot filling plays a decisive role in the accuracy of the dialogue system. Therefore, in the dialogue system, how to improve the performance and efficiency of intent detection and slot filling becomes crucial.

[0004] Although early inventions often modeled the two problems separately, there is a strong correlation between the two tasks, and the two tasks often need to be performed simultaneously in industrial practice, so a large number of joint models have been invented to achieve better results. Compared with two isolated models, the joint model significantly reduces model parameters and training time, and can often use the correlation between the two tasks to improve each other's classification performance. Therefore, there are still inventions based on a single task, but the joint model has become the main solution in the dialogue system.

[0005] However, existing technical solutions often do not consider the use of pre-trained knowledge and cannot meet the needs of high-precision NLU scenarios. Summary of the invention

[0006] The purpose of this application is to provide a natural language understanding method with bidirectional interaction between intent and slot values, which takes into account the close correlation between intent recognition and slot filling tasks, and uses pre-trained knowledge to improve the recognition accuracy of the two tasks. It can be applied to actual industrial scenarios of task-based dialogue systems.

[0007] In order to achieve the above object, the solution provided by the present invention is as follows:

[0008] A natural language understanding method for bidirectional interaction of intent slot values, comprising:

[0009] The BERT semantic encoding module is used to perform text preprocessing on the user's input speech and perform semantic encoding to obtain a semantic representation vector H, which includes the intention semantic representation vector H I and slot semantic representation vector H S ;

[0010] The intention vector H I Input to the intention attention module to generate the semantically enhanced intention representation vector H IW ;

[0011] Through the self-attention semantic interaction module, the intention semantic representation vector H is enhanced by bidirectional interaction semantics. IW and slot semantic representation vector H S , obtain the final intent semantic representation vector and slot semantic representation vector;

[0012] The final intent representation vector and slot semantic representation vector are input into the CRF semantic decoding module to obtain the intent and slot value of the final utterance, wherein the CRF semantic decoding module includes two branches, one branch includes an intent fully connected neural network module, and the other branch includes a slot fully connected neural network module and a CRF slot decoding module;

[0013] Among them, the intention is to represent the vector H I Input to the intention attention module to generate the semantically enhanced intention representation vector H IW ,include:

[0014] Get the weight W of the intent fully connected neural network in the CRF semantic decoding module I , and the intention semantic representation vector H I Perform matrix multiplication and then pass through the softmax function to obtain the softmax output α:

[0015] α=softmax(H I W I );

[0016] Then α and weight W I Multiply it with the intention semantic representation vector H I Add them together to obtain the intention semantic representation vector H after semantic enhancement by the intention attention module IW :

[0017] H IW =H I +aW I ;

[0018] The intention semantic representation vector H is enhanced by the self-attention semantic interaction module and the bidirectional interaction semantics. IW and slot semantic representation vector H S , obtain the final intent semantic representation vector and slot semantic representation vector, including:

[0019] Intent semantic representation vector H enhanced by connection semantics IW and slot intention semantic representation vector H S , get the semantic representation vector H IS :

[0020] H IS =[H IW , H S ];

[0021] H IS Obtain the Q, K, and V matrices through h linear mappings.

[0022]

[0023]

[0024]

[0025] in and is the parameter of the linear mapping, j∈1...h;

[0026] Perform the scaled dot multiplication attention operation on the obtained Q, K, and V matrices in parallel to obtain the self-attention head:

[0027]

[0028] Where d represents the semantic representation vector H IS Dimensions;

[0029] The obtained h self-attention heads are connected and subjected to a linear mapping to obtain the semantic vector H which is semantically enhanced by the multi-head self-attention module. M :

[0030] H M =concat(h1,h2,…,h h )W O ;

[0031] Where W O is the parameter of this linear mapping;

[0032] The semantic vector H MAdd it to the semantic representation vector H output by the BERT semantic encoding module and perform layer normalization:

[0033] H′=LayerNorm(H+H M );

[0034] The semantic vector is the output of the self-attention semantic interaction module, represents the final intent semantic representation vector, Represents the final slot semantics representation vector.

[0035] Furthermore, the BERT semantic encoding module is used to perform text preprocessing on the user's input speech and perform semantic encoding to obtain the intention semantic representation vector H I and slot semantic representation vector H S ,include:

[0036] The user's input is tokenized, and a special identifier ([CLS]) is added at the beginning of the sentence and a special identifier ([SEP]) is added at the end of the sentence. Finally, a sentence sequence consisting of T+1 words is obtained:

[0037] X=(x0,x1,…,x T );

[0038] where x i , i∈0,1,…,T represents the i-th word in the sentence;

[0039] Then, the processed text data X is semantically encoded, the text data X is input, and the semantic representation vector is output:

[0040] H=[H I , H S ];

[0041] Among them, H I Intent semantic representation vector, H S Represents a slot semantics representation vector.

[0042] Furthermore, the final intent representation vector and slot semantic representation vector are input into the CRF semantic decoding module to obtain the intent and slot value of the final utterance, including:

[0043] The final intent semantic representation vector Input the intent fully connected neural network module, and after a fully connected neural network and softmax function, the final intent semantic representation vector is mapped to the intent label, and the intent label y is output. I :

[0044]

[0045] Where W I and b I The trainable weight parameters and bias parameters in the intention fully connected neural network;

[0046] Representing slot semantics as vector The input slot is fully connected to the neural network module. After a fully connected neural network and softmax function, the final slot semantic representation vector is mapped to the emission probability required for CRF decoding, and the emission probability p is output. S :

[0047]

[0048] Where W S and b S are the trainable weight parameters and bias parameters in the fully connected neural network;

[0049] The emission probability p S Input the CRF slot decoding module and decode it to get the slot label y S :

[0050] y S =CRF(p S ).

[0051] This application proposes a natural language understanding method for bidirectional interaction between intent and slot values. It uses pre-trained knowledge to significantly improve the recognition effect of intent recognition and slot filling, so that the system can be applied to high-precision NLU scenarios. The self-attention mechanism is used to bidirectionally interact with intent and slot information, further improving the prediction accuracy of the system. The intent attention mechanism allows the model to consider the label information of the intent, and the use of CRF allows the module to consider the dependency between slot labels, further improving the prediction accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the natural language understanding method for bidirectional interaction of intent slot values ​​in this application. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The following embodiments do not constitute a limitation of the present invention.

[0054] In the industrial use of task-based dialogue systems, intent recognition and slot filling tasks are often performed simultaneously, and there is an inseparable relationship between the two tasks. Most previous technologies only use a joint loss function to jointly model intent recognition and slot filling without explicitly considering the relationship between the two tasks. The introduction of pre-trained models such as BERT has significantly improved the accuracy of intent recognition and slot filling tasks, allowing the model to not only consider the information in existing data sets, but also utilize the massive pre-trained knowledge in pre-trained models such as BERT, while most existing technologies do not use pre-trained knowledge.

[0055] In one embodiment, Figure 1 As shown in the figure, a natural language understanding method with bidirectional interaction between intent slots and values ​​is proposed, including:

[0056] Step S1: Use the BERT semantic encoding module to perform text preprocessing on the user's input speech and perform semantic encoding to obtain a semantic representation vector H, which includes the intention semantic representation vector H I and slot semantic representation vector H S .

[0057] Specifically, the BERT semantic encoding module first performs text preprocessing, tokenizes the user input, adds a special identifier ([CLS]) at the beginning of the sentence, and adds a special identifier ([SEP]) at the end of the sentence, and finally obtains a sentence sequence consisting of T+1 words:

[0058] X=(x0,x1,…,x T );

[0059] where x i , i∈0,1,…,T represents the i-th word in the sentence.

[0060] Then, the processed text data X is semantically encoded, the text data X is input, and the semantic representation vector is output:

[0061] H=[H I , H S ];

[0062] Among them, H I Intent semantic representation vector, H S Represents a slot semantics representation vector.

[0063] Step S2: The intention representation vector H I Input to the intention attention module to generate the semantically enhanced intention representation vector H IW .

[0064] Specifically, first obtain the weight W of the intent fully connected neural network in the CRF semantic decoding module I , and the intention semantic representation vector H I Perform matrix multiplication and then pass through the softmax function to obtain the softmax output α:

[0065] α=softmax(H I W I );

[0066] Then α and weight W I Multiply it with the intention semantic representation vector H I Add them together to obtain the intention semantic representation vector H after semantic enhancement by the intention attention module IW :

[0067] H IW =H I +aW I ;

[0068] Therefore, the current intent label is integrated into the intent semantic representation vector to obtain a higher quality intent semantic representation vector.

[0069] Step S3: The intention semantic representation vector H is enhanced by the self-attention semantic interaction module through bidirectional interaction semantics IW and slot semantic representation vector H S , obtain the final intent semantic representation vector and slot semantic representation vector.

[0070] The self-attention semantic interaction module of this embodiment performs the following operations:

[0071] Connect the semantically enhanced intent semantic representation vector and the slot intent semantic representation vector to obtain the semantic representation vector H IS :

[0072] H IS =[H IW , H S ];

[0073] H IS Obtain the Q, K, and V matrices through h linear mappings.

[0074]

[0075]

[0076]

[0077] in and are the trainable parameters in the linear mapper, j∈1...h.

[0078] Perform the scaled dot multiplication attention operation on the obtained Q, K, and V matrices in parallel to obtain the self-attention head:

[0079]

[0080] Where d represents the semantic representation vector H IS Dimensions;

[0081] The obtained h self-attention heads are connected and subjected to a linear mapping to obtain the semantic vector H which is semantically enhanced by the multi-head self-attention module. M :

[0082] H M =concat(h1,h2,…,h h )W o ;

[0083] Where W O are trainable parameters in the linear mapper;

[0084] The semantic vector H M Add it to the semantic representation vector H output by the BERT semantic encoding module and perform layer normalization:

[0085] H′=LayerNorm(H+H M );

[0086] The semantic vector is the output of the self-attention semantic interaction module, represents the final intent semantic representation vector, Represents the final slot semantic representation vector, which enables full interaction between the information of intent and slot.

[0087] Step S4: input the final intent representation vector and slot semantic representation vector into the CRF semantic decoding module to obtain the intent and slot value of the final utterance. The CRF semantic decoding module includes two branches, one branch includes an intent fully connected neural network module, and the other branch includes a slot fully connected neural network module and a CRF slot decoding module.

[0088] The CRF semantic decoding module of this embodiment is used to map the intent representation vector and the slot semantic representation vector to the intent label and the slot label to obtain the intent and slot value of the last utterance. CRF takes into account the mutual dependency between labels, further improving the accuracy of the model.

[0089] The CRF semantic decoding module performs the following operations:

[0090] The final intent semantic representation vector Input the intent fully connected neural network module, and after a fully connected neural network and softmax function, the final intent semantic representation vector is mapped to the intent label, and the intent label y is output. I :

[0091]

[0092] Where W I and b I are the trainable weight parameters and bias parameters in the intended fully connected neural network.

[0093] Representing slot semantics as vector The input slot is fully connected to the neural network module. After a fully connected neural network and softmax function, the final slot semantic representation vector is mapped to the emission probability required for CRF decoding, and the emission probability p is output. S :

[0094]

[0095] Where W S and b S are the trainable weight parameters and bias parameters in the fully connected neural network.

[0096] The emission probability p S Input the CRF slot decoding module and decode it to get the slot label y S :

[0097] y S =CRF(p S ).

[0098] When encountering new user sentence input, the BERT semantic encoding module first performs semantic encoding on it and maps the user's sentence into a high-dimensional semantic vector. During this period, the semantic vector incorporates a lot of pre-trained knowledge in the BERT semantic encoding module. After the intention attention module passes the intention label information to the intention representation vector, the self-attention semantic interaction module performs a two-way interaction on the intention and slot information, further improving the quality of the representation vector. Finally, the fully connected neural network and CRF are used to decode the intention and slot, where CRF also considers the mutual dependence between slot labels, further improving the prediction accuracy of the model. Experiments have shown that the network model of this application greatly improves the accuracy of intent recognition and slot filling tasks, and can meet the needs of high-precision task-based dialogue systems.

[0099] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Without departing from the spirit and essence of the present invention, technicians familiar with the field can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the claims attached to the present invention.

Claims

1. A natural language understanding method for bidirectional interaction between intent slot values, characterized in that: The natural language understanding method for bidirectional interaction of intent slot values ​​includes: The BERT semantic encoding module is used to perform text preprocessing on the user's input speech and perform semantic encoding to obtain a semantic representation vector H, which includes the intention semantic representation vector H I and slot semantic representation vector H S ; The intention vector H I Input to the intention attention module to generate the semantically enhanced intention representation vector H IW ; Through the self-attention semantic interaction module, the intention semantic representation vector H is enhanced by bidirectional interaction semantics. IW and slot semantic representation vector H S , obtain the final intent semantic representation vector and slot semantic representation vector; The final intent representation vector and slot semantic representation vector are input into the CRF semantic decoding module to obtain the intent and slot value of the final utterance, wherein the CRF semantic decoding module includes two branches, one branch includes an intent fully connected neural network module, and the other branch includes a slot fully connected neural network module and a CRF slot decoding module; Among them, the intention is to represent the vector H I Input to the intention attention module to generate the semantically enhanced intention representation vector H IW ,include: Get the weight W of the intent fully connected neural network in the CRF semantic decoding module I , and the intention semantic representation vector H I Perform matrix multiplication and then pass through the softmax function to obtain the softmax output α: α=softmax(H I W I ); Then α and weight W I Multiply it with the intention semantic representation vector H I Add them together to obtain the intention semantic representation vector H after semantic enhancement by the intention attention module IW : H IW =H I +αW I ; The intention semantic representation vector H is enhanced by the self-attention semantic interaction module and the bidirectional interaction semantics. IW and slot semantic representation vector H S , obtain the final intent semantic representation vector and slot semantic representation vector, including: Intent semantic representation vector H enhanced by connection semantics IW and slot intention semantic representation vector H S , get the semantic representation vector H IS : H IS =[H IW ,H S ]; H IS Obtain the Q, K and V matrices through h linear mappings; Q j =H IS W j Q ; K j =H IS W j K ; V j =H IS W jV ; Where W j Q , W j K and W j V is the parameter of the linear mapping, j∈1…h; Perform the scaled dot multiplication attention operation on the obtained Q, K, and V matrices in parallel to obtain the self-attention head: Where d represents the semantic representation vector H IS Dimensions; The obtained h self-attention heads are connected and subjected to a linear mapping to obtain the semantic vector H after semantic enhancement by the multi-head self-attention module. M : H M =concat(h1,h2,…,h h )W O ; Where W O is the parameter of this linear mapping; The semantic vector H M Add it to the semantic representation vector H output by the BERT semantic encoding module and perform layer normalization: H′=LayerNorm(H+H M ); The semantic vector is the output of the self-attention semantic interaction module, represents the final intent semantic representation vector, Represents the final slot semantics representation vector.

2. The natural language understanding method for bidirectional interaction of intent slot values ​​according to claim 1 is characterized in that: The BERT semantic encoding module is used to perform text preprocessing on the user's input speech and perform semantic encoding to obtain the intention semantic representation vector H I and slot semantic representation vector H S ,include: Segment the user's input utterance, add a special identifier CLS at the beginning of the sentence, and add a special identifier SEP at the end of the sentence, and finally obtain a sentence sequence consisting of t+1 words: X=(x0,x1,…,x T ): where x i ,i∈0,1,…,T represents the i-th word in the sentence; Then, the processed text data X is semantically encoded, the text data X is input, and the semantic representation vector is output: H=[H I ,H S ]; Among them, H I Intent semantic representation vector, H S Represents a slot semantics representation vector.

3. The natural language understanding method for bidirectional interaction of intent slot values ​​according to claim 1 is characterized in that: The final intent representation vector and slot semantic representation vector are input into the CRF semantic decoding module to obtain the intent and slot value of the final utterance, including: The final intent semantic representation vector Input the intent fully connected neural network module, and after a fully connected neural network and softmax function, the final intent semantic representation vector is mapped to the intent label, and the intent label y is output. I : Where W I and b I The trainable weight parameters and bias parameters in the intended fully connected neural network; Representing slot semantics as vector The input slot is fully connected to the neural network module. After a fully connected neural network and softmax function, the final slot semantic representation vector is mapped to the emission probability required for CRF decoding, and the emission probability p is output. S : Where W S and b S are the trainable weight parameters and bias parameters in the fully connected neural network; The emission probability p S Input the CRF slot decoding module and decode it to get the slot label y S : the S =CRF(p S )。

Images