A multi-intention natural language understanding method, system, device and storage medium

Abstract

The present invention relates to the field of artificial intelligence, and in particular to a multi-intent natural language understanding method, system, device and storage medium. By crawling data and defining multi-intent labels and slot labels based on the crawled data, a multi-intent natural language understanding dataset is constructed based on the defined multi-intent labels and slot labels, and contextual semantic relationship embedding, multi-intent label-aware embedding and slot label-aware embedding, position-aware interactive attention network and a lightweight multi-intent natural language understanding joint model are sequentially constructed based on the multi-intent natural language understanding dataset. The position-aware interactive attention network is used to realize bidirectional interaction between the two subtasks of multi-intent recognition and slot filling, making full use of the semantic relationship between the two subtasks. The two subtasks can achieve bidirectional guidance and bidirectional improvement, thereby improving the accuracy of multi-intent natural language understanding and solving the problems of low accuracy and large number of parameters in multi-intent natural language understanding.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and specifically to a multi-intent natural language understanding method, system, device and storage medium, and in particular to a multi-intent natural language understanding and lightweight method, system device and storage medium based on location-aware interactive attention. Background Art

[0002] With the rapid development of computer technology, human-computer dialogue systems have become a research hotspot in natural language processing and are widely used in fields such as intelligent healthcare, personal assistants, and intelligent customer service. Natural language understanding is a core module of dialogue systems. Its goal is to recognize unstructured natural language and convert it into structured semantic representations that computers can understand, thus supporting subsequent modules of task-based human-computer dialogue. Natural language understanding typically consists of two subtasks: intent recognition and slot filling. Intent recognition is often considered a sentence-level classification task, focusing on identifying the overall intent of a sentence; slot filling involves extracting key information from a sentence, known as semantic slots. The categories of intents and semantic slots are predefined. However, in real applications, a single user input sentence often contains multiple intents, making multi-intent natural language understanding a current research hotspot.

[0003] In the early days, rule-based approaches were often used to solve natural language understanding problems. These methods required manual definition of rules and templates, processing user input text through methods such as keyword extraction or grammatical structure analysis, and screening candidate results through methods such as similarity matching to extract possible intent and slot information from user sentences. However, due to the diversity of natural languages, rule-based approaches cannot effectively handle complex and diverse user input, nor can they effectively associate contextual information. Furthermore, the definition of rules and templates is complex, making this approach time-consuming and labor-intensive, and difficult to maintain and reuse.

[0004] Currently, the most widely used natural language understanding methods are mostly based on pre-trained models, treating the two subtasks of multi-intent recognition and slot filling as independent tasks and handling them separately. However, this approach fails to consider the close connection between the two subtasks, fails to fully explore the semantic relationship between the two subtasks, and fails to fully utilize the labeled data information, resulting in model accuracy that is insufficient to meet user needs.

[0005] In multi-intent natural language understanding, each clause fragment in the input sentence contains rich semantic information, so multi-intent natural language understanding is position-dependent, see Figure 1For example, the multi-intent information "Navigation" in a sentence depends on the clause in the first half, while "Buy_ticket" depends on the clause in the second half. However, current attention-based models perform parallel computations on each word in the input sentence, failing to consider and model the positional information between words in the input sentence. This inability to effectively extract local information leads to poor model accuracy.

[0006] While multi-intent natural language understanding models based on pre-trained language models offer significant performance improvements over traditional neural network models, they are still very large, time-consuming to train, and slow to infer, making them difficult to deploy and apply effectively. Therefore, reducing the size of pre-trained language models while maintaining performance—a lightweight approach—has been a hot topic in the deep learning field in recent years. Summary of the Invention

[0007] In response to the problems of low accuracy and large number of parameters in multi-intent natural language understanding in the prior art, the present invention provides a multi-intent natural language understanding method, system, device and storage medium.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] The present invention provides a multi-intent natural language understanding method, comprising the following steps:

[0010] S1: Crawl data and define multi-intent labels and slot labels based on the crawled data. Based on the defined multi-intent labels and slot labels, build a multi-intent natural language understanding dataset.

[0011] S2: Build contextual semantic relationship embeddings using a multi-intent natural language understanding dataset;

[0012] S3: Using contextual semantic relationship embedding, we construct multi-intent label-aware embedding and slot label-aware embedding.

[0013] S4: Build a location-aware interactive attention network using multi-intent label-aware embedding and slot label-aware embedding;

[0014] S5: Use the location-aware interactive attention network to construct predicted multi-intent labels and predicted slot label sequences;

[0015] S6: Build a lightweight multi-intent natural language understanding joint model using the predicted multi-intent labels and the predicted slot label sequences;

[0016] S7: Use a lightweight multi-intent natural language understanding joint model to perform multi-intent natural language understanding.

[0017] Furthermore, the specific method of S1 is:

[0018] Design a crawler program to crawl data information from the corresponding website;

[0019] Clean and organize the crawled data information;

[0020] Filter the cleaned and organized data information to filter out the text of the target data set content;

[0021] Define multi-intent labels and slot labels based on the text of the target dataset content, and annotate the text of the target dataset content to generate the required multi-intent natural language understanding dataset.

[0022] Furthermore, the specific method of S2 is:

[0023] Based on the multi-intent natural language understanding dataset, the pre-trained model BERT is used as the word embedding layer of the text to obtain the word embedding of the text;

[0024] The BiLSTM model and attention mechanism are used to extract features from the word embedding of the text to obtain contextual semantic relationship embedding.

[0025] Furthermore, the specific method of S3 is:

[0026] The contextual semantic relationship embedding in S2 is used as the query vector, and the attention mechanism is used to calculate the multi-intent label attention embedding and the slot label attention embedding;

[0027] Add the multi-intent label attention embedding to the contextual semantic relationship embedding. On the basis of preserving the contextual semantic relationship, establish the connection between the contextual semantic relationship embedding and the multi-intent label to obtain the multi-intent label perception embedding.

[0028] The slot label attention embedding is added to the contextual semantic relationship embedding. On the basis of retaining the contextual semantic relationship, the connection between the contextual semantic relationship embedding and the slot label is established to obtain the slot label aware embedding.

[0029] Furthermore, the specific method of S4 is:

[0030] Perform linear transformations on the multi-intent label-aware embedding and the slot label-aware embedding to obtain the query vector Query of the multi-intent label-aware embedding I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S ;

[0031] Embed the query vector Query with multi-intent label awareness I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S Split into num head Attention heads are used to obtain the query vector of the multi-intent label-aware embedding of the multi-head attention mechanism. Key Vector Sum vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism Key Vector Sum vector

[0032] Multi-intent label-aware query embedding for multi-head attention mechanism using rotational position encoding and key vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism and key vector Perform position encoding to obtain a query vector with position-aware multi-intent label-aware embedding and key vector and query vectors with position-aware slot label-aware embeddings and key vector Among them, i represents the number of the attention head, i∈[1,2,…,num head ];

[0033] Embed query vectors with location-aware multi-intent label awareness As the query vector, the key vector with position-aware slot label-aware embedding As the key vector, the slot label perception embedding value vector of the multi-head attention mechanism As the value vector, the attention mechanism is used to calculate the preliminary slot-aware multi-intent embedding;

[0034] The preliminary slot-aware multi-intent embedding is added to the multi-intent label-aware embedding, and the slot-aware multi-intent embedding is obtained through linear layers and layer normalization.

[0035] The query vector is embedded with position-aware slot label awareness As the query vector, the key vector with location-aware multi-intent tag-aware embedding As the key vector, the value vector of the multi-intent label perception embedding of the multi-head attention mechanism As the value vector, the attention mechanism is used to calculate the preliminary multi-intent-aware slot embedding;

[0036] The preliminary multi-intent-aware slot embedding is added to the slot label-aware embedding, and the multi-intent-aware slot embedding is obtained through a linear layer and layer normalization to complete the construction of the location-aware interactive attention network.

[0037] Furthermore, the specific method of S5 is:

[0038] Reduce the dimensionality of the slot-aware multi-intent embedding obtained from the location-aware interactive attention network;

[0039] Use a multi-intent classifier to classify the reduced-dimensional slot-aware multi-intent embedding; make the multi-intent classifier output a vector with the number of multi-intent labels, where each dimension corresponds to an intent label;

[0040] Record the dimensions in the multi-intent classifier output vector whose values ​​are greater than 0.5, obtain the multi-intent labels corresponding to the dimension vectors whose values ​​are greater than 0.5, that is, the intent of the sentence, and construct the predicted multi-intent labels;

[0041] Use fully connected layers to process multi-intent-aware slot embedding;

[0042] The MCRF mechanism is used to establish the dependency relationship between the processed multi-intent-aware slot embedding and the slot label, constraining the illegal path sequence generated during the decoding process. The MCRF decoder is then used for decoding to construct the predicted slot label sequence.

[0043] Furthermore, the specific method of S6 is:

[0044] For the multi-intent recognition task, a binary cross-entropy loss function is used to calculate the loss between the predicted multi-intent labels and the actual labeled multi-intent labels. For the slot filling task, a cross-entropy loss function is used to calculate the loss between the predicted slot label sequence and the actual labeled slot label sequence. The weighted sum of the two losses is used as the total loss of the joint model.

[0045] Select the optimizer and choose Adam optimizer for network optimization;

[0046] Train, validate, test, and save the joint model. Use the training set in the constructed multi-intent dataset to train the network. Validate on the validation set and adjust parameters based on the validation results to achieve better network performance. Select the joint model with the best performance in the test set and save it to complete the construction of the multi-intent natural language understanding joint model.

[0047] Use SVD technology to decompose the dot product calculation of the query vector Query and the key vector Key of the attention head in the pre-trained model BERT;

[0048] Get a lightweight attention operation method;

[0049] A lightweight attention operation method is used to replace the original attention calculation method in each attention head of each layer in the pre-trained BERT model to build a lightweight multi-intent natural language understanding joint model.

[0050] A multi-intent natural language understanding system, comprising:

[0051] Dataset construction module: crawls data, defines multi-intent labels and slot labels based on the crawled data, and builds a multi-intent natural language understanding dataset based on the defined multi-intent labels and slot labels;

[0052] Contextual semantic relationship embedding module: used to build contextual semantic relationship embedding using multi-intent natural language understanding dataset;

[0053] Tag-aware embedding module: used to build multi-intent tag-aware embedding and slot tag-aware embedding by using contextual semantic relationship embedding;

[0054] Location-aware interactive attention network building module: used to build a location-aware interactive attention network using multi-intent label-aware embedding and slot label-aware embedding;

[0055] Multi-intent label and slot label sequence acquisition module: used to construct predicted multi-intent labels and predicted slot label sequences using the location-aware interactive attention network;

[0056] Lightweight multi-intent natural language understanding joint model construction module: This module is used to build a lightweight multi-intent natural language understanding joint model using predicted multi-intent labels and predicted slot label sequences.

[0057] Multi-intent natural language understanding module: used to perform multi-intent natural language understanding using a lightweight multi-intent natural language understanding joint model.

[0058] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0059] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention provides a multi-intent natural language understanding method. The method crawls data and defines multi-intent labels and slot labels based on the crawled data. According to the defined multi-intent labels and slot labels, a multi-intent natural language understanding dataset is constructed. Contextual semantic relationship embedding, multi-intent label-aware embedding and slot label-aware embedding, position-aware interactive attention network, predicted multi-intent labels and predicted slot label sequences are sequentially constructed according to the multi-intent natural language understanding dataset, and a lightweight multi-intent natural language understanding joint model is constructed. The position-aware interactive attention network is used to realize bidirectional interaction between the two subtasks of multi-intent recognition and slot filling of sentences, and the semantic relationship between the two subtasks is fully utilized. Bidirectional guidance and bidirectional improvement can be achieved between the two subtasks, thereby improving the accuracy of multi-intent natural language understanding. Rotational position encoding is used to incorporate position information into the query vector and key vector in the attention mechanism. The position dependence problem in the multi-intent natural language understanding task is used to model the relative position relationship between words in the text, thereby fully mining semantic information and local information and improving the performance of the model. Finally, the pre-trained language model used in the obtained multi-intent natural language understanding joint model is lightweighted using tensor decomposition technology, and a lightweight multi-intent natural language understanding joint model is constructed. This solves the problem of large number of parameters in the multi-intent natural language understanding model based on the pre-trained model, improves the model's inference speed, reduces the model's computing cost and hardware requirements, and facilitates the model's deployment and use.

[0062] The present invention provides a multi-intent natural language understanding system. Through the setting of a dataset construction module, a contextual semantic relationship embedding module, a label-aware embedding module, a position-aware interactive attention network construction module, a multi-intent label and slot label sequence acquisition module, a lightweight multi-intent natural language understanding joint model construction module and a multi-intent natural language understanding module, data crawling and construction of a multi-intent natural language understanding dataset are realized. Contextual semantic relationship embedding, multi-intent label-aware embedding and slot label-aware embedding, a position-aware interactive attention network, predicted multi-intent labels and predicted slot label sequences and a lightweight multi-intent natural language understanding joint model are sequentially constructed according to the multi-intent natural language understanding dataset, and finally the understanding of multi-intent natural language is realized. The system model makes full use of the association between the two subtasks in multi-intent natural language understanding and realizes two-way guidance of the two subtasks; it can extract local information more accurately on the basis of extracting global information of the text, and the understanding accuracy of multi-intent natural language is high; and lightweight processing of the multi-intent natural language understanding joint model is realized, which speeds up the inference speed of the model, reduces the computing cost and hardware requirements of the model, and facilitates the deployment and use of the model.

[0063] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0064] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This figure illustrates multi-intent natural language understanding in the prior art.

[0066] Figure 2 This is a flow chart of a multi-intent natural language understanding method of the present invention.

[0067] Figure 3 Construct a location-aware interactive attention network graph in the present invention.

[0068] Figure 4 A simplified diagram of the joint model for multi-intent natural language understanding based on location-aware interactive attention.

[0069] Figure 5 This is a structural diagram of a multi-intention natural language understanding system of the present invention. DETAILED DESCRIPTION

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0071] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0072] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0073] In the description of the embodiments of the present invention, it should be noted that if the terms "upper," "lower," "horizontal," "inner," etc. appear, the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the inventive product is typically placed when in use. These terms are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0074] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0075] In the description of the embodiments of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0076] The present invention will be further described in detail below with reference to specific embodiments, which are intended to explain the present invention rather than to limit it.

[0077] See also Figures 2 to 4 The present invention discloses a multi-intent natural language understanding method, comprising the following steps:

[0078] S1: Crawl data and define multi-intent labels and slot labels based on the crawled data. Based on the defined multi-intent labels and slot labels, build a multi-intent natural language understanding dataset. Specifically:

[0079] Design a crawler program to crawl data information from the corresponding website;

[0080] Clean and organize the crawled data information;

[0081] Filter the cleaned and organized data information to filter out the text of the target data set content;

[0082] Define multi-intent labels and slot labels based on the text of the target dataset content, and annotate the defined target dataset content to generate the required multi-intent natural language understanding dataset; when annotating the text, a method combining automatic machine annotation and manual annotation can be used.

[0083] For example: Write a crawler script to obtain data. Write a crawler script to obtain data from the user comment section and user communication section of the corresponding travel website, clean and organize the data, and manually screen user questions that meet the requirements. Based on the data after manual screening, define multiple intent types and entity types for the text. For example, based on the relevant characteristics of the tourism field, the intent types that can be defined include ticket purchase, navigation, location, etc.; for entities, there are types such as place, person, and time. Use natural language processing data automatic annotation tools such as doccano to automatically annotate the collected text. Complete the annotation task through manual verification, correction, and supplementation of the annotation results.

[0084] S2: Using the multi-intent natural language understanding dataset, we construct contextual semantic relationship embeddings, specifically:

[0085] Based on the multi-intent natural language understanding dataset, the pre-trained model BERT is used as the word embedding layer of the text to obtain the word embedding of the text, specifically:

[0086] The input text is passed through the pre-trained model BERT to obtain the vector representation of the text x = [x CLS ,x1,x2,…,x seq_len ],x∈R (seq_len+1)×dim . Where [x CLS ] represents the embedding representation of the entire sentence, [x1,x2,…,x seq_len ] represents the embedding representation of the words in the sentence, seq_len represents the number of words in the text, and dim represents the embedding dimension of the word.

[0087] The BiLSTM model and attention mechanism are used to extract features from the word embedding of the text to obtain the contextual semantic relationship embedding, specifically:

[0088] The output of the text embedding representation is except [x CLS ] is fed into the BiLSTM model to obtain a new text embedding Use attention mechanism to embed new text Perform feature extraction; construct an embedded representation of contextual semantic relations X = [X1, X2, ..., X seq_len ],as follows:

[0089] X=Attention(x * ,x *,x * ) (1)

[0090]

[0091] in, seq_len represents the number of words in the text, and dim represents the embedding dimension of the word.

[0092] S3: Using contextual semantic relationship embedding, we construct multi-intent label-aware embedding and slot label-aware embedding. Specifically:

[0093] The contextual semantic relationship embedding in S2 is used as the query vector, and the attention mechanism is used to calculate the multi-intent label attention embedding and slot label attention embedding as follows:

[0094] The contextual semantic relationship obtained in step S2 is embedded in X as the query vector of the attention operation, Set a parameter matrix W I As the key vector and value vector of multi-intent label attention, Multi-intent label attention embedding for text The calculation method is:

[0095]

[0096]

[0097] Among them, seq_len represents the length of the sentence, dim represents the embedding dimension; num I Indicates the number of multi-intent labels,

[0098] The multi-intent label attention embedding is added to the contextual semantic relationship embedding. On the basis of preserving the contextual semantic relationship, the connection between the contextual semantic relationship embedding and the multi-intent label is established to obtain the multi-intent label perception embedding. The method is:

[0099]

[0100] Among them, H I for multi-intent label-aware embedding, Attentional embedding for multi-intent labels.

[0101] The slot label attention embedding is added to the contextual semantic relationship embedding. On the basis of preserving the contextual semantic relationship, the connection between the contextual semantic relationship embedding and the slot label is established to obtain the slot label aware embedding. The method is as follows:

[0102] Embed the contextual semantic relationship X as the query vector for attention operation, Set a parameter matrix W S As the key vector and value vector of slot label attention, Among them, num S Indicates the number of multi-intent labels, slot label attention embedding of text Calculation method:

[0103]

[0104]

[0105] Among them, seq_len represents the length of the sentence, dim represents the embedding dimension, and num S Indicates the number of slot labels.

[0106] The contextual semantic relationship embedding representation is added to the slot label attention embedding to construct the slot label aware embedding Here’s how:

[0107]

[0108] Among them, H S for slot label-aware embedding, Attention embedding for slot labels.

[0109] S4: Utilize multi-intent label-aware embedding and slot label-aware embedding to build a location-aware interactive attention network. Specifically:

[0110] Perform linear transformations on the multi-intent label-aware embedding and the slot label-aware embedding to obtain the query vector Query of the multi-intent label-aware embedding I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S , the method is as follows:

[0111] The multi-intent label-aware embedding is subjected to three linear transformations as follows:

[0112]

[0113]

[0114]

[0115] in, and Both represent parameter matrices, and Both represent paranoid matrices, Query I is the query vector aware of multiple intent labels, Key I is the key vector of multi-intent label awareness, Value I is the value vector of multi-intent label perception, seq_len represents the length of the text, and dim represents the embedding dimension.

[0116] The slot label-aware embedding is subjected to three linear transformations as follows:

[0117]

[0118]

[0119]

[0120] in, and Both represent parameter matrices, and Both represent paranoid matrices, Query S is the slot label-aware query vector, Key S is the slot label-aware key vector, Value S is the slot label-aware value vector, seq_len represents the length of the text, and dim represents the embedding dimension.

[0121] Embed the query vector Query with multi-intent label awareness I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S Split into num head Attention heads are used to obtain the query vector of the multi-intent label-aware embedding of the multi-head attention mechanism. Key Vector Sum value vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism Key Vector Sum vector Here’s how:

[0122] Embed the query vector Query with multi-intent label awareness I , key vector Key I Sum value vector Value I Split into multiple attention heads, as follows:

[0123]

[0124]

[0125]

[0126] Among them, num head Represents the number of attention heads, and the function f(x,n) represents splitting the tensor x into n tensors in the last dimension. The query vector is embedded by the multi-intent label-aware multi-head attention mechanism, Key vector for multi-intent label-aware embedding of multi-head attention mechanism The value vector for the multi-intent label-aware embedding of the multi-head attention mechanism seq_len represents the length of the text, and dim represents the embedding dimension.

[0127] Slot label-aware embedded query vector Query S , key vector Key S Sum value vector Value S Split into num head The attention head is as follows:

[0128]

[0129]

[0130]

[0131] Among them, num head Represents the number of attention heads, and the function f(x,n) represents splitting the tensor x into n tensors in the last dimension. is the query vector of the slot label-aware embedding of the multi-head attention mechanism, The key vector for the slot label-aware embedding of the multi-head attention mechanism The value vector of the slot label-aware embedding of the multi-head attention mechanism

[0132] Rotational position encoding uses the rotation angle between vectors to represent the relative relationship between features, thereby establishing the dependency relationship between word positions. In the subsequent attention operation, the relative position information is implicitly inferred through the rotated query vector and key vector. and key vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism and key vector Perform position encoding to obtain a query vector with position-aware multi-intent label-aware embedding and key vector and query vectors with position-aware slot label-aware embeddings and key vector Among them, i represents the number of the attention head, i∈[1,2,…,num head ].

[0133] Multi-intent label-aware query embedding with multi-head attention mechanism and key vector Perform position encoding to obtain a query vector with position-aware multi-intent label-aware embedding and key vector Here’s how:

[0134]

[0135]

[0136]

[0137]

[0138]

[0139] in, is the mth query vector with location-aware multi-intent label-aware embedding, is the mth key vector of the location-aware multi-intent tag-aware embedding; m represents the position of the word embedding in the sentence, and d represents the dimension of the word embedding; and Respectively and The mth vector in ; express and The corresponding rotation matrix; express and The rotation matrix corresponding to the medium dimension [2r-1,2r]; express and The rotation parameters corresponding to the medium dimension [2r-1,2r]; express and The corresponding rotation parameter matrix.

[0140] Slot label-aware embedding of query vectors with multi-head attention mechanism and key vector Perform position encoding to obtain a query vector with position-aware slot label-aware embedding and key vector as follows:

[0141]

[0142]

[0143]

[0144]

[0145]

[0146] in, is the mth query vector with position-aware slot label-aware embedding, is the mth key vector of the position-aware slot label-aware embedding; m represents the position of the word embedding in the sentence, and d represents the dimension of the word embedding; and Respectively and The mth vector in ; express and The corresponding rotation matrix; express and The rotation matrix corresponding to the medium dimension [2r-1,2r]; express and The rotation parameters corresponding to the medium dimension [2r-1,2r]; express and The corresponding rotation parameter matrix.

[0147] Embed query vectors with location-aware multi-intent label awareness As the query vector, the key vector with position-aware slot label-aware embedding As the key vector, the value vector of the position-aware slot label-aware embedding As the value vector, the interactive attention mechanism is used to calculate the preliminary slot-aware multi-intent embedding as follows:

[0148]

[0149]

[0150]

[0151] in, Indicates that attention operation is applied to each attention head. Represents the result of each attention head operation, Concat represents the splicing operation, H′ I For preliminary slot-aware multi-intent embedding;

[0152] The preliminary slot-aware multi-intent embedding is added to the multi-intent label-aware embedding, and the slot-aware multi-intent embedding is obtained through linear layers and layer normalization as follows:

[0153] H intent =LayerNorm(FFN I (H′ I +H I )) (34)

[0154] Among them, H intent For slot-aware multi-intent embedding, FFN I Represents a feedforward neural network, and LayerNorm represents a normalization layer.

[0155] The query vector is embedded with position-aware slot label awareness As the query vector, the key vector with location-aware multi-intent tag-aware embedding As the key vector, the value vector with location-aware multi-intent tag-aware embedding As the value vector, use the attention mechanism to calculate the preliminary multi-intent-aware slot embedding as follows:

[0156]

[0157]

[0158]

[0159] Among them, H′ SFor preliminary multi-intent perception slot embedding, Indicates that attention operation is applied to each attention head, Concat indicates concatenation operation, Represents the result of each attention head operation, H′ S For preliminary multi-intent perception slot embedding;

[0160] The preliminary multi-intent-aware slot embedding is added to the slot label-aware embedding, and the multi-intent-aware slot embedding is obtained by post-linear layer and layer normalization to complete the construction of the location-aware interactive attention network. The method is as follows:

[0161] H slot =LayerNorm(FFN S (H′ S +H S )) (38)

[0162] Among them, H slot For multi-intent-aware slot embedding, FFN S Represents a feedforward neural network, and LayerNorm represents a normalization layer.

[0163] S5: Use the location-aware interactive attention network to construct the predicted multi-intent labels and predicted slot label sequences, specifically:

[0164] Reduce the dimensionality of slot-aware multi-intent embeddings in the location-aware interactive attention network;

[0165] Use a multi-intent classifier to classify the slot-aware multi-intent embedding for dimensionality reduction; make the multi-intent classifier output a vector with the number of multi-intent labels, where each dimension corresponds to an intent label;

[0166] Record the dimensions whose values ​​in the multi-intent classifier output vector are greater than 0.5, and obtain the multi-intent labels corresponding to the dimension vectors whose values ​​are greater than 0.5, which are the predicted multi-intents of the text. The method is:

[0167] h intent =maxpool(H intent ) (39)

[0168]

[0169] Among them, the slot-aware multi-intent embedding H obtained by maximum pooling processing intent , h intent To obtain the multi-intention information embedding representation of the text, represents a weight matrix, Represents a bias matrix, num intentIndicates the number of multi-intent labels, Represents the predicted multi-intent label matrix, where each dimension corresponds to the probability of a multi-intent label. The multi-intent probability value greater than 0.5 is selected as the text prediction intent to construct the predicted multi-intent label.

[0170] Use the fully connected layer to process the multi-intent perception slot embedding, and use the fully connected layer to process the multi-intent perception slot embedding H slot Perform a linear transformation as follows:

[0171] O S =H Slot W S +b S (41)

[0172] Among them, num slot is the number of slot labels, is the multi-intent-aware slot embedding after encoding by the MCRF layer, where each dimension represents the probability of a slot label. represents the weight matrix, represents the bias matrix.

[0173] Using MCRF, the dependency relationship between the processed multi-intent perception slot embedding and the slot label is established to constrain the illegal path sequence generated during the decoding process, and MCRF is used to decode to obtain the predicted slot label sequence, that is, the MCRF mechanism is used to decode the O S Decode and get the predicted slot label sequence as follows:

[0174]

[0175] Where P represents all possible sequences and I represents all illegal sequences. i-1 ,y i ,O S ) indicates that the MCRF mechanism calculates i-1 to y i y represents the current label sequence, and y′ represents the legal slot label sequence. Represents the predicted slot label sequence.

[0176] S6: Using the predicted multi-intent labels and the predicted slot label sequence, a lightweight multi-intent natural language understanding joint model is constructed. Specifically:

[0177] For the multi-intent recognition task, the binary cross entropy loss function is used to calculate the loss between the predicted multi-intent label and the actual labeled multi-intent label; for the slot filling task, the cross entropy loss function is used to calculate the loss between the predicted slot label sequence and the actual labeled slot label sequence; the sum of the two is the total loss of the entire model, where α is the weight coefficient. The details are as follows:

[0178] Set the loss function of the joint model:

[0179]

[0180]

[0181] l SLU =l ID +αl SF (45)

[0182] in, A tensor representation of the multi-intent labels that represent the actual annotations of the text, A tensor representation of the multi-intent label probabilities of the model's predicted text, Represents the loss of the multi-intent recognition task; Represents the tensor representation of the slot label actually labeled for the j-th word in the predicted text, The tensor representation of the slot label probability predicted by the j-th word model, represents the loss of the slot filling task; l SLU The weighted sum of the losses of the multi-intent recognition task and the slot filling task is used to represent the total loss of the joint model, and α is the weight coefficient.

[0183] Select the optimizer and choose Adam optimizer for network optimization;

[0184] Perform training, verification, testing, and model saving. Use the training set in the constructed multi-intent dataset to train the model; perform verification on the verification set and adjust parameters based on the verification results to obtain better network performance; select the joint model with the best performance in the test set and save it to build a multi-intent natural language understanding joint model.

[0185] The SVD decomposition technique is used to decompose the dot product calculation of the query vector Query and the key vector Key of the attention head in the pre-trained model BERT. Assuming X is the input of the attention mechanism, the original attention mechanism is as follows:

[0186]

[0187] Q=XW Q +b Q (47)

[0188] K=XW K +b K (48)

[0189] W Q represents the weight coefficient of the query vector in the attention mechanism, b Q represents the bias of the query vector in the attention mechanism; W K represents the weight coefficient of the key vector in the attention mechanism, b Q Represents the bias of the key vector in the attention mechanism. QK T Represents the dot product operation between the query vector and the key vector.

[0190] Among them, the dot product operation QK of the original query vector and key vector in the pre-trained model BERT T The calculation method is:

[0191]

[0192] Only keep the first two terms on the right side of the equal sign and discard the last two terms on the right side of the equal sign to get the dot product operation QK of the new query vector and key vector T The calculation is as follows:

[0193]

[0194] Through SVD decomposition technology, processing (50) obtains a lightweight attention calculation method, using f SVD The SVD decomposition method is used to process the formula (50). Operation,, the decomposition results are as follows:

[0195]

[0196] Where M is a diagonal matrix. The dot product operation QK of the new query vector and key vector T The calculation is as follows:

[0197]

[0198] The simplified method of attention calculation is as follows:

[0199]

[0200] Get the lightweight calculation method Light_Attention of the attention mechanism. Q represents the bias of the query vector in the attention mechanism; W K represents the weight coefficient of the key vector in the attention mechanism, b Q Represents the bias of the key vector in the attention mechanism; In the original attention mechanism A lightweight calculation method after the SVD decomposition method; dim represents the embedding dimension.

[0201] Use the decomposed attention head calculation method Light_Attention to replace the original attention method of each attention head in each layer of the original pre-trained model to build a lightweight multi-intent natural language understanding joint model.

[0202] S7: Use the lightweight multi-intent natural language understanding joint model to perform multi-intent natural language understanding.

[0203] See also Figure 5 The present invention provides a multi-intent natural language understanding system, comprising:

[0204] Dataset construction module: crawls data, defines multi-intent labels and slot labels based on the crawled data, and builds a multi-intent natural language understanding dataset based on the defined multi-intent labels and slot labels;

[0205] Contextual semantic relationship embedding module: used to build contextual semantic relationship embedding using multi-intent natural language understanding dataset;

[0206] Tag-aware embedding module: used to build multi-intent tag-aware embedding and slot tag-aware embedding by using contextual semantic relationship embedding;

[0207] Location-aware interactive attention network building module: used to build a location-aware interactive attention network using multi-intent label-aware embedding and slot label-aware embedding;

[0208] Multi-intent label and slot label sequence acquisition module: used to construct predicted multi-intent labels and predicted slot label sequences using the location-aware interactive attention network;

[0209] Lightweight multi-intent natural language understanding joint model construction module: This module is used to build a lightweight multi-intent natural language understanding joint model using predicted multi-intent labels and predicted slot label sequences.

[0210] Multi-intent natural language understanding module: used to perform multi-intent natural language understanding using a lightweight multi-intent natural language understanding joint model.

[0211] The present invention provides a terminal device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of each of the aforementioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in each of the aforementioned apparatus embodiments are implemented.

[0212] The computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to accomplish the present invention.

[0213] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0214] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0215] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0216] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0217] In summary, the present invention provides a multi-intent natural language understanding system. Through the setting of a dataset construction module, a contextual semantic relationship embedding module, a label-aware embedding module, a position-aware interactive attention network construction module, a multi-intent label and slot label sequence acquisition module, a lightweight multi-intent natural language understanding joint model construction module and a multi-intent natural language understanding module, data crawling and multi-intent natural language understanding dataset construction are realized. Contextual semantic relationship embedding, multi-intent label-aware embedding and slot label-aware embedding, position-aware interactive attention network, predicted multi-intent labels and predicted slot label sequences, and a lightweight multi-intent natural language understanding joint model are sequentially constructed according to the multi-intent natural language understanding dataset, and finally the understanding of multi-intent natural language is realized. The system model makes full use of the association between the two subtasks in multi-intent natural language understanding and realizes two-way guidance of the two subtasks; it can extract local information more accurately on the basis of extracting global information of the text, and the understanding accuracy of multi-intent natural language is high; and it realizes lightweight processing of the multi-intent natural language understanding joint model, accelerates the reasoning speed of the model, reduces the computing cost and hardware requirements of the model, and facilitates the deployment and use of the model.

[0218] The above description is merely a preferred embodiment of the present invention and is not intended to impose any limitation on the technical solution of the present invention. Those skilled in the art should understand that, without departing from the spirit and principles of the present invention, the technical solution can also be subjected to several simple modifications and replacements, and these modifications and replacements are also within the scope of protection covered by the claims.

Claims

1. A multi-intent natural language understanding method, characterized in that: The following steps are involved: S1: Crawl data and define multi-intent labels and slot labels based on the crawled data. Based on the defined multi-intent labels and slot labels, build a multi-intent natural language understanding dataset. S2: Build contextual semantic relationship embeddings using a multi-intent natural language understanding dataset; S3: Using contextual semantic relationship embedding, we construct multi-intent label-aware embedding and slot label-aware embedding. S4: Utilize multi-intent label-aware embedding and slot label-aware embedding to build a location-aware interactive attention network. The specific method is as follows: Perform linear transformations on the multi-intent label-aware embedding and the slot label-aware embedding to obtain the query vector Query of the multi-intent label-aware embedding I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S ; Embed the query vector Query with multi-intent label awareness I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S Split into num head Attention heads are used to obtain the query vector of the multi-intent label-aware embedding of the multi-head attention mechanism. Key Vector Sum vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism Key Vector Sum vector Multi-intent label-aware query embedding for multi-head attention mechanism using rotational position encoding and key vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism and key vector Perform position encoding to obtain a query vector with position-aware multi-intent label-aware embedding and key vector and query vectors with position-aware slot label-aware embeddings and key vector Among them, i represents the number of the attention head, i∈[1,2,…,num head ]; Embed query vectors with location-aware multi-intent label awareness As the query vector, the key vector with position-aware slot label-aware embedding As the key vector, the slot label perception embedding value vector of the multi-head attention mechanism As the value vector, the attention mechanism is used to calculate the preliminary slot-aware multi-intent embedding; the preliminary slot-aware multi-intent embedding is added to the multi-intent label-aware embedding, and the slot-aware multi-intent embedding is obtained through linear layers and layer normalization; The query vector is embedded with position-aware slot label awareness As the query vector, the key vector with location-aware multi-intent tag-aware embedding As the key vector, the value vector of the multi-intent label perception embedding of the multi-head attention mechanism As the value vector, the attention mechanism is used to calculate the preliminary multi-intent-aware slot embedding. The preliminary multi-intent-aware slot embedding is added to the slot label-aware embedding, and the multi-intent-aware slot embedding is obtained through a linear layer and layer normalization to complete the construction of the location-aware interactive attention network. S5: Use the location-aware interactive attention network to construct predicted multi-intent labels and predicted slot label sequences; S6: Build a lightweight multi-intent natural language understanding joint model using the predicted multi-intent labels and the predicted slot label sequences; S7: Use a lightweight multi-intent natural language understanding joint model to perform multi-intent natural language understanding.

2. The multi-intent natural language understanding method according to claim 1, characterized in that The specific method of S1 is: Design a crawler program to crawl data information from the corresponding website; Clean and organize the crawled data information; Filter the cleaned and organized data information to filter out the text of the target data set content; Define multi-intent labels and slot labels based on the text of the target dataset content, and annotate the text of the target dataset content to generate the required multi-intent natural language understanding dataset.

3. The multi-intent natural language understanding method according to claim 1, characterized in that: The specific method of S2 is: Based on the multi-intent natural language understanding dataset, the pre-trained model BERT is used as the word embedding layer of the text to obtain the word embedding of the text; The BiLSTM model and attention mechanism are used to extract features from the word embedding of the text to obtain contextual semantic relationship embedding.

4. The multi-intent natural language understanding method according to claim 1, characterized in that The specific methods of S3 are: The contextual semantic relationship embedding in S2 is used as the query vector, and the attention mechanism is used to calculate the multi-intent label attention embedding and the slot label attention embedding; Add the multi-intent label attention embedding to the contextual semantic relationship embedding. On the basis of preserving the contextual semantic relationship, establish the connection between the contextual semantic relationship embedding and the multi-intent label to obtain the multi-intent label perception embedding. The slot label attention embedding is added to the contextual semantic relationship embedding. On the basis of retaining the contextual semantic relationship, the connection between the contextual semantic relationship embedding and the slot label is established to obtain the slot label aware embedding.

5. The multi-intent natural language understanding method according to claim 1, characterized in that: The specific method of S5 is: The slot-aware multi-intent embedding obtained from the location-aware interactive attention network is reduced in dimension; a multi-intent classifier is used to classify the reduced slot-aware multi-intent embedding; Make the multi-intent classifier output a vector with the dimension of the number of multi-intent labels, where each dimension corresponds to an intent label; Record the dimensions in the multi-intent classifier output vector whose values ​​are greater than 0.5, obtain the multi-intent labels corresponding to the dimension vectors whose values ​​are greater than 0.5, that is, the intent of the sentence, and construct the predicted multi-intent labels; Use fully connected layers to process multi-intent-aware slot embedding; The MCRF mechanism is used to establish the dependency relationship between the processed multi-intent-aware slot embedding and the slot label, constraining the illegal path sequence generated during the decoding process. The MCRF decoder is then used for decoding to construct the predicted slot label sequence.

6. The multi-intent natural language understanding method according to claim 1, characterized in that: The specific method of S6 is: For the multi-intent recognition task, a binary cross-entropy loss function is used to calculate the loss between the predicted multi-intent labels and the actual labeled multi-intent labels. For the slot filling task, a cross-entropy loss function is used to calculate the loss between the predicted slot label sequence and the actual labeled slot label sequence. The weighted sum of the two losses is used as the total loss of the joint model. Select the optimizer and choose Adam optimizer for network optimization; Train, validate, test, and save the joint model. Use the training set in the constructed multi-intent dataset to train the network. Validate on the validation set and adjust parameters based on the validation results to achieve better network performance. The joint model with the best performance in the test set is selected and saved to complete the construction of the multi-intent natural language understanding joint model. SVD technology is used to decompose the dot product calculation of the query vector Query and the key vector Key of the attention head in the pre-trained BERT model. Get a lightweight attention operation method; A lightweight attention operation method is used to replace the original attention calculation method in each attention head of each layer in the pre-trained BERT model to build a lightweight multi-intent natural language understanding joint model.

7. A multi-intent natural language understanding system, characterized in that: include: Dataset construction module: crawls data, defines multi-intent labels and slot labels based on the crawled data, and builds a multi-intent natural language understanding dataset based on the defined multi-intent labels and slot labels; Contextual semantic relationship embedding module: used to build contextual semantic relationship embedding using multi-intent natural language understanding dataset; Tag-aware embedding module: used to build multi-intent tag-aware embedding and slot tag-aware embedding by using contextual semantic relationship embedding; Location-aware interactive attention network building module: This module is used to build a location-aware interactive attention network using multi-intent label-aware embedding and slot label-aware embedding. The specific method for building a location-aware interactive attention network is as follows: Perform linear transformations on the multi-intent label-aware embedding and the slot label-aware embedding to obtain the query vector Query of the multi-intent label-aware embedding I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S ; Embed the query vector Query with multi-intent label awareness I , key vector Key I Sum value vector Value I And the query vector Query of slot label-aware embedding S , key vector Key S Sum value vector Value S Split into num head Attention heads are used to obtain the query vector of the multi-intent label-aware embedding of the multi-head attention mechanism. Key Vector Sum vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism Key Vector Sum vector Multi-intent label-aware query embedding for multi-head attention mechanism using rotational position encoding and key vector And the query vector of the slot label-aware embedding of the multi-head attention mechanism and key vector Perform position encoding to obtain a query vector with position-aware multi-intent label-aware embedding and key vector and query vectors with position-aware slot label-aware embeddings and key vector Among them, i represents the number of the attention head, i∈[1,2,…,num head ]; Embed query vectors with location-aware multi-intent label awareness As the query vector, the key vector with position-aware slot label-aware embedding As the key vector, the slot label perception embedding value vector of the multi-head attention mechanism As the value vector, the attention mechanism is used to calculate the preliminary slot-aware multi-intent embedding; the preliminary slot-aware multi-intent embedding is added to the multi-intent label-aware embedding, and the slot-aware multi-intent embedding is obtained through linear layers and layer normalization; The query vector is embedded with position-aware slot label awareness As the query vector, the key vector with location-aware multi-intent tag-aware embedding As the key vector, the value vector of the multi-intent label perception embedding of the multi-head attention mechanism As the value vector, the attention mechanism is used to calculate the preliminary multi-intent-aware slot embedding. The preliminary multi-intent-aware slot embedding is added to the slot label-aware embedding, and the multi-intent-aware slot embedding is obtained through a linear layer and layer normalization to complete the construction of the location-aware interactive attention network. Multi-intent label and slot label sequence acquisition module: used to construct predicted multi-intent labels and predicted slot label sequences using the location-aware interactive attention network; Lightweight multi-intent natural language understanding joint model construction module: This module is used to build a lightweight multi-intent natural language understanding joint model using predicted multi-intent labels and predicted slot label sequences. Multi-intent natural language understanding module: used to perform multi-intent natural language understanding using a lightweight multi-intent natural language understanding joint model.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images