A new intent data recognition method based on Mahalanobis distance and contrastive learning

Abstract

The present invention provides a method for identifying new intent data based on Mahalanobis distance and contrastive learning. The present invention introduces contrastive learning in the training process of the classification model to obtain more complete sample features, and uses Mahalanobis distance as a scoring function based on the feature vector of the sample to prevent the loss of features after passing through the classification layer, which helps the model improve its ability to identify new intent samples.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a new intention data recognition method based on Mahalanobis distance and contrastive learning. Background Art

[0002] With the development of artificial intelligence (AI) technology, task-oriented, conversational intelligent assistants have emerged, such as mobile voice assistants, e-commerce intelligent customer service, and smart speakers, greatly facilitating people's lives. Intent detection is a key module in task-oriented dialogue systems (TADSs). It is responsible for identifying the intent contained in the user's input and executing subsequent actions based on the recognition results. Therefore, the accuracy of intent recognition directly affects the execution of subsequent steps and user satisfaction with the system. However, in real-world application scenarios, user input may contain completely new intents that have never been seen by the system and are beyond its recognition capabilities. These are called new intents. Correctly identifying new intents prevents the system from executing incorrect actions and providing irrelevant responses, thereby improving the user experience.

[0003] The current mainstream intent recognition technology solution is to convert the new intent recognition task into a text classification task. First, a text multi-classification model is trained using existing intent samples and corresponding labels, and the confidence score obtained by the text classification model is used as the scoring function for the new intent recognition task. If the sample confidence score is lower than a certain threshold, it will be regarded as a new intent sample.

[0004] [Patent 1] CN111382270A Intent recognition method, device, equipment and storage medium based on text classifier.

[0005] The invention uses a text classification model to classify intent, and at the same time determines whether the confidence score obtained by the text classification model is greater than a threshold to determine whether the current intent is a new intent.

[0006] [Paper 2] Out-of-domain Detection for Natural Language Understanding inDialog Systems.

[0007] This paper generates negative samples and introduces them into the training process of the text classification model to enhance the model's ability to recognize new intents.

[0008] [Paper 3] Modeling Discriminative Representations for Out-of-DomainDetection with Supervised Contrastive Learning.

[0009] This paper introduces comparative learning between samples and samples of other categories to enhance the classification ability of text classification models.

[0010] Patent 1 obtains a confidence score through a text classification model as a new intent scoring function, and sets a threshold to detect new intents. However, text classification models based on neural networks often face the problem of overconfidence, that is, even if the input sample does not belong to a known intent, some features are similar to the known intent, resulting in the new intent sample still being classified into the wrong category with a high degree of confidence. This is due to the limited nature of the neural network training set. The classification model will use as few features as possible required for classification, while ignoring other important category features. Since the classification model has never seen a new intent sample, it is difficult to capture the important features of the new intent, and it is easy to classify it into similar intent categories (such as playing music and playing movies).

[0011] To address this issue, Paper 2 introduces negative samples during model training to enhance the model's ability to capture category features. The model uses GAN technology to generate fake negative samples and introduces them into the training process, requiring the model to have as large an entropy as possible for the negative sample predictions, that is, the confidence score obtained by the model is as small as possible, thereby alleviating the problem of overconfidence in the model. However, it is difficult to guarantee the quality of negative samples generated using GAN, that is, whether they are truly new intent samples. Paper 3 introduces contrastive learning during the training process of the classification model to obtain more discriminative features. It uses samples of the same category as positive samples and samples of other categories as negative samples for contrastive learning, thereby capturing more discriminative features between categories. However, the capture of this feature is still limited to distinguishing different features between samples of known intent categories, and cannot capture complete sample features to distinguish new intent samples that are relatively close to known intents. Summary of the Invention

[0012] Purpose of the invention: The technical problem to be solved by the present invention is how to learn more complete sample features to distinguish known intents from new intents, and to avoid feature loss as much as possible when calculating the scoring function, so that the model has better recognition ability for new intent samples, especially new intent samples that are very similar to known intent samples.

[0013] The present invention specifically provides a new intention data recognition method based on Mahalanobis distance and contrastive learning, comprising the following steps:

[0014] Step 1: Input all samples and annotated labels in the training set;

[0015] Step 2: construct positive and negative samples for the known intent samples in the training set for comparative learning training;

[0016] Step 3: Combine contrastive learning to train the intent classification model and obtain sample representation through the classification model;

[0017] Step 4, calculate the category center;

[0018] Step 5: Calculate the minimum Mahalanobis distance between the sample and the category center and determine whether it is a new intent sample.

[0019] Step 2 includes:

[0020] Step 2-1, identify the slots of all input samples and obtain the slot type of the sample;

[0021] Step 2-2, pair the samples with the same slot type in the training set, that is, sample x1 = T1 (s1 = v1, s2 = v2), sample x2 = T2 (s1 = v3, s2 = v4), where s i Represents the i-th slot type, v i Represents the value of the i-th slot, T i Represents the i-th sentence template;

[0022] Step 2-3, construct a positive sample for sample x1 Construct positive samples for sample x2 Construct negative samples for sample x1 Construct negative samples for sample x2

[0023] Step 3 includes:

[0024] Step 3-1: Input the original training set D = {(x, y)} and the contrastive learning training set D′ = (x, x + , x - );

[0025] Step 3-2: The intent recognition model performs forward calculations;

[0026] Step 3-3, calculate the cross entropy CE between the predicted probability distribution of the sample and the true label;

[0027] Step 3-4, calculate the gradient of the loss function value for each model parameter, and use the backpropagation algorithm to update the model parameters;

[0028] Steps 3-5: Use the validation set to evaluate model performance;

[0029] Step 3-6: Determine whether the model performance has improved. If so, return to step 3-2 to continue iterative training. Otherwise, proceed to step 3-7.

[0030] Steps 3-7, end the training model.

[0031] Step 3-1 includes: the original training set D is used for training the classification task, x = {w1, w2, ..., w n} is the original sample, which contains n words w1, w2, ..., w n , y is the corresponding intention label;

[0032] Contrastive learning training set D' is used for contrastive learning training, x + , x - They are respectively the positive and negative samples corresponding to the original sample x; among them, the intention sample label is one-hot encoded and recorded as the set Where m is the number of known intentions, y i represents the i-th intent label, The position of the intent label corresponding to the current positive sample is 1, and the values ​​of the other positions are 0; the original input samples are divided into training set and validation set in a ratio of 4:1, and the training set is input into the model for training;

[0033] Step 3-2 includes: getting word w i word embeddings i , that is, sample x′={e1,e2,...,e n};

[0034] Use the gated recurrent unit GRU (Gate Recurrent Unit) network encoding to input the original sample x′, that is, embed each word into a gated unit, and get the output of each word, which is the hidden layer representation of each word state, that is, {h1, h2, .., h n}=GRU(e1,e2,...,e n ), where h i Represents the i-th word e i The output of GRU() represents the embedding of a series of words into the input gated recurrent unit network, i.e. h t =z t ⊙h t-1 +(1-z t )⊙g(e t , h t-1 ,θ),z t ∈[0, 1] is the update gate, z t =σ(W z x t +U z h t-1 +b z ), Wz , U z and b z is a trainable parameter; the function g(e t , h t-1 ,θ)=tanh(W h e t +U h (r t ·h t-1 )+b h ), where W h , U h and b h is a trainable parameter; r t ∈[0,1] is the reset gate, r t =σ(W r e t +U r ht -1 +b r ), where W r , U r and b r is a trainable parameter;

[0035] Take the mean of the hidden layer representations of all states as the representation of the sample

[0036] The feature space of the sample is restricted by L2 regularization, and the sample modulus is limited to α, that is, Get the original sample representation h x and positive and negative sample representation The original sample is represented by h x Input linear layer and SoftMax layer to get label distribution p={p1,p2,…,p m}, where p m Indicates that the model predicts the sample as the mth intent label y m The probability of m is the number of intentions;

[0037] Step 3-3 includes: using the following formula to calculate the cross entropy between the predicted probability distribution of the sample and the true label

[0038] Calculate sample x and positive and negative samples x + 、x - The triplet loss TL(x, x + , x - )=max(d(x,x + )-d(x,x - )+m,0), where the distance from sample x to sample y where h x =(a1, a2, ..., a d), h y =(b1, b1, ..., b d ), d is the dimension of h, and m is a hyperparameter; the loss function of the final model is L = CE + λ · TL, where λ is a hyperparameter;

[0039] Step 4 includes: the category center of category i is expressed as Among them, N c is the number of samples of category i in the training set, For sample x i The sentence expresses .

[0040] Step 4 includes: calculating the minimum Mahalanobis distance from the sample to the category center: first calculate the covariance of the training set Where c represents the category label;

[0041] Calculate the minimum Mahalanobis distance from sample x to the category center where min c Represents the category c that minimizes the Mahalanobis distance. If M(x) is greater than the selected threshold, the sample x is judged to be a new intent sample.

[0042] The present invention introduces contrastive learning in the training process of the classification model to obtain more complete sample features, and uses the Mahalanobis distance as a scoring function based on the feature vector of the sample to prevent the loss of features after passing through the classification layer, which helps the model improve its ability to identify new intent samples.

[0043] Beneficial effects:

[0044] From a technical perspective, the technical solution of the present invention (1) introduces contrastive learning into model training, which can learn more complete features. (2) Uses Mahalanobis distance as a scoring function to avoid loss of learned features after passing through the classification layer.

[0045] From an application perspective, the technical solution of the present invention (1) utilizes the slot type information inherent in the dataset to automatically construct positive and negative samples for feature learning. (2) The model through comparative learning can learn more complete features, better distinguish new intents similar to known intents, avoid system output errors, and optimize the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0047] Figure 1 It is a flow chart of the present invention.

[0048] Figure 2 It is a flowchart of constructing positive and negative samples.

[0049] Figure 3 It is the model training flow chart. DETAILED DESCRIPTION

[0050] like Figure 1 As shown, the present invention provides a new intent data recognition method based on Mahalanobis distance and contrastive learning, comprising:

[0051] Step 1, input all samples and manually annotated labels in the training set. To train a new intent model, it is necessary to determine the intent label system in advance, and use the corpus and labels in the system as known intents for model learning. The present invention uses the SNIPS dataset, which is a dataset obtained after a voice assistant corpus is converted into text. There are 7 categories in total, namely "play music", "ask the weather", "add to playlist", "book rating", "restaurant reservation", "search for creative works" and "search for movie information". In particular, "search for creative works" and "add to playlist" are regarded as new intent samples here, which is a more challenging set of settings. The data is marked with the slot type for each piece of data, such as the singer's name and the song title.

[0052] Step 2: construct positive and negative samples for the known intent samples in the training set for comparative learning and training. The specific positive and negative sample construction process is as follows: Figure 2 As shown:

[0053] Step 2-1: Identify the slots of all input samples and obtain the slot types they possess. For the sample "Can you play Jay Chou's music "Chrysanthemum Terrace"?", it has two slots: "Can you play [singer name]'s music "[song title]?" The value of slot "Singer name" is Jay Chou, and the value of slot "Song title" is "Chrysanthemum Terrace."

[0054] Step 2-2, pair the samples with the same slot type in the training set, that is, samples x1 = T1 (s1 = v1, s2 = v2), x2 = T2 (s1 = v3, s2 = v4). i Represents the slot type, v i Represents the slot value, p i Represents a sentence pattern (excluding the slot value). For example, Sample 1 (Can you play Jay Chou's music "Chrysanthemum Terrace") and Sample 2 (Play JJ Lin's music "Jiangnan") are a pair of samples. They both have the same slot type: artist name, song title. Samples with the same slot type often have semantically similar sentence patterns. Replacing their slot values ​​will produce a new sample.

[0055] Step 2-3: In order to make the model pay attention to the complete semantic information, samples with different expressions but the same semantic information are considered as positive samples. If the slot values ​​of two samples are the same and the sentence patterns are similar, they are considered to have similar semantics. This is to construct a positive sample for sample x1. Construct positive samples for sample x2 If the slot values ​​of two samples are different, they represent different semantics even if the sentence patterns are the same. This is to construct a negative sample for sample x1. Sample x2 constructs negative samples For example, for the example "Can you play Jay Chou's music [Chrysanthemum Terrace]?", the positive example is "Play Jay Chou's music [Chrysanthemum Terrace]," and the negative example is "Can you play JJ Lin's music [Jiangnan]?" By enabling the model to distinguish between the original and negative examples, it can focus on more complete semantic information. Table 1 shows an example of a sample used for contrastive learning.

[0056] Table 1

[0057]

[0058]

[0059] Step 3: Combine contrastive learning to train the intent classification model and obtain sample representation through the classification model. The specific training process is as follows: Figure 3 As shown:

[0060] Step 3-1, input original sample x=w1,w2,...,w n and the corresponding label y are used to train the classification model, inputting the original sample x and the corresponding positive and negative samples x + , x - Used for contrastive learning training. Among them, the intent sample labels are one-hot encoded and recorded as a set Where m is the number of known intentions, y i Indicates the i-th intent label. The position corresponding to the intent label of the current positive sample in y is 1, and the values ​​of the other positions are 0. The input samples are divided into training set and validation set at a ratio of 4:1. The training set is input into the model for training, and the validation set is used in the subsequent part.

[0061] Step 3-2, the intent recognition model performs forward calculation: first obtain the word w i word embeddings i , that is, sample x′={e1,e2,...,e n};

[0062] Use the gated recurrent unit GRU (Gate Recurrent Unit) network encoding to input the original sample x′, that is, embed each word into e iInput into a gated unit and get the output h i , is the hidden layer representation of each word state, that is, {h1, h2, .., h n}=GRU(e1,e2,...,e n ), where GRU() represents embedding a series of words into the input gated recurrent unit network, i.e. h t =z t ⊙h t-1 +(1-z t )⊙g(e t , h t-1 ,θ),z t ∈[0, 1] is the update gate, z t =σ(W z x t +U z h t-1 +b z ), W z , U z and b z is a trainable parameter; the function g(e t , h t-1 ,θ)=tanh(W h e t +U h (r t ·h t-1 )+b h ), where W h , U h and b h is a trainable parameter; r t ∈[0,1] is the reset gate, r t =σ(W r e t +U r h t-1 +b r ), where W r , U r and b r is a trainable parameter. The mean of the hidden layer representations of all states is taken as the representation of the sample

[0063] The feature space of the sample is restricted by L2 regularization, and the sample modulus is limited to α, that is, Get the original sample representation h x and positive and negative sample representation The original sample is represented by h x Input linear layer and SoftMax layer to get label distribution p={p1,p2,...,p m}, where p m Indicates that the model predicts the sample as the mth intent label ym The probability of m is the number of intentions;

[0064] Step 3-3, calculate the cross entropy between the predicted probability distribution of the sample and the true label, denoted as Calculate the triplet loss (TripletLoss) between the sample and the positive and negative samples, denoted as TL(x, x + , x - )=max(d(x,x + )-d(x,x - )+m,0), where d(x,y)=||h x -h y ||2 is the distance from sample x to sample y, and m is a hyperparameter. The loss function of the final model is L = CE + λ·TL, where λ is a hyperparameter.

[0065] In steps 3-4, the gradient of the loss function value for each model parameter is calculated, and the model parameters are updated using the backpropagation algorithm.

[0066] In steps 3-5, the model performance is evaluated using the validation set. Here, the accuracy of the model is evaluated.

[0067] In step 3-6, determine whether the model performance has improved. If so, return to step 3-2 to continue iterative training. Otherwise, proceed to step 3-7.

[0068] Steps 3-7, end the training model.

[0069] Step 4: Get the sample representation h x , calculate the category center. The category center of category i is expressed as Among them, N c is the number of samples of category i in the training set.

[0070] Step 5: Calculate the minimum Mahalanobis distance from the sample to the class center. First calculate the covariance of the training set, Where c represents the category label. Calculate the minimum Mahalanobis distance from sample x to the category center If M(x) is greater than the selected threshold, it is judged as a new intention sample.

[0071] In this embodiment, the experimental results of the present invention on the test set are shown in Table 2 below:

[0072] Table 2

[0073] MODEL AUROC (%) AUPR (%) MAH 93.76 84.64 MAH+L2-norm 96.59 91.74 MAH+L2-norm+Contrastive Loss 96.73 92.31

[0074] MAH refers to the result obtained using the Mahalanobis distance as the scoring function based on the classification model, MAH+L2-norm refers to the result after L2 regularization of the feature space, and MAH+L2-norm+Contrastive Loss refers to the final solution after adding contrastive learning. AUROC (Area Under Receiver Operating Characteristic Curve) refers to the area under the receiver operating characteristic curve, and AUPR (Area Under Precision Recall Curve) refers to the area under the precision-recall curve. Experimental results show that the effectiveness of the Mahalanobis distance is significantly improved after regularization of the feature space. Furthermore, introducing contrastive learning into the model further improves the results.

[0075] The present invention provides a novel intent data recognition method based on Mahalanobis distance and contrastive learning. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A new intent data recognition method based on Mahalanobis distance and contrastive learning, characterized in that: The following steps are involved: Step 1: Input all samples in the training set and manually annotated labels; To train a new intent model, we first define an intent labeling system. The corpus and labels in this system serve as known intents for model learning. We use the SNIPS dataset, which is a text-based dataset of voice assistant corpus. The dataset contains seven categories: play music, ask about the weather, add to playlist, rate a book, reserve a restaurant, search for creative works, and search for movie information. We consider searching for creative works and adding to playlists as examples of new intents. Each piece of data is labeled with the slot type, including the artist name and song title. Step 2: construct positive and negative samples for the known intent samples in the training set for comparative learning training; Step 3: Combine contrastive learning to train the intent classification model and obtain sample representation through the classification model; Step 4, calculate the category center; Step 5: Calculate the minimum Mahalanobis distance between the sample and the category center and determine whether it is a new intent sample; Step 2 includes: Step 2-1, identify the slots of all input samples and obtain the slot type of the sample; Step 2-2, pair the samples with the same slot type in the training set, that is, sample x1 = T1(s1=v1, s2=v2), sample x2=T2(s1=v3, s2=v4), where s i Represents the i-th slot type, v i Represents the value of the i-th slot, T i Represents the i-th sentence template; Step 2-3, construct a positive sample for sample x1 Construct positive samples for sample x2 Construct negative samples for sample x1 Construct negative samples for sample x2 Step 3 includes: Step 3-1: Input the original training set D = {(x, y)} and the contrastive learning training set D ′ =(x,x + ,x - ); Step 3-2: The intent recognition model performs forward calculation; Step 3-3, calculate the cross entropy CE between the predicted probability distribution of the sample and the true label; Step 3-4, calculate the gradient of the loss function value for each model parameter, and use the backpropagation algorithm to update the model parameters; Steps 3-5: Use the validation set to evaluate model performance; Step 3-6: Determine whether the model performance has improved. If so, return to step 3-2 to continue iterative training. Otherwise, proceed to step 3-7. Step 3-7, end the training model; Step 3-1 includes: the original training set D is used for training the classification task, x = {w1, w2, ..., w n } is the original sample, which contains n words w1,w2,…,w n , y is the corresponding intention label; Contrastive learning training set D' is used for contrastive learning training, x + ,x - They are respectively the positive and negative samples corresponding to the original sample x; among them, the intention sample label is one-hot encoded and recorded as the set Where m is the number of known intentions, y i represents the i-th intent label, The position of the intent label corresponding to the current positive sample is 1, and the values ​​of the other positions are 0; the original input samples are divided into training set and validation set in a ratio of 4:1, and the training set is input into the model for training; Step 3-2 includes: getting word w i word embeddings i , that is, sample x′={e1,e2,…,e n }; Use the Gated Recurrent Unit (GRU) network to encode the input original sample x. ′ , that is, embed each word into a gated unit and get the output of each word, which is the hidden layer representation of each word state, namely {h1, h2, .., h n }=GRU(e1,e2,…,e n ), where h i Represents the i-th word e i The output of GRU() represents the embedding of a series of words into the input gated recurrent unit network, i.e. h t =z t ⊙h t-1 +(1-z t )⊙g(e t ,h t-1 ,θ),z t ∈[0,1] is the update gate, z t =σ(W z x t +u z h t-1 +b z ),W z ,U z and b z is a trainable parameter; the function g(e t ,h t-1 ,θ)=tanh(W h e t +U h (r t ·h t-1 )+b h ), where W h ,U h and b h is a trainable parameter; r t ∈[0,1] is the reset gate, r t =σ(W r e t +U r h t-1 +b r ), where W r ,U r and b r is a trainable parameter; Take the mean of the hidden layer representations of all states as the representation of the sample The feature space of the sample is restricted by L2 regularization, and the sample modulus is limited to α, that is, Get the original sample representation h x and positive and negative sample representation The original sample is represented by h x Input linear layer and SoftMax layer to get label distribution p={p1,p2,…,p m }, where p m Indicates that the model predicts the sample as the mth intent label y m The probability of m is the number of intentions; Step 3-3 includes: using the following formula to calculate the cross entropy between the predicted probability distribution of the sample and the true label Calculate sample x and positive and negative samples x + 、x - The triplet loss TL(x, x + , x - )=max(d(x,x + )-d(x,x - )+m,0), where the distance from sample x to sample y where h x =(a1,a2,…,a d ),h y =(b1,b1,...,b d ), d is the dimension of h, and m is a hyperparameter; the loss function of the final model is L = CE + λ · TL, where λ is a hyperparameter; Step 4 includes: the category center of category i is expressed as Among them, N c is the number of samples of category i in the training set, For sample x i The sentence indicates; Step 4 includes: calculating the minimum Mahalanobis distance from the sample to the category center: first calculate the covariance of the training set Where c represents the category label; Calculate the minimum Mahalanobis distance from sample x to the category center where min c represents the category c that minimizes the Mahalanobis distance; if M(x) is greater than the selected threshold, the sample x is judged to be a new intention sample.

Images