Cloze-style data augmentation method for small sample intent recognition

Abstract

One or more embodiments of the present application provide a cloze-style data enhancement method for small-sample intent recognition, including: constructing a cloze-style data enhancement task for intent recognition based on a pre-trained language model, first using an unsupervised learning method to make the data enhancement result semantically similar to the original input sentence, and then using a supervised contrastive learning method based on metric classification to make intent samples of the same category closer to each other in the embedding space, and intent samples of different categories farther away from each other. The contrastive learning method includes prototype-level contrastive learning and instance-level contrastive learning. Meaningful data is generated without destroying the syntactic structure and adding noise, making full use of limited data and obtaining separable embeddings. A better distance distribution is obtained in the embedding space, thereby improving the performance of the metric-based classification method.

Description

Technical Field

[0001] The present invention belongs to the technical field of intent recognition, and in particular relates to a cloze-type data enhancement method for small-sample intent recognition. Background Art

[0002] Intent recognition aims to identify a user's underlying intent from their utterances and is a key component of task-oriented dialogue systems. However, a practical challenge is that the number of intent categories is growing faster than the rate of human-annotated data, resulting in only a small amount of data available for many new intent categories. This data shortage causes traditional deep neural networks to overfit on small amounts of training data, severely impacting practical applications. Therefore, some researchers have proposed few-shot learning to address this data scarcity issue. One effective approach is text data augmentation, but it often generates noisy or meaningless data.

[0003] In real-world applications, new intent categories emerge rapidly, and there is limited well-labeled data, making it difficult to directly apply these to optimize existing deep neural networks. These networks typically incorporate a pre-trained language model as their backbone to encode text data into continuous, low-dimensional vectors, such as BERT and RoBERTa. Such models often have complex, multi-layered architectures, resulting in a significant number of parameters. If traditional training paradigms are used to directly update the parameters of deep neural network models using a small amount of training data, the model will only capture local features, resulting in insufficient generalization and overfitting, meaning that it performs well on the training set but poorly on the test set. To address this issue, Snell et al. proposed a few-shot learning (FSL) strategy to help models achieve generalization with limited data. These researchers formulate few-shot intent recognition as a meta-learning problem. They simulate the few-shot scenario through a series of small meta-tasks. This approach has been widely used in few-shot text classification tasks such as relation classification, event detection, and intent detection.

[0004] A major challenge is that meta-learning-based few-shot learning methods are still prone to overfitting to biased distributions due to limited training samples. Some researchers have attempted to prevent overfitting through data augmentation methods. One key idea is back-translation: translating the input text's representation in another language into the original language. Another common approach is to leverage external knowledge bases to obtain representations that are semantically similar to the original sentence. Specifically, Dopierre et al. introduced multiple knowledge bases to generate different paraphrased sentences of the original input, rather than reordering the words in the sentence. However, while back-translation can generate different representations with the same semantics, it performs poorly for short texts. The representations it generates are often similar or even identical to the original input sentence. Regarding paraphrase generation methods, we believe that they are not suitable for text augmentation in all fields because corresponding external knowledge bases are not always available. Furthermore, previous data augmentation-related methods, such as the method proposed by Liu et al., have low training efficiency and are difficult to scale to tasks with a large number of intents. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a cloze-type data enhancement method for small sample intent recognition, so as to solve at least one of the above-mentioned problems existing in the prior art.

[0006] Based on the above-mentioned purpose, one or more embodiments in the present application provide a cloze-type data enhancement method for small-sample intent recognition, which includes: constructing a cloze-type intent recognition task based on a pre-trained language model for data enhancement, first using an unsupervised learning method to make the data enhancement result semantically similar to the original input sentence, and then using a supervised contrastive learning method based on metric classification to make intent samples of the same category closer to each other in the embedding space, and intent samples of different categories farther away from each other, and the contrastive learning method includes prototype-level contrastive learning and instance-level contrastive learning.

[0007] Based on the above technical solution of the present invention, the following improvements can also be made:

[0008] Optionally, the unsupervised learning method includes: using a pre-trained language model as a feature extractor, using markers to mask a set proportion of input words, and predicting these masked markers based on the semantics of the context of the input sentence; the feature extractor encodes the sentence in the cloze-type intent recognition task into a hidden layer vector representation by adding two special markers, where the two special markers respectively represent the beginning and end of the sentence in the cloze-type intent recognition task. This process is expressed by the following formula:

[0009]

[0010] Where F(·) is the feature extractor, T is the cloze template, [MASK] is the mark of the masked word, Pat(T, x) is the data augmentation pattern, x is the input sentence, represents the hidden layer vector, [CLS] and [SEP] represent the start and end respectively;

[0011] The hidden layer vector As the representation of the masked word [MASK], It is regarded as a sentence representation generated according to the pattern Pat(T, x), which is semantically similar to the input sentence x. Formula (1) is repeated on all input samples to obtain the corresponding data enhancement results.

[0012] Optionally, the pre-trained language model is constrained by a loss function to weaken vectors that do not match the semantics of the input sentence, ultimately obtaining appropriate data augmentation results.

[0013] Optionally, without introducing any external knowledge and labels, the input sentence x is fed into the pre-trained language model to obtain its low-dimensional vector representation, which can be expressed as:

[0014]

[0015] Among them, the hidden layer vector Represented as the entire sentence x, through the following loss function:

[0016]

[0017] Used to reduce and The distance between them.

[0018] Optionally, after completing the unsupervised cloze-style data augmentation, a metric-based prototype network is used as a classifier to test the effect of data augmentation.

[0019] Optionally, first calculate the average representation of samples in the same category and use it as the prototype of the category:

[0020]

[0021] Among them, c i represents the prototype representation of category i, K i Represents the support dataset under the current meta-task T The number of samples of category i in , is the representation of the kth sentence in category i; through this prototype representation, the average distance from the samples of the same category to its center can be the shortest; similarly, based on Through the prototype calculation formula (4), the enhanced prototype c′ can be obtainedi .

[0022] Optionally, in order to make the final prototype cover the common features of its category more comprehensively, the prototype of the input sample and the prototype of the data augmentation result are weighted as follows:

[0023]

[0024] Where α is a weighting coefficient to control the respective contributions from the original input data and the enhanced data;

[0025] Given a score function s(·,·), the prototype network predicts the label x of the query instance by computing the softmax distribution of the similarity between the query embedding vector and the prototype Q , the formula is as follows:

[0026]

[0027] Among them, y is the predicted label, x Q Is the current meta-task The query instance in the query set Q, j is the true label, Denote the final prototype based on the initial and augmented data of category j, and choose the cosine similarity as s(·,·); learning is done by minimizing the negative log probability:

[0028]

[0029] to proceed.

[0030] Optionally, a contrastive learning-based loss is introduced for prototype-level contrastive learning, so as to separate prototypes of different categories as much as possible and make the average representations of different categories far away from each other, which is expressed by the following formula:

[0031]

[0032] Among them, s(·,·) is the same similarity measurement function as in the formula, so s(c i , c i ) is a constant 1, simplifying formula (4) to the following form:

[0033]

[0034] Where e is a constant, It is the contrast loss of prototype level.

[0035] Optionally, instance-level contrastive learning is introduced into prototype-level contrastive learning to make instances of the same category close to each other, which is expressed by the following formula:

[0036]

[0037] in, express and The positive examples include the vector representation of the original corpus and the enhanced vector representation, which belong to the same category as them. The similarity between the sample vector representations of the same category can be improved, while the similarity between the sample vectors of different categories can be improved by minimizing the loss To reduce.

[0038] The present invention provides a cloze-style data augmentation method for small-sample intent recognition. Specifically, it proposes a data augmentation method suitable for short texts that does not require any knowledge-based input. We consider the pretrained language model itself as a knowledge base, trained on a large text corpus, and therefore capable of performing simple tasks. A cloze task similar in format to the pretrained language model is constructed for data augmentation to fully leverage the knowledge inherent in the pretrained language model. The model-restored hidden state vector of the "[MASK]" marker is used as the data augmentation result for the input sentence, rather than a true sentence consisting of a series of words. Furthermore, to ensure that the data augmentation result is not meaningless noise, an unsupervised learning method is used to make it semantically similar to the original input sentence. Subsequently, to maximize the use of the small number of samples in the meta-task, a supervised contrastive learning strategy is employed to bring samples of the same category closer together in the embedding space and samples of different categories further apart. This generates meaningful data without destroying syntactic structure or adding noise, fully utilizing limited data and obtaining separable embeddings. A better distance distribution is achieved in the embedding space, thereby improving the performance of metric-based classification methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a model overall framework diagram of a cloze-style data enhancement method for small sample intent recognition according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the performance of different templates of a cloze-style data augmentation method for small-sample intent recognition in the "5-way 1-shot" and "5-way 5-shot" meta-tasks on the CLINC-150 and BANKING-77 datasets. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0042] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in one or more embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] refer to Figure 1 and Figure 2 One or more embodiments of the present application provide a cloze-type data enhancement method for small-sample intent recognition, including constructing a cloze-type data enhancement task based on a pre-trained language model for intent recognition, first using an unsupervised learning method to make the data enhancement result semantically similar to the original input sentence, and then using a supervised contrastive learning method for metric-based classification, so that intent samples of the same category are closer to each other in the embedding space, and intent samples of different categories are farther away from each other, and the contrastive learning method includes prototype-level contrastive learning and instance-level contrastive learning.

[0044] As an optional embodiment, the unsupervised learning method includes: using a pre-trained language model as a feature extractor, using markers to mask a set proportion of input words, and predicting these masked markers based on the semantics of the context of the input sentence; the feature extractor encodes the sentence in the cloze-style intent recognition task into a hidden layer vector representation by adding two special markers, where the two special markers respectively represent the beginning and end of the sentence in the cloze-style intent recognition task. This process is expressed by the following formula:

[0045]

[0046] Among them, F(·) is the feature extractor, T is the cloze template, [MASK] is the mark of the masked word, Pat(T,x) is the data augmentation pattern, x is the input sentence, represents the hidden layer vector, [CLS] and [SEP] represent the start and end respectively;

[0047] The hidden layer vector As the representation of the masked word [MASK], It is regarded as a sentence representation generated according to the pattern Pat(T,x), which is semantically similar to the input sentence x. Formula (1) is repeated on all input samples to obtain the corresponding data enhancement results.

[0048] An auxiliary cloze template T is introduced to construct the data enhancement pattern Pat, as shown below:

[0049] T=The sentence:′__′means[MASK].#

[0050] Pat(T,x)=The sentence:′x′means[MASK]:#

[0051] As an optional embodiment, pre-trained language models do not always generate vectors that fully match the semantics of the input sentence. Therefore, a method is needed to constrain the model to reduce this mismatch and ultimately obtain appropriate data augmentation results. Without introducing any external knowledge or labels, an unsupervised learning method is designed that leverages the model's inherent semantic understanding capabilities to enable it to produce appropriate results as much as possible. The pre-trained language model is constrained using a loss function to reduce vectors that do not match the semantics of the input sentence, ultimately obtaining appropriate data augmentation results.

[0052] Without introducing any external knowledge and labels, the input sentence x is fed into the pre-trained language model to obtain its low-dimensional vector representation, which can be expressed as:

[0053]

[0054] Among them, the hidden layer vector Represented as the entire sentence x, through the following loss function:

[0055]

[0056] Used to reduce and The distance between them.

[0057] As an optional embodiment, after unsupervised cloze-style data augmentation, a metric-based prototype network is used as a classifier to test the effectiveness of data augmentation. The prototype network first calculates the average representation of samples in the same category and uses it as the prototype of the category:

[0058]

[0059] Among them, c i represents the prototype representation of category i, Ki Represents the support dataset under the current meta-task T The number of samples of category i in , is the representation of the kth sentence in category i; through this prototype representation, the average distance from the samples of the same category to its center can be the shortest; similarly, based on By using the prototype calculation formula (4), we can get the enhanced prototype c′ i .

[0060] Optionally, in order to make the final prototype cover the common features of its category more comprehensively, the prototype of the input sample and the prototype of the data augmentation result are weighted as follows:

[0061]

[0062] Where α is a weighting coefficient to control the respective contributions from the original input data and the enhanced data;

[0063] Given a score function s(·,·), the prototype network predicts the label x of the query instance by computing the softmax distribution of the similarity between the query embedding vector and the prototype Q , the formula is as follows:

[0064]

[0065] Among them, y is the predicted label, x Q Is the current meta-task The query instance in the query set Q, j is the true label, Denote the final prototype based on the initial and augmented data of category j, and choose the cosine similarity as s(·,·); learning is done by minimizing the negative log probability:

[0066]

[0067] Since the prototype network predicts labels by measuring the distance between the query instance and the prototype, a proper distance distribution is crucial to improve the intent recognition performance.

[0068] As an alternative embodiment, considering that the prototype is calculated using all samples of the corresponding category in the current meta-task, the prototype can represent the common characteristics of samples in this category. At the same time, considering that the prototype network is metric-based, an intuitive idea to improve classification accuracy is to increase the distance between prototypes of different categories in the embedding space.

[0069] Therefore, a loss based on contrastive learning is introduced for prototype-level learning in order to separate prototypes of different categories as much as possible. Specifically, the goal is to make the similarity of prototype embeddings of different categories as small as possible, which can be expressed as:

[0070]

[0071] Among them, s(·,·) is the same similarity measurement function as in the formula, so s(c i , c i ) is a constant 1, simplifying formula (4) to the following form:

[0072]

[0073] Where e is a constant, is the prototype-level contrastive loss. We expect prototypes of different categories to be distant from each other. However, performing contrastive learning directly at the prototype level only distances the average representations of different categories from each other. This approach does not guarantee that samples of the same category are close to each other, and thus does not sufficiently improve intent recognition accuracy.

[0074] As an optional embodiment, to further improve the performance of intent recognition, instance-level contrastive learning is introduced. This strategy not only keeps instances of different categories away from each other, but also brings instances of the same category closer together. The specific formula is as follows:

[0075]

[0076] in, express and The positive examples include the vector representation of the original corpus and the enhanced vector representation, which belong to the same category as them. The similarity between the sample vector representations of the same category can be improved, while the similarity between the sample vectors of different categories can be improved by minimizing the loss To reduce.

[0077] In order to verify the technical effect of the embodiment, the following experiments were designed:

[0078] Experimental setup

[0079] Two public intent recognition datasets, CLINC-150 and BANKING-77, are used to evaluate the capabilities of our model and the baseline models in the discussion. CLINC-150 consists of 150 intent categories from 10 daily life domains, with 150 samples in each category. In addition, there are some intent sentences in the dataset that are marked as "out of scope", which are considered to be noise with multiple unknown categories. In order to accurately test the performance of the discussed model, we deleted these samples marked as "out of scope" and only used well-labeled samples for training and testing. BANKING-77 is a single-domain dataset for intent recognition, containing 13,083 samples from 77 categories in the banking domain. Table 1 provides statistics for CLINC-150 and BANKING-77.

[0080] Table 1 Statistics of datasets CLINC-150 and BANKING-77

[0081]

[0082] Model Summary

[0083] The effectiveness of our proposed model is verified by comparing it with the following baseline models:

[0084] Prototypical Networks: A metric-based few-shot classification model that uses the distance between samples in the embedding space to measure their similarity. It considers the prototype label closest to the query sample as its category prediction.

[0085] GCN: A graph convolutional network-based method for few-shot classification, which treats few-shot learning as a supervised information transfer task and can be trained in an end-to-end manner.

[0086] Matching Networks: A few-shot classification framework that trains a network to map a small support set of labeled and unlabeled instances to their labels and avoids relying on fine-tuning to adapt to new categories.

[0087] Problem Setting

[0088] RQ1: Can our proposed method outperform these competitive baseline models on the intent recognition task?

[0089] RQ2: Which module of our proposed CDA plays a greater role in improving recognition accuracy?

[0090] RQ3: What is the impact of different templates on model performance?

[0091] Model Settings

[0092] Following the common practice of few-shot learning experiments, we discuss two meta-tasks with different numbers of samples, including "5-way 1-shot" and "5-way 5-shot". For all models involved in the discussion, we apply the same feature extractor (i.e., bert-base-uncased) to encode the input sentences to ensure fair performance comparison.

[0093] Based on RQ1, we examine the intent recognition capabilities of the two meta-tasks on CLINC-150 and BANKING-77. The overall intent recognition performance of all discussed models is shown in Table 2.

[0094] Table 2. Overall performance (%) and 95% confidence intervals for the two types of meta-tasks on the test set. The results produced by the best performer in each column are bolded. The results produced by the best baseline are underlined.

[0095]

[0096]

[0097] First, we can see that regardless of the dataset, all models perform better on the meta-task when the number of individual category samples is large. This is because as the number of individual category samples increases, the total number of samples available to the model also increases, and the common features obtained from the samples are closer to the true common features.

[0098] Next, we focus on analyzing the performance of the baselines. We find that MatchNet achieves the highest accuracy in the "5-way 1-shot" meta-task on both datasets, while ProtoNet achieves the best performance in the "5-way 5-shot" meta-task on both datasets. MatchNet's advantage in the "1-shot" meta-task can be explained by the fact that temporary similarity matching calculations significantly improve model performance. For ProtoNet, its advantage in the "5-shot" meta-task is due to its ability to fuse features from instances in the same category to capture their commonalities.

[0099] Next, we focus on the performance of our proposed model. Comparing baseline models with the CDA model, we find that on the CLINC-150 dataset, CDA-PC and CDA-IC outperform almost all baseline models discussed. However, on the BANKING-77 dataset, CDA-PC performs weaker than MatchNet and ProtoNet on the "5-way 1-shot" and "5-way 5-shot" meta-tasks, respectively. This is because samples from the same category in the CLINC-150 dataset are short sentences and more similar than those in BANKING-77. Therefore, the data augmentation results are very close to the initial input sentences, which helps the model capture their common features. Furthermore, because the BANKING-77 dataset is more specialized than CLINC-150, the pre-trained language model has less relevant knowledge than the CLINC-150 dataset. Directly using augmented samples to calculate category prototypes would be equivalent to introducing noise, which would weaken the characteristics of the category itself and thus reduce recognition performance.

[0100] To address the challenges of CDA-PC applications, CDA-IC utilizes an instance-level contrastive learning strategy to improve small-sample intent recognition performance. The advantage of CDA-IC can be explained by its instance-level contrastive learning strategy, which treats both the initial data and the corresponding augmented data of the same category as positive examples. This approach allows each sample to interact with more data, not only shortening the spatial distance between initial input data of the same category but also making the augmented data semantically closer to the original data of the same category.

[0101] On the CLINC-150 dataset, CDA-IC achieves a 4.36% improvement in accuracy on the "5-way 1-shot" meta-task and a 4.91% improvement on the "5-way 5-shot" meta-task compared to the best baseline model. On the BANKING-77 dataset, the accuracy on the "5-way 1-shot" meta-task improves by 1.69% and the accuracy on the "5-way 5-shot" meta-task improves by 1.86%.

[0102] Ablation experiments

[0103] To answer RQ2, we analyze the importance of different modules in our CDA-IC model by removing the two basic components of CDA-IC, namely the instance-level contrastive learning module and the unsupervised learning module. The results of the ablation experiment are shown in Table 3:

[0104] Table 3 Ablation experiments of CDA-IC on the “5-way 1-shot” and “5-way 5-shot” meta-tasks on the CLINC-150 and BANKING-77 datasets. The independent module with the largest reduction in each column is then used mark.

[0105]

[0106] Clearly, removing any component of CDA-IC leads to performance degradation, demonstrating the crucial role of the unsupervised learning module and the instance-level contrastive learning module in improving few-shot intent recognition. In particular, across both meta-task types, removing the instance-level contrastive learning module leads to the most severe performance degradation, regardless of the dataset. For example, on the CLINC-150 dataset, the CDA-IC model without the instance-level contrastive learning module experiences a 3.63% and 3.82% performance drop on the "5-way 1-shot" meta-task and the "5-way 5-shot" meta-task, respectively. On the BANKING-77 dataset, the CDA-IC model without the instance-level contrastive learning module achieves a 4.16% and 4.44% performance drop on the "5-way 1-shot" meta-task and the "5-way 5-shot" meta-task, respectively.

[0107] Furthermore, it is noteworthy that each module has its own unique contribution. Specifically, in the "5-way 1-shot" meta-task, removing the unsupervised learning module results in a greater performance drop than in the "5-way 5-shot" meta-task. This suggests that when features are insufficient, the unsupervised learning module plays a more significant role and is more conducive to improving small-sample intent recognition performance. Furthermore, in the "5-way 5-shot" meta-task, the instance-level contrastive learning module plays a more significant role than in the "5-way 1-shot" meta-task. This phenomenon can be explained by the fact that, in this case, the bottleneck limiting performance is no longer the lack of features, but rather the exploitation of commonalities within the same category and uniqueness across different categories. The instance-level contrastive learning module not only shortens the distance between samples of the same category in the embedding space, but also increases the distance between vector representations of different categories, thereby exploiting commonalities within the same category and uniqueness across different categories.

[0108] The impact of different templates

[0109] To answer RQ3, we design three different templates and apply them to data augmentation. All discussed template types are listed in Table 4.

[0110] Table 4 Three templates for cloze data augmentation

[0111]

[0112] Since our proposed model is based on a pre-trained language model, it needs to use templates to generate semantically similar data. Because different templates use different words and punctuation marks, i.e. tokens, the semantic vectors obtained from the pre-trained language model are also different. For details, see Figure 2 .

[0113] Figure 2 It shows the performance of different templates in the "5-way 1-shot" and "5-way 5-shot" meta-tasks on the CLINC-150 and BANKING-77 datasets.

[0114] Clearly, it shows that different templates do lead to significant changes in model performance. Specifically, in the "5-way 1-shot" meta-task performed on the CLINC-150 dataset, the performance difference between different templates is close to 1%. In addition, as Figure 2 As shown in Figure 5, on the “5-way 5-shot” meta-task performed on the BANKING-77 dataset, the performance difference between different templates can even reach 1.3%.

[0115] Judging from the overall trend, the length of the template has no direct relationship with the effect of data increment. In detail, although Template 2 is the shortest, its performance on the CLINC-150 dataset is not the worst. Its performance on the "5-way 1-shot" meta-task is very close to that of Template 3 and better than Template 1. It is worth noting that compared with the other two templates, Template 3 performs best on all tasks of the CLINC-150 and BANKING-77 datasets. This phenomenon can be explained by the fact that Template 3 has the clearest semantic guidance for the [MASK] tag. When the original input sentence is filled in the template, Template 3 clearly states that [MASK] represents the intention of the input sentence. Therefore, the generated semantic embedding vector is more directional.

[0116] In summary, the design of the template has a significant impact on the performance of data augmentation. A good template can provide appropriate semantic guidance and effectively improve the performance of data augmentation.

[0117] We propose a Cloze-style Data Augmentation (CDA) model for few-shot intent recognition. Inspired by the task of pre-training language models, we design a template-based unsupervised data augmentation strategy, hoping to produce meaningful data without destroying the syntactic structure and adding noise. Furthermore, to make full use of the limited data and obtain separable embeddings, we perform contrastive learning between the original data and the augmented data. As a result, each sample can interact with samples from all remaining categories, thereby distinguishing the embeddings of different categories in the embedding space. Experimental results on the CLINC-150 and BANKING-77 datasets illustrate the effectiveness against all discussed baselines. In addition, ablation studies show that the contrastive module is the most important component of the entire model.

[0118] It will be understood by those skilled in the art that the above-mentioned device may only include components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figures.

[0119] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the above embodiments or technical features in different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0120] The one or more embodiments of this application are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this application should be included within the scope of protection of this disclosure.

Claims

1. A cloze-style data augmentation method for small sample intent recognition, characterized by: include: A cloze-style data augmentation task is constructed based on a pre-trained language model for intent recognition. Unsupervised learning methods are first used to make the data augmentation result semantically similar to the original input sentence. Then, supervised contrastive learning methods are used based on metric classification to make intent samples of the same category closer to each other in the embedding space and intent samples of different categories farther away from each other. Contrastive learning methods include prototype-level contrastive learning and instance-level contrastive learning. The unsupervised learning method includes: using a pre-trained language model as a feature extractor, using markers to mask a set proportion of input words, and predicting these masked markers based on the semantics of the input sentence context; the feature extractor encodes sentences in a cloze-style intent recognition task into hidden layer vector representations by adding two special markers, where the two special markers respectively represent the beginning and end of the sentence in the cloze-style intent recognition task. This process is expressed by the following formula: Where F(·) is the feature extractor, T is the cloze template, [MASK] is the mark of the masked word, Pat(T,x) is the data augmentation pattern, x is the input sentence, represents the hidden layer vector, [CLS] and [SEP] represent the start and end respectively; The hidden layer vector As the representation of the masked word [MASK], It is considered as a sentence representation generated according to the pattern Pat(T,x), which is semantically similar to the input sentence x. Formula (1) is repeated on all input samples to obtain the corresponding data enhancement results. Without introducing any external knowledge and labels, the input sentence x is fed into the pre-trained language model to obtain its low-dimensional vector representation, which can be expressed as: Among them, the hidden layer vector Represented as the entire sentence x, through the following loss function: Used to reduce and The distance between them.

2. The cloze-type data augmentation method for small sample intent recognition according to claim 1, characterized in that: The pre-trained language model is constrained by the loss function to weaken the vectors that do not match the semantics of the input sentence, and ultimately obtain appropriate data enhancement results.

3. The cloze-type data enhancement method for small sample intent recognition according to claim 1, characterized in that: After completing the unsupervised cloze-style data augmentation, a metric-based prototype network is used as a classifier to test the effect of data augmentation.

4. The cloze-type data enhancement method for small sample intent recognition according to claim 3, characterized in that: First, calculate the average representation of samples in the same category and use it as the prototype of the category: Among them, c i represents the prototype representation of category i, K i Represents the support dataset under the current meta-task T The number of samples of category i in , Represents the representation of the kth sentence in category i; through this prototype representation, the average distance from the samples of the same category to its center can be the shortest; similarly, based on Through the prototype calculation formula (4), the enhanced prototype c can be obtained i ′ .

5. The cloze-type data enhancement method for small sample intent recognition according to claim 4, characterized in that: In order to make the final prototype cover the common features of its category more comprehensively, the prototype of the input sample and the prototype of the data enhancement result are weighted as follows: Where α is a weighting coefficient to control the respective contributions from the original input data and the enhanced data; Given a score function s(·,·), the prototype network predicts the label x of the query instance by computing the softmax distribution of the similarity between the query embedding vector and the prototype Q , the formula is as follows: Among them, y is the predicted label, x Q Is the current meta-task Query set The query instance in , j is the true label, Denote the final prototype based on the initial and augmented data of category j, and choose the cosine similarity as s(·,·); learning is done by minimizing the negative log probability: to proceed.

6. The cloze-type data enhancement method for small sample intent recognition according to claim 1, characterized in that: A contrastive learning-based loss is introduced for prototype-level contrastive learning, so as to separate prototypes of different categories as much as possible and make the average representations of different categories far away from each other, which is expressed by the following formula: Among them, s(·,·) is the same similarity measurement function as in the formula, so s(c i ,c i ) is a constant 1, simplifying formula (4) to the following form: Where e is a constant, It is the contrast loss of prototype level.

7. The cloze-type data enhancement method for small sample intent recognition according to claim 6, characterized in that: Based on the prototype-level contrastive learning, we introduce instance-level contrastive learning to make instances of the same category close to each other, which can be expressed by the following formula: in, express and The positive examples include the vector representation of the original corpus and the enhanced vector representation, which belong to the same category as them. The similarity between the sample vector representations of the same category can be improved, while the similarity between the sample vectors of different categories can be improved by minimizing the loss To lower.

Images