A text data optimization method for small sample intent recognition

Abstract

The present invention provides a text data optimization method for small sample intent recognition, comprising the following steps: step 1, constructing a training text data set; step 2, dividing the training tasks into levels in a stepwise manner according to the number of intents they have; step 3, sampling a batch of small sample intent recognition training samples; step 4, using two or more tasks sampled from the same batch for parallel training on a meta-learning model based on metric learning; step 5, judging whether the training is terminated; step 6, ending the model training. The present invention can apply the parallelized training meta-learning model to the situation where the number of intents of training tasks for small sample intent recognition in real scenarios is inconsistent.

Description

Technical Field

[0001] The present invention belongs to the field of intent recognition in natural language processing, and in particular, relates to a text data optimization method for small sample intent recognition. Background Art

[0002] Intent Detection is one of the key technologies for building a human-computer interactive dialogue system. Intent detection means that a computer program can analyze and process the semantic information contained in the input dialogue text and finally determine the intent category to which it belongs.

[0003] In the current popular human-computer dialogue system platform, users are often required to create new dialogue tasks and provide relevant annotated data. However, due to the high cost of data annotation, many users are unable to provide a large amount of annotated text, and each intent often has only a dozen or even a few samples. In this case, the intent recognition task belongs to few-shot intent detection, which needs to be treated as a few-shot learning problem.

[0004] Conventional machine learning and deep learning methods often suffer from overfitting under small data and cannot meet usage requirements. Meta-learning is a machine learning method used to solve the problem of small sample learning. The goal of meta-learning is to train neural network models on various learning tasks so that the model can solve new learning tasks using only a small number of training samples. The various learning tasks used by meta-learning when training models are usually also small sample learning tasks, which are often set as classification tasks with the same number of classification categories in scientific research. When meta-learning is applied to small sample intent recognition, each training task in the training phase is a small sample intent recognition task.

[0005] However, due to the particularity of the real-world application scenarios of small-sample intent recognition, the number of intents, that is, the number of classification categories, of each intent recognition task is usually inconsistent. Therefore, when meta-learning is applied to small-sample intent recognition tasks, parallel training cannot be performed due to the different numbers of intents in the training tasks in the training set.

[0006] At present, there is a problem of inconsistent number of intents in intent recognition tasks when using meta-learning training. The first type of method is to focus only on the ideal situation where the number of training tasks is the same, in which case parallel training can be performed naturally. The second type is to uniformly adopt a serial training method for training tasks with different numbers of intents, and only train a single task in each step.

[0007] [Paper 1] Model-Agnostic Meta-Learning for Fast Adaptation of DeepNetworks.

[0008] The training tasks used for meta-learning in this paper are a set of subcategories with a fixed number of samples from all categories of a classification task with a larger number of categories. Training tasks with the same number of categories can be naturally trained in parallel.

[0009]

Paper 2

[0010] The paper contains experiments on the small sample intent recognition dataset ODIC, which integrates all the intents in the training set into a unified set, and samples 5 intents from the total intent set as a training task in each training step. This method is the same as the paper 1 and can also be naturally trained in parallel.

[0011] [Paper 2] FewJoint: A Few-shot Learning Benchmark for Joint Language Understanding.

[0012] The training data of this paper comes from real-life scenarios. The total number of intents in the training tasks of small sample intent recognition varies, ranging from 1 to 7. The training method used in this paper is to train only a single sample of a single task in each step, which is a serial training method.

[0013] Paper 1 represents an idealized meta-learning application scenario in the study, in which all training tasks have the same number of categories. However, in actual scenarios, due to differences in scenarios and the high cost of sample annotation, the number of classification categories of the labeled tasks available for training is often different. Therefore, the ideal solution represented by Paper 1 cannot be used in some real-world scenarios.

[0014] Paper 2 integrates all the intents of the existing annotated data into a total intent set and then samples intents from it to generate intent recognition tasks for training. This method does not take into account the impact of the intents in different original scenarios or tasks forming an intent recognition task on the model learning results. For example, if a sentence input a belongs to intent A in scenario 1 and intent B in scenario 2, if intent A and intent B are sampled to form a single intent recognition task, it will be contrary to the actual data distribution to still use the original annotations. On the other hand, the intents in the small sample intent recognition task sampled by this method may not come from the same field or scenario at all. In other words, although the training method of Paper 2 can be parallel, there will be differences between the data distribution learned by its training model and the actual data distribution, which is not conducive to the final performance of the model in actual scenarios.

[0015] Paper 3 uses a serial training method for training tasks. If only serial training can be used in model training, on the one hand, the model training cycle will take several times longer than parallel training, which is not conducive to production iteration; on the other hand, the computing power of GPUs has been improved to a very high level in recent years, and the inability to conduct parallel training may also cause a serious waste of existing computing power. Summary of the invention

[0016] Purpose of the invention: The technical problem to be solved by the present invention is that meta-learning cannot be directly parallelized when the number of intents in the training tasks of small sample intent recognition in real scenarios is inconsistent. The present invention divides the training tasks into different levels according to the number of intents they have, and then samples the same number of intents for the training tasks of the same level in the following manner to perform parallel training. Ultimately, the above problem is solved to a certain extent.

[0017] The present invention specifically provides a text data optimization method for small sample intent recognition, comprising the following steps:

[0018] Step 1: Build a training text dataset;

[0019] Step 2: The training tasks are divided into different levels according to the number of intentions they have.

[0020] Step 3, sampling a batch of small sample intention recognition training samples;

[0021] Step 4: Perform parallel training on two or more tasks sampled from the same batch on the meta-learning model based on metric learning;

[0022] Step 5, determine whether the training is terminated;

[0023] Step 6: End model training.

[0024] Step 1 includes: defining a training text dataset as S = {T1, T2, ..., T n}, where T i is the i-th small sample intention recognition task in the real dialogue scene, n is the total number of training tasks in S, and i ranges from 1 to n;

[0025] Each small-shot intent recognition task is defined as Where Intent ij T i Corresponding to an intention in a dialogue scenario, C i T i The number of intents contained;

[0026] For each task an intent is defined as Where quxery ijk To be marked as Intent ij A conversation text of N ij Intent ij The total number of annotation texts included.

[0027] Step 2 includes: constructing a level list levels = {L1, L2, ..., L M}, L M Indicates the Mth level;

[0028] The maximum level of the number of intentions a training task has is the level of the training task. i The number of intentions contained C i ≥L l And C i <L l+1 , then T i The corresponding level is L l , if C i ≥L M Then T i The corresponding level is L M ;

[0029] According to each task T i The number of intentions contained divides all tasks in the training text dataset S into different subsets, that is, constructs the partition of the training text dataset S in Indicates that it corresponds to level L M A subset of .

[0030] Step 3 includes:

[0031] Step 3-1, determine the level L of the current batch to be sampled l ;

[0032] Step 3-2, at Level L l A collection of tasks Sample a task T i ;

[0033] Step 3-3, in Task T i Sample L l intention;

[0034] Step 3-4, for L l intents, and k dialogue texts are sampled for each intent;

[0035] Step 3-5, determine whether the sampling of this batch is completed: if the corresponding batch size of the level has been sampled training task samples, execute step 3-6, otherwise return to step 3-2 to continue sampling;

[0036] Step 3-6, return the training data sampled from the current batch: The training data D of the current batch contains tasks, each task contains L l intents, each of which contains k dialogue texts labeled as the intent;

[0037] Split the training data D into support set D support and query set D query The splitting method is to split the k dialogue texts of each intent under the training data D into k support and k query Two subsets of conversation texts, where k = k support +k query .

[0038] In step 4, the meta-learning model based on metric learning includes an encoder and a similarity metric function.

[0039] Step 4 specifically includes the following steps:

[0040] Step 4-1, use the encoder to encode the input text: for the support set D support and query set D query All the dialogue texts in , use the text encoder to encode each dialogue text into a vector representation of a specific dimension. In the text encoding process, the support set D support The data representation shape in is L l , k support , sen_len, word_emb], where sen_len is the maximum length of the input text limited by the encoder, and word_emb is the dimension size of the vector representation corresponding to each word;

[0041] After being encoded by the encoder, we get where sen_emb is the dimension of the sentence vector representation of each dialogue text; for the query set D query The encoding shape is Data representation;

[0042] Step 4-2, query set sentence representation and support set D support The intention is to measure the similarity:

[0043] Use the similarity measurement function to calculate the similarity between two vector representations;

[0044] The support set D support The sentence vectors corresponding to each intent in are integrated into a single intent vector, that is, converted into Where intent_emb is the k support The average value of the sen_emb calculations is obtained to obtain a prototype representation for each intent in each task;

[0045] The query set D query The sentence vector representation of all dialogue texts of each task in is calculated similarly to the prototype representation of the intent under the corresponding task, and the similarity result is The penultimate dimension L l That is, each dialogue text is predicted to be the score of each intent in its source task. l The vector composed of scores is abbreviated as pred;

[0046] Step 4-3, calculate the loss: use the multi-classification cross entropy function as the loss function loss, that is, loss = Cross_Entropy(label, pred), where loss is the loss value, label is the label vector, and Cross_Entropy() is the cross entropy function;

[0047] Use the similarity results predicted in step 4-2 to perform cross entropy calculation with the actual label to obtain the prediction loss value of the model. For the query set D in the training phase query Each conversation text in is represented as a one-hot vector as the annotation vector, and the one-hot vector dimension is L l The true intention of each dialogue text is marked as 1, and the rest are marked as 0;

[0048] Step 4-4, gradient regression: Based on the loss calculated in step 4-3, use the back propagation algorithm to calculate the gradient and perform reverse regression to update the model;

[0049] Step 4-5, single-step training is completed.

[0050] In step 4-2, the similarity between two vector representations is calculated using the following formula:

[0051] score ab =F(sen_emb a , sen_emb b )

[0052] where sen_emb a and sen_emb b is the sentence vector representation of two sentences, F() is the similarity measurement function, score ab is the final calculated similarity score of the two input texts.

[0053] In step 4-2, the similarity measurement function is a layer of linear neural network. After the two sentence vectors are concatenated and passed into the linear neural network, a decimal between 0 and 1 is obtained after the activation layer and the softmax() function is calculated. Finally, the vector pred is a vector in which each component is positive and the sum of each component is 1. The specific formula of the single-layer linear neural network is:

[0054] score ab =relu(Wx ab +bias)

[0055] Among them, relu() is the rectified linear unit activation function, W is the weight matrix, bias is the bias vector, and x ab It is the vector after the concatenation of sentence vectors a and b.

[0056] In step 4-2, the cross entropy function is denoted as L cross_entropy , the calculation formula is as follows:

[0057]

[0058] Where N is the number of dialogue texts in the query set, N = L l *k query ; L l is the intention number; y ic is a symbolic function, taking values ​​of 0 or 1. It is 1 when the dialogue text i belongs to the intention c, otherwise it is 0, that is, the component of the vector label corresponding to the position of the intention c; p ic is the probability that dialogue text i is predicted as intent c.

[0059] Step 5 includes: judging whether the model training is terminated according to the preset termination condition, and executing step 6 if it is terminated, otherwise returning to step 3 to continue training.

[0060] Beneficial effects: From a technical perspective, the method of the present invention downsamples the task intents belonging to the same intent number level by first dividing the levels, so that the training tasks with different intent numbers used in meta-learning can also be trained in a parallel manner. At the same time, the training tasks with different intent numbers are divided into levels in a step-by-step manner according to the number of intents, which ensures that the downsampling ratio is not too small, thereby avoiding the problem of low utilization of training data and insufficient learning caused by a low downsampling ratio.

[0061] From the application level, the present invention can apply the parallel training meta-learning model to the situation where the number of intentions in the training task of small sample intent recognition in real-life scenarios is inconsistent. The present invention changes the model training method from serial to parallel, which can greatly reduce the time cost of training the model and make full use of existing computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0063] Figure 1 It is a flow chart of the method of the present invention.

[0064] Figure 2 It is a flow chart for sampling a batch of training data.

[0065] Figure 3 This is a single-step parallel computation flowchart of the model based on metric learning.

[0066] Figure 4 It is a schematic diagram of the text encoder calculation process. DETAILED DESCRIPTION

[0067] like Figure 1 As shown, the present invention provides a text data optimization method for small sample intent recognition, comprising:

[0068] Step 1: Construct a training text dataset. Define the training text dataset as S = {T1, T2, ..., T n}, where T i is a small sample intent recognition task in a real dialogue scenario, and n is the total number of training tasks in S. Each small sample intent recognition task is defined as Where Intent ij T i Corresponding to an intention in a dialogue scenario, C i T i The number of intentions contained. In particular, for different T p and T q, which correspond to the number of intentions C contained p and C q Not necessarily the same. For each task, an intent is defined as Where quxery ijk To be marked as Intent ij A conversation text of N ij Intent ij The total number of annotation texts included is shown in Table 1:

[0069] Table 1

[0070]

[0071] Among them, "Mobile Application", "Geographic City" and "Family Names" are the three small sample intent recognition tasks in the data set used in this embodiment, which can be recorded as T1, T2 and T3 respectively. It can be seen that "Mobile Application" contains three intents, "Geographic City" contains two intents, and the "Family Names" task contains four intents. The three intents "Download", "Start" and "Query" contained in "Mobile Application" can be recorded as Intent 1_1 , Intent 1_2 and Intent 1_3 In the dataset, each intent of each task has several annotated dialogue texts, such as "download dictionary software" is annotated as Intent 1_1 A dialogue text, which can be recorded as quxery 1_1_1 .

[0072] Step 2: Divide the training tasks into levels in a step-by-step manner according to the number of intentions they have. First, construct a level list, i.e., levels = {L1, L2, ..., L M The maximum value of the number of intentions in a training task is the level of the training task. i The number of intentions contained C i ≥L l And C i <L l+1 , then T i The corresponding level is L l , if C i ≥L M Then T i The corresponding level is L M . So we can use each task T i The number of intentions contained in S divides all tasks in S into different subsets, that is, constructs the partition of S in Indicates that it corresponds to level LM According to the long-tail distribution of the number of training task intentions and the characteristics of computer computing media, the level list used in this embodiment consists of powers of 2: {2, 4, 8, 16, 32, 64}, that is, the entire data set is divided into 6 mutually exclusive subsets S2, S4, S8, S9, S10, S11, S12, S13, S14, S15, S16, S17, S18, S19, S20, S21, S22, S23, S24, S25, S26, S27, S28, S29, S30, S31, S32, S33, S34, S35, S36, S37, S38, S39, S40, S41, S42, S43, S44, S45, S46, S47, S48, S49, S50, S51, S52, S53, S54, S55, S56, S57, S58, S59, S60, S61, S62, S63, S64, S65 16 , S 32 and S 64 In the example described in step 1, "mobile phone applications" and "geographic cities" are classified into level 2, namely S2, and "family names" are classified into level 4, namely S4.

[0073] Step 3: Sample a batch of small sample intention recognition training samples. Sample multiple training tasks to form a batch as training samples for single-step model training. The training between the training tasks in this batch is parallel training. The specific sampling process is as follows: Figure 2 As shown:

[0074] Step 3-1, determine the level L of the current batch to be sampled l The training tasks of a sampled batch all belong to the same level. Therefore, before performing actual task sampling, the number of levels L used in the current batch must be determined. l The specific determination method may be to use all levels in sequence in a circular manner, or to determine the level by weight according to the number of tasks included in the level. This embodiment adopts the latter.

[0075] Step 3-2, at Level L l A collection of tasks Sample a task T i . From all the L l A collection of tasks A specific task T is sampled by weighted sampling i The weight is calculated based on the number of intentions C each task has. i .

[0076] Step 3-3, in Task T i Sample L l Intentions. Defined by the level division, the task T can be determined i At least L l Intention. For task T i All the intents contained in the data can be weighted according to the number of dialogue texts contained in each intent, and the intents can be weighted by weighted sampling. l For example:

[0077] In a certain step of the training process of this embodiment, the level to be sampled is 2, and the task "Mobile Application" listed in step 1 is sampled from set S2. "Mobile Application" contains 3 intents. In order to be able to train in parallel with other tasks, 2 intents need to be sampled from the 3 intents of "Mobile Application" as training data for the current step.

[0078] Step 3-4, for this L l In meta-learning, when training on a single training task, a limited number of labeled samples are generally used, and the number of labeled samples used is k. If task L l Intent ij Number of annotation texts contained N ij If N is greater than k, k items are sampled without replacement. If the annotated text is really limited, N ij If k is less than k, sampling with replacement can be used. In this embodiment, k is 10 for subsequent support set and query set use.

[0079] Step 3-5, determine whether the sampling of this batch is completed. If the corresponding batch size of the level has been sampled training task samples, execute steps 3-6, otherwise return to step 3-2 to continue sampling. The batch size corresponding to each level It can be specifically set according to the level size and GPU memory. In this embodiment, the batch size corresponding to the level list is {256, 64, 16, 4, 1}.

[0080] Step 3-6, return the training data sampled from the current batch. The training data D of the current batch contains tasks, each task contains L l Intents, each of which contains k dialogue texts labeled as the intent. Meta-learning training requires the training data to be divided into a support set and a query set. Therefore, the training data D needs to be split into a support set D support and D query The splitting method is to split the k dialogue texts of each intention under D into k support and k quxery Two subsets of conversation texts, where k = k support +k quxery In this embodiment, k support and k quxery All of them are 5, that is, for each intent, the 10 labeled texts sampled in steps 3-4 are assigned to the support set D support 5 annotated texts, query set D query There are 5 marked texts.

[0081] In the example described in step 1, the two tasks "Mobile Application" and "Geographic City" cannot be directly trained in parallel at the same time because they contain different numbers of intents. However, after being divided into the same level set S2 in step 2, when they are sampled at the same time in step 3, they can be trained in parallel in the subsequent steps.

[0082] Step 4: Use two or more tasks sampled from the same batch to perform parallel training on the meta-learning model based on metric learning. The meta-learning model based on metric learning consists of two parts: an encoder and a similarity measurement function. The encoder encodes the input text into a vector representation, and the similarity measurement function determines the degree of similarity between the vector representation of the unlabeled text and the labeled text. This step will use the data sampled in step 3 to perform single-step training on the model. The actual training of the model involves the forward calculation and back propagation of the neural network. The specific single-step calculation process of the model is as follows: Figure 3 shown.

[0083] Step 4-1, use the encoder to encode the input text. support and D query For all the conversation texts in the collection, a text encoder is used to encode each conversation text into a vector representation of a specific dimension. In the text encoding process, each word in each conversation text corresponds to a unique dictionary number, through which each word corresponds to a trainable word vector representation, that is, Where x represents a word, id() is a dictionary function, is the word vector mapping function. Support set D support The data representation shape in is Where sen_len is the maximum length of the input text limited by the encoder, and word_emb is the dimension size of the vector corresponding to each word. After encoding by the encoder, we get where sen_emb is the dimension of the sentence vector representation of each dialogue text. Similarly, for the query set D query The encoding is as follows In this embodiment, the pre-trained language model Bert is used to encode the dialogue text, such as Figure 4 As shown. The Bert model takes the original characters of the dialogue text as input, inserts a [CLS] character before each input text, and finally uses the output vector corresponding to the [CLS] character after integrating the semantic information of the whole sentence as the sentence vector representation of each dialogue text. The maximum input text length sen_len used in this embodiment is 30, the word vector size word_emb is 256, and the sentence vector size sen_emb is 256. When the current step sampling is level 2, the corresponding batch size is 256. For the support set Dsupport For example, the input of the encoder is 2560 (256*2*5) annotated texts, that is, a tensor with a shape of [256, 2, 5, 30, 256]; the output is the corresponding 2560 sentence vectors, that is, a tensor with a shape of [256, 2, 5, 256].

[0084] Step 4-2: measure the similarity between the query set sentence representation and the support set intent representation. The similarity measurement uses a function to calculate the similarity between the two vector representations, namely score. ab =F(sen_emb a , sen_emb b ), where sen_emb a and sen_emb b is the sentence vector representation of two sentences, F() is the similarity measurement function, score ab is the final calculated similarity score of the two input texts, which is a decimal between zero and one. First, we need to integrate the multiple sentence vectors corresponding to each intent in the support set into a single intent vector, that is, convert it into Where intent_emb is the k support The sen_emb is calculated by averaging, and a prototype representation of each intent in each task is obtained. Then the sentence vector representation of all dialogue texts of each task in the query set is calculated with the prototype representation of the intent under the corresponding task. The similarity result is as follows: The penultimate dimension L l That is, each dialogue text is predicted to be the score of each intent in its source task. l The similarity score vector composed of scores can be abbreviated as pred. The similarity measurement function used in this embodiment is a one-layer linear neural network. After the two sentence vectors are concatenated and passed into the neural network, a decimal between 0 and 1 is obtained after the activation layer and the softmax() function is calculated. Finally, the vector pred is a vector in which each component is positive and the sum of each component is 1. The specific formula of the single-layer linear neural network used in this embodiment is:

[0085] score ab = re1u(Wx ab +bias)

[0086] Among them, relu() is the rectified linear unit (ReLU) activation function, W is the weight matrix, bias is the bias vector, and x ab is the concatenation of sentence vectors a and b. W and bias are the parameters to be trained.

[0087] Step 4-3, calculate the loss. This embodiment uses multi-classification cross entropy as the loss function, that is, loss = Cross_Entropy(label, pred), where loss is the loss value, label is the label vector, pred is the similarity score vector predicted in step 4-2, and Cross_Entropy() is the cross entropy function. Use the similarity score predicted in step 4-2 and the true label to perform cross entropy calculation to obtain the predicted loss value of the model. For each conversation text in the query set during the training phase, it is represented as a one-hot vector, that is, a vector with only one bit being 1 and all other bits being 0 as the label vector. The one-hot vector dimension is L l The true intention corresponding to each dialogue text is marked as 1, and the rest are marked as 0. Therefore, the data representation shape of the true label is the same as the similarity shape, and the cross entropy calculation can be performed. The specific formula of the cross entropy function is:

[0088]

[0089] Where N is the number of query set dialogue texts, N = L l *k query ; L l is the intention number; y ic is a symbolic function, taking values ​​of 0 or 1. It is 1 when the dialogue text i belongs to the intention c, otherwise it is 0, that is, the component of the vector label corresponding to the position of the intention c; p ic is the probability that dialogue text i is predicted as intent c, that is, the component corresponding to the position of intent c in vector pred.

[0090] For example:

[0091] For the annotated text "What cities are there in Jiangsu Province" mentioned in step 1, according to its intended meaning, as shown in Table 2:

[0092] Table 2

[0093] text Intent: contains city query Intent: Query the province Which cities are there in Jiangsu Province? 1 0

[0094] Among them, 1 indicates that the corresponding text belongs to the intent, and 0 indicates that the corresponding text does not belong to the intent.

[0095] The label vector label of "What cities are there in Jiangsu Province" is a one-hot vector [1, 0] of length 2. In a certain step of the training process of this embodiment, the similarity score vector pred of "What cities are there in Jiangsu Province" output by the model is [0.7, 0.3]. The loss value calculated by the cross entropy function for only this label text is 0.07745. During the training process, the loss values ​​calculated for multiple label texts trained in parallel are averaged and used as the training loss value for a single step.

[0096] Step 4-4, gradient regression. According to the loss calculated in step 4-3, the back propagation algorithm is used to calculate the gradient and perform back regression, thereby updating the parameters of the encoder and the similarity metric function. During the gradient back propagation process, the learning rate of the neural network can be dynamically adjusted according to the training process. In this embodiment, the optimization algorithm Adam algorithm is used.

[0097] Step 4-5, single-step training is completed.

[0098] Step 5, determine whether the training is terminated. Determine whether the model training is terminated according to the preset termination conditions. If it is terminated, execute step 6, otherwise return to step 3 to continue training. The termination condition can be a certain number of training steps, or it can be that the model has reached a certain evaluation standard on the validation set. The termination condition used in this embodiment is that the prediction accuracy and recall rate of the model on the validation set have reached the preset results. When the training is terminated, the optimal parameters of the encoder and the similarity measurement function are obtained, which can be applied to the actual intelligent dialogue system.

[0099] Step 6: End model training.

[0100] For a small sample intent recognition training task set S with different intent numbers, a new task set S' can be sampled using downsampling so that each task in S' has the same number of intents. Each task T' in S' has a one-to-one correspondence with a task T in S, and the intent set contained in T' is a subset of the intent set contained in T. The training tasks in the original set S cannot be trained in parallel, but the training tasks in S' can be trained in parallel.

[0101] If the minimum number of intents is downsampled for all training tasks, the downsampling rate of some tasks will be extremely low, resulting in low utilization of training data and insufficient learning. For example, if there is a task T with 50 intents, and only 2 intents are sampled each time task T is used, the model will not be able to learn knowledge containing relative information between most intents. Therefore, the present invention adopts a level-based division method, which can alleviate this problem on the one hand, and introduce parallel training on the other hand.

[0102] The present invention provides a text data optimization method for small sample intent recognition. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A text data optimization method for small sample intent recognition, characterized in that: The following steps are involved: Step 1, build a training text dataset; Step 2: The training tasks are divided into different levels according to the number of intentions they have. Step 3, sampling a batch of small sample intention recognition training samples; Step 4: Perform parallel training on two or more tasks sampled from the same batch on the meta-learning model based on metric learning; Step 5, determine whether the training is terminated; Step 6, end model training; Step 1 includes: defining a training text data set as S = {T1, T2, ..., T n }, where T i is the i-th small sample intention recognition task in the real dialogue scene, n is the total number of training tasks in S, and i ranges from 1 to n; Each small-shot intent recognition task is defined as Among them, ntent ij T i Corresponding to an intention in a dialogue scenario, C i T i The number of intents contained; For each task an intent is defined as The query ijk To be marked as Intent ij A dialogue text of N ij for intention, ntent ij The total number of annotation texts included; Step 2 includes: constructing a level list levels = {L1, L2, ..., L M }, L M Indicates the Mth level; The maximum level of the number of intentions a training task has is the level of the training task. i The number of intentions contained C i ≥L l And C i <L l+1 , then T i The corresponding level is L l , if C i ≥L M Then T i The corresponding level is L M ; According to each task T i The number of intentions contained divides all tasks in the training text dataset S into different subsets, that is, constructs the partition of the training text dataset S in Indicates that it corresponds to level L M A subset of Step 3 includes: Step 3-1, determine the level L of the current batch to be sampled l ; Step 3-2, at Level L l A collection of tasks Sample a task T i ; Step 3-3, in Task T i Sample L l intention; Step 3-4: for L l intents, sample k dialogue texts for each intent; Step 3-5, determine whether the sampling of this batch is completed: if the sampling has been completed, the corresponding batch size of the level training task samples, execute step 3-6, otherwise return to step 3-2 to continue sampling; Step 3-6, return the training data sampled from the current batch: The training data D of the current batch contains tasks, each task contains L l intents, each of which contains k dialogue texts labeled as the intent; Split the training data D into support set D support and query set D query The splitting method is to split the k dialogue texts of each intent under the training data D into k support and k query Two subsets of conversation texts, where k = k support +k query ; Step 4 specifically includes the following steps: Step 4-1, use the encoder to encode the input text: for the support set D support and query set D query All the dialogue texts in , use the text encoder to encode each dialogue text into a vector representation of a specific dimension. In the text encoding process, the support set D support The data representation shape in is Where sen_len is the maximum length of the input text limited by the encoder, and word_emb is the dimension size of the vector representation corresponding to each word; After being encoded by the encoder, we get where sen_emb is the dimension of the sentence vector representation of each dialogue text; the query set Dquery is encoded to obtain a shape of Data representation; Step 4-2, query set sentence representation and support set D support The intention is to measure the similarity: Use the similarity measurement function to calculate the similarity between two vector representations; The support set D support The sentence vectors corresponding to each intent in are integrated into a single intent vector, that is, converted into Where intent_emb is the k support The average value of the sen_emb calculations is obtained to obtain a prototype representation for each intent in each task; The query set D query The sentence vector representation of all dialogue texts of each task in is calculated similarly to the prototype representation of the intent under the corresponding task, and the similarity result is The penultimate dimension L l That is, each dialogue text is predicted to be the score of each intent in its source task. l The vector composed of scores is abbreviated as pred; Step 4-3, calculate the loss: use the multi-classification cross entropy function as the loss function loss, that is, loss = Cross_Entropy(label, pred), where loss is the loss value, label is the label vector, and Cross_Entropy() is the cross entropy function; Use the similarity results predicted in step 4-2 to perform cross entropy calculation with the actual label to obtain the prediction loss value of the model. For the query set D in the training phase query Each conversation text in is represented as a one-hot vector as the annotation vector. The dimension of the one-hot vector is L1. The true intention corresponding to each conversation text is marked as 1, and the rest of the positions are marked as 0. Step 4-4, gradient regression: Based on the loss calculated in step 4-3, use the back propagation algorithm to calculate the gradient and perform reverse regression to update the model; Step 4-5, single-step training is completed.

2. The method according to claim 1, characterized in that In step 4, the meta-learning model based on metric learning includes an encoder and a similarity metric function.

3. The method according to claim 2, characterized in that In step 4-2, the similarity between two vector representations is calculated using the following formula: score ab =F(Sen_emb a ,sen_emb b ) where sen_emb a and sen_emb b is the sentence vector representation of two sentences, F() is the similarity measurement function, score ab is the final calculated similarity score of the two input texts.

4. The method according to claim 3, characterized in that In step 4-2, the similarity measurement function is a layer of linear neural network. After the two sentence vectors are concatenated and passed into the linear neural network, a decimal between 0 and 1 is obtained after the activation layer and the softmax() function is calculated. Finally, the vector pred is a vector in which each component is positive and the sum of each component is 1. The specific formula of the single-layer linear neural network is: score ab =relu(Wx ab +bias) Among them, relu() is the rectified linear unit activation function, W is the weight matrix, bias is the bias vector, and x ab It is the vector after the concatenation of sentence vectors a and b.

5. The method according to claim 4, characterized in that In step 4-2, the cross entropy function is denoted as L cross_entropy , the calculation formula is as follows: Where N is the number of dialogue texts in the query set, N = L l *k query ; L l is the intention number; y ic is a symbolic function, taking values ​​of 0 or 1. It is 1 when the dialogue text i belongs to the intention c, otherwise it is 0, that is, the component of the vector label corresponding to the position of the intention c; p ii is the probability that dialogue text i is predicted as intent c.

6. The method according to claim 5, characterized in that Step 5 includes: judging whether the model training is terminated according to the preset termination condition, and executing step 6 if it is terminated, otherwise returning to step 3 to continue training.

Images