A continuous small sample intent recognition method based on natural language prompt mechanism

Abstract

The present invention discloses a method for continuous small-sample intent recognition based on a natural language prompt mechanism. The present invention includes the following steps: 1) Teacher model preparation: Initialize the teacher model from zero on a new task, and perform fine-tuning based on a contrastive learning method; 2) Perform knowledge distillation: Freeze the teacher model, use new task data, perform knowledge distillation on the model trained on the old task, and transfer the teacher representation; 3) Perform memory replay: Use a small amount of data saved from the old task to perform contrastive learning replay on the current student model. Compared with the existing technology, the present invention, for the first time, proposes a parameter-efficient natural language prompt mechanism for the problem of small-sample intent recognition in a continuous learning scenario. Combined with memory replay and knowledge distillation strategies, it can greatly alleviate the problems of knowledge forgetting and negative knowledge transfer in the continuous learning process.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and specifically relates to the application of continuous learning and prompted learning in small-sample intent recognition tasks in continuous learning scenarios. Background Art

[0002] Intent recognition is a core component of task-based dialogue systems. It aims to understand the purpose behind user utterances and accurately classify them into different intents. Traditional intent recognition systems are trained using large amounts of labeled data to identify a set of predefined intent categories. However, in real-world scenarios, these systems often face the need for continuous intent recognition. Furthermore, annotated data for new intents in diverse domains is often scarce. A naive approach to detecting new intents is to recollect annotated data and retrain the model, but this is both time-consuming and impractical for deployed dialogue systems. Therefore, in real-world scenarios, it is imperative to build an intent recognition system that can continuously identify new intents using minimal labeled data.

[0003] Existing technologies are often limited to continuous learning scenarios or small sample sizes, and no technology has yet considered the sparsity and domain diversity of new intent data in continuous learning scenarios. In this practical scenario, the core issue is how to accurately identify new intents while mitigating catastrophic forgetting. Summary of the Invention

[0004] The purpose of this invention is to provide a continuous small sample intention recognition method based on a natural language prompt mechanism to address the novel and practical problem of small sample intention recognition in a continuous learning scenario.

[0005] Inspired by cued learning, this paper designs a prototype network architecture that combines parameterized natural language cues and a replay strategy, and uses a knowledge distillation strategy to solve the problem of negative knowledge transfer in the continuous learning process, achieving a dynamic balance between learning new tasks and preventing the forgetting of old tasks.

[0006] The technical solutions specifically adopted in the present invention are as follows:

[0007] A continuous small-sample intent recognition method based on a natural language prompt mechanism, the steps are as follows:

[0008] S1: On a new task in the intent recognition scenario, a teacher model with the same structure as the student model is initialized. Both the student and teacher models use SimCSE-RoBERTa as the original encoder, and introduce learnable prefix cue parameters into each multi-head attention layer within the original encoder and connect them with the original key-value representation to form a prefix cue-guided multi-head attention layer. At the same time, a learnable adapter structure is inserted after each feed-forward layer within the original encoder. The teacher model is trained using a contrastive learning method based on the intent recognition task dataset corresponding to the new task. During the training process, the original encoder parameters need to be frozen, and only the newly introduced network structure relative to the original encoder is fine-tuned.

[0009] S2: Freeze the teacher model trained in S1 and use it to perform knowledge distillation on the student model trained on the old task based on the intent recognition task dataset corresponding to the new task. This results in the mean squared error loss between the different representations output by the teacher and student models for the same sample.

[0010] S3: When training the student model on a new task, perform memory replay of contrastive learning on the current student model using data cached from the old task to calculate the memory replay loss based on contrastive learning; the cached data from the old task are the two samples in each category that are closest and farthest from the corresponding category representation in cosine distance, which are saved after the student model is trained on the old task;

[0011] S4: Iteratively train the student model for the new task through the knowledge distillation of S2 and the memory replay task of S3. The total loss of the training is the weighted sum of the mean square error loss and the memory replay loss.

[0012] S5: After the new task training is completed, all training samples of each category in the new task are fed into the encoder and the sample representation is output. The average of all sample representations in each category is used as the representation of this category. The two samples in each category that are closest and farthest from the category representation in cosine distance are cached for memory replay during the next new task training.

[0013] S6: During testing or actual application, the user's query text is input into the current latest student model to obtain the query representation through encoding. The cosine similarity between the query representation and all category representations is calculated respectively, and the category most similar to the query representation is used as the predicted intent category.

[0014] Preferably, in S1, in the process of fine-tuning the network structure newly introduced by the teacher model relative to the original encoder, for each iteration step, N categories need to be randomly sampled, and K user speech text samples are randomly sampled for each category. The sampled N*K user speech text samples are sent to the teacher model to obtain N*K sample representations, and the comparative learning method is used to bring the sample representations of the same category closer and the sample representations of different categories farther apart.

[0015] Preferably, in S2, during the knowledge distillation process, for each iterative step, N categories are randomly sampled on the intent recognition task dataset corresponding to the new task, and K user speech text samples are sampled for each category. The sampled N*K user speech text samples are respectively sent to the teacher model and the student model, and sample representations are obtained for each model, and the mean square error loss between the output representations of the same sample in different models is calculated.

[0016] Preferably, in S3, in each iterative step of the new task training process for the student model, an old task is randomly sampled, and N categories are randomly selected from the cache, and two user speech text samples saved in each category are taken out, and the sampled N*2 user speech text samples are sent to the student model, and the contrastive learning method is used to bring the two sample representations of the same category closer, and to pull the sample representations of different categories farther apart, and the memory replay loss based on contrastive learning is calculated.

[0017] Preferably, in the model structure of the student model and the teacher model, the learnable prefix hint parameter introduced in each multi-head attention layer is The two introduced parameters concatenate the key-value representations of the original multi-head attention layer in SimCSE-RoBERTa to form a prefix-cue-guided multi-head attention layer:

[0018]

[0019] Among them: head i represents the i-th attention head in the prefix cue-guided multi-head attention layer, X∈R m×d is the input text sequence representation, and is the parameter matrix; l represents the length of the prefix hint parameter, d h is the hidden layer dimension of each attention head, and d is the hidden layer dimension of the multi-head attention layer.

[0020] Preferably, in the model structure of the student model and the teacher model, the learnable adapter structure inserted after the feedforward layer is composed of the lower projection matrix W down ∈R d×r , nonlinear activation function ReLU(·) and the upper projection matrix Wup ∈R r ×d The output h′ of each feed-forward layer in the model after inserting the learnable adapter structure is expressed as:

[0021] h′←h+ReLU(hW down )W up

[0022] Where: h represents the original feedforward layer output in SimCSE-RoBERTa, and r is a dimensional hyperparameter.

[0023] Preferably, in the intention recognition task dataset, each user speech text sample is represented as x=w1w2w3…, and each sample is preprocessed into the same input format [CLS]w1w2w3…[SEP] by adding [CLS] and [SEP]; after inputting it into the teacher model or student model, the representation of the [CLS] position is taken out as the sample representation f(x) of the sample.

[0024] Preferably, in S1, when the teacher model is trained based on the contrastive learning method on the basis of the intention recognition task dataset corresponding to the new task, the contrastive learning loss L is used. sim The calculation formula is as follows:

[0025]

[0026] in represents the teacher model, x i represents the i-th user utterance text sample, represents the category representation of the j-th category, which is obtained by averaging the K sample representations of each class; sim(·) is the cosine similarity function, τ is the temperature hyperparameter; I(·) is the indicator function, which takes 1 if the equation in the brackets holds and takes 0 otherwise.

[0027] As a preference, in said S2, the mean square error loss L dis The calculation formula is as follows:

[0028]

[0029] in represents the student model;

[0030] In S3, the formula for calculating the memory replay loss based on contrastive learning is as follows:

[0031]

[0032] Preferably, in S4, the calculation formula of the total loss function used when training the encoder model is:

[0033] L=λ dis L dis +λ mem L mem

[0034] where λ dis and λ mem Represent the mean square error loss L dis and memory replay loss L mem weight values; in each iteration step, both weight values ​​need to be dynamically determined by solving the Pareto optimal solution.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] Compared to existing technologies, this invention integrates natural language cues into continuous learning and mitigates negative knowledge transfer through knowledge distillation strategies, significantly improving intent recognition accuracy and mitigating catastrophic forgetting. Furthermore, during continuous learning, the number of model fine-tuning parameters is significantly reduced, and only a single model copy needs to be maintained, making training more efficient and significantly reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the steps of a continuous small-sample intent recognition method based on a natural language prompt mechanism;

[0038] Figure 2 It is a framework diagram of the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1 As shown, in a preferred embodiment of the present invention, a continuous small sample intention recognition method based on a natural language prompt mechanism is provided, and its steps are shown as S1 to S6:

[0041] S1: On a new task in the intent recognition scenario, a teacher model with the same structure as the student model is initialized; both the student model and the teacher model use SimCSE-RoBERTa as the original encoder, and introduce a learnable prefix prompt parameter into each multi-head attention layer inside the original encoder and connect it with the original key-value representation to form a prefix prompt-guided multi-head attention layer. At the same time, a learnable adapter structure is inserted after each feed-forward layer inside the original encoder; the teacher model is trained based on the intent recognition task dataset corresponding to the new task using a contrastive learning method, and the original encoder parameters need to be frozen during the training process, and only the newly introduced network structure relative to the original encoder is fine-tuned.

[0042] In this embodiment, in the model structure of the above student model and teacher model, the learnable prefix hint parameter introduced in each multi-head attention layer is The two introduced parameters concatenate the key-value representations of the original multi-head attention layer in SimCSE-RoBERTa to form a prefix-cue-guided multi-head attention layer:

[0043]

[0044] Among them: head i represents the i-th attention head in the prefix cue-guided multi-head attention layer, X∈R m×d is the input text sequence representation, and is the parameter matrix; l represents the length of the prefix hint parameter, d h is the hidden layer dimension of each attention head, and d is the hidden layer dimension of the multi-head attention layer.

[0045] In this embodiment, in the model structure of the above-mentioned student model and teacher model, the learnable adapter structure inserted after the feedforward layer is composed of the lower projection matrix W down ∈R d×r , nonlinear activation function ReLU(·) and the upper projection matrix W up ∈R r×d The output h′ of each feed-forward layer in the model after inserting the learnable adapter structure is expressed as:

[0046] h′←h+ReLU(hW down )W up

[0047] Where: h represents the original feedforward layer output in SimCSE-RoBERTa, and r is a dimensional hyperparameter.

[0048] It should be noted that since the student model in the present invention is continuously learning on different tasks, the introduced adapter and prefix hint parameters are shared between each task, and the parameters of the original encoder need to be kept frozen during the continuous learning process.

[0049] In this embodiment, in the above S1, in the process of fine-tuning the network structure newly introduced by the teacher model relative to the original encoder, for each iteration step, N categories need to be randomly sampled, and K user speech text samples are randomly sampled for each category. The sampled N*K user speech text samples are sent to the teacher model to obtain N*K sample representations, and the comparative learning method is used to bring the sample representations of the same category closer and the sample representations of different categories farther apart.

[0050] In this embodiment, in the above S1, when the teacher model is trained based on the contrastive learning method on the basis of the intention recognition task dataset corresponding to the new task, the contrastive learning loss L is used. sim The calculation formula is as follows:

[0051]

[0052] in represents the teacher model, x i represents the i-th user utterance text sample, represents the category representation of the j-th category, which is obtained by averaging the K sample representations of each category; sim(·) is the cosine similarity function, τ is the temperature hyperparameter (set to 0.07 in this embodiment); I(·) is the indicator function, which takes 1 if the equation in the brackets holds and takes 0 otherwise.

[0053] In this embodiment, each task corresponds to a new intent recognition task dataset. Therefore, each user utterance text sample in the intent recognition task dataset corresponding to each task can be represented as x = w1w2w3…. Each sample is preprocessed into the same input format [CLS]w1w2w3…[SEP] by adding [CLS] and [SEP]. After inputting it into the teacher model or student model, the representation at the [CLS] position is taken as the sample representation f(x) of the sample. In the present invention, each user utterance text in the intent recognition task dataset is a user utterance text in the task-based dialogue system.

[0054] S2: Freeze the teacher model trained in S1, and based on the intent recognition task dataset corresponding to the new task, use the teacher model to perform knowledge distillation on the student model trained on the old task, and obtain the mean squared error loss between the different representations output by the teacher model and the student model for the same sample.

[0055] In this embodiment, in S2 above, during the knowledge distillation process, for each iterative step, N categories need to be randomly sampled on the intent recognition task dataset corresponding to the new task, and K user speech text samples are sampled for each category. The sampled N*K user speech text samples are respectively sent to the teacher model and the student model, and each obtains a sample representation, and the mean square error loss between the output representations of the same sample in different models is calculated.

[0056] In this embodiment, the above mean square error loss L dis The calculation formula is as follows:

[0057]

[0058] in represents the student model;

[0059] S3: When training the student model for a new task, memory replay based on contrastive learning is performed on the current student model using the data cached from the old task to calculate the memory replay loss based on contrastive learning; the data cached in the old task are the two samples in each category that are closest and farthest from the corresponding category representation in cosine distance, which are saved after the student model is trained on the old task.

[0060] In this embodiment, in the above S3, in each iterative step of the new task training process for the student model, an old task needs to be randomly sampled, and N categories are randomly selected from the cache. Two user speech text samples saved in each category are taken out, and the sampled N*2 user speech text samples are sent to the student model. The contrastive learning method is used to bring the two sample representations of the same category closer, and to pull the sample representations of different categories farther apart, and the memory replay loss based on contrastive learning is calculated.

[0061] In this embodiment, the calculation formula for the memory replay loss based on contrastive learning is as follows:

[0062]

[0063] S4: The student model is iteratively trained on the new task through the knowledge distillation of S2 and the memory replay task of S3. The total loss of the training is the weighted sum of the mean square error loss and the memory replay loss.

[0064] In this embodiment, in the above S4, the calculation formula of the total loss function used when training the encoder model can be expressed as follows:

[0065] L=λ dis L dis +λ mem L mem

[0066] where λ dis and λ mem Represent the mean square error loss L dis and memory replay loss L mem weight values; in each iteration step, both weight values ​​need to be dynamically determined by solving the Pareto optimal solution.

[0067] S5: After the new task training is completed, all training samples of each category in the new task are sent to the encoder and the sample representation is output. The average value of all sample representations in each category is used as the representation of this category; the two samples in each category that are closest and farthest from the category representation in cosine distance are cached for memory replay during the next new task training.

[0068] It should be noted that after the new task training is completed in step S5, two samples are cached for each category. These two samples will be used as the "cached data in the old task" for memory replay in the aforementioned step S3 in the next round of new task training.

[0069] S6: During testing or actual application, the user's query text is input into the current latest student model to obtain the query representation through encoding. The cosine similarity between the query representation and all category representations is calculated respectively, and the category most similar to the query representation (the category with the highest cosine similarity) is used as the predicted intent category.

[0070] It can be seen that in the above steps S1 to S6, the student model can continuously learn on new tasks, where the adapter and prefix hint parameters introduced in the original encoder are shared between each task, while the original pre-trained parameters in the original encoder remain frozen during the continuous learning process.

[0071] The continuous small sample intention recognition method based on the natural language prompt mechanism described in S1 to S6 above is applied to a specific example to demonstrate its specific implementation method and technical effect.

[0072] Example

[0073] This embodiment is mainly applied to a continuous learning small-sample intent recognition benchmark dataset consisting of 9 tasks. Each task contains a small-sample intent recognition dataset, where each category in each dataset contains 5 or 10 training samples (denoted as 5-shot and 10-shot settings). Details are shown in Table 1.

[0074] Table 1 Basic information of the dataset (5-shot / 10-shot)

[0075]

[0076] like Figure 1 As shown, this embodiment trains and tests on each task according to the process described in S1 to S6 above. After training on each task, the test accuracy is evaluated on all previously seen tasks. In order to measure the performance of the model at each time step, this embodiment also uses three additional quantitative indicators for evaluation: the average accuracy A after training on the tth task, t , average forgetting rate F t and learning accuracy LA t Indicator. Let a t,i represents the test accuracy on the i-th task after training the t-th task. These indicators are defined as follows:

[0077]

[0078]

[0079] Among them, A t measures the average performance over all previously seen tasks. t LA measures how well the model has forgotten all previously seen tasks after learning the t-th task. t This measures how well the model learns when it sees a new task.

[0080] After training 9 tasks, this embodiment evaluated the performance of all previously visited tasks, and the results are shown in Table 2. LC represents a method that uses RoBERTa as an encoder and adds Linear and Softmax layers on the top layer to perform classification. L-DNNC is a classic method in small-sample intent recognition, which concatenates the category name and query text and feeds them into the classification model to perform binary classification, indicating whether they belong to the same category. L-PN is a matching-based prototype network baseline in small-sample learning, and PN-AGEM represents a prototype network baseline that uses the classic anti-forgetting baseline AGEM. PN-Joint stores all data from all previously seen tasks and uses all data to train the entire prototype network when learning new tasks. It is the upper bound of the performance of this setting.

[0081] Table 2 Test accuracy of each task after training 9 tasks

[0082]

[0083] As shown in Table 2, compared to other baseline methods, our method outperforms the previous baseline in terms of average test accuracy across all tasks (85.61% and 88.16% in the 5-shot and 10-shot settings, respectively). At the same time, our method still achieves better performance in tasks accessed earlier, which demonstrates the superiority of our method in avoiding catastrophic forgetting.

[0084] This example also evaluates the performance indicators of each task after training based on the performance characterization at each time step. The results are shown in Table 3. PLE represents this method, while SC, S-PN, and S-PLE represent single-task training methods in which a model is trained separately for each task.

[0085] Table 3 Average accuracy, average forgetting rate and average learning accuracy at each time step

[0086]

[0087] As shown in Table 3, our method outperforms previous baselines in the 5-shot setting in terms of average accuracy and average forgetting rate, while also achieving competitive performance in the 10-shot setting and exhibiting less forgetting than other baselines.

[0088] In addition, this example conducts an ablation study on different components of the model. w / o prefix means removing prefix hints, w / o memory means not using the memory replay strategy, w / o TKT means not using the knowledge distillation strategy, w / o PSR means randomly sampling old task data for caching, and w / o DWR means optimizing by fixing the weights of the two tasks, for example: dis =0.9,λ mem =0.1. The ablation experiment results are shown in Table 4.

[0089] Table 4. Ablation experiment results (5-shot / 10-shot)

[0090]

[0091] As can be seen in Table 4, although removing prefix hints, memory replay, or knowledge distillation achieves better performance in some areas, performance in other areas degrades significantly. However, combining these components allows this method to achieve good results across all quantitative metrics.

[0092] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A continuous small sample intent recognition method based on natural language prompt mechanism, characterized by: Here are the steps: S1: For a new intent recognition task, a teacher model with the same structure as the student model is initialized. Both the student and teacher models use SimCSE-RoBERTa as the original encoder. Learnable prefix cue parameters are introduced into each multi-head attention layer within the original encoder and concatenated with the original key-value representation to form a prefix cue-guided multi-head attention layer. A learnable adapter structure is also inserted after each feed-forward layer within the original encoder. The teacher model is trained based on the intent recognition task dataset corresponding to the new task using a contrastive learning approach, and the original encoder parameters need to be frozen during the training process, and only the newly introduced network structure relative to the original encoder is fine-tuned; S2: Freeze the teacher model trained in S1 and use it to perform knowledge distillation on the student model trained on the old task based on the intent recognition task dataset corresponding to the new task. This results in the mean squared error loss between the different representations output by the teacher and student models for the same sample. S3: When training the student model on a new task, perform memory replay of contrastive learning on the current student model using data cached from the old task to calculate the memory replay loss based on contrastive learning; the cached data from the old task are the two samples in each category that are closest and farthest from the corresponding category representation in cosine distance, which are saved after the student model is trained on the old task; S4: Iteratively train the student model for the new task through the knowledge distillation of S2 and the memory replay task of S3. The total loss of the training is the weighted sum of the mean square error loss and the memory replay loss. S5: After the new task training is completed, all training samples of each category in the new task are fed into the encoder and the sample representation is output. The average of all sample representations in each category is used as the representation of this category. The two samples in each category that are closest and farthest from the category representation in cosine distance are cached for memory replay during the next new task training. S6: During testing or actual application, the user's query text is input into the current latest student model to obtain the query representation through encoding. The cosine similarity between the query representation and all category representations is calculated respectively, and the category most similar to the query representation is used as the predicted intent category.

2. The continuous small sample intent recognition method based on the natural language prompt mechanism according to claim 1 is characterized in that: In S1, in the process of fine-tuning the network structure newly introduced by the teacher model relative to the original encoder, for each iteration step, N categories need to be randomly sampled, and K user speech text samples are randomly sampled for each category. The sampled N*K user speech text samples are sent to the teacher model to obtain N*K sample representations. The comparative learning method is used to bring the sample representations of the same category closer and the sample representations of different categories farther apart.

3. The continuous small sample intent recognition method based on natural language prompt mechanism as claimed in claim 2 is characterized in that: In S2, during the knowledge distillation process, for each iterative step, N categories must be randomly sampled from the intent recognition task dataset corresponding to the new task, and K user speech text samples are sampled from each category. The sampled N*K user speech text samples are sent to the teacher model and the student model respectively, and sample representations are obtained for each model. The mean square error loss between the output representations of the same sample in different models is calculated.

4. The continuous small sample intent recognition method based on natural language prompt mechanism according to claim 3 is characterized in that: In S3, in each iterative step of the new task training process for the student model, an old task is randomly sampled, and N categories are randomly selected from the cache. Two user speech text samples saved in each category are taken out, and the sampled N*2 user speech text samples are sent to the student model. The contrastive learning method is used to bring the two sample representations of the same category closer, and to pull the sample representations of different categories farther apart, and the memory replay loss based on contrastive learning is calculated.

5. The method for continuous small sample intent recognition based on natural language prompt mechanism according to claim 4 is characterized in that: In the model structure of the student model and the teacher model, the learnable prefix hint parameter introduced in each multi-head attention layer is The two introduced parameters concatenate the key-value representations of the original multi-head attention layer in SimCSE-RoBERTa to form a prefix-cue-guided multi-head attention layer: Among them: head i represents the i-th attention head in the prefix cue-guided multi-head attention layer, X∈R m×d is the input text sequence representation, and is the parameter matrix; l represents the length of the prefix hint parameter, d h is the hidden layer dimension of each attention head, and d is the hidden layer dimension of the multi-head attention layer.

6. The continuous small sample intent recognition method based on natural language prompt mechanism according to claim 5 is characterized in that: In the model structure of the student model and the teacher model, the learnable adapter structure inserted after the feedforward layer is represented by the lower projection matrix W down ∈R d×r , nonlinear activation function ReLU(·) and the upper projection matrix W up ∈R r×d The output h′ of each feed-forward layer in the model after inserting the learnable adapter structure is expressed as: h′←h+ReLU(hW down )W up Where: h represents the original feedforward layer output in SimCSE-RoBERTa, and r is a dimensional hyperparameter.

7. The continuous small sample intent recognition method based on natural language prompt mechanism according to claim 1 is characterized in that: In the intention recognition task dataset, each user speech text sample is represented as x=w1w2w3…, and each sample is preprocessed into the same input format [CLS]w1w2w3…[SEP] by adding [CLS] and [SEP]; after inputting it into the teacher model or student model, the representation of the [CLS] position is taken out as the sample representation f(x) of the sample.

8. The continuous small sample intent recognition method based on natural language prompt mechanism according to claim 6 is characterized in that: In S1, when the teacher model is trained based on the contrastive learning method on the basis of the intention recognition task dataset corresponding to the new task, the contrastive learning loss L is used. sim The calculation formula is as follows: in represents the teacher model, x i represents the i-th user utterance text sample, represents the category representation of the j-th category, which is obtained by averaging the K sample representations of each class; sim(·) is the cosine similarity function, τ is the temperature hyperparameter; I(·) is the indicator function, which takes 1 if the equation in the brackets holds and takes 0 otherwise.

9. The method for continuous small sample intent recognition based on natural language prompt mechanism according to claim 6, characterized in that: In S2, the mean square error loss L dis The calculation formula is as follows: in represents the student model; In S3, the formula for calculating the memory replay loss based on contrastive learning is as follows: 。 10. The continuous small sample intent recognition method based on natural language prompt mechanism according to claim 1 is characterized in that: In S4, the calculation formula of the total loss function used when training the encoder model is: L=λ dis L dis +λ mem L mem where λ dis and λ mem Represent the mean square error loss L dis and memory replay loss L mem weight values; in each iteration step, both weight values ​​need to be dynamically determined by solving the Pareto optimal solution.

Images