Pseudo-tag-based intention recognition model training method, intention recognition method and device

Abstract

The invention provides a pseudo-tag-based intention recognition model training method, an intention recognition method and an intention recognition device. The method comprises the following steps: inputting a sample text into a language model to extract a feature vector; clustering the sample text based on the feature vector, taking a clustering result as a pseudo tag, and calculating normalized mutual information of the real tag and the pseudo tag of the labeled sample text; determining a confidence score corresponding to each sample; the confidence score is used for quantifying noise in the pseudo tag, screening a high-confidence sample and taking the corresponding pseudo tag as a self-supervision signal, and iteratively optimizing the language model until convergence; after iteration, clustering is initialized again, a clustering result is updated, and mutual information and the number of iterations are normalized; when the number of iterations reaches an upper limit or the normalized mutual information amplification is smaller than a threshold value, training is terminated, and the language model is determined as an intention recognition model; the problem that the new intention recognition capability of the model is reduced due to continuous propagation and accumulation of noise pseudo labels can be solved; and the new intention recognition capability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a pseudo-label-based intent recognition model training method, an intent recognition method, and a device. Background Art

[0002] In today's digital age, dialogue systems, thanks to their convenient interactive methods, have been widely used in customer service, intelligent assistants, recommendation systems, and other fields. For these dialogue systems, accurately understanding user intent is key to providing an efficient and high-quality interactive experience. However, as user needs continue to evolve and diversify, relying solely on a limited number of predefined intent categories is no longer sufficient to meet the complex and ever-changing needs of real-world scenarios. It is against this backdrop that the task of New Intent Discovery (NID) has emerged, and its importance is growing. By discovering and adapting to dynamically changing user needs, NID can significantly improve the service quality and user satisfaction of dialogue systems.

[0003] Semi-supervised learning strategies are currently gaining popularity in the field of novel intent discovery. Self-training (pseudo-labeling) is a commonly used approach for novel intent discovery, often combined with clustering and representation learning. Specifically, this approach first trains an initial model using a small amount of labeled data. This model is then used to make predictions on a large amount of unlabeled data, generating pseudo-labels. Finally, this pseudo-labeled data is added to the training set to further optimize model performance.

[0004] However, this semi-supervised approach also presents a new challenge: the potential for noise in the pseudo-labels. During the clustering phase, these noisy pseudo-labels can cause samples to be misassigned to different clusters, undermining the accuracy and rationality of the clustering results. During the representation learning phase, they can contaminate the learned feature representations, causing the model to learn erroneous information. More seriously, as training iterations proceed, noisy pseudo-labels continue to propagate and accumulate, significantly reducing the model's ability to recognize new intents. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a pseudo-label-based intent recognition model training method, intent recognition method, and apparatus to eliminate or improve one or more deficiencies in the prior art. This method addresses the problem that as training iterations progress, noisy pseudo-labels continue to propagate and accumulate, significantly reducing the model's ability to recognize new intents.

[0006] One aspect of the present invention provides a method for training an intent recognition model based on pseudo labels, the method comprising the following steps:

[0007] Input the sample text into the pre-trained language model to extract the feature vector; the sample text includes labeled sample text corresponding to known intent categories, unlabeled sample text corresponding to known intent categories, and unlabeled sample text corresponding to unknown intent categories;

[0008] Perform clustering on the sample text based on the feature vector, use the clustering results as pseudo labels, and calculate the normalized mutual information between the true labels and pseudo labels of the annotated sample text;

[0009] Determine a confidence score for each sample in the sample text; the confidence score is obtained by normalizing and weighting preset indicators to quantify the degree of noise in the pseudo-label; the preset indicators include the distance of the sample text to the center of the cluster to which it belongs, the information entropy of the pseudo-label distribution of the annotated sample text, the frequency of the sample text being assigned to the same cluster in a preset number of clustering operations, and the rate of change of the pseudo-label distribution of each cluster in successive clustering operations;

[0010] High-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo labels are used as self-supervisory signals, and the language model is iteratively optimized through contrastive learning until convergence. The size of the dynamic confidence threshold is positively correlated with the number of iterations; different clusters correspond to different dynamic confidence thresholds.

[0011] The centroid of each cluster is used as the new cluster center, re-clustering and updating the clustering results, normalized mutual information and number of iterations;

[0012] When the number of iterations after the update reaches the preset upper limit or the increase in normalized mutual information is less than the preset threshold, the training is terminated and the current language model is determined as the intent recognition model; otherwise, the step of determining the confidence score corresponding to each sample in the sample text is executed.

[0013] In some embodiments of the present invention, high-confidence sample texts with confidence scores greater than a dynamic confidence threshold and corresponding pseudo labels are used as self-supervisory signals, and a language model is iteratively optimized through contrastive learning until convergence, including:

[0014] Perform data augmentation on high-confidence sample texts, generate corresponding enhanced versions, and construct positive and negative sample pairs;

[0015] Input the positive sample pairs and negative sample pairs into the language model to extract deep semantic features and obtain training results;

[0016] Input the training results into the cross entropy loss function and the supervised contrast loss function, sum the output values ​​of the two loss functions to get the total loss value;

[0017] The gradient is calculated through backpropagation, and the optimizer is used to update the language model parameters to minimize the total loss value. The above process is iterated until the model converges.

[0018] In some embodiments of the present invention, the language model pre-training process includes:

[0019] The sample text is input into the initial language model for feature extraction to obtain the initial feature vector; the sample text includes annotated sample text and unannotated sample text; the initial language model includes an input embedding layer, a feature extraction layer, a mean pooling layer, a nonlinear projection layer and a dynamic task head; the input embedding layer is used to convert the input text into an initial vector representation; the feature extraction layer is used to process the initial vector representation and extract deep semantic features to obtain a token representation; the mean pooling layer is used to perform a mean pooling operation on the token representation to obtain a sentence-level semantic vector; the nonlinear projection layer is used to map the sentence-level semantic vector to a low-dimensional intent representation space to obtain the corresponding intent representation; the dynamic task head is used to embed and output the predicted intent label required for contrastive learning;

[0020] Perform data augmentation on each sample text, generate corresponding augmented samples, construct positive sample pairs and negative sample pairs, and mix the annotated augmented samples and the unannotated augmented samples in a preset ratio to obtain at least one training batch;

[0021] Input at least one training batch into the initial language model to obtain the enhanced feature vector and predicted intent label output by the dynamic task head;

[0022] Input the initial feature vector, predicted intent label, and enhanced feature vector into a preset loss function to obtain a loss value; the preset loss function includes a preset semi-supervised contrast loss function and a cross entropy loss function;

[0023] The gradient is calculated through backpropagation, and the optimizer is used to update the initial language model parameters to minimize the loss value. The above process is iterated until the model converges.

[0024] When the initial language model converges, the dynamic task head is removed to obtain the pre-trained language model.

[0025] In some embodiments of the present invention, before each round of iterative training of the initial language model based on the preset semi-supervised contrastive loss function, the method further includes:

[0026] Use labeled sample text to perform supervised contrastive learning on the pre-trained language model. Input the true intent label and predicted intent label corresponding to the labeled sample text into the preset supervised contrastive loss function to obtain the supervised contrastive loss value.

[0027] The gradient is calculated through backpropagation, and the optimizer is used to update the model parameters of the initial language model to minimize the supervised contrast loss value. The above process is iterated until the model converges.

[0028] In some embodiments of the present invention, the feature extraction layer includes at least one Transform layer; and using an optimizer to update the initial language model parameters includes: updating the parameters of the last Transform layer in the feature extraction layer and the parameters of the dynamic task head through the optimizer.

[0029] In some embodiments of the present invention, before using the high-confidence samples with confidence scores greater than the dynamic confidence threshold and the corresponding pseudo labels as self-supervisory signals, the process further includes:

[0030] When the pseudo-label distribution change rate is less than or equal to the preset change rate threshold, the K sample texts with the lowest confidence scores are determined as samples to be corrected; where K is an integer greater than 1;

[0031] For each sample to be corrected, the pseudo-label probability distribution of the adjacent sample texts in the feature space is retrieved by the K-nearest neighbor algorithm, and the pseudo-label with the highest frequency is selected as the candidate pseudo-label from the pseudo-label probability distribution;

[0032] If there is a candidate pseudo label in the historical pseudo label corresponding to the sample to be corrected, the historical pseudo label is retained as the pseudo label of the sample to be corrected.

[0033] In some embodiments of the present invention, a dynamic confidence threshold is generated based on a global threshold and a local threshold. In the first iteration, the global threshold is initialized to the average confidence score of the unlabeled sample texts, and the local threshold is initialized to the average confidence score corresponding to all sample texts in each cluster. In subsequent iterations, the global threshold and the local threshold of the previous round are smoothly updated by exponential moving average.

[0034] Before using high-confidence samples with confidence scores greater than the dynamic confidence threshold and the corresponding pseudo labels as self-supervisory signals, it also includes:

[0035] The global threshold of the previous round and the local threshold of the previous round are smoothly updated by exponential moving average;

[0036] Normalize each updated local threshold;

[0037] Multiply the normalized local threshold by the updated global threshold to obtain the dynamic confidence threshold of each cluster in the current iteration round.

[0038] Another aspect of the present invention provides an intention recognition method, the method comprising:

[0039] Get the text to be recognized input by the user;

[0040] The text to be recognized is input into the intent recognition model to obtain the intent label corresponding to the text to be recognized; the intent recognition model is trained using the above-mentioned pseudo-label-based intent recognition model training method.

[0041] Another aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the electronic device implements the steps of the previous pseudo-label-based intent recognition model training method and intent recognition method.

[0042] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the aforementioned pseudo-label-based intent recognition model training method and intent recognition method.

[0043] The present invention has at least the following beneficial effects:

[0044] The pseudo-label-based intent recognition model training method, intent recognition method, and device of the present invention can solve the problem that as training iterates, noisy pseudo-labels will continue to propagate and accumulate, resulting in a significant decrease in the model's ability to recognize new intents. By comprehensively utilizing the spatial and temporal feature information of sample texts to dynamically optimize pseudo-labels, the spatial dimension is evaluated through the reliability of sample texts and the cohesion of intent categories to accurately locate noisy labels. The temporal dimension is analyzed through the stability and consistency of pseudo-label distribution to capture the data evolution law, and a spatiotemporal fusion pseudo-label quality evaluation mechanism is constructed, breaking through the limitations of traditional single indicators. It can comprehensively capture the semantic characteristics of user intent, reduce the negative impact of noise on model performance, avoid the problem of pseudo-label noise interfering with model training and intent recognition, provide the model with more reliable self-supervision signals, and improve the accuracy, reliability, and ability to recognize new intents. At the same time, by dynamically optimizing pseudo-labels and combining with a dynamic confidence threshold adjustment strategy to dynamically screen high-confidence pseudo-labels, it is possible to further suppress noise propagation, reduce the negative impact on model performance, and further improve the accuracy and reliability of the model in recognizing new intents.

[0045] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0046] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0048] Figure 1 A flowchart of a pseudo-label-based intent recognition model training method provided by one embodiment of the present invention.

[0049] Figure 2 A schematic diagram of sample text and intent labels provided in one embodiment of the present invention.

[0050] Figure 3 A schematic diagram of a framework for training an intent recognition model based on pseudo labels provided by one embodiment of the present invention.

[0051] Figure 4 A schematic diagram of a framework for pseudo-label denoising provided by an embodiment of the present invention.

[0052] Figure 5 A schematic diagram of spatial perspective indicators and temporal perspective indicators provided by an embodiment of the present invention.

[0053] Figure 6 A flowchart of an intent recognition method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0055] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0056] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0057] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0058] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0059] The following is a detailed introduction to the pseudo-label-based intent recognition model training method provided in this application.

[0060] The execution subject of the pseudo-label-based intent recognition model training method provided in this application is an electronic device, which can be a terminal such as a computer, a mobile phone, a tablet computer, a camera, or a server. This embodiment does not limit the implementation method of the electronic device.

[0061] This embodiment provides a pseudo-label-based intent recognition model training method. Figure 1 As shown, the method at least includes steps S101 to S106:

[0062] Step S101: Input the sample text into a pre-trained language model to extract a feature vector.

[0063] In some embodiments of the present invention, the sample text includes annotated sample text corresponding to a known intent category, unannotated sample text corresponding to a known intent category, and unannotated sample text corresponding to an unknown intent category.

[0064] For example: Reference Figure 2 For example, the known intent categories include "auto-recharge," "transfer," "forgot secret," "exchange rate," and "age limit." The text "My password has been changed" is labeled with the intent of "balance inquiry" and serves as a labeled sample text. The text "I haven't received my credit card yet" is labeled with the intent of "card arrival" and serves as a labeled sample text. Unlabeled sample text may include the text "I want to know how to check my balance," which has the same intent as the known intent category "balance inquiry" but is unlabeled. The text "I recently found my card! Can you help me rebuild it?" has no corresponding known intent as a label, so the intent of this sample text belongs to the unknown intent category.

[0065] In some embodiments of the present invention, the sample text is a sample text in an intent benchmark dataset, which includes but is not limited to the Banking Conversational Dataset (BANKING), the Multi-Domain Intent Classification Dataset (CLINC), and the Technical Question and Answer Dataset (StackOverflow).

[0066] Among them, the banking dialogue dataset contains 13,083 banking customer service inquiries, covering 77 intent categories; the multi-domain intent classification dataset contains 22,500 cross-domain queries, covering 150 intent categories; the technical question and answer dataset contains 20,000 technical question titles, covering 20 intent categories.

[0067] In some embodiments of the present invention, 25%, 50%, and 75% of the categories in each dataset are randomly selected as known intent classes. For each known intent class, 10% of the sample texts are labeled, and the remaining unlabeled sample texts contain both known intents and new intents.

[0068] In some embodiments of the present invention, the number of unlabeled sample texts is greater than the number of labeled sample texts. By combining unlabeled and labeled sample texts to train the language model, the language model's ability to recognize known intents can be effectively improved, while also discovering potential new intents.

[0069] In some embodiments of the present invention, a language model refers to a model used to extract features from sample text, obtained by pre-training an initial language model. Because the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model has demonstrated excellent performance in natural language processing (NLP) tasks, in some embodiments of the present invention, the pre-trained BERT model is used as the initial language model.

[0070] In actual implementation, the initial language model can also be other language models or improved models of the BERT model, such as lightweight BERT (ALite BERT, ALBERT), enhanced BERT (Decoding-enhanced BERT with disentangled attention, DeBERTa), text-to-text transfer transformer (Text-to-Text Transfer Transformer, T5), etc. This embodiment does not limit the model type of the initial language model.

[0071] Specifically, the pre-training phase of the language model includes at least steps S1011 to S1016:

[0072] Step S1011: Input the sample text into the initial language model for feature extraction to obtain an initial feature vector. The sample text includes annotated sample text annotated with intent labels and unannotated sample text unannotated with intent labels.

[0073] In some embodiments of the present invention, the pre-training stage includes a semi-supervised pre-training stage. The semi-supervised pre-training stage refers to a stage of performing semi-supervised training on the extracted initial intent representation.

[0074] like Figure 3 As shown, sample text from the intent benchmark dataset is input into the initial language model. The initial language model then extracts features from both labeled and unlabeled sample text to generate an initial feature vector, which serves as the initial intent representation. After extracting the initial intent representation, the semi-supervised pre-training phase begins. The goal of the semi-supervised pre-training phase is to fully utilize the limited amount of labeled sample text to provide a good initial representation for subsequent clustering operations.

[0075] During the semi-supervised pre-training phase, data augmentation (e.g., random dropout) is first applied to both labeled and unlabeled sample texts to generate augmented sample pairs. The augmented samples from the labeled and unlabeled samples are then mixed into the same training batch, and feature representations are optimized using a contrastive learning mechanism. During the contrastive learning phase, the dynamic, learnable projection head in the initial language model maps the language model outputs into a contrastive space, generating the embeddings required for contrastive learning.

[0076] Specifically, in some embodiments of the present invention, the initial language model includes an input embedding layer, a feature extraction layer, a mean pooling layer, a nonlinear projection layer, and a dynamic task head. Different task-specific projection heads are added as dynamic task heads at different training stages. The dynamic task heads added during the pre-training stage are used to generate the embeddings required for contrastive learning and output the predicted intent labels corresponding to the sample text.

[0077] The input embedding layer converts the input text into an initial vector representation. The feature extraction layer processes the initial vector representation and extracts deep semantic features to obtain token representations.

[0078] After the sample text is input into the initial language model, the input embedding layer receives the input sample text and uses a tokenizer to segment it into a series of tokens. The [CLS] and [SEP] tags are added to the segmented sequence. The segmented sequence is converted into a numerical form that can be processed by the initial language model, namely the initial vector representation, and input into the feature extraction layer. The [CLS] tag is usually added at the beginning of the sequence to indicate the classification information of the entire sequence; the [SEP] tag is used to separate different sentences.

[0079] In some embodiments of the present invention, the feature extraction layer is composed of at least one Transformer layer. Each Transformer layer extracts deep semantic features and outputs the hidden state of the last Transformer layer, including [CLS] and the representation of each token, which is then input into the mean pooling layer.

[0080] The mean pooling layer is used to perform mean pooling on the token representation to obtain a sentence-level semantic vector. Specifically, the mean pooling layer can be expressed as follows:

[0081] x i =mean-pooling([CLS,T1,...,T L ])

[0082] Where x i Represents the sentence-level semantic vector corresponding to the i-th sample text input into the initial language model; [CLS, T1, ..., T L ] represents the token representation corresponding to the i-th sample text, where CLS represents the text classification vector.

[0083] The nonlinear projection layer includes a fully connected layer and an activation function layer, which is used to map the sentence-level semantic vector to a low-dimensional intent representation space, enhance semantic differentiation, and obtain the corresponding intent representation. Specifically, the nonlinear projection layer can be expressed as follows:

[0084] I i =h(x i )=σ(W h x i +b h )

[0085] Where, I i represents the intention representation corresponding to the i-th sample text; σ is the ReLU activation function, W h represents the weight matrix; b h represents the bias term.

[0086] Step S1012: perform data augmentation on each sample text, generate corresponding augmented samples, construct positive sample pairs and negative sample pairs, and mix the annotated augmented samples and the unannotated augmented samples in a preset ratio to obtain at least one training batch.

[0087] The preset ratio refers to the preset mixing ratio of labeled augmented samples to unlabeled augmented samples. In the same training batch, the number of labeled augmented samples is the same as or close to the number of unlabeled augmented samples. Based on this, the preset ratio can be 1:1, 1:1.1, or 1:1.15, etc. This embodiment does not limit the value of the preset ratio.

[0088] In some embodiments of the present invention, the annotated augmented samples and the unannotated augmented samples are mixed in mini-batches. The batch size of each training batch can be 32, 64, or 128. This embodiment does not impose a limit on the size of the training batch.

[0089] Step S1013: Input at least one training batch into the initial language model to obtain the enhanced feature vector and predicted intent label output by the dynamic task head.

[0090] In step S1014, the initial feature vector, predicted intent label, and enhanced feature vector are input into a preset loss function to obtain a loss value. The preset loss function includes a preset semi-supervised contrastive loss function and a cross-entropy loss function. By jointly optimizing the preset semi-supervised contrastive loss function and the cross-entropy loss function, the model effectively utilizes the unlabeled sample text. This combined strategy enables the model to effectively utilize information from unlabeled samples, eliminating the need to rely on a large number of labeled samples, laying a solid feature foundation for subsequent clustering tasks.

[0091] Among them, the preset loss function can be expressed by the following formula:

[0092]

[0093] Where, represents the preset semi-supervised contrast loss function; represents the cross entropy loss function, where z l Indicates the true intention label corresponding to the annotated enhanced sample, y l Indicates the predicted intent label corresponding to the annotated augmented sample.

[0094] In some embodiments of the present invention, a preset semi-supervised contrast loss function is used to fuse the supervisory signal of the labeled sample text with the self-supervised signal of the unlabeled sample text, thereby providing enhanced intent representation initialization. Specifically, the preset semi-supervised contrast loss function Expressed as:

[0095]

[0096] Where z i represents the feature vector corresponding to the i-th sample text; z l represents a set of labeled samples; Z u represents the set of unlabeled samples; P(i) represents the index set of all positive samples that belong to the same intent category as the i-th sample text; p represents the positive sample index; z p represents the feature vector of the positive sample with positive sample index p; j represents the index of the negative sample; z jRepresents the feature vector of the negative sample with negative sample index j; n represents the size of the training batch; τ represents the temperature parameter used to control the distribution of similarity; sim() represents similarity.

[0097] In step S1015 , the gradient is calculated by back propagation, and the optimizer is used to update the initial language model parameters to minimize the loss value. The above process is iterated until the model converges.

[0098] In some embodiments of the present invention, the optimizer only updates the parameters of the last Transformer layer of the feature extraction layer and the dynamic task head. This allows for efficient adaptation to downstream tasks while preserving the general knowledge of the initial language model and avoiding excessive computational costs.

[0099] Specifically, the optimizer is used to update the initial language model parameters, including: updating the parameters of the last Transform layer in the feature extraction layer and the parameters of the dynamic task head through the optimizer.

[0100] Step S1016: When the initial language model converges, the dynamic task head is removed to obtain a pre-trained language model.

[0101] After the initial language model converges, the dynamic task head is removed, retaining the main body of the initial language model. The model is converted into a general feature extractor to obtain a pre-trained language model. The features extracted by the pre-trained language model are used in subsequent clustering iterations.

[0102] However, in subsequent clustering iterations, there's a possibility that labeled and unlabeled examples from the same intent category will be mixed and assigned to different clusters. This phenomenon is known as catastrophic forgetting. Therefore, to mitigate this problem, some embodiments of the present invention include a supervised contrastive learning phase before clustering.

[0103] The supervised contrastive learning phase uses labeled sample text for supervised contrastive learning. At the beginning of each iteration, supervised contrastive learning is performed using labeled sample text data to strengthen the model's memory of known intent categories.

[0104] Specifically, before each round of iterative training of the initial language model based on the preset semi-supervised contrastive loss function, it also includes: using labeled sample text to perform supervised contrastive learning on the pre-trained language model, inputting the true intent label and predicted intent label corresponding to the labeled sample text into the preset supervised contrastive loss function to obtain the supervised contrastive loss value; calculating the gradient through backpropagation, and using the optimizer to update the model parameters of the initial language model to minimize the supervised contrastive loss value, and iterating the above process until the model converges.

[0105] Among them, the preset supervised contrast loss function can be expressed by the following formula:

[0106]

[0107] Where Z i represents the feature vector corresponding to the i-th sample text; P(i) represents the index set of all positive samples belonging to the same intent category as the i-th sample text; p represents the positive sample index; z p represents the feature vector of the positive sample with positive sample index p; j represents the index of the negative sample; z j Represents the feature vector of the negative sample with negative sample index j; n represents the size of the training batch; τ represents the temperature parameter used to control the distribution of similarity; sim() represents similarity.

[0108] Step S102 : performing a clustering operation on the sample text based on the feature vector, taking the clustering result as a pseudo label, and calculating the normalized mutual information between the true label and the pseudo label of the annotated sample text.

[0109] After extracting the feature vector of the sample text through the language model, the K-means algorithm is used to perform initial clustering on the sample text based on the extracted feature vector, dividing the sample text into K different clusters, calculating the distance between each sample text and the cluster center, and assigning the sample text to the nearest cluster. The clustering result obtained is used as the initial pseudo-label of the sample text.

[0110] After the initial clustering is complete, the normalized mutual information (NMI) between the true and pseudo labels of the annotated sample text is calculated to evaluate the similarity between the clustering results and the true labels. A higher NMI value indicates a closer clustering result to the true labels and a better clustering effect.

[0111] Step S103: Determine the confidence score corresponding to each sample in the sample text. The confidence score is obtained by normalizing and weighting the preset indicators and is used to quantify the degree of noise in the pseudo-label.

[0112] In actual implementation, the generated pseudo-labels may contain a large amount of noisy labels, which can make the learning process unreliable. Specifically, during the clustering phase, the noise in the pseudo-labels may cause sample texts to be incorrectly assigned to the wrong clusters, distorting the clustering results. This distorted clustering further affects model learning, causing the model to learn incorrect patterns. Furthermore, during the representation learning phase of the language model, noise further contaminates the learned representations, reducing their discriminative power and reliability, making it difficult for the model to accurately recognize and classify new intents.

[0113] To address the above issues, existing pseudo-label denoising research has mainly focused on two aspects: optimizing the training process and improving the quality of the dataset. However, when applying these traditional methods to new intent discovery tasks, they have obvious limitations. From the perspective of evaluation metrics, existing evaluation metrics are often too simple and single, and cannot fully capture the complexity of user intent scenarios. For example, in practical applications, the data distribution between different intent categories is often unbalanced. Some common intent categories may have a large amount of data, while some rare intent categories have sparse data. Existing evaluation metrics find it difficult to effectively process and evaluate this unbalanced data distribution, resulting in the inability to accurately identify and remove noisy pseudo-labels.

[0114] In order to solve the above-mentioned problems of the prior art, in some embodiments of the present invention, multi-angle preset indicators are used to analyze and evaluate pseudo labels.

[0115] like Figure 4 As shown, the preset indicators include indicators from a spatial perspective and indicators from a temporal perspective. The indicators from a spatial perspective include a reliability indicator and a cohesion indicator, and the indicators from a temporal perspective include a consistency indicator and a stability indicator.

[0116] In some embodiments of the present invention, clustering results are considered from both spatial and temporal perspectives, with cluster-level and instance-level evaluation metrics designed for each perspective. By integrating spatiotemporal features, the pseudo-label corresponding to each sample text is scored to obtain a confidence score. Sample texts are then filtered based on the confidence score, thereby achieving pseudo-label denoising. This process can effectively mitigate the impact of noise propagation on model performance, improving the model's reliability and discriminative ability.

[0117] From a spatial perspective, the reliability of a sample is evaluated by the confidence of the sample text relative to its neighbors, and the cohesion of known intent categories is evaluated by the correlation between different sample texts. From a temporal perspective, the category consistency of a single sample text and the temporal distribution stability of sample texts in the same intent category are examined.

[0118] In the spatial perspective, Figure 5 As shown in , the reliability index refers to the distance between the sample text and the center of the cluster to which it belongs. By calculating the distance between the sample text and the center of the cluster, it reflects the degree to which each sample text belongs to its corresponding cluster. Specifically, the reliability index can be expressed by the following formula:

[0119]

[0120] Among them, R(x i , t) is the sample text x iReliability index in round t of clustering operation; D represents the number of dimensions; x i，j Represents the sample text x i The j-th dimension feature of c i，j Represents sample text x i The j-th dimension feature of the cluster center to which it belongs.

[0121] like Figure 5 As shown in , the cohesion index refers to the information entropy of the pseudo-label distribution of annotated sample texts. For each known intent category, we first obtain the distribution of its pseudo-labels, and then calculate the information entropy of the annotated sample texts under this intent category. Specifically, the cohesion index can be expressed as follows:

[0122]

[0123] Where H(c) is the cohesion index of intent category c; N c is the number of sample texts under intent category c; p i Represents the pseudo label probability of the i-th sample text under intent category c.

[0124] In the time perspective, Figure 5 As shown, the consistency index refers to the frequency with which sample texts are assigned to the same cluster in a preset number of rounds of clustering operations. The preset number of rounds refers to a pre-set number of clustering rounds, which may be 3, 4, or 5 rounds as a window. This embodiment does not limit the value of the preset number of rounds.

[0125] By calculating the consistency index, we assess how often the same sample text is assigned to the same cluster across multiple clustering operations. If a sample text is consistently assigned to the same cluster within a time window, the clustering result is likely more accurate. Conversely, if a sample text changes frequently within a time window, it may be affected by noise, resulting in a lower consistency index.

[0126] Specifically, the cluster consistency index can be expressed as follows:

[0127]

[0128] Among them, C(x i , w) is the sample text x i Cluster consistency indicator within window w; n max (x i , t′) is the sample text x i The maximum number of times a cluster is assigned to the same cluster in t′ rounds of clustering; n(x i , j, t′) is the sample text x iThe number of times a cluster is assigned to the jth cluster in round t′; K represents the total number of clusters within the window w.

[0129] like Figure 5 As shown in Figure 2, the stability metric refers to the rate of change in the pseudo-label distribution of each cluster across successive clustering operations. The stability of each cluster is quantified by comparing its pseudo-label distribution across successive clustering iterations. In other words, if the sample text within an intent category remains completely consistent across successive clustering operations, it demonstrates high stability. By identifying intent categories that maintain a stable distribution during the clustering process, reliable signals are provided for model training.

[0130] Specifically, the stability index can be expressed by the following formula:

[0131]

[0132] Where T(c, t) represents the stability of the intent category c in the t-th clustering operation; N c，t represents the number of sample texts of graph category c in the t-th clustering operation; p i，t represents the pseudo label probability that the i-th sample text belongs to the intent category c in the t-th clustering operation; q i，t-1 represents the pseudo label probability that the i-th sample text belongs to the intent category c in the t-1-th clustering operation.

[0133] After obtaining each preset indicator for all sample texts, each preset indicator is normalized between 0 and 1 and weighted aggregated to obtain a confidence score for each sample text to quantify the degree of noise in the pseudo-label. Specifically, the confidence score can be expressed as follows:

[0134] S(x i ,w,t)=[α·Norm(R(x i ,t))+(1-α)·Norm(C(x i ,w))·[β·Norm(T(c,t))+(1-β)·Norm(H(c))]

[0135] Where α and β represent weight coefficients; R(x i , t) is the sample text x i The reliability index of the clustering operation in round t; T(c, t) represents the stability of the intent category c in the clustering operation in round t; C(x i , w) is the sample text x i The clustering consistency indicator within the window w; H(c) is the cohesion indicator of the intent category c.

[0136] In addition, to further improve the quality of pseudo-labels, after obtaining the confidence score corresponding to each sample text, the sample texts are sorted according to the confidence score, and the pseudo-labels of the last K sample texts in the sort are reassigned, where K is an integer greater than 1.

[0137] In some embodiments of the invention, by analyzing the proximity and historical clustering information of the last K sample texts, neighboring sample voting is performed to correct low-confidence sample texts. This is combined with historical clustering information to correct and reallocate pseudo-labels, correcting some extremely inaccurate pseudo-labels and improving overall pseudo-label quality. This can enhance the model's ability to adapt to complex data distributions while mitigating the adverse effects of noisy labels.

[0138] First, identify the K sample texts with the lowest confidence scores from the sample texts and add them to the set to be corrected as the sample texts to be corrected. Among them, the set to be revised It can be expressed as:

[0139]

[0140] Among them, X is the sample text set; x i represents the i-th sample text in the sample text set; s k Represents sample text x k The confidence score of ; bottomK represents the number of sample texts to which pseudo labels are reassigned.

[0141] After determining the set to be modified Afterwards, for each sample text x in the collection i , use the K-Nearest Neighbor Algorithm (KNN) to analyze the neighbor pseudo-label distribution of these samples, and update the set to be corrected based on the most frequent label in the neighbor pseudo-label distribution . When determining new pseudo-labels, the algorithm considers the historical clustering information of the sample text. If the most frequent label in the distribution of neighbor pseudo-labels is consistent with the historical clustering information, the label corresponding to the historical clustering information is retained. Otherwise, the most frequent label in the distribution of neighbor pseudo-labels is used as the new pseudo-label. This process helps correct possible erroneous pseudo-labels and enhances the language model's understanding of the data clustering structure, thereby improving the language model's performance in new intent discovery tasks.

[0142] Specifically, for the set to be corrected Each sample text x in i , the reallocated pseudo-label PL(x i ) can be expressed as follows:

[0143]

[0144] In the formula, M(x i ) indicates that in the sample text x i The pseudo-label category with the highest frequency among the neighbors of i ) is the sample text x i Neighbor set of (determined by K nearest neighbor algorithm); L(x j ) is the sample text x i The pseudo label of H(x i ) is the same as the sample text x i Pseudo labels related to historical clustering information.

[0145] In addition, when the change rate of the pseudo-label distribution between the current clustering iteration round and the previous clustering iteration round reaches the preset change rate threshold, the reallocation of pseudo-labels is stopped to prevent the model from frequently changing the pseudo-labels of the same sample text during training, thereby causing instability in subsequent language model learning and affecting the convergence of the language model.

[0146] Specifically, after determining the confidence score corresponding to each sample in the sample text, it also includes: when the pseudo-label distribution change rate is less than or equal to a preset change rate threshold, determining the K sample texts with the lowest confidence scores as samples to be corrected; wherein K is an integer greater than 1; for each sample to be corrected, retrieving the pseudo-label probability distribution of the neighboring sample texts in the feature space through the K-nearest neighbor algorithm, and selecting the pseudo-label with the highest frequency from the pseudo-label probability distribution as a candidate pseudo-label; when there is a candidate pseudo-label in the historical pseudo-label corresponding to the sample to be corrected, retaining the historical pseudo-label as the pseudo-label of the sample to be corrected.

[0147] In step S104 , high-confidence samples with confidence scores greater than a dynamic confidence threshold and corresponding pseudo labels are used as self-supervisory signals, and the language model is iteratively optimized through contrastive learning until convergence.

[0148] After each clustering operation, the confidence score corresponding to each sample text is calculated, and the confidence score is compared with the dynamic confidence threshold corresponding to the current clustering round to screen out high-confidence sample texts, thereby denoising the pseudo labels.

[0149] Traditional methods typically consider sample confidence scores statically, ignoring the dynamic changes in feature representations and decision boundaries during intent modeling. In real-world new intent discovery tasks, feature representations and decision boundaries evolve as models train and data is continuously updated. Static confidence score evaluation methods are unable to adapt to these changes, impacting pseudo-label denoising effectiveness and model performance.

[0150] In order to solve the problems in the traditional method, in some embodiments of the present invention, the size of the dynamic confidence threshold is positively correlated with the number of iterations, and different clusters correspond to different dynamic confidence thresholds.

[0151] Specifically, the dynamic confidence threshold is generated based on a global threshold and a local threshold. The global threshold reflects the distribution of confidence scores across the entire sample text dataset and is used to measure the overall credibility of the sample text. The local threshold reflects the distribution of confidence scores within each cluster and is used to measure the credibility of the sample text in a specific cluster.

[0152] In the first round of iteration, the global threshold is initialized to the average confidence score of the unlabeled sample texts, and the local threshold is initialized to the average confidence score corresponding to all sample texts in each cluster.

[0153] Due to the huge amount of data, calculating the confidence scores of all unlabeled data in each iteration or even each training epoch can be very time-consuming. Based on this, in subsequent iterations, the global threshold and local threshold of the previous round are smoothly updated by the Exponential Moving Average (EMA).

[0154] Among them, the global threshold can be expressed by the following formula:

[0155]

[0156] Where, τ t represents the global threshold corresponding to the tth iteration; λ represents the momentum decay of EMA, and λ∈(0,1); N represents the number of sample texts, s i Indicates the confidence of the i-th sample text.

[0157] The local threshold adjusts the global threshold in a class-specific manner to account for variability within each intent class and potential class adjacency issues. To estimate the learning state for each intent class c, the expected value of the model's prediction for each intent class is calculated, which can be expressed as follows:

[0158]

[0159] in, represents the local threshold for intent category c in the tth iteration; represents a vector containing the predicted expected values ​​of all intent categories; The set of sample texts representing intent category c; s i，c Represents the confidence score of the i-th sample text in intent category c.

[0160] By combining the global threshold and the local threshold, the dynamic confidence threshold τ corresponding to the final intent category c is t (c) is expressed as:

[0161]

[0162] Among them, τ t (c) represents the dynamic confidence threshold corresponding to intent category c in the tth iteration; MaxNorm represents maximum normalization; represents the local threshold for intent category c in the tth iteration; τ t Represents the global threshold corresponding to the tth iteration.

[0163] Specifically, before using high-confidence samples and corresponding pseudo-labels with confidence scores greater than the dynamic confidence threshold as self-supervision signals, the process also includes: smoothly updating the global threshold and local threshold of the previous round through exponential moving average; normalizing each updated local threshold; and multiplying the normalized local threshold by the updated global threshold to obtain the dynamic confidence threshold of each cluster in the current iteration round.

[0164] After obtaining the dynamic confidence threshold corresponding to each intent category, sample texts are filtered based on the dynamic confidence threshold. Specifically, for each intent category c, sample texts with confidence scores exceeding the dynamic confidence threshold are first selected. Then, a certain number of sample texts with high confidence scores are filtered from each cluster through Top-K, and a new subset is constructed for subsequent self-supervised training.

[0165] The denoised and reallocated pseudo-labels are used as self-supervisory signals. The language model's intent representation space is iteratively optimized through contrastive learning. The model parameters of the language model are updated until the language model converges to form an intent recognition model. New intents can be discovered through the intent recognition model.

[0166] Specifically, high-confidence sample texts with confidence scores greater than a dynamic confidence threshold and corresponding pseudo-labels are used as self-supervisory signals, and the language model is iteratively optimized through contrastive learning until convergence, including: performing data augmentation on high-confidence sample texts, generating an enhanced version corresponding to each sample text and constructing positive sample pairs and negative sample pairs; inputting the positive sample pairs and negative sample pairs into the language model to extract deep semantic features and obtain training results; inputting the training results into the cross-entropy loss function and the supervised contrast loss function, summing the output values ​​of the two types of loss functions to obtain the total loss value; calculating the gradient through backpropagation, and using the optimizer to update the language model parameters to minimize the total loss value, and iterating the above process until the model converges.

[0167] Among them, the cross entropy loss function can be expressed by the following formula:

[0168]

[0169] Where N is the number of sample texts in a training batch; K is the number of intent categories; j is the jth intent category; z is the feature vector corresponding to the original sample text; z′ is the feature vector corresponding to the enhanced version of the sample text; y is the feature vector corresponding to the original sample text; a Pseudo labels representing sample texts; Represents the cross entropy loss corresponding to the original sample text; represents the cross entropy loss corresponding to the enhanced version of the sample text; z i Represents the feature vector corresponding to the i-th sample text; y i a Represents the pseudo label corresponding to the i-th sample text.

[0170] At the same time, the supervised contrast loss is introduced to improve the model's ability to distinguish subtle semantic differences by maximizing the similarity with the enhanced version of the sample text and minimizing the similarity with the sample text of different intent categories. Specifically, the supervised contrast loss function can be expressed as follows:

[0171]

[0172] Where z i represents the feature vector corresponding to the i-th sample text; P(i) represents the index set of all positive samples belonging to the same intent category as the i-th sample text; p represents the positive sample index; z p represents the feature vector of the positive sample with positive sample index p; j represents the index of the negative sample; z j Represents the feature vector of the negative sample with negative sample index j; n represents the size of the training batch; τ represents the temperature parameter used to control the distribution of similarity; sim() represents similarity.

[0173] The total loss function that sums the output values ​​of the two types of loss functions is expressed as:

[0174]

[0175] Where, Cross entropy loss function; represents the supervised contrastive loss function.

[0176] By simultaneously optimizing two losses, combining instance-level and cluster-level information for learning, and using aligned objectives to train the model, we achieve cluster-friendly representations. By iteratively alternating clustering and representation learning, the model gradually adapts to the complexity and variability of the data, gradually improving its performance and ultimately achieving better discrimination between different intent categories.

[0177] In addition, before iteratively training the pre-trained language model, a new projection head for contrastive learning needs to be added to the pre-trained language model as a dynamic task head. To prevent feature changes from affecting pseudo-label stability, only the parameters for the newly added dynamic task head are optimized when optimizing the language model parameters.

[0178] Specifically, an optimizer is used to update the language model parameters to minimize the total loss value, including: freezing the model parameters of the nonlinear projection layer in the language model, and updating the parameters of the projection head used for contrastive learning through the optimizer.

[0179] Step S105 , taking the centroid of each cluster as a new cluster center, re-clustering and updating the clustering results, normalized mutual information and number of iterations.

[0180] In some embodiments of the present invention, after each clustering iteration, the centroid of each cluster, ie, the average feature vector of all sample texts in the cluster, is calculated; the calculated centroid is saved and used for initialization of the next round of clustering.

[0181] In the next round of clustering, the centroid saved in the previous round of clustering is used as the initial cluster center, and the clustering operation is re-executed on the sample text to obtain a new clustering result, so as to reduce the dependence on the initial center selection and improve the stability and efficiency of clustering.

[0182] After obtaining the new clustering results, a new pseudo label is assigned to each sample text according to the new clustering results, the normalized mutual information is updated, the value of the iteration counter is increased, and the currently completed clustering round is recorded.

[0183] Step S106: When the updated number of iterations reaches a preset upper limit and the normalized mutual information increase is less than a preset threshold, the training is terminated and the current language model is determined as the intent recognition model.

[0184] The preset upper limit refers to a preset upper limit for the number of iterations. When the number of iterations reaches the preset upper limit, training is terminated, the current language model is determined as the intent recognition model, and the model parameters and related data are saved.

[0185] Alternatively, check the changes in normalized mutual information for multiple consecutive rounds (for example, 5 rounds, 6 rounds, or 7 rounds, etc.). If the increase is less than a preset threshold (such as 0.01), terminate the training, determine the current language model as the intent recognition model, and save the model parameters and related data.

[0186] When the updated number of iterations does not reach the preset upper limit and the normalized mutual information increase is greater than the preset threshold, the step of determining the confidence score corresponding to each sample in the sample text is executed, and the sample text is screened for subsequent model training steps.

[0187] For example, taking 5 rounds as an example, in the first round, the normalized mutual information is 0.70; in the second round, the normalized mutual information is 0.72, which changes to 0.02; in the third round, the normalized mutual information is 0.73, which changes to 0.01; in the fourth round, the normalized mutual information is 0.735, which changes to 0.005; in the fifth round, the normalized mutual information is 0.74, which changes to 0.005; that is, in these 5 rounds, the preset termination condition is not met, and the confidence score corresponding to each sample text is continued to be calculated. ; Or, in the first round, the normalized mutual information is 0.71; in the second round, the normalized mutual information is 0.72, which changes to 0.01; in the third round, the normalized mutual information is 0.73, which changes to 0.01; in the fourth round, the normalized mutual information is 0.735, which changes to 0.005; in the fifth round, the normalized mutual information is 0.74, which changes to 0.005; that is, in these 5 rounds, the preset termination condition is met, the current language model is determined to be the intent recognition model, and the iterative training is terminated.

[0188] After obtaining the intent recognition model, the trained model is fine-tuned on the validation set, and the model performance is evaluated on the test set. The Adjusted Rand Index (ARI) and accuracy are used as evaluation indicators to verify the effectiveness of the method in discovering new intents on three intent benchmark datasets (including the banking field dialogue dataset BANKING, the multi-domain intent classification dataset CLINC, and the technical question and answer dataset StackOverflow).

[0189] The verification results show that the model training method provided in this embodiment is significantly better than the existing most advanced new intent discovery method USNID under various experimental settings.

[0190] On the BANKING dataset, in a scenario where the proportion of known intent categories is low (25%), the intent recognition model provided in this embodiment effectively suppresses noise propagation through a spatiotemporal denoising mechanism. Its accuracy is 2.53% higher than the 65.85% of the existing most advanced new intent discovery method USNID, verifying its robustness under limited annotated data.

[0191] On the CLINC dataset, the intent recognition model provided by this embodiment demonstrates stronger generalization capabilities, and due to its domain diversity (covering 10 domains), it is easier to distinguish intent boundaries, and the spatiotemporal feature fusion mechanism effectively captures cross-domain semantic differences.

[0192] For the StackOverflow dataset, the intent recognition model provided by this embodiment achieved an accuracy of 88.53% with 75% labeled samples, a 2.87% improvement over the USNID dataset's 85.66%. This dataset is densely packed with technical terms and has fuzzy intent boundaries, making it difficult for traditional methods to accurately cluster it. By using pseudo-label reallocation to correct the misassignment of low-confidence samples, this model demonstrates a unique advantage in identifying intent from technical questions, achieving a 3.96% improvement in its adjusted Rand Index over the USNID dataset's 74.90%.

[0193] In summary, the pseudo-label-based intention recognition model training method provided in this embodiment inputs the sample text into the pre-trained language model to extract the feature vector; performs clustering operation on the sample text based on the feature vector, uses the clustering result as the pseudo label, and calculates the normalized mutual information between the true label of the labeled sample text and the pseudo label; determines the confidence score corresponding to each sample in the sample text; the confidence score is obtained by normalizing and weighting the preset indicators, which is used to quantify the degree of noise in the pseudo label; the preset indicators include the distance from the sample text to the center of the cluster to which it belongs, the pseudo label distribution information entropy of the labeled sample text, The frequency of sample texts being assigned to the same cluster in a preset number of rounds of clustering operations and the rate of change of pseudo-label distribution for each cluster in continuous clustering operations; using high-confidence samples with confidence scores greater than the dynamic confidence threshold and the corresponding pseudo-labels as self-supervisory signals, and iteratively optimizing the language model through comparative learning until convergence; the size of the dynamic confidence threshold is positively correlated with the number of iterations; different clusters have different dynamic confidence thresholds; re-clustering through the centroid-guided mechanism, updating the clustering results, normalized mutual information, and number of iterations; when the number of iterations after the update reaches the preset upper limit or the increase in normalized mutual information is less than the preset threshold If the training is in the state of being unsatisfactory, the training is terminated and the current language model is determined as the intent recognition model; otherwise, the step of determining the confidence score corresponding to each sample in the sample text is performed. This can solve the problem that as the training is iterated, noisy pseudo-labels will continue to propagate and accumulate, resulting in a significant decrease in the model's ability to recognize new intents. By comprehensively utilizing the spatial and temporal features of the sample text, the pseudo-labels are dynamically optimized. The spatial dimension is evaluated by the reliability of the sample text and the cohesion of the intent category to accurately locate the noisy labels. The temporal dimension is analyzed by the stability and consistency of the pseudo-label distribution to capture the data evolution law. A spatiotemporal fusion pseudo-label quality evaluation mechanism is constructed, which breaks through the limitations of traditional single indicators, can comprehensively capture the semantic characteristics of user intent, reduce the negative impact of noise on model performance, avoid the problem of pseudo-label noise interfering with model training and intent recognition, provide the model with more reliable self-supervision signals, and improve the accuracy, reliability and ability of new intent recognition. At the same time, by dynamically optimizing pseudo-labels and combining with a dynamic confidence threshold adjustment strategy to dynamically screen high-confidence pseudo-labels, it can further suppress the propagation of noise, reduce the negative impact on model performance, and further improve the accuracy and reliability of the model in recognizing new intents.

[0194] Figure 6 This is a flow chart of the intention recognition method provided in the embodiment of the present application. Figure 6 As shown, the intention recognition method includes at least the following steps S601 to S602:

[0195] Step S601: Obtain the text to be recognized input by the user.

[0196] Step S602: input the text to be recognized into the intention recognition model to obtain the intention label corresponding to the text to be recognized.

[0197] The intent labels include user-annotated intent labels and unannotated intent labels. The intent recognition model is trained using the pseudo-label-based intent recognition model training method described in the above embodiment. For details, refer to the above embodiment and will not be repeated here.

[0198] In summary, the intent recognition method provided in this embodiment inputs user-entered text to be recognized into an intent recognition model to obtain an intent label corresponding to the text to be recognized. The intent recognition model is trained using the pseudo-label-based intent recognition model training method described in the above embodiment, improving the accuracy and reliability of identifying new intents in the text to be recognized.

[0199] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the method described above.

[0200] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0201] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0202] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0203] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0204] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A pseudo-label-based intent recognition model training method, characterized in that: The method comprises the following steps: Inputting sample text into a pre-trained language model to extract a feature vector; the sample text includes labeled sample text corresponding to a known intent category, unlabeled sample text corresponding to the known intent category, and unlabeled sample text corresponding to an unknown intent category; Performing a clustering operation on the sample text based on the feature vector, using the clustering result as a pseudo label, and calculating the normalized mutual information between the true label and the pseudo label of the annotated sample text; Determining a confidence score corresponding to each sample in the sample text; the confidence score is obtained by normalizing and weighting preset indicators and is used to quantify the degree of noise in the pseudo-label; the preset indicators include the distance between the sample text and the center of the cluster to which it belongs, the pseudo-label distribution information entropy of the annotated sample text, the frequency of the sample text being assigned to the same cluster in a preset number of rounds of clustering operations, and the pseudo-label distribution change rate of each cluster in consecutive clustering operations; High-confidence samples with confidence scores greater than a dynamic confidence threshold and corresponding pseudo labels are used as self-supervisory signals, and the language model is iteratively optimized through contrastive learning until convergence; the size of the dynamic confidence threshold is positively correlated with the number of iterations; different clusters correspond to different dynamic confidence thresholds; Taking the centroid of each cluster as a new cluster center, re-clustering and updating the clustering result, the normalized mutual information and the number of iterations; When the number of iterations after the update reaches a preset upper limit or the increase in normalized mutual information is less than a preset threshold, the training is terminated and the current language model is determined as the intent recognition model; otherwise, the step of determining the confidence score corresponding to each sample in the sample text is executed.

2. The method according to claim 1, characterized in that The method uses high-confidence sample texts with confidence scores greater than a dynamic confidence threshold and corresponding pseudo labels as self-supervisory signals, and iteratively optimizes the language model through contrastive learning until convergence, including: Performing data enhancement on the high-confidence sample text, generating a corresponding enhanced version and constructing positive sample pairs and negative sample pairs; Inputting the positive sample pair and the negative sample pair into the language model to extract deep semantic features and obtain a training result; Input the training results into the cross entropy loss function and the supervised contrast loss function, and sum the output values ​​of the two loss functions to obtain a total loss value; The gradient is calculated by backpropagation, and the language model parameters are updated using an optimizer to minimize the total loss value. The above process is iterated until the model converges.

3. The method according to claim 1, characterized in that The pre-training process of the language model includes: The sample text is input into the initial language model for feature extraction to obtain an initial feature vector; the sample text includes annotated sample text and unannotated sample text; the initial language model includes an input embedding layer, a feature extraction layer, a mean pooling layer, a nonlinear projection layer and a dynamic task head; the input embedding layer is used to convert the input text into an initial vector representation; the feature extraction layer is used to process the initial vector representation and extract deep semantic features to obtain a token representation; the mean pooling layer is used to perform a mean pooling operation on the token representation to obtain a sentence-level semantic vector; the nonlinear projection layer is used to map the sentence-level semantic vector to a low-dimensional intent representation space to obtain a corresponding intent representation; the dynamic task head is used to embed and output a predicted intent label required for contrastive learning; Performing data augmentation on each sample text in the sample text to generate corresponding augmented samples, constructing positive sample pairs and negative sample pairs, and mixing the annotated augmented samples and the unannotated augmented samples in a preset ratio to obtain at least one training batch; Inputting the at least one training batch into the initial language model to obtain an enhanced feature vector and a predicted intent label output by the dynamic task head; Inputting the initial feature vector, the predicted intent label, and the enhanced feature vector into a preset loss function to obtain a loss value; the preset loss function includes a preset semi-supervised contrast loss function and a cross entropy loss function; Calculating gradients through backpropagation, using an optimizer to update the initial language model parameters to minimize the loss value, and iterating the above process until the model converges; When the initial language model converges, the dynamic task head is removed to obtain the pre-trained language model.

4. The method according to claim 3, characterized in that Before performing each round of iterative training on the initial language model based on the preset semi-supervised contrastive loss function, the method further includes: Using the annotated sample text to perform supervised contrastive learning on the pre-trained language model, inputting the true intent label and the predicted intent label corresponding to the annotated sample text into a preset supervised contrastive loss function to obtain a supervised contrastive loss value; The gradient is calculated by backpropagation, and the model parameters of the initial language model are updated using an optimizer to minimize the supervised contrast loss value. The above process is iterated until the model converges.

5. The method according to claim 3, characterized in that The feature extraction layer includes at least one Transform layer; and the updating of the initial language model parameters using the optimizer includes: updating the parameters of the last Transform layer in the feature extraction layer and the parameters of the dynamic task head through the optimizer.

6. The method according to claim 1, characterized in that Before using the high-confidence samples with confidence scores greater than the dynamic confidence threshold and the corresponding pseudo labels as self-supervisory signals, the method further includes: When the pseudo-label distribution change rate is less than or equal to a preset change rate threshold, K sample texts with the lowest confidence scores are determined as samples to be corrected; wherein K is an integer greater than 1; For each sample to be corrected, the pseudo-label probability distribution of the adjacent sample texts in the feature space is retrieved by the K-nearest neighbor algorithm, and the pseudo-label with the highest frequency is selected as the candidate pseudo-label from the pseudo-label probability distribution; If the candidate pseudo label exists in the historical pseudo label corresponding to the sample to be corrected, the historical pseudo label is retained as the pseudo label of the sample to be corrected.

7. The method according to claim 1, characterized in that The dynamic confidence threshold is generated based on the global threshold and the local threshold. In the first round of iteration, the global threshold is initialized to the average confidence score of the unlabeled sample texts, and the local threshold is initialized to the average confidence score corresponding to all sample texts in each cluster. In subsequent iterations, the global threshold and the local threshold of the previous round are smoothly updated by exponential moving average. Before using the high-confidence samples with confidence scores greater than the dynamic confidence threshold and the corresponding pseudo labels as self-supervisory signals, the method further includes: Smoothing the global threshold value of the previous round and the local threshold value of the previous round by using the exponential moving average; Normalize each updated local threshold; Multiply the normalized local threshold by the updated global threshold to obtain the dynamic confidence threshold of each cluster in the current iteration round.

8. A method for identifying intention, characterized in that: The method comprises: Get the text to be recognized input by the user; The text to be recognized is input into an intention recognition model to obtain an intention label corresponding to the text to be recognized; the intention recognition model is trained using the method described in any one of claims 1 to 7.

9. An electronic device comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the electronic device implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images