Pseudo-label based intent recognition model training method, intent recognition method and device

Abstract

The application provides an intent recognition model training method and device based on pseudo labels, and an intent recognition method and device, which comprises the following steps: inputting sample text into a language model to extract a feature vector; clustering the sample text based on the feature vector, taking the clustering result as a pseudo label, and calculating the normalized mutual information between the real label of the labeled sample text and the pseudo label; determining the confidence score corresponding to each sample; the confidence score is used to quantify the noise in the pseudo label, filter high-confidence samples, and take the corresponding pseudo label as a self-supervised signal to iteratively optimize the language model until convergence; after iteration, reinitialize the clustering, update the clustering result, the normalized mutual information, and the iteration number; when the iteration number reaches an upper limit or the normalized mutual information increment is less than a threshold, terminate the training and determine the language model as an intent recognition model; the problem that noise pseudo labels continuously spread and accumulate, leading to a decline in the ability of the model to recognize new intents, can be solved; and the ability of the model to recognize new intents is improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method for training an intent recognition model based on pseudo-labels, an intent recognition method, and an apparatus. Background Technology

[0002] In today's digital age, dialogue systems, with their convenient interaction methods, have been widely used in customer service, intelligent assistants, recommendation systems, and other fields. For these dialogue systems, accurately understanding user intent is the core of providing an efficient and high-quality interactive experience. However, as user needs continue to evolve and diversify, relying solely on a limited set of predefined intent categories is no longer sufficient to meet the complex and ever-changing demands of real-world scenarios. It is against this backdrop that the New Intent Discovery (NID) task has emerged, and its importance is increasingly evident. By discovering and adapting to dynamically changing user needs, NID can significantly improve the service quality and user satisfaction of dialogue systems.

[0003] Currently, semi-supervised learning strategies are becoming a research hotspot in the field of novel intent discovery. Among them, self-training (pseudo-labeling) is one of the commonly used methods in novel intent discovery tasks, which is usually combined with clustering and representation learning. Specifically, this method first trains an initial model using a small amount of labeled data, then uses the model to predict a large amount of unlabeled data to generate pseudo-labels, and finally adds these pseudo-labeled data to the training set to further optimize the model performance.

[0004] However, this semi-supervised approach also presents new challenges, namely the potential introduction of noise through pseudo-labels. These noisy pseudo-labels can lead to incorrect assignment of samples to different clusters during the clustering stage, compromising the accuracy and rationality of the clustering results. During the representation learning stage, they can contaminate the learned feature representations, causing the model to learn incorrect information. More seriously, as training iterates, noisy pseudo-labels continue to propagate and accumulate, significantly reducing the model's ability to recognize new intentions. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a training method, an intent recognition method, and an apparatus for intent recognition models based on pseudo-labels, to eliminate or improve one or more defects existing in the prior art. This addresses the problem that as training iteratively progresses, noisy pseudo-labels continuously propagate and accumulate, leading to a significant decrease in the model's ability to recognize new intents.

[0006] One aspect of the present invention provides a method for training an intent recognition model based on pseudo-labels, the method comprising the following steps:

[0007] The sample text is input into a pre-trained language model to extract feature vectors; the sample text includes labeled sample text corresponding to known intent categories, unlabeled sample text corresponding to known intent categories, and unlabeled sample text corresponding to unknown intent categories.

[0008] Clustering is performed on the sample text based on the feature vectors, the clustering results are used as pseudo-labels, and the normalized mutual information between the real labels and pseudo-labels of the labeled sample text is calculated.

[0009] Determine the confidence score for each sample in the sample text; the confidence score is obtained by normalizing and weighting the aggregation of preset indicators, and is used to quantify the noise level in the pseudo-labels; the preset indicators include the distance from the sample text to the center of its cluster, the pseudo-label distribution information entropy of the labeled sample text, the frequency of the sample text being assigned to the same cluster in a preset number of clustering operations, and the pseudo-label distribution change rate of each cluster in continuous clustering operations;

[0010] High-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels are used as self-supervised signals. The language model is iteratively optimized through comparative learning until convergence. The dynamic confidence threshold is positively correlated with the number of iterations. Different clusters correspond to different dynamic confidence thresholds.

[0011] Use the centroid of each cluster as the new cluster center, re-cluster, and update the clustering results, normalized mutual information, and number of iterations;

[0012] If the number of iterations after the update reaches the preset upper limit or the increase in normalized mutual information is less than the preset threshold, the training is terminated and the current language model is identified as the intent recognition model; otherwise, the step of determining the confidence score corresponding to each sample in the sample text is executed.

[0013] In some embodiments of the present invention, high-confidence sample texts with confidence scores greater than a dynamic confidence threshold and their corresponding pseudo-labels are used as self-supervised signals. The language model is iteratively optimized through contrastive learning until convergence, including:

[0014] Data augmentation is performed on high-confidence sample texts to generate corresponding augmented versions and construct positive and negative sample pairs.

[0015] The positive and negative sample pairs are input into the language model to extract deep semantic features, and the training results are obtained.

[0016] The training results are input into the cross-entropy loss function and the supervised contrastive loss function. The output values ​​of the two loss functions are summed to obtain the total loss value.

[0017] The gradient is calculated through backpropagation, and the language model parameters are updated using an optimizer to minimize the total loss. This process is repeated iteratively until the model converges.

[0018] In some embodiments of the present invention, the pre-training process of the language model includes:

[0019] The sample text is input into the initial language model for feature extraction, resulting in an initial feature vector. The sample text includes both labeled and unlabeled samples. The initial language model comprises an input embedding layer, a feature extraction layer, a mean pooling layer, a non-linear projection layer, and a dynamic task head. The input embedding layer transforms the input text into an initial vector representation. The feature extraction layer processes the initial vector representation, extracting deep semantic features to obtain a token representation. The mean pooling layer performs mean pooling on the token representation to obtain a sentence-level semantic vector. The non-linear projection layer maps the sentence-level semantic vector to a low-dimensional intent representation space to obtain the corresponding intent representation. The dynamic task head is used to generate the embeddings required for contrastive learning and output predicted intent labels.

[0020] Data augmentation is performed on each sample text in the sample text to generate corresponding augmented samples. Positive sample pairs and negative sample pairs are constructed, and labeled augmented samples and unlabeled augmented samples are mixed according to a preset ratio to obtain at least one training batch.

[0021] Input at least one training batch into the initial language model to obtain the enhanced feature vector and predicted intent label output by the dynamic task head;

[0022] The initial feature vector, the predicted intent label, and the enhanced feature vector are input into the preset loss function to obtain the loss value; the preset loss function includes the preset semi-supervised contrastive loss function and the cross-entropy loss function;

[0023] The gradient is calculated by backpropagation, and the initial language model parameters are updated by an optimizer to minimize the loss value. The above process is iteratively executed until the model converges.

[0024] If the initial language model converges, the dynamic task head is removed to obtain a pre-trained language model.

[0025] In some embodiments of the present invention, before performing each round of iterative training on the initial language model based on a preset semi-supervised contrastive loss function, the method further includes:

[0026] Supervised contrastive learning is performed on the pre-trained language model using labeled sample text. The real intent label and predicted intent label corresponding to the labeled sample text are input into the preset supervised contrastive loss function to obtain the supervised contrastive loss value.

[0027] The gradient is calculated through backpropagation, and the model parameters of the initial language model are updated using an optimizer to minimize the supervised contrastive loss value. The above process is iteratively executed until the model converges.

[0028] In some embodiments of the present invention, the feature extraction layer includes at least one Transform layer; updating the initial language model parameters using an optimizer includes updating the parameters of the last Transform layer in the feature extraction layer and the parameters of the dynamic task head through the optimizer.

[0029] In some embodiments of the present invention, before using high-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels as self-supervised signals, the method further includes:

[0030] If the rate of change of the pseudo-label distribution is less than or equal to a preset rate of change threshold, the K sample texts with the lowest confidence scores are identified as samples to be corrected; where K is an integer greater than 1.

[0031] For each sample to be corrected, the pseudo-label probability distribution of neighboring sample texts in the feature space is retrieved using the K-nearest neighbor algorithm, and the pseudo-label with the highest frequency is selected as the candidate pseudo-label from the pseudo-label probability distribution.

[0032] If candidate pseudo-labels exist in the historical pseudo-labels corresponding to the sample to be corrected, the historical pseudo-labels are retained as pseudo-labels for the sample to be corrected.

[0033] In some embodiments of the present invention, the dynamic confidence threshold is generated based on a global threshold and a local threshold. In the first iteration, the global threshold is initialized to the average confidence score of the unlabeled sample text, and the local threshold is initialized to the average confidence score of all sample texts in each cluster. In subsequent iterations, the global threshold and the local threshold of the previous round are updated smoothly by exponential moving average.

[0034] Before using high-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels as self-supervised signals, the process also includes:

[0035] The global threshold and the local threshold of the previous round are updated smoothly by using an exponential moving average.

[0036] Normalize each updated local threshold;

[0037] Multiply the normalized local threshold by the updated global threshold to obtain the dynamic confidence threshold for each cluster in the current iteration.

[0038] Another aspect of the present invention provides an intent recognition method, the method comprising:

[0039] Obtain the text to be recognized by the user input;

[0040] The text to be recognized is input into the intent recognition model to obtain the intent label corresponding to the text to be recognized; the intent recognition model is trained using the above-mentioned pseudo-label-based intent recognition model training method.

[0041] Another aspect of the present invention provides an electronic device including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the electronic device implements the steps of the aforementioned pseudo-label-based intent recognition model training method and intent recognition method.

[0042] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the aforementioned pseudo-label-based intent recognition model training method and intent recognition method.

[0043] The present invention has at least the following beneficial effects:

[0044] The present invention provides a pseudo-label-based intent recognition model training method, intent recognition method, and apparatus. These methods address the problem that as training iteratively progresses, the propagation and accumulation of noisy pseudo-labels significantly reduces the model's ability to recognize new intents. By dynamically optimizing pseudo-labels using both spatial and temporal features of the sample text, the spatial dimension is evaluated through the reliability of the sample text and the cohesion of the intent category, accurately locating noisy labels. The temporal dimension is analyzed through the stability and consistency of pseudo-label distribution, capturing data evolution patterns and constructing a spatiotemporally integrated pseudo-label quality evaluation mechanism. This overcomes the limitations of traditional single-indicator methods, comprehensively capturing the semantic features of user intent, reducing the negative impact of noise on model performance, and avoiding the interference of pseudo-label noise with model training and intent recognition. It provides the model with more reliable self-supervised signals, improving the accuracy, reliability, and ability to recognize new intents. Furthermore, by dynamically optimizing pseudo-labels and combining this with a dynamic confidence threshold adjustment strategy, high-confidence pseudo-labels are dynamically selected, further suppressing noise propagation, mitigating the negative impact on model performance, and further improving the model's accuracy and reliability in recognizing new intents.

[0045] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0046] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0048] Figure 1 This is a flowchart of a pseudo-label-based intent recognition model training method provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of sample text and intent tags provided in an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the framework for training an intent recognition model based on pseudo-labels according to an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram of a pseudo-tag denoising framework provided in an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram illustrating spatial and temporal perspective indicators provided in an embodiment of the present invention.

[0053] Figure 6 A flowchart of an intent recognition method provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0055] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0056] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0057] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0058] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0059] The following section provides a detailed introduction to the training method for the pseudo-label-based intent recognition model provided in this application.

[0060] The execution subject of the pseudo-label-based intent recognition model training method provided in this application is an electronic device, which can be a terminal such as a computer, mobile phone, tablet computer, or camera, or it can be a server. This embodiment does not limit the implementation method of the electronic device.

[0061] This embodiment provides a method for training an intent recognition model based on pseudo-labels, such as... Figure 1 As shown, the method includes at least steps S101 to S106:

[0062] Step S101: Input the sample text into the pre-trained language model to extract the feature vector.

[0063] In some embodiments of the present invention, the sample text includes labeled sample text corresponding to known intent categories, unlabeled sample text corresponding to known intent categories, and unlabeled sample text corresponding to unknown intent categories.

[0064] For example: Reference Figure 2 Taking known intent categories including "automatic top-up," "transfer," "forgot password," "exchange rate," and "age restriction" as examples, the text "My password has been changed" is labeled as the intent "balance inquiry," serving as an labeled sample text; the text "I haven't received my credit card yet" is labeled as the intent "card arrived," also serving as a labeled sample text. Unlabeled sample text may contain the text "I want to know how to check my balance," which is the same as the known intent category "balance inquiry" but is not labeled; the text "I recently found my card! Can you help me rebuild it?" has no corresponding known intent as a label, therefore the intent of this sample text is an unknown intent category.

[0065] In some embodiments of the present invention, the sample text is sample text from an intent benchmark dataset. This intent benchmark dataset includes, but is not limited to, the banking domain dialogue dataset (BANKING), the multi-domain intent classification dataset (CLINC), and the technical question-answering dataset (StackOverflow).

[0066] Among them, the banking domain dialogue dataset contains 13,083 customer service queries in the banking domain, covering 77 intent categories; the multi-domain intent classification dataset contains 22,500 cross-domain queries, covering 150 intent categories; and the technical question-and-answer dataset contains 20,000 technical question titles, covering 20 intent categories.

[0067] In some embodiments of the present invention, 25%, 50%, and 75% of the categories are randomly selected as known intent classes in each dataset. For each known intent class, 10% of the sample text is labeled, and the remaining unlabeled sample text contains both known intents and new intents.

[0068] In some embodiments of the present invention, the number of unlabeled sample texts is greater than that of labeled sample texts. By combining unlabeled and labeled sample texts to train the language model, the ability of the language model to recognize known intents can be effectively improved, while also discovering potential new intents.

[0069] In some embodiments of the present invention, the language model refers to the model used to extract features from sample text, obtained by training the initial language model through a pre-training phase. Since the pre-trained Bidirectional Encoder Representations from Transformers (BERT) model has demonstrated superior performance in Natural Language Processing (NLP) tasks, in some embodiments of the present invention, the pre-trained BERT model is used as the initial language model.

[0070] In practice, the initial language model can also be other language models or improved models of BERT, such as lightweight BERT (ALite BERT, ALBERT), enhanced BERT (Decoding-enhanced BERT with disentangled attention, DeBERTa), text-to-text transfer transformer (T5), etc. This embodiment does not limit the model type of the initial language model.

[0071] Specifically, the pre-training phase of the language model includes at least steps S1011 to S1016:

[0072] Step S1011: Input the sample text into the initial language model for feature extraction to obtain the initial feature vector. The sample text includes labeled sample text with intent tags and unlabeled sample text without intent tags.

[0073] In some embodiments of the present invention, the pre-training phase includes a semi-supervised pre-training phase. A semi-supervised pre-training phase refers to the phase of semi-supervised training of the extracted initial intent representation.

[0074] like Figure 3 As shown, sample text from the intent benchmark dataset is input into the initial language model. The initial language model extracts features from both labeled and unlabeled sample texts, generating initial feature vectors as the initial intent representation. After extracting the initial intent representation, the model proceeds to the semi-supervised pre-training stage. The purpose of the semi-supervised pre-training stage is to fully utilize the limited number of labeled sample texts to provide a good initial representation for subsequent clustering operations.

[0075] In the semi-supervised pre-training phase, firstly, data augmentation (such as random dropout) is applied to both labeled and unlabeled sample texts to generate augmented sample pairs. Then, the augmented samples from labeled and unlabeled sample texts are mixed into the same training batch, and the feature representation is optimized through a contrastive learning mechanism. In the contrastive learning phase, the language model output is mapped to the contrastive space using the dynamically learnable projector in the initial language model, generating the embeddings required for contrastive learning.

[0076] Specifically, in some embodiments of the present invention, the initial language model includes an input embedding layer, a feature extraction layer, an average pooling layer, a nonlinear projection layer, and a dynamic task head. Different task-specific projection heads are added as dynamic task heads at different training stages. The dynamic task heads added during the pre-training stage are used to generate the embeddings required for contrastive learning, as well as the predicted intent labels corresponding to the output sample text.

[0077] The input embedding layer transforms the input text into an initial vector representation. The feature extraction layer processes the initial vector representation, extracting deep semantic features to obtain the token representation.

[0078] After the sample text is input into the initial language model, the input embedding layer receives the input sample text and uses a tokenizer to segment it into a series of tokens. [CLS] and [SEP] markers are added to the segmented sequence. The segmented sequence is then converted into a numerical form that the initial language model can process, i.e., an initial vector representation, and input into the feature extraction layer. The [CLS] marker is typically added at the beginning of the sequence to represent the classification information of the entire sequence; the [SEP] marker is used to separate different sentences.

[0079] In some embodiments of the present invention, the feature extraction layer consists of at least one Transform layer. Deep semantic features are extracted through each Transformer layer, and the hidden state of the last Transformer layer, including [CLS] and the representation of each token, is output and fed into the mean pooling layer.

[0080] The mean pooling layer is used to perform mean pooling on the token representation to obtain a sentence-level semantic vector. Specifically, the mean pooling layer can be represented by the following formula:

[0081] x i =mean-pooling([CLS,T1,...,T L ])

[0082] In the formula, x i This represents the sentence-level semantic vector corresponding to the i-th sample text input to the initial language model; [CLS, T1, ..., T L ] represents the token representation corresponding to the i-th sample text, where CLS represents the text classification vector.

[0083] The nonlinear projection layer, comprising a fully connected layer and an activation function layer, maps sentence-level semantic vectors to a low-dimensional intent representation space, enhancing semantic discriminability and yielding the corresponding intent representation. Specifically, the nonlinear projection layer can be represented by the following equation:

[0084] I i =h(x i )=σ(W h x i +b h )

[0085] In the formula, I i Let W represent the intent representation corresponding to the i-th sample text; σ is the ReLU activation function, and W... h b represents the weight matrix; h This indicates the bias term.

[0086] Step S1012: Perform data augmentation on each sample text to generate corresponding augmented samples, construct positive sample pairs and negative sample pairs, and mix labeled augmented samples and unlabeled augmented samples according to a preset ratio to obtain at least one training batch.

[0087] The preset ratio refers to the pre-set mixing ratio of labeled augmented samples and unlabeled augmented samples. In the same training batch, the number of labeled augmented samples is the same as or close to the number of unlabeled augmented samples. Based on this, the preset ratio can be 1:1, 1:1.1, or 1:1.15, etc. This embodiment does not limit the value of the preset ratio.

[0088] In some embodiments of the present invention, labeled augmented samples and unlabeled augmented samples are mixed in mini-batches. The batch size can be 32, 64, or 128; this embodiment does not limit the batch size.

[0089] Step S1013: Input at least one training batch into the initial language model to obtain the enhanced feature vector and predicted intent label of the dynamic task head output.

[0090] Step S1014 involves inputting the initial feature vector, predicted intent label, and enhanced feature vector into a preset loss function to obtain the loss value. The preset loss function includes a preset semi-supervised contrastive loss function and a cross-entropy loss function. By jointly optimizing the preset semi-supervised contrastive loss function and the cross-entropy loss function, the model effectively utilizes unlabeled sample text. This combined strategy enables the model to effectively utilize information from unlabeled samples, thus eliminating the need to rely on a large number of labeled samples and laying a solid feature foundation for subsequent clustering tasks.

[0091] The preset loss function can be expressed by the following formula:

[0092]

[0093] In the formula, This indicates a predefined semi-supervised contrastive loss function; Let z represent the cross-entropy loss function, where z l This indicates the true intent label corresponding to the annotated augmented sample, y l This indicates that there are pre-defined labels for the augmented samples.

[0094] In some embodiments of the present invention, a preset semi-supervised contrastive loss function is used to fuse the supervised signal of labeled sample text with the self-supervised signal of unlabeled sample text, thereby providing enhanced intent representation initialization. Specifically, the preset semi-supervised contrastive loss function... Represented as:

[0095]

[0096] In the formula, z i z represents the feature vector corresponding to the i-th sample text; l Z represents the set of labeled samples; u Let z represent the unlabeled sample set; P(i) represent the set of indices of all positive samples belonging to the same intent category as the i-th sample text; p represents the positive sample index; z p Let p represent the feature vector of the positive sample with index p; j represents the index of the negative sample; z represents the feature vector of the positive sample with index p. jrepresents the feature vector of the negative sample with index j; n represents the size of the training batch; τ represents the temperature parameter used to control the distribution of similarity; sim() represents similarity.

[0097] Step S1015: Calculate the gradient through backpropagation, update the initial language model parameters using the optimizer to minimize the loss value, and iteratively execute the above process until the model converges.

[0098] In some embodiments of the present invention, the optimizer only updates the parameters of the last Transformer layer of the feature extraction layer and the dynamic task head. This allows for efficient adaptation to the needs of downstream tasks while preserving the general knowledge of the initial language model, and avoids excessive computational costs.

[0099] Specifically, the initial language model parameters are updated using an optimizer, including updating the parameters of the last Transform layer in the feature extraction layer and the parameters of the dynamic task head.

[0100] Step S1016: If the initial language model converges, remove the dynamic task head to obtain the pre-trained language model.

[0101] After the initial language model converges, the dynamic task head is removed, retaining only the backbone of the initial language model. This transforms the model into a general feature extractor, resulting in a pre-trained language model. The features extracted by the pre-trained language model are used for subsequent clustering iterations.

[0102] However, in subsequent clustering iterations, there is a possibility that labeled and unlabeled sample texts from the same intent category may be mixed and assigned to different clusters; this phenomenon is known as catastrophic forgetting. Therefore, to mitigate the catastrophic forgetting problem, in some embodiments of the present invention, a supervised contrastive learning phase is included before performing clustering operations.

[0103] The supervised contrastive learning phase refers to the phase where supervised contrastive learning is performed using labeled sample text. At the beginning of each iteration, supervised contrastive learning is first performed using labeled sample text data to strengthen the model's memory of known intent categories.

[0104] Specifically, before each iteration of training the initial language model based on the preset semi-supervised contrastive loss function, the process includes: using labeled sample text to perform supervised contrastive learning on the pre-trained language model, inputting the real intent label and predicted intent label corresponding to the labeled sample text into the preset supervised contrastive loss function to obtain the supervised contrastive loss value; calculating the gradient through backpropagation, using an optimizer to update the model parameters of the initial language model to minimize the supervised contrastive loss value, and iteratively executing the above process until the model converges.

[0105] The pre-defined supervised comparison loss function can be expressed by the following formula:

[0106]

[0107] In the formula, Z i Let z represent the feature vector corresponding to the i-th sample text; P(i) represents the set of indices of all positive samples belonging to the same intent category as the i-th sample text; p represents the positive sample index; z p Let p represent the feature vector of the positive sample with index p; j represents the index of the negative sample; z represents the feature vector of the positive sample with index p. j represents the feature vector of the negative sample with index j; n represents the size of the training batch; τ represents the temperature parameter used to control the distribution of similarity; sim() represents similarity.

[0108] Step S102: Perform clustering operation on the sample text based on the feature vector, use the clustering result as pseudo label, and calculate the normalized mutual information between the real label and the pseudo label of the labeled sample text.

[0109] After extracting the feature vectors of the sample text through the language model, the K-means algorithm is used to perform initial clustering of the sample text based on the extracted feature vectors, dividing the sample text into K different clusters. The distance between each sample text and the center of the cluster is calculated, and the sample text is assigned to the nearest cluster. The resulting clustering results are used as the initial pseudo-labels of the sample text.

[0110] After initial clustering, the similarity between the clustering results and the true labels is evaluated by calculating the normalized mutual information (NMI) between the real and pseudo labels of the labeled sample text. A higher NMI value indicates that the clustering results are closer to the true labels, and the better the clustering effect.

[0111] Step S103: Determine the confidence score corresponding to each sample in the sample text. The confidence score is obtained by normalizing and weighting the preset indicators, and is used to quantify the noise level in the pseudo-labels.

[0112] In practice, the generated pseudo-labels may contain a large number of noisy labels, which can render the learning process unreliable. Specifically, during the clustering stage, noise in the pseudo-labels may cause sample texts to be incorrectly assigned to incorrect clusters, thus distorting the clustering results. This distorted clustering result will further affect the model's learning, causing the model to learn incorrect patterns. Furthermore, during the representation learning stage of the language model, noise can further contaminate the learned representations, reducing their discriminative power and reliability, making it difficult for the model to accurately identify and classify novel intentions.

[0113] To address the aforementioned issues, existing research on pseudo-label denoising primarily focuses on two aspects: optimizing the training process and improving dataset quality. However, these traditional methods exhibit significant limitations when applied to novel intent discovery tasks. From an evaluation metric perspective, existing metrics are often too simplistic and singular, failing to adequately capture the complexity of user intent scenarios. For instance, in practical applications, data distribution across different intent categories is typically unbalanced; some common intent categories may have abundant data, while rare intent categories have scarce data. Existing evaluation metrics struggle to effectively handle and evaluate this imbalanced data distribution, resulting in an inability to accurately identify and remove noisy pseudo-labels.

[0114] To address the problems of the prior art, in some embodiments of the present invention, multi-angle preset indicators are used to analyze and evaluate pseudo-labels.

[0115] like Figure 4 As shown, the preset indicators include indicators from a spatial perspective and indicators from a temporal perspective. Among them, the indicators from a spatial perspective include reliability indicators and cohesion indicators, while the indicators from a temporal perspective include consistency indicators and stability indicators.

[0116] In some embodiments of this invention, feature information is considered from both spatial and temporal perspectives regarding the clustering results, and cluster-level and instance-level evaluation metrics are designed for each perspective. By fusing spatiotemporal features, the pseudo-labels corresponding to each sample text are scored to obtain a confidence score. Sample texts are then filtered based on the confidence score, thereby achieving denoising of pseudo-labels. This process can effectively mitigate the impact of noise propagation on model performance and improve the model's reliability and discriminative ability.

[0117] From a spatial perspective, the reliability of a sample is assessed by the confidence level of the sample text relative to its neighbors, and the cohesion of known intent categories is evaluated by the correlation between different sample texts. From a temporal perspective, the category consistency of individual sample texts and the stability of the distribution of sample texts within the same intent category over time are examined.

[0118] In a spatial perspective, such as Figure 5 As shown, the reliability index refers to the distance from a sample text to the center of its corresponding cluster. By calculating the distance between a sample text and the cluster center, it reflects the degree to which each sample text belongs to its corresponding cluster. Specifically, the reliability index can be expressed by the following formula:

[0119]

[0120] Where R(x) i ,t) represents the sample text x iThe reliability metric for t rounds of clustering operations; D represents the number of dimensions; x i，j Indicates sample text x i The j-th dimension feature; c i，j Indicates sample text x i The j-th dimension feature of the cluster center to which it belongs.

[0121] like Figure 5 As shown, the cohesion metric refers to the information entropy of the pseudo-label distribution of labeled sample text. For each known intent category, the distribution of its pseudo-labels is first obtained, and then the information entropy of the labeled sample text under that intent category is calculated. Specifically, the cohesion metric can be expressed by the following formula:

[0122]

[0123] In the formula, H(c) is the cohesion index of intent category c; N c p represents the number of sample texts under intent category c; i This represents the pseudo-label probability of the i-th sample text under intent category c.

[0124] From a time perspective, such as Figure 5 As shown, the consistency index refers to the frequency with which sample texts are assigned to the same cluster in a preset number of clustering rounds. The preset number of rounds refers to a pre-defined number of clustering rounds, which can be a window, including 3, 4, or 5 rounds, etc. This embodiment does not limit the value of the preset number of rounds.

[0125] The consistency index is used to assess how frequently the same sample text is assigned to the same cluster in multiple clustering operations. If a sample text is consistently assigned to the same cluster within a time window, the clustering results are likely more accurate; conversely, if a sample text changes frequently within a time window, it may be affected by noise, resulting in a lower consistency index.

[0126] Specifically, the clustering consistency index can be expressed by the following formula:

[0127]

[0128] Wherein, C(x) i w) represents the sample text x i Clustering consistency metrics within window w; n max (x i ,t′) is the sample text x i The maximum number of times n(x) is assigned to the same cluster in round t′ of clustering; i (j, t′) represents the sample text x iThe number of times a cluster is assigned to the j-th cluster in round t′; K represents the total number of clusters within window w.

[0129] like Figure 5 As shown, the stability index refers to the rate of change in the pseudo-label distribution of each cluster during successive clustering operations. The stability of each cluster is quantified by comparing the pseudo-label distribution of each cluster in successive clustering iterations. In other words, high stability is indicated when the sample text within an intent category remains completely consistent across successive clustering operations. Identifying intent categories that maintain stable assignments during clustering provides a reliable signal for model training.

[0130] Specifically, the stability index can be expressed by the following formula:

[0131]

[0132] In the formula, T(c, t) represents the stability of the intention category c in the t-th clustering operation; N c，t p represents the number of sample texts in graph category c during the t-th clustering operation; i，t q represents the pseudo-label probability that the i-th sample text belongs to intention category c in the t-th clustering operation; i，t-1 Let represent the pseudo-label probability that the i-th sample text belongs to the intent category c in the (t-1)-th clustering operation.

[0133] After obtaining each preset index from all sample texts, each preset index in the range of 0 to 1 is normalized and then weighted and aggregated to obtain a confidence score for each sample text, thus quantifying the noise level in the pseudo-labels. Specifically, the confidence score can be expressed by the following formula:

[0134] S(x i ,w,t)=[α·Norm(R(x i ,t))+(1-α)·Norm(C(x i ,w))·[β·Norm(T(c,t))+(1-β)·Norm(H(c))]

[0135] In the formula, α and β represent weighting coefficients; R(x i ,t) represents the sample text x i The reliability metric for clustering operations in round t; T(c,t) represents the stability of the intention category c in the t-th clustering operation; C(x i w) represents the sample text x i The clustering consistency index within window w; H(c) is the cohesion index of intent category c.

[0136] Furthermore, to further improve the quality of pseudo-labels, after obtaining the confidence score for each sample text, the sample texts are sorted according to their confidence scores, and the pseudo-labels for the last K sample texts in the sort are reassigned. Here, K is an integer greater than 1.

[0137] In some embodiments of the invention, by analyzing the proximity and historical clustering information of the last K sample texts, neighboring sample voting is used to correct low-confidence sample texts, and pseudo-labels are corrected and redistributed in conjunction with historical clustering information to correct some extremely inaccurate pseudo-labels, thereby improving the overall quality of pseudo-labels. This can enhance the model's ability to adapt to complex data distributions while mitigating the adverse effects of noisy labels.

[0138] First, identify the K sample texts with the lowest confidence scores from the sample texts, and add them to the set to be corrected. In the middle. Among them, the set to be corrected. It can be represented as:

[0139]

[0140] Where X is the set of sample texts; x i s represents the i-th sample text in the sample text set; k Indicates sample text x k The confidence score; bottomK represents the number of sample texts with reassigned pseudo-labels.

[0141] After determining the set to be corrected Then, for each sample text x in the set i The K-Nearest Neighbor Algorithm (KNN) is used to analyze the distribution of pseudo-labels of the neighbors of these samples, and the set to be corrected is updated based on the most frequent label in the pseudo-label distribution of the neighbors. The system identifies pseudo-labels for the sample text. When determining new pseudo-labels, it considers the historical clustering information of the sample text. If the most frequent label in the neighbor pseudo-label distribution matches the historical clustering information, the label corresponding to the historical clustering information is retained; otherwise, the most frequent label in the neighbor pseudo-label distribution is used as the new pseudo-label. This process helps correct potentially erroneous pseudo-labels and enhances the language model's understanding of the data clustering structure, thereby improving the language model's performance in new intent discovery tasks.

[0142] Specifically, for the set to be corrected Each sample text x in i The reassigned pseudo-label PL(x) i This can be expressed as:

[0143]

[0144] In the formula, M(x) i ) indicates that in the sample text x i The most frequently occurring pseudo-label category among the neighbors, N(x) i ) is the sample text x i The neighbor set of L(x) (determined by the K-nearest neighbor algorithm); j ) is the sample text x i pseudo-tags, H(x) i ) is related to the sample text x i Pseudo-labels related to historical clustering information.

[0145] In addition, if the rate of change of pseudo-label distribution between the current clustering iteration round and the previous clustering iteration round reaches a preset rate of change threshold, the reallocation of pseudo-labels will be stopped to prevent the model from frequently changing the pseudo-labels of the same sample text during training, which would lead to instability in subsequent language model learning and affect the convergence of the language model.

[0146] Specifically, after determining the confidence score corresponding to each sample in the sample text, the process further includes: if the rate of change of the pseudo-label distribution is less than or equal to a preset rate of change threshold, identifying the K sample texts with the lowest confidence scores as samples to be corrected; where K is an integer greater than 1; for each sample to be corrected, using the K nearest neighbor algorithm to retrieve the pseudo-label probability distribution of neighboring sample texts in the feature space, and selecting the pseudo-label with the highest frequency from the pseudo-label probability distribution as a candidate pseudo-label; if there are candidate pseudo-labels in the historical pseudo-labels corresponding to the sample to be corrected, retaining the historical pseudo-labels as pseudo-labels for the sample to be corrected.

[0147] Step S104: High-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels are used as self-supervised signals. The language model is iteratively optimized through comparative learning until convergence.

[0148] After each clustering operation, the confidence score corresponding to each sample text is calculated. The confidence score is compared with the dynamic confidence threshold corresponding to the current clustering round, and sample texts with high confidence are selected to remove noise from the pseudo-labels.

[0149] Traditional methods typically consider sample confidence scores statically, neglecting the dynamic changes in feature representations and decision boundaries during intent modeling. In real-world novel intent discovery tasks, feature representations and decision boundaries change as the model is trained and data is continuously updated. Static confidence score evaluation methods cannot adapt to these changes in a timely manner, impacting the effectiveness of pseudo-label denoising and overall model performance.

[0150] To address the problems in traditional methods, in some embodiments of the present invention, the dynamic confidence threshold is positively correlated with the number of iterations, and different dynamic confidence thresholds correspond to different clusters.

[0151] Specifically, the dynamic confidence threshold is generated based on a global threshold and local thresholds. The global threshold reflects the distribution of confidence scores across the entire sample text dataset, measuring the overall confidence of the sample texts. The local threshold reflects the distribution of confidence scores within each cluster, measuring the confidence of the sample texts within a specific cluster.

[0152] In the first iteration, the global threshold is initialized to the average confidence score of the unlabeled sample text, and the local threshold is initialized to the average confidence score of all sample texts in each cluster.

[0153] Due to the massive amount of data, calculating the confidence scores of all unlabeled data in each iteration or even at each training epoch can be extremely time-consuming. Therefore, in subsequent iterations, the global and local thresholds from the previous iteration are smoothly updated using an exponential moving average (EMA).

[0154] The global threshold can be expressed by the following formula:

[0155]

[0156] In the formula, τ t Let represent the global threshold corresponding to the t-th iteration; λ represents the momentum decay of the EMA, and λ∈(0,1); N represents the count of sample texts, s i This represents the confidence level of the i-th sample text.

[0157] Local thresholds adjust the global threshold in a class-specific manner to address variability and potential class adjacency issues within each intent class. To estimate the learning state for each intent class c, the expected value of the model's prediction for each intent class is calculated, which can be expressed as:

[0158]

[0159] in, This represents the local threshold for intent category c in the t-th iteration; This represents a vector containing the predicted expected values ​​for all intent categories; The set of sample texts representing intent category c; s i，c The confidence score represents the i-th sample text in intent category c.

[0160] By combining global and local thresholds, the final dynamic confidence threshold τ corresponding to the intent category c is obtained. t (c) is represented as:

[0161]

[0162] Where, τ t (c) represents the dynamic confidence threshold corresponding to intent category c in the t-th iteration; MaxNorm represents maximum normalization; τ represents the local threshold for intent category c in the t-th iteration; t This represents the global threshold corresponding to the t-th iteration.

[0163] Specifically, before using high-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels as self-supervised signals, the process further includes: smoothly updating the global threshold and local threshold of the previous round using an exponential moving average; normalizing each updated local threshold; and multiplying the normalized local threshold by the updated global threshold to obtain the dynamic confidence threshold of each cluster in the current iteration round.

[0164] After obtaining the dynamic confidence threshold for each intent category, sample texts are filtered based on the dynamic confidence threshold. Specifically, for each intent category c, sample texts with confidence scores exceeding the dynamic confidence threshold are first selected, and then a certain number of sample texts with high confidence scores are selected from each cluster using Top-K filtering to construct a new subset for subsequent self-supervised training.

[0165] The denoised and reassigned pseudo-labels are used as self-supervised signals. The intent representation space of the language model is iteratively optimized through contrastive learning, and the model parameters of the new language model are updated until the language model converges to obtain the intent recognition model. New intents are then discovered through the intent recognition model.

[0166] Specifically, high-confidence sample texts with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels are used as self-supervised signals. The language model is iteratively optimized through contrastive learning until convergence, including: data augmentation of the high-confidence sample texts, generating augmented versions for each sample text and constructing positive and negative sample pairs; inputting the positive and negative sample pairs into the language model to extract deep semantic features and obtain training results; inputting the training results into the cross-entropy loss function and the supervised contrastive loss function, summing the output values ​​of the two loss functions to obtain the total loss value; calculating the gradient through backpropagation, using an optimizer to update the language model parameters to minimize the total loss value, and iteratively executing the above process until the model converges.

[0167] The cross-entropy loss function can be expressed by the following formula:

[0168]

[0169] In the formula, N represents the number of sample texts in a training batch; K represents the number of intent categories; j represents the j-th intent category; z represents the feature vector corresponding to the original sample text; z′ represents the feature vector corresponding to the enhanced version of the sample text; y a Pseudo-tags representing sample text; This represents the cross-entropy loss corresponding to the original sample text; z represents the cross-entropy loss corresponding to the enhanced version of the sample text; i y represents the feature vector corresponding to the i-th sample text; i a This represents the pseudo-label corresponding to the i-th sample text.

[0170] Simultaneously, a supervised contrastive loss is introduced to improve the model's ability to distinguish subtle semantic differences by maximizing the similarity with the enhanced version of the sample text and minimizing the similarity with sample texts from different intent categories. Specifically, the supervised contrastive loss function can be expressed as follows:

[0171]

[0172] In the formula, z i Let z represent the feature vector corresponding to the i-th sample text; P(i) represents the set of indices of all positive samples belonging to the same intent category as the i-th sample text; p represents the positive sample index; z p Let p represent the feature vector of the positive sample with index p; j represents the index of the negative sample; z represents the feature vector of the positive sample with index p. j represents the feature vector of the negative sample with index j; n represents the size of the training batch; τ represents the temperature parameter used to control the distribution of similarity; sim() represents similarity.

[0173] The total loss function, which sums the outputs of the two types of loss functions, is expressed as:

[0174]

[0175] In the formula, Cross-entropy loss function; This represents the supervised comparison loss function.

[0176] By simultaneously optimizing two losses, combining instance-level and cluster-level information for learning, and using alignment as a target to train the model, cluster-friendly representations are obtained. Then, through an iterative process, clustering and representation learning are alternately performed, allowing the model to gradually adapt to the complexity and variability of the data, progressively improving its performance, and ultimately achieving a better ability to distinguish different intent categories.

[0177] In addition, before iteratively training the pre-trained language model, a new projection head for contrastive learning needs to be added as a dynamic task head. Furthermore, to avoid feature variations affecting pseudo-label stability, only the parameters of the newly added dynamic task head are optimized when optimizing the language model's parameters.

[0178] Specifically, the language model parameters are updated using an optimizer to minimize the total loss, including: freezing the model parameters of the non-linear projection layer in the language model and updating the parameters of the projection head used for contrastive learning using the optimizer.

[0179] Step S105: Use the centroid of each cluster as the new cluster center, re-cluster, and update the clustering results, normalized mutual information, and iteration count.

[0180] In some embodiments of the present invention, after each clustering iteration, the centroid of each cluster is calculated, that is, the average feature vector of all sample texts in the cluster; the calculated centroid is saved for the initialization of the next round of clustering.

[0181] In the next round of clustering, the centroids saved from the previous round of clustering are used as the initial cluster centers. The clustering operation is re-executed on the sample text to obtain new clustering results, thereby reducing the dependence on the selection of initial centers and improving the stability and efficiency of clustering.

[0182] After obtaining the new clustering results, new pseudo-labels are assigned to each sample text based on the new clustering results, the normalized mutual information is updated, the value of the iteration count counter is increased, and the current number of clustering rounds completed is recorded.

[0183] Step S106: If the number of iterations after the update reaches the preset upper limit and the increase in normalized mutual information is less than the preset threshold, terminate the training and determine the current language model as the intent recognition model.

[0184] The preset upper limit refers to the maximum number of iterations set in advance. When the preset upper limit is reached, training is terminated, the current language model is identified as the intent recognition model, and the model parameters and related data are saved.

[0185] Alternatively, examine the changes in normalized mutual information over multiple consecutive rounds (e.g., 5, 6, or 7 rounds). If the increase is less than a preset threshold (e.g., 0.01), terminate the training, identify the current language model as the intent recognition model, and save the model parameters and related data.

[0186] If the number of iterations after the update does not reach the preset upper limit and the increase in normalized mutual information is greater than the preset threshold, the step of determining the confidence score corresponding to each sample in the sample text is executed, and the sample text is selected for subsequent model training steps.

[0187] For example, taking 5 rounds as an example, in the first round, the normalized mutual information is 0.70; in the second round, the normalized mutual information is 0.72, changing to 0.02; in the third round, the normalized mutual information is 0.73, changing to 0.01; in the fourth round, the normalized mutual information is 0.735, changing to 0.005; in the fifth round, the normalized mutual information is 0.74, changing to 0.005. That is, in these 5 rounds, the preset termination condition is not met, and the confidence score corresponding to each sample text continues to be calculated. Alternatively, in the first round, the normalized mutual information is 0.71; in the second round, it is 0.72, changing to 0.01; in the third round, it is 0.73, changing to 0.01; in the fourth round, it is 0.735, changing to 0.005; and in the fifth round, it is 0.74, changing to 0.005. That is, in these 5 rounds, if the preset termination condition is met, the current language model is determined as the intent recognition model, and the iterative training is terminated.

[0188] After obtaining the intent recognition model, the trained model is fine-tuned on the validation set, and the model performance is evaluated on the test set. The Adjusted Rand Index (ARI) and accuracy are adjusted as evaluation metrics to verify the effectiveness of the method of this invention in discovering new intents on three intent benchmark datasets (including the banking domain dialogue dataset BANKING, the multi-domain intent classification dataset CLIC, and the technical question answering dataset StackOverflow).

[0189] The verification results show that the model training method provided in this embodiment is significantly better than the existing state-of-the-art novel intent discovery method USNID under various experimental settings.

[0190] On the BANKING dataset, in scenarios where the proportion of known intent categories is low (25%), the intent recognition model provided in this embodiment effectively suppresses noise propagation through a spatiotemporal denoising mechanism. Its accuracy is 2.53% higher than the existing state-of-the-art new intent discovery method USNID (65.85%), verifying its robustness under limited labeled data.

[0191] On the CLIC dataset, the intent recognition model provided in this embodiment exhibits stronger generalization ability and is more likely to distinguish intent boundaries due to its domain diversity (covering 10 domains), while the spatiotemporal feature fusion mechanism effectively captures cross-domain semantic differences.

[0192] For the StackOverflow dataset, the intent recognition model provided in this embodiment achieves an accuracy of 88.53% in a scenario with 75% labeled samples, a 2.87% improvement over USNID's 85.66%. This dataset is characterized by dense technical terminology and ambiguous intent boundaries, making accurate clustering difficult with traditional methods. By correcting the misassignment of low-confidence samples through pseudo-label reassignment, the model demonstrates a unique advantage in identifying intent related to technical issues, with its adjusted RAND index improving by 3.96% compared to USNID's 74.90%.

[0193] In summary, the pseudo-label-based intent recognition model training method provided in this embodiment involves inputting sample text into a pre-trained language model to extract feature vectors; performing clustering operations on the sample text based on the feature vectors, using the clustering results as pseudo-labels, and calculating the normalized mutual information between the real labels and pseudo-labels of the labeled sample text; determining the confidence score corresponding to each sample in the sample text; the confidence score is obtained by normalizing and weighting the aggregation of preset indicators, used to quantify the noise level in the pseudo-labels; the preset indicators include the distance from the sample text to the center of its cluster, the pseudo-label distribution information entropy of the labeled sample text, etc. The frequency of sample texts being assigned to the same cluster in a preset number of clustering rounds and the rate of change in the distribution of pseudo-labels for each cluster in consecutive clustering operations are considered. High-confidence samples with confidence scores greater than the dynamic confidence threshold and their corresponding pseudo-labels are used as self-supervised signals, and the language model is iteratively optimized through contrastive learning until convergence. The dynamic confidence threshold is positively correlated with the number of iterations; different clusters correspond to different dynamic confidence thresholds. A centroid-guided re-clustering mechanism is used to update the clustering results, normalized mutual information, and number of iterations. The optimization process is considered when the updated number of iterations reaches a preset upper limit or the increase in normalized mutual information is less than a preset threshold. If the training terminates and the current language model is identified as the intent recognition model, then the step of determining the confidence score corresponding to each sample in the sample text is executed. This addresses the problem that as training iterates, noise pseudo-labels continue to spread and accumulate, leading to a significant decrease in the model's ability to recognize new intents. By comprehensively utilizing the spatial and temporal features of the sample text to dynamically optimize pseudo-labels, the spatial dimension is evaluated through the reliability of the sample text and the cohesion of the intent category, accurately locating noise labels. The temporal dimension is analyzed through the stability of the pseudo-label distribution and the consistency of the allocation, capturing the data evolution pattern and constructing a spatiotemporal fusion pseudo-label quality evaluation mechanism. This breaks through the limitations of traditional single indicators, comprehensively capturing the semantic features of user intents, reducing the negative impact of noise on model performance, avoiding the problem of pseudo-label noise interfering with model training and intent recognition, providing the model with more reliable self-supervised signals, and improving the accuracy, reliability, and ability to recognize new intents. At the same time, by dynamically optimizing pseudo-labels and combining a dynamic confidence threshold adjustment strategy, high-confidence pseudo-labels are dynamically selected, which can further suppress noise propagation, reduce the negative impact on model performance, and further improve the model's accuracy and reliability in recognizing new intents.

[0194] Figure 6 This is a flowchart illustrating the intent recognition method provided in an embodiment of this application. Figure 6 As shown, the intent recognition method includes at least the following steps S601 to S602:

[0195] Step S601: Obtain the text to be recognized input by the user.

[0196] Step S602: Input the text to be recognized into the intent recognition model to obtain the intent label corresponding to the text to be recognized.

[0197] The intent labels include user-pre-labeled intent labels and unlabeled intent labels. The intent recognition model is trained using the pseudo-label-based intent recognition model training method shown in the above embodiments. Details are as described in the above embodiments and will not be repeated here.

[0198] In summary, the intent recognition method provided in this embodiment inputs the text to be recognized by the user into the intent recognition model to obtain the intent label corresponding to the text to be recognized. The intent recognition model is trained using the pseudo-label-based intent recognition model training method shown in the above embodiment, thereby improving the accuracy and reliability of recognizing new intents in the text to be recognized.

[0199] Corresponding to the above method, the present invention also provides an electronic device, which includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the electronic device performs the steps of the method as described above.

[0200] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0201] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0202] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0203] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0204] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pseudo-label based intent recognition model training method, characterized in that, The method comprises the following steps: The method comprises the following steps: Inputting the sample text into a pre-trained language model to extract a feature vector; the sample text comprises labeled sample text corresponding to a known intent category, unlabeled sample text corresponding to the known intent category, and unlabeled sample text corresponding to an unknown intent category; Performing a clustering operation on the sample text based on the feature vector, taking the clustering result as a pseudo label, and calculating the normalized mutual information between the true label and the pseudo label of the labeled sample text; Determine the confidence score corresponding to each sample in the sample text; the confidence score is obtained by normalizing and weighting the preset indicators, and is used to quantify the noise level in the pseudo label; the preset indicators include the distance of the sample text to the center of the cluster to which it belongs, the pseudo label distribution information entropy of the labeled sample text, the frequency of the sample text being assigned to the same cluster in a preset number of clustering operations, and the pseudo label distribution change rate of each cluster in consecutive clustering operations; Taking the high-confidence sample and the corresponding pseudo label with a confidence score greater than a dynamic confidence threshold as a self-supervised signal, iteratively optimizing the language model through contrastive learning until convergence; the size of the dynamic confidence threshold is positively correlated with the number of iterations; the dynamic confidence thresholds of different clusters are different; Taking the centroid of each cluster as a new cluster center, re-clustering and updating the clustering result, the normalized mutual information and the number of iterations; In the case where the updated number of iterations reaches a preset upper limit or the normalized mutual information increment is less than a preset threshold, terminate the training and determine the current language model as an intent recognition model; otherwise, perform the step of determining the confidence score corresponding to each sample in the sample text; The dynamic confidence threshold is generated based on a global threshold and a local threshold; in the first iteration, the global threshold is initialized as the average of the confidence scores of the unlabeled sample texts, and the local threshold is initialized as the average of the confidence scores of all sample texts in each cluster; in subsequent iterations, the global threshold of the previous round and the local threshold of the previous round are updated by exponential moving average smoothing; Before taking the high-confidence sample and the corresponding pseudo label with a confidence score greater than a dynamic confidence threshold as a self-supervised signal, it further comprises: Smoothly updating the global threshold of the previous round and the local threshold of the previous round by the exponential moving average; Normalizing each updated local threshold; 2. The method of claim 1, wherein, Multiplying the normalized local threshold by the updated global threshold to obtain the dynamic confidence threshold of each cluster in the current iteration round. Taking the high-confidence sample text and the corresponding pseudo label with a confidence score greater than a dynamic confidence threshold as a self-supervised signal, iteratively optimizing the language model through contrastive learning until convergence, comprises: Performing data augmentation on the high-confidence sample text to generate a corresponding augmented version and constructing positive sample pairs and negative sample pairs; Inputting the positive sample pairs and the negative sample pairs into the language model to extract deep semantic features and obtain a training result; The training result is input into a cross-entropy loss function and a supervised contrast loss function, the output values of the two loss functions are summed to obtain a total loss value; The gradient is calculated by back propagation, the language model parameters are updated by an optimizer to minimize the total loss value, and the above process is iteratively performed until the model converges.

3. The method of claim 1, wherein, The pre-training process of the language model comprises: The sample text is input into an initial language model for feature extraction to obtain an initial feature vector; the sample text includes labeled sample text and unlabeled sample text; the initial language model comprises an input embedding layer, a feature extraction layer, a mean pooling layer, a nonlinear projection layer and a dynamic task head; the input embedding layer is used to convert the input text into an initial vector representation; the feature extraction layer is used to process the initial vector representation to extract deep semantic features to obtain a token representation; the mean pooling layer is used to perform a mean pooling operation on the token representation to obtain a sentence-level semantic vector; the nonlinear projection layer is used to map the sentence-level semantic vector to a low-dimensional intent representation space to obtain a corresponding intent representation; and the dynamic task head is used for embedding and output prediction intent label required for contrast learning; Data augmentation is performed on each sample text in the sample text to generate corresponding augmented samples, positive sample pairs and negative sample pairs are constructed, and labeled augmented samples and unlabeled augmented samples are mixed according to a preset ratio to obtain at least one training batch; The at least one training batch is input into the initial language model to obtain an augmented feature vector and a predicted intent label output by the dynamic task head; The initial feature vector, the predicted intent label and the augmented feature vector are input into a preset loss function to obtain a loss value; the preset loss function comprises a preset semi-supervised contrast loss function and a cross-entropy loss function; The gradient is calculated by back propagation, the initial language model parameters are updated by an optimizer to minimize the loss value, and the above process is iteratively performed until the model converges. In the case where the initial language model converges, the dynamic task head is removed to obtain the pre-trained language model.

4. The method of claim 3, wherein, Before each round of iterative training of the initial language model based on the preset semi-supervised contrast loss function, further comprising: The pre-trained language model is subjected to supervised contrast learning using the labeled sample text, and a real intent label corresponding to the labeled sample text and the predicted intent label are input into a preset supervised contrast loss function to obtain a supervised contrast loss value; The gradient is calculated by back propagation, the model parameters of the initial language model are updated by an optimizer to minimize the supervised contrast loss value, and the above process is iteratively performed until the model converges.

5. The method of claim 3, wherein, The feature extraction layer comprises at least one Transform layer; and the updating of the initial language model parameters by the optimizer comprises updating the parameters of the last Transform layer in the feature extraction layer and the parameters of the dynamic task head by the optimizer.

6. The method of claim 1, wherein, Before the high-confidence samples with a confidence score greater than a dynamic confidence threshold and the corresponding pseudo labels are used as self-supervised signals, further comprising: In a case where the pseudo label distribution change rate is less than or equal to a preset change rate threshold, K sample texts with the lowest confidence scores are determined as to-be-corrected samples; wherein the K is an integer greater than 1; For each to-be-corrected sample, a K-nearest neighbor algorithm is used to retrieve a pseudo label probability distribution of a neighboring sample text in a feature space, and a pseudo label with the highest frequency in the pseudo label probability distribution is selected as a candidate pseudo label; In a case where the candidate pseudo label exists in a historical pseudo label corresponding to the to-be-corrected sample, the historical pseudo label is retained as a pseudo label of the to-be-corrected sample.

7. An intention recognition method characterized by, The method comprises: obtaining a to-be-recognized text input by a user; inputting the to-be-recognized text into an intent recognition model to obtain an intent label corresponding to the to-be-recognized text; the intent recognition model is trained by the method of any one of claims 1 to 6.

8. An electronic device comprising a processor, a memory, and a computer program / instructions stored on the memory, wherein, The processor is configured to execute the computer programs / instructions, and when the computer programs / instructions are executed, the electronic device implements the steps of the method of any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer programs / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 7.

Images