New intention discovery method based on dual-temperature confidence weight cluster representation learning

Abstract

The invention discloses a new intention discovery method based on dual-temperature confidence weight cluster representation learning, and the method comprises the steps: firstly dividing a multi-field intention text data set into a training set and a test set, enabling a sample in the training set to be semi-supervised data, and then carrying out the data enhancement of the sample in the training set, and generating new sample data; secondly, a new intention discovery model is constructed, the new intention discovery model comprises a converter-based bidirectional encoder representation basic model BERT-base and K-Means clustering operation, and a new intention discovery label is output and obtained; and finally, iteratively updating the new intention discovery model through the data in the training set, and testing the new intention discovery model through the test set. According to the method, the distance between the samples in the same cluster can be shortened, the distance between the samples in different clusters can be shortened, and meanwhile, the problem of false label noise influence is solved through double temperature and confidence weight.

Description

Technical Field

[0001] This invention belongs to the field of novel intent discovery based on deep learning, and particularly relates to a novel intent discovery method based on dual-temperature confidence weight cluster representation learning. Background Technology

[0002] In the development of task-oriented dialogue systems, designing and identifying dialogue intents are crucial steps. Traditional task-oriented dialogue systems are typically based on a predefined set of intent categories, which reflect the specific interaction needs the system can handle. However, in practical applications, due to the diversity and constant change of user needs, it is difficult to fully cover user input using predefined categories. For example, over time, users may raise new needs that cannot be correctly categorized by the existing intent set. This scenario not only affects the stability of the dialogue system but also limits its adaptability to complex environments. Therefore, to address this issue, researchers have proposed the Novel Intent Discovery (NID) task.

[0003] In recent years, with the development of deep learning technology, architectures such as DPN (Decoupled Prototype Network), DCSC (Deep Contrastive Semi-Supervised Clustering), CLNN (Closest Neighbor Contrastive Learning), and DWGF (Diffusion Weighted Graph Framework) have been widely applied to novel intent discovery and have achieved good classification results. Despite these advancements, some problems remain: 1. Existing methods lack intra-cluster compactness and inter-cluster separation, failing to capture cluster-friendly representations because they exhibit poor ability to effectively control and coordinate intra- and inter-cluster distances. Summary of the Invention

[0004] This invention addresses the aforementioned technical problems by proposing a novel intent discovery method based on dual-temperature confidence weight cluster representation learning. The architecture of this invention consists of three parts: a dual-temperature confidence weight intra-cluster contrastive learning module, a class center separation module, and a momentum-updated class center module. The dual-temperature confidence weight intra-cluster contrastive learning module constructs a similarity matrix by combining features extracted from BERT (a transformer-based bidirectional encoder representation) with the class centers obtained from clustering. Then, it performs contrastive learning based on the confidence weights obtained from the similarity matrix and the dual temperatures. The class center separation module maximizes the distance between clusters and improves distribution uniformity by separating the class centers. The momentum-updated class center module changes the class centers using momentum to adapt them to changes in intent representation and cluster distribution. By combining dual-temperature, confidence, and cluster representation learning, this invention enhances the model's ability to extract text features, making similar types of text more compact and separating different types of text, thus effectively solving the problems existing in novel intent discovery.

[0005] The specific steps of the method of the present invention include:

[0006] S1. Data Preprocessing: The multi-domain intent text dataset is divided into a training set and a test set. The samples in the training set are semi-supervised data. Then, data augmentation is performed on the samples in the training set to generate new sample data.

[0007] The semi-supervised data consists of a portion of labeled data and a portion of unlabeled data.

[0008] The data augmentation described herein involves randomly replacing certain words in the text with certain words from the word segmenter's vocabulary with a certain probability.

[0009] S2. After preprocessing the samples in the training set in step S1, input them into the BERT-base (transformer-based bidirectional encoder representation base version) model. The BERT-base model consists of k layers of encoders. The input data passes through the k layers of encoders to obtain sample features, and then passes through the average pooling layer to obtain the mean of the sample features.

[0010] The first encoder layer consists of a multi-head self-attention layer (this layer allows the model to focus on tokens (word vectors) at different positions when processing the sequence and calculate attention weights between tokens, thereby capturing dependencies in the input sequence), a feedforward network layer (which further transforms the output of the self-attention mechanism to extract higher-level features), a residual connection layer (which alleviates the gradient vanishing problem in deep networks by adding the output of the feedforward network layer to the input of the self-attention layer and then passing it to subsequent layers), and a normalization layer (used to stabilize the training process; it is usually located after the outputs of the self-attention layer and the feedforward network layer, helping to eliminate numerical differences between different layers and improve the training efficiency and stability of the model). They are executed sequentially and work together to encode the input sequence.

[0011] The second to kth layers are also composed of the above four layers. The input data first passes through the multi-head self-attention layer, then through the residual connection layer and the normalization layer, and then through the feedforward network layer, the residual connection layer and the normalization layer.

[0012] The average pooling layer is formed by summing the features of all tokens in the input text and averaging them.

[0013] S3. Initialize the cluster center parameters, then perform K-Means clustering on the training data before data augmentation to obtain the cluster center coordinates and pseudo-labels for the unlabeled data. Obtain the adjusted cluster center coordinates, multiply the cluster center coordinates by the cluster center parameters, and obtain the new intent discovery label.

[0014] The pseudo-label represents assigning a label to unlabeled data for self-supervised learning.

[0015] S4. Normalize the mean of the sample features in step S2 and the cluster center coordinates in step S3, and then calculate the similarity between the two using cosine similarity to form an initial similarity matrix.

[0016] S5. Extract the maximum value of the similarity matrix in step S4 as the initial confidence score of the sample, and normalize the score. The result is used to guide the subsequent temperature parameter setting.

[0017] The temperature parameter is used to adjust the model's focus on samples with high confidence and samples with low confidence.

[0018] The maximum value of the similarity matrix is ​​the maximum probability value of each sample belonging to a certain class. This is used as its initial confidence. Normalization is mainly to obtain high-confidence samples and low-confidence samples. The pseudo-labels of high-confidence samples are instructive for the model, while low-confidence samples are given lower weights to reduce the impact of noise and uncertainty.

[0019] S6. Based on the normalized confidence scores of the samples in step S5, set a confidence threshold. For samples with confidence scores higher than the threshold, set their temperature parameter to a predefined low value to enhance the model's ability to distinguish these samples. For samples with confidence scores lower than or equal to the threshold, set their temperature parameter to a predefined high value to encourage the model to distinguish these samples more precisely. Then, recalculate the similarity matrix using the adjusted temperature parameter: multiply the elements of the similarity matrix by the corresponding temperature parameter to obtain a new similarity matrix;

[0020] S7. Calculate the contrastive learning loss based on the similarity matrix between the sample and the cluster center obtained in step S6. The numerator is the similarity between the sample and its cluster center, the denominator is the similarity between the sample and all cluster centers, and the coefficient is the confidence weight.

[0021] S8. Based on the cluster center coordinates obtained in step S3, calculate the similarity matrix between the cluster centers using cosine similarity.

[0022] S9. Based on the cluster center coordinates obtained in step S3, calculate the smooth inverse distance between the cluster centers to prevent gradient explosion caused by excessively small distances.

[0023] The smoothed inverse distance is obtained by adding a very small number to the Euclidean distance and taking the square root.

[0024] S10. Improve the cosine similarity between class centers and combine it with the smooth inverse distance to create an adaptive constraint weight matrix.

[0025] The improved cosine similarity refers to adjusting the cosine similarity by taking a negative number and adding 1, and then taking the logarithm of that value.

[0026] S11. Calculate the class center separation loss based on the similarity matrix and adaptive constraint weight matrix obtained in step S10. The numerator is the sum of the ratios of the similarity and adaptive constraint weights, and the denominator is the number of classes minus 1.

[0027] S12. Sum the sample data from the data augmentation training, take the average value, and obtain the new cluster center coordinates. Then, use the new cluster centers to update the original cluster center coordinates.

[0028] S13. Based on the training process from S1 to S12, the model learns better parameters, adjusts the contrastive learning loss function and class center separation loss, optimizes the new intent discovery model to distinguish between known intents and new intents, and obtains the category of new intents.

[0029] The beneficial effects of this invention are as follows: BERT's encoder extracts features from the input text and obtains the intent feature representation of the sentence text through an average pooling layer. K-means is used to cluster the semantic features, obtaining the coordinates of the cluster centers and pseudo-labels for unlabeled samples. The clustering effect is improved through a dual-temperature confidence weight intra-cluster contrastive learning module, a cluster center separation module, and a momentum update cluster center module. This invention has the following benefits: 1. The dual-temperature confidence weight intra-cluster contrastive learning module can shorten the distance between samples within the same cluster and widen the distance between samples in different clusters, while simultaneously solving the problem of pseudo-label noise through dual-temperature and confidence weights; 2. The cluster center separation module can maximize the distance between clusters and improve distribution uniformity; 3. The momentum update cluster center module allows the cluster centers to adapt to changes in intent representation and cluster distribution. Attached Figure Description

[0030] Figure 1 This is a novel intent discovery method based on dual-temperature confidence weight cluster representation learning;

[0031] Figure 2 This is a diagram illustrating the results of the method of the present invention on the normalized mutual information of the CLIC dataset;

[0032] Figure 3 This is a graph showing the results of adjusting the Rand index using the method of this invention on the CLIC dataset;

[0033] Figure 4 This is a graph showing the results of the method of the present invention on the clustering accuracy of the CLIC dataset. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0035] As a preferred embodiment of the present invention, a stroke image segmentation method based on multi-attention and dilated convolution is provided, the architecture of which is as follows: Figure 1 As shown, it includes the following steps:

[0036] S1. Data preprocessing: The dataset is divided into a training set and a test set. The samples in the training set are semi-supervised data. Then, data augmentation is performed on the samples in the training set to generate new sample data.

[0037] The data augmentation involves randomly replacing certain words in the text with certain words in the word segmenter's vocabulary with a probability of 0.25. The input data consists of input_ids, input_mask, and segment_ids.

[0038] S2. The samples in the training set are preprocessed in step S1 and then input into the BERT-base model. The BERT-base model consists of k encoder layers. The input data passes through the k encoder layers to obtain the sample features, and then passes through the average pooling layer to obtain the mean of the sample features.

[0039] The first layer encoder consists of a multi-head self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer.

[0040] The second to k layers are also composed of the above four layers. The input data first passes through the multi-head self-attention layer, then through the residual connection layer and the normalization layer, and then through the feedforward network layer, the residual connection layer and the normalization layer. In this embodiment, k=12.

[0041] The average pooling layer calculates the average of the features of all tokens in the input text to obtain Zi, using the following formula:

[0042] z i =mean-pooling(BERT(x) i ))

[0043] In this embodiment x i This represents the i-th text data in a batch of text data, where mean-pooling is the average pooling layer, and z i It is x i The result after average pooling.

[0044] S3. Initialize the cluster center parameters so that they are considered as learnable parameters of the model. Then, perform K-Means clustering on the training set data before data augmentation to obtain the cluster center coordinates and pseudo-labels for the unlabeled data. Obtain the adjusted cluster center coordinates, and multiply the cluster center coordinates by the cluster center parameters to obtain the new intent discovery labels.

[0045] The pseudo-label represents assigning a label to unlabeled data for self-supervised learning.

[0046] S4. Normalize the sample features from step S2 and the cluster center coordinates from step S3, and then calculate the similarity between the two using cosine similarity to form an initial similarity matrix S. The size of S is N×K. In this embodiment, N is the number of samples and K is the number of classes.

[0047] S5. Extract the maximum value of the similarity matrix from step S4 as the initial confidence score C for the sample, and normalize the score. The result is used to guide subsequent temperature parameter settings. The calculation formula is as follows:

[0048]

[0049] C = [c1, c2, ..., c N ]

[0050] c i =confidence score(i)

[0051] In this embodiment, i is the data in the i-th row of the similarity matrix, j is the data in columns 1 to K, and c i is the maximum value in the i-th row, and C is the maximum value in each row of the similarity matrix.

[0052] The maximum value of the similarity matrix is ​​the maximum probability value of each sample belonging to a certain class. This is used as its initial confidence. Normalization is mainly to obtain high-confidence samples and low-confidence samples. The pseudo-labels of high-confidence samples are instructive for the model, while low-confidence samples are given lower weights to reduce the impact of noise and uncertainty.

[0053] S6. Based on the confidence scores of the samples in step S5, set the temperature parameter of high-confidence samples to low value and the temperature parameter of low-confidence samples to high value to obtain the temperature parameter t. Recalculate the similarity matrix after temperature adjustment. The calculation formula is as follows:

[0054] t = where(C>c threshold ,t high ,t low )

[0055] In this embodiment c thresholdTemperature threshold hyperparameter, t high High temperature hyperparameter, t low Ground temperature exceeds parameters, and c threshold =0.25, t high =0.75, t low =0.1, the WHERE function selects values ​​in C that are greater than c. threshold For t high less than or equal to c threshold For t low t is the result of C after applying the WHERE function.

[0056] S7. Calculate the contrastive learning loss based on the similarity matrix between the sample and the cluster center obtained in step S6. The numerator is the similarity between the sample and its cluster center, the denominator is the similarity between the sample and all cluster centers, and the coefficient is the confidence weight.

[0057] S8. Based on the cluster center coordinates obtained in step S3, calculate the similarity matrix between the cluster centers using cosine similarity.

[0058] S9. Based on the cluster center coordinates obtained in step S3, calculate the Gaussian kernel distance between the cluster centers to capture more complex distribution relationships.

[0059] The formula for calculating the Gaussian kernel distance is as follows:

[0060]

[0061] In this embodiment, σ is a hyperparameter, x and y are both class center coordinates, and exp is an exponential function.

[0062] S10. Calculate the class center separation loss based on the similarity matrix and Gaussian kernel distance obtained in step S9. The numerator is the sum of the ratios of the aforementioned similarity and adaptive constraint weights, and the denominator is the number of classes minus 1.

[0063] S11. Calculate the new cluster center coordinates based on the sample features, and then update the original cluster center coordinates using the new cluster centers. The calculation formula is as follows:

[0064] C updated =momentum·C old +(1-momentum)·C new

[0065] In this embodiment, momentum is the momentum update hyperparameter and momentum = 0.9, C old It is the old class center coordinate, C new It is the new class center coordinate, C updated These are the class center coordinates after momentum update.

[0066] Example:

[0067] The steps in this embodiment are the same as those in the specific implementation method, and will not be repeated here. The following section demonstrates some of the implementation processes and results.

[0068] This invention is implemented on the CLIN and BANKING datasets. The CLIN dataset contains 18,000 training samples and 2,250 test samples. The training set contains 1,344 labeled samples and 16,656 unlabeled samples, with 113 known classes and 37 unknown classes. The BANKING dataset contains 9,003 training samples and 3,080 test samples. The training set contains 673 labeled samples and 8,330 unlabeled samples, with 58 known classes and 19 unknown classes. For the model training process, the text batch size for each iteration is set to 128, and the Adam optimizer is used to train the model with a learning rate of 0.00001.

[0069] To verify the effectiveness of the technology of this invention, Figure 2 , Figure 3 , Figure 4 The results of this invention's technique on the CLIC dataset for three metrics are shown, and compared with methods such as DAC, DSSCC, PTJN, DPN, DCSC, CLNN, and DWGF. Compared with these methods, this invention's technique can achieve a more compact distribution within clusters and higher separation between clusters in the clustering space, improving classification accuracy. This demonstrates the effectiveness of the designed dual-temperature confidence weight cluster representation learning module, class center separation module, and momentum update class center module.

[0070] To verify the effectiveness of the present invention, Tables 1 and 2 use the metrics Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Clustering Accuracy (ACC) to compare the present invention with existing technologies. As shown in the comparison in Table 1 on the BANKING dataset and in the comparison in Table 2 on the CLINK dataset, the present invention achieves better results on multiple metrics, thus proving its effectiveness.

[0071] Table 1. Quantitative Comparison Results of Invention Technology and Existing Technology

[0072]

[0073] Table 2. Quantitative Comparison Results of Invention Technology and Existing Technology

[0074]

[0075]

Claims

1. A novel intent discovery method based on dual-temperature confidence weight cluster representation learning, characterized in that, Specifically, the following steps are included: S1. Divide the multi-domain intent text dataset into a training set and a test set. The samples in the training set are semi-supervised data. Then, perform data augmentation on the samples in the training set to generate new sample data. S2. Construct a new intent discovery model, including a bidirectional encoder representation of the base model BERT-base based on a transformer and K-Means clustering operations, and output new intent discovery labels; S3. Iteratively update the new intent discovery model using data from the training set, and then test it using the test set.

2. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 1, characterized in that, The semi-supervised data packet contains labeled data and unlabeled data; The data augmentation described involves randomly replacing words in the text with words from the word segmenter's vocabulary.

3. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 2, characterized in that, The specific implementation process of the new intent discovery model is as follows: S2-1. Input the samples in the data-augmented training set into the BERT-base model, which is based on a transformer-based bidirectional encoder, to obtain the sample features. Then, the mean of the sample features is obtained through an average pooling layer. S2-2. Initialize the cluster center parameters, then perform K-Means clustering on the training set data before data augmentation to obtain the cluster center coordinates and pseudo-labels for the unlabeled data. Obtain the adjusted cluster center coordinates, multiply the cluster center coordinates by the cluster center parameters, and obtain the new intent discovery label.

4. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 3, characterized in that, The bidirectional encoder based on the transformer represents the basic model BERT-base, which consists of k layers of encoders. Input data is passed through the k layers of encoders to obtain sample features. The first encoder layer consists of a multi-head self-attention layer, a feedforward network layer, a residual connection layer, and a normalization layer cascaded together to complete the encoding process of the input sequence. The second to k layers of the encoder first pass through a multi-head self-attention layer, then through a residual connection layer and a normalization layer, followed by a feedforward network layer, a residual connection layer, and a normalization layer.

5. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 4, characterized in that, The specific implementation process of iteratively updating the new intent discovery model using data from the training set is as follows: S3-1. Normalize the mean of the sample features in step S2-1 and the cluster center coordinates in step S2-2, and then calculate the similarity between the two using cosine similarity to form an initial similarity matrix T1; extract the maximum value of the similarity matrix as the initial confidence score of the sample, and normalize the score. S3-2. Based on the confidence scores of the normalized samples, set a confidence threshold. For samples with confidence scores higher than the threshold, set their temperature parameter to A; for samples with confidence scores lower than or equal to the threshold, set their temperature parameter to B, and A is less than B. Then, recalculate the similarity matrix using the adjusted temperature parameters: multiply the elements of the similarity matrix by the corresponding temperature parameters to obtain a new similarity matrix T2. S3-3. Calculate the contrastive learning loss based on the new similarity matrix T2; S3-4. Based on the cluster center coordinates obtained in step S2-2, calculate the similarity matrix T3 between the cluster centers using cosine similarity, and calculate the smooth inverse distance between the cluster centers. S3-5. Improve the cosine similarity between class centers, and combine the improved cosine similarity with the smooth inverse distance to obtain the adaptive constraint weight matrix; S3-6. Calculate the class center separation loss based on the similarity matrix T3 and the adaptive constraint weight matrix; S3-7. Summate the sample data from the data augmentation training, take the average value to obtain the new cluster center coordinates, then update the original cluster center coordinates through the new cluster center momentum, adjust the contrastive learning loss function and the cluster center separation loss, and optimize the new intent discovery model.

6. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 5, characterized in that, The contrastive learning loss is specifically defined as follows: the numerator is the similarity of a sample to its class center, the denominator is the similarity of a sample to all class centers, and the coefficient is the confidence weight. The class center separation loss has a numerator that is the sum of the similarity and adaptive constraint weights mentioned above, and a denominator that is the number of classes minus 1.

7. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 5, characterized in that, The smoothed inverse distance is obtained by adding a very small number to the Euclidean distance and taking the square root.

8. The novel intent discovery method based on dual-temperature confidence weight cluster representation learning according to claim 5, characterized in that, The improvement of cosine similarity in step S3-5 is specifically achieved by adjusting the cosine similarity by taking a negative number and adding 1, and then taking the logarithm of that value.

Images