A short text clustering method based on an outlier-free pseudo-labeling algorithm

Abstract

The present invention belongs to the field of deep clustering, and specifically relates to a short text clustering method based on a non-outlier pseudo-label algorithm. The method uses a masked pre-trained language model to obtain a feature extraction model for feature extraction to improve the quality of the text features of a data set. By combining contrastive learning and clustering learning based on a non-outlier pseudo-label algorithm, the quality of negative example pairs in contrastive learning is increased, and the problem that the effect of contrastive learning is made worse by inherent false negative examples is effectively alleviated; at the same time, contrastive learning is used to shorten the distance between outliers and non-outliers, and the distance between non-outliers is shortened by clustering heads, so as to obtain a smaller intra-class distance and a larger inter-class distance, thereby achieving an improvement in clustering effect.

Description

Technical Field

[0001] The present invention belongs to the field of deep clustering, and particularly relates to a short text clustering method based on an outlier-free pseudo-label algorithm. Background Art

[0002] Short Text Clustering (STC) is a key task in unsupervised learning, and its goal is to cluster unlabeled short texts into different clusters. With the development of technology and the popularity of social media, short texts such as online reviews, Weibo posts, and search terms are growing rapidly. How to organize these texts according to specific topics or event comments is an important and crucial step in data mining tasks. The common approach is to use methods such as text clustering to effectively organize short text information to complete tasks such as text summarization, public opinion analysis, and event monitoring.

[0003] Affected by the characteristics of short texts, such as less semantic information, higher noise, and higher dimensions, traditional clustering methods perform poorly in short text clustering tasks. This is because traditional clustering methods often rely on the distance matrix in the data space, and obtaining a better distance matrix of the data often requires extracting better feature vectors from short texts.

[0004] To solve this problem, in previous deep learning methods, some researchers used neural networks to enrich sparse feature representations to obtain better feature vectors; at the same time, some researchers tried to add additional knowledge to short texts to enrich text representations to obtain better feature vectors. However, the text feature representations generated by these two methods still have problems such as low data purity and high crossover between clusters. In view of this, some scholars proposed pre-training SentenceBERT on a specific dataset using a Mask Language Model (MLM) to improve the initial text feature representation of SentenceBERT and increase the purity of text feature representations. However, when clustering, the overlap between its clusters is still very serious.

[0005] In recent years, contrastive learning based on deep learning has made rapid progress. Contrastive learning is a paradigm that enables models to learn to compare. It has shined in the field of self-supervised learning and has achieved breakthrough success in image text clustering tasks and sentence representation tasks. Contrastive learning can disperse data feature representations in the data space to alleviate the overlap problem between data clusters. Its basic idea is to bring the feature vectors of positive pairs closer and the feature vectors of negative pairs farther apart. The positive pairs are composed of text data enhanced from the same source text, and the negative pairs are composed of text data enhanced from different source texts. However, since we do not have the true labels of the data, the inherent false negative pairs in contrastive learning will make the contrastive learning effect worse and unstable, thereby affecting the accuracy of clustering. The inherent false negative pairs refer to the source texts of the two enhanced texts of the negative pairs belonging to the same semantic cluster, so we cannot pull such text feature vectors apart because they belong to a semantic cluster and should be pulled closer. Summary of the invention

[0006] The purpose of the present invention is to propose a short text clustering method based on a non-outlier pseudo-label algorithm to solve the problem that inherent false negative pairs will make the effect of contrastive learning worse and unstable, thereby affecting the accuracy of clustering.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] A short text clustering method based on a non-outlier pseudo-label algorithm comprises the following steps:

[0009] Step 1: For the source text dataset, use mask pre-training SentenceBERT to obtain a feature extraction model for the source text dataset; the source text dataset consists of given n short texts;

[0010] Step 2: Extract feature vectors. The feature vector extraction method is as follows:

[0011] 2.1. A batch of text data is randomly selected from the source text data set, and its corresponding enhanced text data is obtained; the enhanced text data includes weakly enhanced text data and strongly enhanced text data;

[0012] 2.2. Input the text data, weakly enhanced text data and strongly enhanced text selected in step 2.1 into the feature extraction model obtained in step 1 for feature extraction, and obtain the source text feature vector, the weakly enhanced text feature vector and the strongly enhanced text feature vector;

[0013] Step 3: Calculate the pseudo labels of non-outliers. The calculation method of the pseudo labels of non-outliers is as follows:

[0014] 3.1. Cluster the source text feature vectors obtained in step 2.2 into K cluster classes using a clustering algorithm, and obtain the initial pseudo-labels for each cluster class;

[0015] 3.2. Detect the outlier data in each cluster class using an isolation forest outlier detection algorithm, and remove the detected outlier data to obtain the final pseudo-labels;

[0016] Step 4. Based on the source text feature vectors obtained in step 2, use a clustering network to calculate the probability distribution values of the text data in K cluster classes, and use the probability distribution values and the pseudo-labels obtained in step 3 to perform clustering to obtain clustering labels; based on the enhanced text feature vectors obtained in step 2, use a contrastive learning network to obtain more discriminative feature vectors. In this process, use the pseudo-labels obtained in step 3 to alleviate the influence of inherent false negative examples; adopt a joint strategy to optimize the losses of the clustering network and the contrastive learning network, and according to the optimization results, use backpropagation and stochastic gradient descent algorithms to update the parameters of the contrastive learning network, the clustering network, and the feature extraction model obtained in step 1;

[0017] Step 5. Input the text data to be processed into the updated feature extraction model for feature extraction, input it into the updated clustering model to obtain the probability distribution values of the text belonging to K cluster classes, and select the index corresponding to the maximum probability distribution value as the clustering label of the text.

[0018] Further, step 1 includes the following steps:

[0019] First, randomly mask the words in the dataset, then mark the masked words, and train SentenceBERT based on the marked dataset; and in step 1, use a dataset with a masking rate of 15% for training.

[0020] Further, the methods for obtaining the weakly augmented data and strongly augmented data in step 2.1 are as follows:

[0021] 2.1.1. Use a pre-trained roberta model to obtain the top n words with the highest matching probability for the input short text;

[0022] 2.1.2. Based on the context environment, randomly replace 20% of the words in each short text using the top n words obtained in 2.1.1 to obtain the weakly augmented data;

[0023] Use the top n words obtained in 2.1.1 to perform synonym replacement on 20% of the words in a short text, random deletion on 20% of the words, and random swapping on 20% of the words in turn to obtain the strongly augmented text of the source text.

[0024] Further, the method of tuning the losses of the clustering network and the contrast learning network using a joint strategy in step 4 includes the following steps:

[0025] For the clustering network, a weighted cross-entropy loss function is used to calculate its loss

[0026]

[0027] In formula (1), denotes the index set of the data belonging to the non-outlier points in batch , w k ∝1 / N k is the weight parameter representing the k-th cluster and the size of this cluster is N k , and this weight parameter can prevent the large-cluster data from destroying the feature representation in the latent space. p i (·) represents the probability distribution of data x i ;

[0028] For the contrast learning network, the average value of the minimum losses of the weakly augmented text data and the strongly augmented text data is used as its loss The positive example pairs in the contrast learning network are composed of the strongly augmented text data and the weakly augmented text data augmented from the same source text, and the negative example pairs are composed of the strongly augmented text data and the weakly augmented text data augmented from different source texts; the specific calculation process is as follows:

[0029] Calculate the minimum loss of the weakly augmented text data

[0030]

[0031] Formula (2), denotes the index set of all data that forms a negative example pair with data x i , is a dot product formula, and τ is the temperature parameter;

[0032] Calculate the minimum loss of the strongly augmented text data

[0033]

[0034] The overall loss of the contrast learning network is:

[0035]

[0036] In formula (4), M represents the size of the batch;

[0037] Joint strategy is expressed as:

[0038]

[0039] In formula (5), λ represents a monotonically decreasing function with a range of [5,15]. The expression of λ is:

[0040]

[0041] In formula (6), l represents the current iteration number, and L represents the total iteration number.

[0042] The present invention provides a short text clustering method based on non-outlier pseudo-labels, which uses a masked pre-trained language model to obtain a feature extraction model for feature extraction to improve the quality of the text feature of the data set. By combining contrastive learning and clustering learning based on a non-outlier pseudo-label algorithm, the quality of negative pairs in contrastive learning is increased, and contrastive learning is used to shorten the distance between outliers and non-outliers, and the distance between non-outliers is shortened by clustering heads. In this way, a smaller intra-class distance and a larger inter-class distance are obtained, thereby improving the clustering effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of the overall framework of short text clustering based on the non-outlier pseudo-labeling algorithm. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0045] like Figure 1 As shown, the present embodiment provides a short text clustering method based on a non-outlier pseudo-label algorithm, comprising the following steps:

[0046] Step 1: Source text dataset Use mask pre-training SentenceBERT to obtain a feature extraction model for the source text dataset; source text dataset It is composed of n given short texts. The steps of using mask to train SentenceBERT in this embodiment are:

[0047] First, the data set The words in the dataset are randomly masked, the masked words are marked, the ids of the marked words are recorded, and SentenceBERT is trained based on the marked dataset; and in step 1, a dataset with a masking rate of 15% is used for training.

[0048] Step 2: Extract feature vectors. The feature vector extraction method is as follows:

[0049] 2.1. From the source text dataset A batch of text data randomly selected from Obtain its corresponding enhanced text data. The enhanced text includes weakly enhanced text data and strongly enhanced data M represents the number of texts included in a batch. The methods for obtaining weakly enhanced data and strongly enhanced data are as follows:

[0050] 2.1.1. Use a pre-trained roberta model to obtain the top n words with the highest matching probability with the input short text;

[0051] 2.1.2. Based on the context environment, randomly replace 20% of the words in each short text with the top n words obtained in step 2.1.1 to obtain weakly enhanced text

[0052] Use the top n words obtained in step 2.1.1 to perform synonym replacement, random deletion, and random swapping on 20% of the words in a short text in sequence, so as to obtain the strongly enhanced text of the source text

[0053] 2.2. Input the text data selected in step 2.1 Weakly enhanced text data and strongly enhanced text data into the feature extraction model obtained in step 1 for feature extraction, and obtain the source text feature vector, weakly enhanced text feature vector, and strongly enhanced text feature vector.

[0054] In this embodiment, the symbol f θ (·) represents SentenceBERT before pre-training, and f θ ′(·) represents the feature extraction model obtained in step 1. The source text feature vector extracted by the feature extraction model is h i = f θ ′(x i ), and the feature vectors of the two enhanced texts are and where

[0055] Step 3. Calculate the non-outlier pseudo-labels. The calculation method of the non-outlier pseudo-labels is as follows:

[0056] 3.1. Use a clustering algorithm to cluster the source text feature vector h i = f θ ′(x i ) obtained in step 2.2 into K cluster classes, and obtain the initial pseudo-labels of each cluster class where y i ∈{0,...,K - 1}.

[0057] 3.2. Use the Isolation Forest outlier detection algorithm to detect the outlier data in each cluster class, and remove the detected outlier data, so as to obtain the final pseudo labels. In this embodiment, by removing the outlier data, the effectiveness of the data is improved, and thus the clustering accuracy and efficiency are improved.

[0058] Step 4. Based on the source text feature vector h i obtained in Step 2, use the clustering network to calculate the probability distribution values of the text data x i in K cluster classes, and use the probability distribution values and the pseudo labels obtained in Step 3 to perform clustering to obtain clustering labels; based on the enhanced text feature vector obtained in Step 2, use the contrastive learning network to obtain a more discriminative feature vector. In this process, use the pseudo labels obtained in Step 3 to mitigate the influence of inherent false negative examples; adopt a joint strategy to optimize the losses of the clustering network and the contrastive learning network, and according to the optimization results, use the backpropagation and stochastic gradient descent algorithms to update the parameters of the contrastive learning network, the clustering network, and the feature extraction model obtained in Step 1.

[0059] In this embodiment, the clustering network g C (·) is a three-layer non-linear perceptron, and the number of nodes in each layer is 768, 768, and K in sequence, and there is a Relu activation function between layers. Among them, K represents the number of classes into which the data set is to be clustered. The clustering network g C (·) maps the source text feature vector h i from 768 dimensions to a K-dimensional subspace, which is formally expressed as p i = g C (h i ), and the vector p i represents the probability distribution values of the text x i belonging to K classes respectively. The clustering network uses a weighted cross-entropy loss function to calculate its loss

[0060] In formula (1),

[0061] In the formula, represents the index set of the data belonging to the non-outlier data in the batch , w k ∝ 1 / N k is the weight parameter representing the kth cluster and the size of this cluster is N k , and this weight parameter can prevent the large cluster data from destroying the feature representation of the latent space during the training process.

[0062] The contrastive learning network used is also a three-layer non-linear perceptron, with the number of nodes in each layer being 768, 768, and 128 in sequence, and there is a Relu activation function between layers. The role of this head is to map the feature vector of the enhanced text from 768 dimensions to a 128-dimensional subspace, formally expressed as The contrastive learning network pulls the feature vectors of the positive example pairs of text closer and pulls the feature representations of the negative example pairs farther apart, so that SentenceBERT can obtain more discriminative feature vectors. The positive example pairs in this network are the weakly enhanced text data i and strongly enhanced text data enhanced from the same source text data x The negative example pairs are composed of the weakly enhanced text data and strongly enhanced text data enhanced from different source texts where i ≠ j. Since there are no true labels for these data, there are cases where the inherent false negative example pairs affect the effect of contrastive learning. For example, for two different source text data in the inherent false negative example pairs that come from the same semantic cluster class, their feature vectors should not be pulled farther apart. To overcome this problem, in this embodiment, the pseudo-labels obtained through step 3 are respectively combined with the weakly enhanced text and strongly enhanced text to minimize the loss.

[0063] The minimization of the loss of the weakly enhanced text is: For:

[0064]

[0065] Among them, represents the index set of all data that forms a negative example pair with the data x i , is a dot product formula, and τ is the temperature parameter set to 0.5;

[0066] In the formula, represents the index set of all data that forms a negative example pair with the text data x i , is a dot product formula; τ is the temperature parameter, set to 0.5;

[0067] The minimum loss of the feature vector of the strongly enhanced text is:

[0068]

[0069] The overall loss of the contrastive learning network is:

[0070] In formula (4),

[0071] Using a combined strategy Dynamically adjust The weights in the combined training, combined strategy Expressed as:

[0072]

[0073] In Equation (5), λ represents a monotonically decreasing function with a value range of [5, 15], and the expression of λ is:

[0074]

[0075] In Equation (6), l represents the current iteration step, and L represents the total number of iteration steps.

[0076] Step 5: Input the text data to be processed into the updated feature extraction model for feature extraction, input it into the updated clustering network to calculate the distribution probability value of the text belonging to the K clusters, and select the index corresponding to the maximum probability as the clustering label of the text.

[0077] In this embodiment, the learning rate of the feature extraction model is 0.000005, the learning rates of the contrast learning network and the clustering network are 0.00005, the number of texts M included in each batch is 800, and the total number of iterations L = 1000, and the total number of iterations L = 1000. During the training process, the pseudo-labels are refined every 50 iterations.

[0078] To more intuitively show the above short text clustering method based on the non-outlier pseudo-label algorithm, the following pseudo-code is used in this embodiment for summary.

[0079]

[0080] As can be seen from the summary content of the above pseudo-code, the short text clustering method based on the non-outlier pseudo-label algorithm provided in this embodiment uses the feature extraction model obtained by the masked pre-trained language model for feature extraction, improving the quality of the text feature vectors of the dataset; by combining the non-outlier pseudo-label algorithm with contrast learning and clustering learning, taking into account both the instance-level data relationships and the cluster-level distinctiveness of the text dataset, making the generated text feature representations have better separability and obtaining better clustering effects.

[0081] Although the above describes the illustrative specific embodiments of the present invention for the understanding of those skilled in the art in this technical field, it should be clear that the present invention is not limited to the scope of the specific embodiments. Any equivalent substitution or equivalent replacement, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

Claims

1. A short text clustering method based on an outlier-free pseudo-label algorithm, characterized in that, it includes the following steps: Step 1: For the source text dataset, use masked pre-training SentenceBERT to obtain a feature extraction model for the source text dataset; the source text dataset consists of n given short texts; Step 2: Extract feature vectors, and the extraction method of the feature vectors is as follows: 2.

1. Randomly select a batch of text data from the source text dataset, and obtain its corresponding augmented text data; the augmented text data includes weakly augmented text data and strongly augmented text data; 2.

2. Input the text data, weakly augmented text data, and strongly augmented text selected in step 2.1 into the feature extraction model obtained in step 1 for feature extraction, and obtain source text feature vectors, weakly augmented text feature vectors, and strongly augmented text feature vectors; Step 3: Calculate the outlier-free pseudo-labels, and the calculation method of the outlier-free pseudo-labels is as follows: 3.

1. Use a clustering algorithm to cluster the source text feature vectors obtained in step 2.2 into K cluster classes, and obtain the initial pseudo-labels for each cluster class; 3.

2. Use an isolation forest outlier detection algorithm to detect the outlier data in each cluster class, and remove the detected outlier data to obtain the final pseudo-labels; Step 4: Based on the source text feature vectors obtained in step 2, use a clustering network to calculate the probability distribution values of the text data in the K cluster classes, and use the probability distribution values and the pseudo-labels obtained in step 3 to perform clustering to obtain clustering labels; based on the augmented text feature vectors obtained in step 2, use a contrastive learning network to obtain more discriminative feature vectors. In this process, use the pseudo-labels obtained in step 3 to alleviate the influence of inherent false negative examples; adopt a joint strategy to optimize the losses of the clustering network and the contrastive learning network, and according to the optimization results, use backpropagation and stochastic gradient descent algorithms to update the parameters of the contrastive learning network, the clustering network, and the feature extraction model obtained in step 1; Step 5: Input the text data to be processed into the updated feature extraction model for feature extraction, input it into the updated clustering model to obtain the probability distribution values of the text belonging to the K cluster classes, and select the index corresponding to the maximum probability distribution value as the clustering label of the text.

2. The short text clustering method based on an outlier-free pseudo-label algorithm according to claim 1, characterized in that, Step 1 includes the following steps: First, randomly mask the words in the dataset, then mark the masked words, and train SentenceBERT based on the marked dataset; And in step 1, a dataset with a masking rate of 15% is used for training.

3. The short text clustering method based on an outlier-free pseudo-label algorithm according to claim 1, characterized in that, The method for obtaining the weakly augmented data and strongly augmented data in step 2.1 is as follows: 2.1.

1. Use a pre-trained roberta model to obtain the top n words with the highest matching probability with the input short text; 2.1.

2. Based on the context environment, randomly replace 20% of the words in each short text with the first n words obtained in 2.1.1 to obtain weakly augmented data; Use the first n words obtained in 2.1.1 to perform synonym replacement on 20% of the words, random deletion on 20% of the words, and random swapping on 20% of the words in a short text in sequence, so as to obtain a strongly augmented text of the source text.

4. A short text clustering method based on an outlier-free pseudo-label algorithm according to any one of claims 1 to 3, characterized in that the method for optimizing the losses of the clustering network and the ratio learning network by adopting a joint strategy in step 4 includes the following steps: For the clustering network, a weighted cross-entropy loss function is used to calculate its loss In formula (1), represents the index set of data belonging to non-outliers in batch , w k ∝1 / N k is the weight parameter representing the k-th cluster and the size of this cluster is N k , and this weight parameter can prevent the data of large clusters from destroying the feature representation of the latent space during training, p i (·) represents the probability distribution of data x i ; For the contrastive learning network, the average of the minimum losses of the weakly augmented text data and the strongly augmented text data is used as its loss. The positive example pairs in the contrastive learning network are composed of strongly augmented text data and weakly augmented text data augmented from the same source text, and the negative example pairs are composed of strongly augmented text data and weakly augmented text data augmented from different source texts; the specific calculation process is as follows: Calculate the minimum loss of weakly augmented text data Equation (2), represents all data indices that form negative example pairs with data x i is a dot product formula, where τ is the temperature parameter; is a dot product formula, and τ is the temperature parameter; Calculate the minimum loss of strongly enhanced text data Overall loss of the contrastive learning network is as follows: In formula (4), M represents the size of the batch; Combined strategy Expressed as: In formula (5), λ represents a monotonically decreasing function with a value range of [5, 15], and the expression of λ is: In formula (6), l represents the current iteration step, and L represents the total number of iteration steps.

Images