Deep semi-supervised text clustering method based on pairwise constraints and cluster guidance
By introducing pairwise constraints and cluster guidance methods into the text depth clustering model, the accuracy and robustness of text clustering are optimized, and the problem of poor clustering accuracy and robustness in the existing technology is solved, and the problem of poor clustering accuracy and robustness is realized. More effective supervision of information utilization and clustering results.
Patent Information
- Application Number
- CN202111574148.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-21
AI Technical Summary
When the existing text-depth clustering model utilizes paired constraints and supervision information of class tags, there is a problem of poor clustering accuracy and robustness, especially ignoring the label matching problem between labeled clusters and labelless clusters. This leads to incorrect supersupervised information being introduced and clustering accuracy is restricted.
A deep semi-supervised text clustering method based on paired constraints and cluster guidance is proposed. By preprocessing and vectorizing the target text data, hidden layer features are learned, and relationship matrix and similarity matrix are calculated under paired constraints. Combining cluster allocation losses for labeled and unlabeled clusters, the joint loss function is optimized to iterative clustering results.
It significantly improves the accuracy and robustness of text clustering, fully explores the supervision information in the tag, strengthens the guidance learning of label-free clusters, avoids incorrect supervision information guidance, and improves the clustering performance of the model.
Smart Images

Figure CN114238638B_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate to the field of data processing technology, and in particular to a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance. Background Art
[0002] At present, faced with the information overload problem brought about by the continuous expansion of social networks, how to reasonably organize and efficiently utilize the valuable information in the text has become a focal issue in the field of data mining. The use of clustering technology for integration and segmentation can assist in the deep mining, analysis and prediction of massive texts. It is also an important prerequisite for the discovery and extraction of key knowledge. It has important applications in many aspects such as entity classification, sentiment analysis, intelligent recommendation, and automatic question and answer.
[0003] Existing clustering methods such as Kmeans are prone to the "curse of dimensionality" problem, and the method of reducing dimension first and then clustering is prone to the "error propagation" phenomenon, that is, the error in the dimension reduction step directly affects the quality of the clustering results. Therefore, in order to obtain the most favorable dimension reduction representation for clustering, the concept of deep clustering was proposed. By jointly optimizing the two tasks of dimension reduction and clustering, better model performance can be obtained than when learning a single task. Early representative methods include deep embedding clustering (DEC) and improved deep embedding clustering (IDEC). However, the above models are all distance-based unsupervised clustering methods, which have certain limitations. In addition, the prior information in the text data has not been fully utilized in the above models. Therefore, in order to fully combine the existing supervised information to guide the feature extraction and clustering process, some semi-supervised clustering models have been proposed one after another. However, they only rely on a voting mechanism based on sample soft assignment to obtain the adaptive labels of each labeled cluster, while ignoring the label assignment problem of the labeled cluster labels. Therefore, when the soft assignment results of some samples are seriously deviated, different clusters are likely to point to the same correction label, resulting in the introduction of incorrect hyper-supervisory information, and the clustering accuracy will be seriously restricted.
[0004] In summary, existing text deep clustering models rarely integrate the dual supervision advantages of pairwise constraints and class labels at the same time, and the existing supervised clustering methods using class labels cannot fully utilize the supervisory information therein and there is incorrect supervisory information guidance.
[0005] It can be seen that there is an urgent need for a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance with high clustering accuracy and robustness. Summary of the invention
[0006] In view of this, an embodiment of the present disclosure provides a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance, which at least partially solves the problem of poor clustering performance and robustness in the prior art.
[0007] The present disclosure provides a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance, including:
[0008] Preprocess and vectorize the target text data to obtain a multi-dimensional vector;
[0009] Learning the hidden features of the target text data according to the multidimensional vector, and inputting the hidden features into a preset algorithm for clustering to obtain initial cluster centers;
[0010] Calculating sample soft distribution and auxiliary distribution according to the hidden layer features and the initial cluster centers, and calculating clustering loss according to the sample soft distribution and the auxiliary target distribution;
[0011] Constructing pairwise constraints according to labeled samples in the target text data, calculating a relationship matrix and a similarity matrix under the pairwise constraints, and generating pairwise constraint losses using cross entropy;
[0012] According to the data labels, all the labeled samples are divided into a plurality of labeled clusters of different types, according to the real-time clustering results, the unlabeled samples in the target text data are divided into a plurality of unlabeled clusters, and the cluster allocation losses of all the labeled clusters and all the unlabeled clusters are calculated, wherein the number of the labeled clusters and the unlabeled clusters is the same as the number of categories of the data labels;
[0013] A joint loss function is calculated according to the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster allocation loss, and a clustering result is obtained when the joint loss function is iterated to reach a preset condition.
[0014] According to a specific implementation of the embodiment of the present disclosure, the step of preprocessing and vectorizing the target text data to obtain a multi-dimensional vector includes:
[0015] Performing text enhancement on the target text data;
[0016] Performing a traditional-simplified conversion operation, a word segmentation operation, and a stop word removal operation on the target text data after the text enhancement to obtain intermediate text data;
[0017] After the intermediate text data is vectorized by using word embedding technology, the multi-dimensional vector is obtained through average pooling operation.
[0018] According to a specific implementation of the embodiment of the present disclosure, the step of constructing pairwise constraints according to the labeled samples in the target text data, calculating the relationship matrix and the similarity matrix under the pairwise constraints, and generating the pairwise constraint loss using the cross entropy includes:
[0019] According to the labeled samples, the label information is converted into the pairwise constraints and the relationship matrix is constructed accordingly;
[0020] Calculate the cosine angle between any two samples in the relationship matrix according to the hidden layer features to obtain the similarity matrix;
[0021] The pairwise constraint loss is calculated using the cross entropy according to the relationship matrix and the similarity matrix.
[0022] According to a specific implementation of an embodiment of the present disclosure, the steps of dividing all the labeled samples into a plurality of different types of labeled clusters according to data labels, dividing the unlabeled samples in the target text data into a plurality of unlabeled clusters according to real-time clustering results, and calculating the cluster allocation losses of all the labeled clusters and all the unlabeled clusters include:
[0023] Converting the labeled samples into the plurality of labeled clusters according to the data labels;
[0024] Dividing the unlabeled samples into a plurality of unlabeled clusters according to the real-time clustering result;
[0025] Traversing all the labeled clusters and all the unlabeled clusters, calculating the probability that the clusters belong to the same category using average cosine similarity, and generating cluster soft assignment;
[0026] Based on the cluster soft assignment, the label assignment results of all the labeled clusters and all the unlabeled clusters are obtained by using the Hungarian algorithm, and the target distribution is constructed based on them;
[0027] The cluster allocation loss is calculated using KL divergence according to the cluster soft allocation and the target distribution.
[0028] According to a specific implementation of an embodiment of the present disclosure, the step of calculating a joint loss function according to the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster assignment loss, and obtaining a clustering result when the joint loss function is iterated to reach a preset condition includes:
[0029] Combining the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster assignment loss to obtain the joint loss function;
[0030] Iteratively optimizing the model parameters of the deep semi-supervised clustering model according to the joint loss function and the stochastic gradient descent algorithm, and generating corresponding iteration results in each iterative calculation;
[0031] When the iteration reaches the preset condition, the last iteration result is used as the clustering result.
[0032] According to a specific implementation manner of the embodiment of the present disclosure, the preset condition includes that two consecutive clustering result changes are less than either a termination threshold or a maximum number of iterations.
[0033] According to a specific implementation of the embodiment of the present disclosure, the expression of the pairwise constraint loss is: Where N1 represents the total number of labeled samples, A={a ij} is the relationship matrix, S = {s ij} is the similarity matrix, D={d ij} is the distance matrix transformed from the similarity matrix, where d ij =(1-s ij ) / 2.
[0034] According to a specific implementation of the embodiment of the present disclosure, the expression of cluster soft allocation is: in, and denote the K labeled clusters and the unlabeled clusters, respectively; a1 and a2 denote the hidden feature representations of the data points in the labeled cluster and the unlabeled cluster, respectively; cos(a1, a2) denotes the cosine similarity between the two; M ij It means that the loss contribution of each cluster is normalized according to the number of data points in each cluster, and its value is cluster L i and U j The product of the number of samples in .
[0035] According to a specific implementation manner of the embodiment of the present disclosure, the expression of the target distribution is J=Hungary(-I), wherein J represents the cluster allocation result calculated based on the Hungary algorithm.
[0036] According to a specific implementation of the embodiment of the present disclosure, the cluster allocation loss is expressed as: It is used to reflect the difference between the average similarity between two clusters and the cluster allocation results, and then use the distribution difference between the two to continuously guide the update of the hidden features of all samples and cluster centers Where K is the number of clusters and N2 is the total number of samples.
[0037] The deep semi-supervised text clustering scheme based on pairwise constraints and cluster guidance in the embodiment of the present disclosure includes: preprocessing and vectorizing the target text data to obtain a multidimensional vector; learning the hidden features of the target text data according to the multidimensional vector, and inputting the hidden features into a preset algorithm for clustering to obtain an initial clustering center; calculating the sample soft distribution and auxiliary distribution according to the hidden features and the initial clustering center, and calculating the clustering loss according to the sample soft distribution and the auxiliary target distribution; constructing pairwise constraints according to the labeled samples in the target text data, and calculating the relationship matrix and the similarity matrix under the pairwise constraints, and using Generate pairwise constraint loss using cross entropy; divide all the labeled samples into multiple different types of labeled clusters according to the data labels, divide the unlabeled samples in the target text data into multiple unlabeled clusters according to the real-time clustering results, and calculate the cluster allocation losses of all the labeled clusters and all the unlabeled clusters, wherein the number of the labeled clusters and the unlabeled clusters is the same as the number of categories of the data labels; calculate a joint loss function based on the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster allocation loss, and obtain a clustering result when the joint loss function iteration reaches a preset condition.
[0038] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the supervisory information in the labels is fully mined, the guided learning between them and the unlabeled clusters is strengthened, the robustness of the deep semi-supervised clustering model is improved, and at the same time the accuracy of text clustering is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 A flowchart of a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided in an embodiment of the present disclosure;
[0041] Figure 2 A framework diagram of a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided in an embodiment of the present disclosure;
[0042] Figure 3 A clustering effect diagram involving a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0043] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0044] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0045] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0046] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0047] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0048] Faced with the information overload problem brought about by the continuous expansion of social networks, how to reasonably organize and efficiently utilize the valuable information in texts has become a focus in the field of data mining. The use of clustering technology for integration and segmentation can assist in the deep mining, analysis and prediction of massive texts. It is also an important prerequisite for the discovery and extraction of key knowledge. It has important applications in many aspects such as entity classification, sentiment analysis, intelligent recommendation and automatic question answering. In particular, text clustering, as one of the important contents of data mining technology, can better mine the most valuable deep information and association rules from massive disordered, noisy, random and variable data by classifying text knowledge with similar themes or semantic representations, thereby effectively improving the management retrieval and information mining performance of massive text data.
[0049] Among them, vector embedding is an important pre-operation of text clustering, which can convert unstructured text into numerical vectors that can be recognized and calculated by computers. However, since the embedded text vectors often have characteristics such as high dimensionality and sparseness, directly using traditional clustering methods such as Kmeans is prone to cause the "dimensionality disaster" problem. The method of reducing dimension first and then clustering is prone to "error propagation", that is, the error in the dimension reduction step directly affects the quality of the clustering results. Therefore, in order to obtain the most favorable dimension reduction representation for clustering, the concept of deep clustering was proposed. By jointly optimizing the two tasks of dimension reduction and clustering, better model performance can be obtained than when learning a single task. Early representative methods include deep embedding clustering (DEC) and improved deep embedding clustering (IDEC).
[0050] However, all of the above models are distance-based unsupervised clustering methods, which have certain limitations. In addition, the prior information in the text data is not fully utilized in the above models. Therefore, in order to fully combine the existing supervision information to guide the feature extraction and clustering process, some semi-supervised clustering models have been proposed, which are mainly divided into two types of supervision: pairwise constraints and class labels. Since the predicted labels of deep clustering models will change all the time during the iteration process and have no practical meaning, that is, the "label shift" phenomenon, most semi-supervised clustering methods use pairwise constraints to guide feature learning and clustering processes, and are easy to convert from labels. By making the features of sample pairs in one class as close as possible or the similarity as close to 1 as possible, and the features of features not in one class as far as possible or the similarity as close to 0 as possible, the introduction of local sample information is achieved. Among them, the semi-supervised deep clustering model based on similarity is relatively more stable in training process than the semi-supervised deep clustering (SDEC) model directly based on feature optimization, and can better improve clustering performance. However, it is not difficult to find that paired constraint supervision can only reflect the correlation information of two samples in label supervision, while ignoring the global cluster information divided by label number. The supervision information is weaker, and when facing the text clustering task in professional fields, since each category of text belongs to the same field, the sample features of different categories will be close to each other, that is, the sample overlap phenomenon, and the space for improving clustering accuracy will become smaller and smaller. Therefore, in order to make full use of the supervision information in the label, a deep semi-supervised clustering method (Semi-DEC) based on label adaptation is proposed to solve the label shift problem in the clustering process. It has stronger feature representation and clustering allocation capabilities, but it only relies on the voting mechanism based on sample soft allocation to obtain the adaptive labels of each labeled cluster, while ignoring the label matching problem between the labeled cluster and the unlabeled cluster. Therefore, when the soft allocation results of some samples are seriously deviated, different clusters are easy to point to the same correction label, which leads to the introduction of incorrect super-supervision information, and the clustering accuracy will be seriously restricted.
[0051] In summary, existing text deep clustering models rarely integrate the dual supervision advantages of pairwise constraints and class labels at the same time, and the existing supervised clustering methods using class labels cannot fully utilize the supervisory information therein and there is incorrect supervisory information guidance.
[0052] The embodiments of the present disclosure provide a pairwise-constrained and cluster-guided deep semi-supervised text clustering method, which can be applied to a text tag clustering process in a text data processing scenario.
[0053] See also Figure 1 , is a flowchart of a deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided by an embodiment of the present disclosure. Figure 1 As shown, the method mainly comprises the following steps:
[0054] S101, preprocessing and vectorizing the target text data to obtain a multi-dimensional vector;
[0055] For example, when it is necessary to cluster text data in a specific field such as the injection molding field, we can collect empirical text descriptions of the mechanism of the injection molding process H(k), k = 1, 2, ..., N3, where H(k) is a sentence of indefinite length, k is the sentence number, and N3 is the total number of sentences. Then, we perform preprocessing operations such as traditional-simplified conversion, Chinese word segmentation, and stop word removal. We use word embedding technology to obtain the multidimensional vector representation corresponding to each injection molding text, and then perform vector normalization. We use EDA (Easy data augmentation) technology such as synonym replacement to obtain the text-enhanced injection molding vector. Among them, N4 is the word vector dimension and the number of vectors is N2.
[0056] S102, learning the hidden features of the target text data according to the multidimensional vector, and inputting the hidden features into a preset algorithm for clustering to obtain initial cluster centers;
[0057] When implementing it, Figure 2 As shown, the injection molding text vector after vectorization, standardization and data enhancement, i.e., v(i), i=1,2,...,N2, is used to transform the original high-dimensional feature space using a stacked autoencoder structure, thereby extracting the key low-dimensional hidden layer feature representation and initial clustering center. The specific process may include:
[0058] First, the entire stacked autoencoder network is pre-trained. Specifically, the pre-training is divided into two processes: encoding and decoding. Set v as the input of the deep autoencoder and z as the intermediate encoding result of the deep autoencoder, that is, the hidden layer feature representation. is the final decoded output of the deep autoencoder. The calculation process is as follows:
[0059] z=f(v)=g1(w1 v+b1) (1)
[0060]
[0061] Among them, f(·) realizes the encoding mapping of input v to hidden layer feature z, and g(·) realizes the encoding mapping of hidden layer feature z to output The decoding reconstruction, g1 and g2 represent the activation functions of the encoder and decoder respectively, both use the ReLU function, θ = [w1, b1, w2, b2] is the model parameter, and the objective function is to minimize the reconstruction error Then, the random gradient descent Adam algorithm is used for fine-tuning until the network converges. After training the first layer, its hidden layer feature z is used as the input of the second layer autoencoder. The second layer is trained according to formula (1) (2), and so on. Finally, the pre-trained network obtains the initial mapping f from the original high-dimensional vector to the hidden layer low-dimensional feature through the encoder layer. θ :v→z, realize the transformation of feature space.
[0062] Then input the cluster number K and N2 data samples Based on the trained network, we get the sample point v i The hidden layer feature representation z i , and then use the Kmeans algorithm to generate K initial cluster centers and real-time clustering results in the hidden feature space Among them C j Represents the sample set of the jth category. The specific method is to first randomly select K cluster centers from N2 data samples Then, the Euclidean distance is used to measure the distance from all data points to the cluster center and divide them into the nearest category λ i =argmin j∈{1,2,...,K} r ij In which r ij represents the Euclidean distance between two points, λ i Represents the classification of the i-th sample, so that the sum of the distances from each data point to the center of the cluster to which it belongs is minimized, and then the mean of each cluster is calculated As the new cluster center, where |C j | represents the number of samples in the jth cluster. Finally, the Kmeans algorithm moves towards The above steps are continuously optimized and iterated in the direction until the cluster center no longer changes or a sufficient number of iterations is reached, and the initial cluster center of the deep embedded clustering model is obtained according to the final clustering result.
[0063] S103, calculating sample soft distribution and auxiliary distribution according to the hidden layer features and the initial cluster center, and calculating clustering loss according to the sample soft distribution and the auxiliary target distribution;
[0064] In the specific implementation, based on the hidden features of the data points initially mapped in the pre-training phase and the calculated initial cluster centers First, the Student's t distribution is used to measure the distance between the hidden layer feature of each data point and the initial cluster center, and the sample soft assignment that represents the probability of the i-th data sample belonging to the j-th class is obtained. Then, the auxiliary target distribution is used to learn the clustering loss from the current high-confidence soft assignment while minimizing the reconstruction error. The above two steps are alternately implemented to achieve joint optimization of clustering performance. The specific calculation process is as follows:
[0065] Once a sample belongs to a certain cluster, the probability of belonging to other clusters is 0, which is the hard assignment commonly used in clustering process. Soft assignment is defined as the probability that all samples belong to each cluster. It is not difficult to find that soft assignment can obtain hard assignment by taking the maximum probability in the category dimension. That is the final clustering result. The specific soft allocation is calculated using Student's t distribution:
[0066]
[0067] Among them, z i represents the feature after dimensionality reduction by the current mapping, μ j Indicates the current cluster center, q ij Characterizes the probability that the i-th data sample belongs to the j-th class, that is, soft assignment.
[0068] The deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided by the embodiment of the present disclosure defines auxiliary distribution based on soft assignment, mainly considering the following three characteristics: 1) improving cluster purity; 2) strengthening the prediction of high-confidence assignment; 3) normalizing the contribution of each cluster center to avoid large categories distorting the feature space. The specific calculation formula is as follows:
[0069]
[0070] in, Indicates sharpening of soft assignments to strengthen the prediction of high-confidence assignments and improve cluster purity, ∑ i q ij Characterize the assignment frequency of each cluster and normalize its contribution accordingly.
[0071] Then the clustering loss L2 can be defined as the KL divergence between the soft distribution Q and the auxiliary distribution P. By allowing the soft distribution to continuously fit and approximate the designed auxiliary distribution, the high-confidence distribution in the soft distribution can be learned, and then the feature mapping f can be further optimized based on the pre-training. θ , hidden layer features and cluster centers The specific formula is as follows:
[0072]
[0073] While the clustering loss is used to fine-tune the entire network, in order to prevent the original high-dimensional feature space from being severely distorted, the reconstruction error can be used To preserve the original local structure, it is jointly optimized with the clustering loss to learn more accurate clustering divisions.
[0074] S104, constructing pairwise constraints according to the labeled samples in the target text data, calculating the relationship matrix and the similarity matrix under the pairwise constraints, and generating the pairwise constraint loss using cross entropy;
[0075] Optionally, the expression of the pairwise constraint loss is
[0076] Where N1 represents the total number of labeled samples, A={a ij} is the relationship matrix, S = {s ij} is the similarity matrix, D={d ij} is the distance matrix transformed from the similarity matrix, where d ij =(1-s ij ) / 2.
[0077] In specific implementation, based on a small number of label samples {v l ,y l}Establish pairwise constraints, where v l Represents a set of labeled samples, and constructs a similarity matrix by performing pairwise calculations y l Its corresponding label is used to establish the relationship matrix That is, when the labels of two samples are consistent, the element a corresponding to the relationship matrix ij is 1, otherwise it is 0, and then calculate the cross entropy of A and S Then, the paired constraint supervision information can be fully exploited.
[0078] S105, dividing all the labeled samples into a plurality of labeled clusters of different types according to the data labels, dividing the unlabeled samples in the target text data into a plurality of unlabeled clusters according to the real-time clustering results, and calculating the cluster allocation losses of all the labeled clusters and all the unlabeled clusters, wherein the number of the labeled clusters and the unlabeled clusters is the same as the number of categories of the data labels;
[0079] Furthermore, the expression of cluster soft allocation is: in, and denote the K labeled clusters and the unlabeled clusters, respectively; a1 and a2 denote the hidden feature representations of the data points in the labeled cluster and the unlabeled cluster, respectively; cos(a1, a2) denotes the cosine similarity between the two; M ij It means that the loss contribution of each cluster is normalized according to the number of data points in each cluster, and its value is cluster L i and U j The product of the number of samples in .
[0080] Optionally, the expression of the target distribution is J=Hungary(-I), where J represents the cluster allocation result calculated based on the Hungary algorithm.
[0081] Optionally, the cluster allocation loss is expressed as It is used to reflect the difference between the average similarity between two clusters and their cluster allocation results, and then use the distribution difference between the two to continuously guide the update of the hidden features of all samples and cluster centers Where K is the number of clusters and N2 is the total number of samples.
[0082] For example, based on the data label y l A small number of labeled samples v l Divided into K different categories of labeled clusters, while ignoring their specific category numbers, most of the unlabeled samples x u Based on the clustering results of the previous period, the K labeled clusters are divided into K unlabeled clusters in real time. The K labeled clusters are then compared with each unlabeled cluster, and the cumulative average of the sample similarities between the two clusters is calculated to obtain the probability that the i-th labeled cluster belongs to the j-th unlabeled cluster. We define this as cluster soft allocation The Hungarian algorithm is then used to obtain the best assignment results for labeled and unlabeled clusters, which are used as the auxiliary target distribution J = Hungary (-I). Then, the KL divergence can be used to generate the cluster assignment loss. Guided joint optimization of labeled and unlabeled samples.
[0083] S106, calculating a joint loss function according to the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster allocation loss, and obtaining a clustering result when the joint loss function iteration reaches a preset condition.
[0084] In specific implementation, after obtaining the clustering loss, the pairwise constraint loss and the cluster allocation loss, the weights of each loss function can be set, and the above objective function L=L1+αL2+βL3+γL4 can be combined to optimize the network parameters, and iterative updates are performed until a certain stop condition is met to obtain the clustering result. The reconstruction error L1 is used to protect the original high-dimensional feature structure, the clustering loss L2 is based on the Euclidean distance to measure and separate high-confidence samples of different classes, the pairwise constraint loss L3 uses cosine similarity to achieve sample feature update learning, and the cluster allocation loss L4 measures the similarity between each labeled and unlabeled cluster from the perspective of the cluster, thereby effectively preventing the overall misclassification of the labeled cluster, while making full use of the pairwise constraint supervision and label information in the labeled samples and propagating them to the unlabeled samples.
[0085] The deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided in this embodiment fully mines the supervisory information in the labels, strengthens the guided learning between them and unlabeled clusters, improves the robustness of the deep semi-supervised clustering model, and significantly improves the accuracy of text clustering.
[0086] Based on the above embodiment, the step S101 described in which the target text data is preprocessed and vectorized to obtain a multi-dimensional vector includes:
[0087] Performing text enhancement on the target text data;
[0088] Performing a traditional-simplified conversion operation, a word segmentation operation, and a stop word removal operation on the target text data after the text enhancement to obtain intermediate text data;
[0089] After the intermediate text data is vectorized by using word embedding technology, the multi-dimensional vector is obtained through average pooling operation.
[0090] For example, a large amount of relevant corpus D(m) in the field of injection molding is collected, m = 1, 2, ..., N5, where D(m) is an article of indefinite length, m is the article number, and N5 is the total number of articles. In the embodiment of the present disclosure, after the same preprocessing such as traditional-simplified conversion, Chinese word segmentation, and stop word removal, the word sequence and vocabulary of the injection molding corpus after segmentation are obtained, which are used for the unsupervised training of the word embedding model to realize the vectorization of all injection molding vocabulary X1 = [x1, x2..., x V ],in represents N4-dimensional word vector, represents matrix data composed of V N4-dimensional word vectors. In the embodiment of the present disclosure, the word vector dimension N4=300. According to the word vector result obtained by training, each target text W={w n},n=1,2,...,N6 is transformed into the corresponding word vector sequence X2={x n}, n = 1, 2, ..., N6, where N6 represents the number of words in a sentence, and then average pooling is performed to obtain a multi-dimensional sentence vector representation of each target text Finally, through vector normalization, that is, subtracting each mean and then dividing by its standard deviation, the final sample vector v(i) representing the semantic information is formed, i=1,2,...,N2.
[0091] On the basis of the above embodiment, step S104, constructing pairwise constraints according to labeled samples in the target text data, calculating the relationship matrix and the similarity matrix under the pairwise constraints, and generating pairwise constraint losses using cross entropy, includes:
[0092] According to the labeled samples, the label information is converted into the pairwise constraints and the relationship matrix is constructed accordingly;
[0093] Calculate the cosine angle between any two samples in the relationship matrix according to the hidden layer features to obtain the similarity matrix;
[0094] The pairwise constraint loss is calculated using the cross entropy according to the relationship matrix and the similarity matrix.
[0095] In specific implementation, a small number of labeled samples can be converted into paired constraint information first, and the corresponding relationship matrix A can be constructed, that is, only focusing on whether the sample pairs are in the same category or not, without paying attention to their specific category labels. Then, the relationship matrix A is used to determine whether the paired samples should be similar, and the hidden feature learning of the paired samples is adjusted and optimized. The specific process is as follows:
[0096] First, a small amount of label information is given Based on whether the labels of two samples are in the same class or not, a pairwise constraint relationship matrix A is constructed as shown in formula (6):
[0097]
[0098] Among them, i,j∈{1,2,...N2}, A={a ij} is a N2×N2 dimensional relationship matrix, which is used to represent the category constraint information of two labeled samples. ij When it is 1, it means that the sample pair is in the same category, and the similarity between the two should be as close to 1 as possible. ij When it is 0, it means that the sample pair is not in the same category, and the similarity between the two is as small as possible. Therefore, in order to more fully reflect the similarity of semantic information in the text vector, the cosine angle can be used to calculate the similarity matrix S corresponding to the relationship matrix A:
[0099]
[0100] Among them, i,j∈{1,2,...N2}, S={s ij} is a N2×N2 dimensional similarity matrix used to represent data point z i and z j The similarity of , and then calculate the distance matrix D = {d ij}, where d ij =(1-s ij ) / 2, so that its value range is [0,1], and the cross entropy function is used to define the pairwise constraint loss as follows:
[0101]
[0102] Through the above supervision steps, when the prior information a ij When it is 1, the clustering process will guide z i and z j Towards dij =0, that is, s ij =1, otherwise the clustering process will guide z i and z j Towards d ij = 1, thereby realizing the introduction of paired constraint supervision information for learning.
[0103] Furthermore, in step S105, all the labeled samples are divided into a plurality of different types of labeled clusters according to the data labels, the unlabeled samples in the target text data are divided into a plurality of unlabeled clusters according to the real-time clustering results, and the cluster allocation losses of all the labeled clusters and all the unlabeled clusters are calculated, including:
[0104] Converting the labeled samples into a plurality of labeled clusters according to the data labels;
[0105] Dividing the unlabeled samples into a plurality of unlabeled clusters according to the initial real-time clustering results;
[0106] Traversing all the labeled clusters and all the unlabeled clusters, calculating the probability that the clusters belong to the same category using average cosine similarity, and generating cluster soft assignment;
[0107] Based on the cluster soft assignment, the label assignment results of all the labeled clusters and all the unlabeled clusters are obtained by using the Hungarian algorithm, and the target distribution is constructed based on them;
[0108] The cluster allocation loss is calculated using KL divergence according to the cluster soft allocation and the target distribution.
[0109] In the specific implementation, considering that on the basis of introducing pairwise constraint information, since it only considers the correlation information between two samples, the clustering accuracy is not greatly improved, in order to make full use of the supervision information in the label, avoid the bad supervision information brought by adaptive labels, and overcome the negative impact of sample overlap, a cluster-cluster game allocation loss can be proposed based on the pairwise constraint loss, so that not only similar labeled samples can be embedded together, but also they can be classified into the correct unlabeled clusters. At the same time, the labeled and unlabeled samples are jointly optimized. The overall process is divided into three steps: cluster soft allocation calculation, label allocation based on the Hungarian algorithm, and KL distribution minimization.
[0110] For example, similar to the soft assignment of samples in the deep embedded clustering model, it mainly represents the probability that a data sample point belongs to a certain cluster, while the cluster soft assignment in the embodiment of the present disclosure represents the probability that a certain labeled cluster belongs to the same category as a certain unlabeled cluster. The concept of cluster is similar to that of class cluster, representing a set of samples of the same category. Therefore, based on a given set of a small number of labeled samples, on the one hand, we can divide them into multiple labeled clusters according to their specific labels, and on the other hand, based on the soft assignment of samples obtained in the above steps, we can take the maximum probability from the perspective of class clusters to obtain the hard assignment results of all samples, that is, the real-time clustering results, and then divide the remaining unlabeled samples into multiple unlabeled clusters according to the real-time clustering results.
[0111] It is not difficult to find that the unlabeled cluster reflects the real-time sample clustering pattern, and the labeled cluster reflects the initially given sample clustering pattern. The traditional adaptive label method is mainly based on the real-time sample clustering results in each labeled cluster to indirectly achieve adaptation with the unlabeled cluster clustering pattern, but it often only considers the soft allocation information within a single cluster, which is easy to introduce incorrect hyper-supervisory information. Therefore, the embodiment of the present disclosure proposes the concept of cluster soft allocation, which directly characterizes the probability of adaptation between the labeled cluster and the unlabeled cluster, avoiding the influence of label shift and considering the information between clusters. The specific calculation formula is as follows:
[0112]
[0113] in, and denote the K labeled clusters and the unlabeled cluster respectively, a1 and a2 denote the hidden feature representation of the data points in the labeled and unlabeled clusters respectively, cos(a1, a2) denotes the cosine similarity between the two, and M ij It means that the loss contribution of each cluster is normalized according to the number of data points in each cluster, and its value is cluster L i and U j The product of the number of samples in .
[0114] Then, the Hungarian algorithm is used to obtain unique assignments, namely the auxiliary target distribution J to guide network learning, which has the following characteristics: 1) avoids repeated label assignments; 2) coordinates all cluster information to make decisions, and then learns cluster loss from the optimal assignment. The specific calculation formula of the auxiliary target distribution J is as follows:
[0115] J=Hungary(-I) (10)
[0116] where I={I ij} represents the cluster soft allocation, J = {J ij} represents the target distribution, which is obtained by finding the optimal label allocation through the Hungarian algorithm.
[0117] Considering the need to make the cluster soft allocation I match the target distribution J as much as possible, the cluster allocation loss is defined as the KL divergence loss of both the soft allocation and the auxiliary target distribution, so as to obtain the optimization direction away from the best adaptation result, and then continuously guide the update of the hidden features of all samples and cluster centers The specific calculation process is as follows:
[0118]
[0119] Where L4 is the calculated cluster loss, which is used to reflect the difference between the average similarity between two clusters and their cluster assignment results, K is the number of clusters, and N2 represents the total number of samples.
[0120] On the basis of the above embodiment, as described in step S106, a joint loss function is calculated according to the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster allocation loss, and a clustering result is obtained when the joint loss function is iterated to reach a preset condition, including:
[0121] Combining the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster assignment loss to obtain the joint loss function;
[0122] Iteratively optimizing the model parameters of the deep semi-supervised clustering model according to the joint loss function and the stochastic gradient descent algorithm, and generating corresponding iteration results in each iterative calculation;
[0123] When the iteration reaches the preset condition, the last iteration result is used as the clustering result.
[0124] Optionally, the preset condition includes that two consecutive clustering result changes are less than either a termination threshold or a maximum number of iterations.
[0125] In specific implementation, the deep semi-supervised joint learning framework corresponding to the deep semi-supervised text clustering method based on pairwise constraints and cluster guidance provided by the embodiment of the present disclosure is as follows: Figure 2 As shown in the figure, by jointly reconstructing the error, clustering loss, pairwise constraint and cluster loss, we can simultaneously achieve the learning of multiple tasks such as preserving the original feature structure, strengthening the high confidence assignment and making full use of the sample-cluster supervision information.
[0126] L=L1+αL2+βL3+γL4 (12)
[0127] The joint objective function is shown in the above formula (12), where the specific loss calculation is as shown in formula (5)(8)(11). Then, the stochastic gradient descent Adam algorithm is used to optimize the model parameters of the above semi-supervised deep embedding clustering framework, mainly including the stacked autoencoder network parameters θ, the hidden layer mapping features and cluster centers The optimization update is performed until two consecutive cluster allocation changes are less than tol% or the maximum number of training times is reached, the cluster learning is terminated, and the clustering result is output.
[0128] In order to illustrate the clustering effect of the deep semi-supervised text clustering method based on pairwise constraints and cluster guidance in the embodiment of the present disclosure, it is considered to compare with four typical deep clustering methods: deep embedding clustering (DEC), improved deep embedding clustering (IDEC), deep semi-supervised embedding clustering (SDEC), and semi-supervised deep embedding clustering based on adaptive label strategy (Semi-DEC). Specifically, the embodiment is composed of N2 short text vectors of 300 dimensions, and the clustering effect obtained is as follows: Figure 3 As shown, (a) and (b) are the clustering results of DEC and IDEC models respectively. It can be seen that the completely unsupervised deep embedding clustering model cannot distinguish overlapping samples well, and the misclassification of categories ☆ and □ is very serious. (c) and (d) represent the results of SDEC and SemiDEC models respectively, which are greatly improved compared with the completely unsupervised IDEC model, but most of the labeled samples are still misclassified and are strongly disturbed by overlapping samples. (e) is a deep semi-supervised clustering model proposed by the deep semi-supervised text clustering method based on pairwise constraints and cluster guidance disclosed in this paper. A cluster allocation loss is proposed based on the pairwise constraint loss, which is better than all benchmark models in both ACC and NMI evaluation indicators.
[0129] In summary, compared with the existing deep clustering models, the deep semi-supervised text clustering method based on pairwise constraints and cluster guidance disclosed in this disclosure has the following advantages:
[0130] The cluster allocation loss proposed in the deep semi-supervised text clustering method based on pairwise constraints and cluster guidance of the present invention can effectively solve the error correction of adaptive labels by considering the global information of labeled clusters and the label adaptation between unlabeled clusters, and avoid introducing incorrect hyper-supervisory information to affect clustering accuracy;
[0131] The deep semi-supervised text clustering method based on pairwise constraints and cluster guidance disclosed in the present invention integrates the dual supervision advantages of pairwise constraints and class labels, converts a small amount of labeled data into features of two dimensions: sample pairwise constraints and cluster information, which can fully utilize the supervised information and jointly optimize it with unsupervised sample features, thereby effectively overcoming the negative impact of sample overlap and significantly improving the text clustering effect.
[0132] The deep semi-supervised text clustering method based on pairwise constraints and cluster guidance disclosed in the present invention aims at the problems of high dimensionality, sparsity, and easy feature overlap of text, and proposes a joint framework for deep semi-supervised text clustering. From the sample perspective, the labeled data adopts pairwise constraint loss, which can realize the learning of sample features in one class or not in one class. From the cluster perspective, the labeled data adopts cluster form and the unlabeled clusters guide each other to learn, which can avoid the wrong allocation of clusters on the basis of pairwise constraint supervision. The two are unified in the original deep embedded clustering framework, which effectively improves the robustness of the model.
[0133] The deep semi-supervised text clustering method based on pairwise constraints and cluster guidance disclosed in the present invention does not make any requirements on the complexity of the feature extraction model. At the same time, the method is also easy to realize the clustering of various modal data such as images and audios, and the transplantation method is simple and easy.
[0134] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A deep semi-supervised text clustering method based on pairwise constraints and cluster guidance, It is characterized in that include: Preprocess and vectorize the target text data to obtain a multi-dimensional vector; Learning the hidden features of the target text data according to the multidimensional vector, and inputting the hidden features into a preset algorithm for clustering to obtain initial cluster centers; Calculating sample soft distribution and auxiliary distribution according to the hidden layer features and the initial cluster centers, and calculating clustering loss according to the sample soft distribution and the auxiliary target distribution; Constructing pairwise constraints according to labeled samples in the target text data, calculating a relationship matrix and a similarity matrix under the pairwise constraints, and generating pairwise constraint losses using cross entropy; According to the data labels, all the labeled samples are divided into a plurality of labeled clusters of different types, according to the real-time clustering results, the unlabeled samples in the target text data are divided into a plurality of unlabeled clusters, and the cluster allocation losses of all the labeled clusters and all the unlabeled clusters are calculated, wherein the number of the labeled clusters and the unlabeled clusters is the same as the number of categories of the data labels; A joint loss function is calculated according to the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster allocation loss, and a clustering result is obtained when the joint loss function is iterated to reach a preset condition.
2. The method according to claim 1, Features The step of preprocessing and vectorizing the target text data to obtain a multi-dimensional vector includes: Performing text enhancement on the target text data; Performing a traditional-simplified conversion operation, a word segmentation operation, and a stop word removal operation on the target text data after the text enhancement to obtain intermediate text data; After the intermediate text data is vectorized by using word embedding technology, the multi-dimensional vector is obtained through average pooling operation.
3. The method according to claim 1, Features The step of constructing pairwise constraints according to the labeled samples in the target text data, calculating the relationship matrix and the similarity matrix under the pairwise constraints, and generating the pairwise constraint loss using the cross entropy comprises: According to the labeled samples, the label information is converted into the pairwise constraints and the relationship matrix is constructed accordingly; Calculate the cosine angle between any two samples in the relationship matrix according to the hidden layer features to obtain the similarity matrix; The pairwise constraint loss is calculated using the cross entropy according to the relationship matrix and the similarity matrix.
4. The method according to claim 3, Features The steps of dividing all the labeled samples into a plurality of labeled clusters of different types according to the data labels, dividing the unlabeled samples in the target text data into a plurality of unlabeled clusters according to the real-time clustering results, and calculating the cluster allocation losses of all the labeled clusters and all the unlabeled clusters include: Converting the labeled samples into a plurality of labeled clusters according to the data labels; Dividing the unlabeled samples into a plurality of unlabeled clusters according to the real-time clustering result; Traversing all the labeled clusters and all the unlabeled clusters, calculating the probability that the clusters belong to the same category using average cosine similarity, and generating cluster soft assignment; Based on the cluster soft assignment, the label assignment results of all the labeled clusters and all the unlabeled clusters are obtained by using the Hungarian algorithm, and the target distribution is constructed based on them; The cluster allocation loss is calculated using KL divergence according to the cluster soft allocation and the target distribution.
5. The method according to claim 4, Features The step of calculating the joint loss function according to the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster allocation loss, and obtaining the clustering result when the joint loss function iteration reaches a preset condition, includes: Combining the reconstruction error, the clustering loss, the pairwise constraint loss and the cluster assignment loss to obtain the joint loss function; Iteratively optimizing the model parameters of the deep semi-supervised clustering model according to the joint loss function and the stochastic gradient descent algorithm, and generating corresponding iteration results in each iterative calculation; When the iteration reaches the preset condition, the last iteration result is used as the clustering result.
6. The method according to claim 5, Features ,The preset conditions include that the changes of clustering results for two consecutive times are less than any of the termination threshold or the maximum number of iterations.
7. The method according to claim 3, Features , the expression of the pairwise constraint loss is Among them, N 1 Represents the total number of labeled samples, A={a ij } is the relationship matrix, S = {s ij } is the similarity matrix, D={d ij } is the distance matrix transformed from the similarity matrix, where d ij =(1-s ij ) / 2.
8. The method according to claim 4, Features , the expression of cluster soft allocation is in, and Respectively represent the K labeled clusters and the unlabeled clusters, a 1 and a 2 Respectively represent the hidden layer feature representation of the data points in the labeled cluster and the unlabeled cluster, cos(a 1 ,a 2 ) represents the cosine similarity between the two, M ij It means that the loss contribution of each cluster is normalized according to the number of data points in each cluster, and its value is cluster L i and U j The product of the number of samples in .
9. The method according to claim 8, Features ,The expression of the target distribution is J=Hungary(-I), where J represents the cluster allocation result calculated based on the Hungarian algorithm.
10. The method according to claim 9, Features ,The expression of the cluster allocation loss is: It is used to reflect the difference between the average similarity between two clusters and the cluster allocation results, and then use the distribution difference between the two to continuously guide the update of the hidden features of all samples and cluster centers Among them, K is the number of clusters, N 2 Represents the total number of samples.
