An adaptive semi-supervised deep clustering method

Abstract

The application discloses a kind of self-adapting semi-supervised deep clustering method, belong to image data clustering analysis technical field, by using labeled data training model;The trained model is again trained to the unlabeled training data, and the classification features of the data are obtained, and its classification features are converted into probability value;The features obtained by labeled data and unlabeled training data in the last hidden layer of model are semi-supervised clustering to obtain the similarity between unlabeled training data and cluster center, and the probability value and the similarity corresponding label consistent are given its pseudo label;Self-adaptively select the data with high confidence from pseudo label data and add to labeled data to participate in the next model iteration training, until model converges, then training ends;Finally, new test data is clustered and tested.Compared with other clustering methods, the application can improve the accuracy of model clustering on different data sets.

Description

Technical Field

[0001] This invention belongs to the field of image data clustering analysis technology, specifically relating to an adaptive semi-supervised deep clustering method. Background Technology

[0002] Traditional clustering methods often separate feature extraction and clustering steps, making it difficult to achieve ideal clustering results for high-dimensional image datasets. Deep learning-based clustering algorithms, however, can use features automatically learned by deep learning models for cluster analysis.

[0003] Clustering can be categorized into unsupervised and semi-supervised clustering based on whether or not labeled data is used. For image data that is difficult to distinguish, unsupervised clustering without data labels, while simple, may lead to unsatisfactory clustering results. Therefore, semi-supervised clustering using prior data can significantly improve clustering performance. In some semi-supervised clustering methods, labeled data is often randomly selected from the original dataset along with their labels to guide clustering, which may not yield optimal results. The key to semi-supervised classification methods in semi-supervised learning lies in acquiring knowledge from unlabeled training data to guide classification; therefore, obtaining good supervisory information from unlabeled training data becomes crucial. However, few methods currently focus on simultaneously extracting reliable information from both labeled and unlabeled training data. Summary of the Invention

[0004] To address the problem of automatically extracting effective features from data, mining effective information from reliable labeled samples and labels, and acquiring useful knowledge from unlabeled training data to guide semi-supervised learning, this invention provides an adaptive semi-supervised deep clustering method.

[0005] To achieve the above objectives, the present invention employs the following technical solutions:

[0006] An adaptive semi-supervised deep clustering method includes the following steps:

[0007] Step 1: Use labeled data to train the neural network model;

[0008] Step 2: Use the neural network model trained in Step 1 to classify the unlabeled training data, obtain the probability value of the category to which each data belongs, and obtain the features of each unlabeled training data.

[0009] Then, the features extracted from the last hidden layer of the neural network model for both labeled and unlabeled training data are subjected to constrained seed K-Means clustering to obtain the Pearson similarity between each unlabeled training data and the cluster center.

[0010] Then, assign pseudo-labels to the unlabeled training data whose labels match the maximum similarity and maximum probability values ​​in each data point; the purpose is to maximize the accuracy of the pseudo-labels, otherwise there may be cases where the pseudo-labels are wrong, but the data is selected because the calculated confidence level is high.

[0011] Step 3: Calculate the confidence level of the pseudo-label data;

[0012] Step 4: Use an adaptive method to select data with high confidence in the pseudo-label data;

[0013] Step 5: Take the data with high confidence in the pseudo-labels from Step 4, along with their pseudo-labels, and use them together with the existing label data as input to the next neural network model.

[0014] Step 6: Iterate through steps 1 to 5. If the difference between the optimal thresholds obtained in two adjacent iterations is less than the set threshold, continue iterating until the convergence condition is met, then stop the iteration loop.

[0015] Step 7: Input the label data obtained in the above steps and the new test data into the neural network model, and perform constrained seed K-Means clustering on the features after dimensionality reduction of the model to obtain the clustering results of the new test data.

[0016] Furthermore, the specific method for using labeled data to train the neural network model in step 1 is as follows: a deep learning neural network model is trained using n labeled data points, represented as:

[0017]

[0018] Among them, D L The label represents the dataset; t represents the number of data points in each class of the labeled data; K represents the number of data categories; x represents the labeled data; the subscript L represents the labeled data and set; the model can be built according to the data dimensions and characteristics, including but not limited to convolutional neural networks, multilayer perceptrons, and graph neural networks.

[0019] D L The corresponding label is Y = {y Li The model applies to data D. L The output is F = {f Li}, i∈{1,2,…,n}, where f Li Representing data x i After inputting the model, the dimensionality-reduced feature of the last layer of the fully connected layer is obtained, and the size of this feature is 1×K; the formula of the model is:

[0020] f Li =h W，b (xi )

[0021] Where W and b are the parameters of the model, h W，n (·) denotes the model function; the loss function used to fine-tune the network model parameters is the cross-entropy loss function, expressed as:

[0022]

[0023] Furthermore, in step 2, the neural network model trained in step 1 is used to classify the unlabeled training data to obtain the probability value of the category to which each data point belongs. The specific method for obtaining the features of each unlabeled training data point is as follows:

[0024] The neural network model trained in step 1 was used on the unlabeled training data D. U ={x U1 x U2 , ..., x Um The data is classified to obtain the dimensionality reduction features f for each data point. Uj The dimensionality-reduced features are transformed into probability values ​​of their respective categories using a normalization method. j∈{1, 2, ..., m}, and find the maximum value among them. Where m represents the number of unlabeled training data.

[0025] Furthermore, in step 2, the specific method for performing constrained seed K-Means clustering on the features extracted from the last hidden layer of the neural network model for both labeled and unlabeled training data to obtain the Pearson similarity between each unlabeled training data point and its cluster center is as follows: The features extracted from the last hidden layer of the model for both labeled and unlabeled training data are input into the constrained seed K-Means clustering, with labeled data used as the indicator for clustering; the similarity of each data point is obtained by calculating the Pearson correlation coefficient between each unlabeled training data point and each cluster center. j∈{1, 2, ..., m}, and find the maximum value among them.

[0026] Furthermore, the specific method for assigning pseudo-labels to unlabeled training data whose labels match the maximum similarity and maximum probability values ​​in each data set in step 2 is as follows: Finally, compare each unlabeled training data set... and We determine whether r and r′ are equal, and assign pseudo-labels to unlabeled training data with equal values.

[0027] Furthermore, the specific method for calculating the confidence level of the pseudo-label data in step 3 is as follows: based on the similarity and probability values ​​of the pseudo-label data, calculate the Euclidean distance between these two values, and use the distance value to reflect the non-confidence value of each pseudo-label data.

[0028]

[0029] Among them, C Uj This represents the non-confidence level; the smaller the value, the higher the confidence level.

[0030] Furthermore, the specific method for selecting data with high confidence levels in pseudo-labeled data using an adaptive approach in step 4 is as follows: First, iterate through all the unconfidence values ​​of the unlabeled training data to obtain the inter-class variance at the corresponding threshold. After one iteration, obtain the threshold T corresponding to the maximum inter-class variance. max Next, select one less than T. max Collect all unlabeled training data corresponding to unconfidence values ​​and count the number of unlabeled training data in each class; if there is a class with more than 10 data, then follow the above method to collect data less than T. max The inter-class variance is calculated by iterating through all the unconfidence values ​​until the number of unlabeled training data in each class is within the range of [0, 10]. Finally, a new optimal threshold T is obtained. max Those less than T max The unlabeled training data corresponding to all non-confidence values ​​are selected pseudo-label data with high confidence.

[0031] Furthermore, an optimal threshold is adaptively and iteratively determined to select unlabeled training data with high confidence. The unconfidence values ​​of all unlabeled training data are iterated over, and in each iteration, a single unconfidence value is selected as the current threshold T. This threshold divides the unconfidence values ​​of the data into two groups, and the inter-class variance between these two groups is calculated separately. The specific formula for calculating the inter-class variance is as follows:

[0032] g = w 0 ×w 1 ×(u 0 -u 1 ) 2

[0033] Among them, w 0 This represents the probability that the number of non-confidence values ​​greater than the threshold T is relative to the total number of non-confidence values; w 1 This represents the probability that the number of non-confidence values ​​less than the threshold T is relative to the total number of non-confidence values; u 0 This represents the average of all non-confidence values ​​greater than the threshold T; u 1 This represents the average of all non-confidence values ​​less than the threshold T; after one iteration, the maximum inter-class variance g is selected.max The corresponding threshold T max This is the optimal threshold obtained in this iteration; then, a threshold less than T is selected. max Collect all unlabeled training data corresponding to unconfidence values ​​and count the number of unlabeled training data in each class; if there is a class with more than 10 data points, then collect data in classes less than T. max The inter-class variance is calculated by iterating through all the unconfidence values ​​until the number of unlabeled training data in each class is within the range of [0, 10]. Finally, a new optimal threshold T is obtained. max Less than T′ max All unlabeled training data corresponding to non-confidence values ​​are selected pseudo-labeled data with high confidence. Finally, the pseudo-labeled data with high confidence are stored in D. P In, it is represented as:

[0034]

[0035] Among them, D P This represents a pseudo-label dataset with high confidence; k represents the number of pseudo-label data points of each class with high confidence selected; and the superscript of x represents the label of that data point.

[0036] Step 4 uses an adaptive method to select pseudo-label data with high confidence levels, ensuring that the number of high-confidence pseudo-label data in each category is the same. This is to guarantee that the experimental results are not affected by differences in the number of data points.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. This invention proposes a novel adaptive semi-supervised deep clustering method that effectively utilizes information from all labeled data and also fully leverages some information from unlabeled training data. It performs semi-supervised clustering by adaptively selecting high-confidence data from the abundant unlabeled training data.

[0039] 2. The method of the present invention can automatically measure the confidence level of pseudo-label data. By obtaining the confidence level of pseudo-label data, high-confidence pseudo-label data can be selected for model training.

[0040] 3. The present invention was tested on a common image dataset and achieved better clustering results than K-Means clustering, constrained seed K-Means clustering and unsupervised deep clustering. Attached Figure Description

[0041] Figure 1 This is a flowchart of the adaptive semi-supervised deep clustering method of the present invention;

[0042] Figure 2 This is a schematic diagram illustrating the analysis and testing results of FashionMNIST data. Detailed Implementation

[0043] Example 1

[0044] An adaptive semi-supervised deep clustering method includes the following steps:

[0045] Step 1: Use labeled data to train the neural network model;

[0046] Training a deep learning neural network model using n labeled data points can be represented as:

[0047]

[0048] Among them, D L The label represents the dataset; t represents the number of data points in each class of the labeled data; K represents the number of data categories; x represents the labeled data; the subscript L represents the labeled data and the set; the model can be built according to the data dimensions and characteristics, including but not limited to convolutional neural networks, multilayer perceptrons, and graph neural networks.

[0049] D L The corresponding label is Y = {y Li The model applies to data D. L The output is F = {f Li}, i∈{1,2,…,n}, where f Li Representing data x i After inputting the model, the dimensionality-reduced feature of the last layer of the fully connected layer is obtained, and the size of this feature is 1×K; the formula of the model is:

[0050] f Li =h W,b (x i )

[0051] Where W and b are the parameters of the model, h W，n (·) denotes the model function; the loss function used to fine-tune the network model parameters is the cross-entropy loss function, expressed as:

[0052]

[0053] Step 2: Use the neural network model trained in Step 1 to classify the unlabeled training data, obtain the probability value of each data category, and obtain the features of each unlabeled training data. Then, perform constrained seed K-Means clustering on the features extracted from the last hidden layer of the neural network model for both labeled and unlabeled training data, and obtain the Pearson similarity between each unlabeled training data and the cluster center. Finally, assign pseudo-labels to the unlabeled training data with the same label as the maximum similarity and the maximum probability value in each data.

[0054] The neural network model trained in step 1 was used on the unlabeled training data D. U ={x U1 x U2 , ..., x Um The data is classified to obtain the dimensionality reduction features f for each data point. Uj The dimensionality-reduced features are transformed into probability values ​​of their respective categories using a normalization method. j∈{1, 2, ..., m}, and find the maximum value among them. Where m represents the number of unlabeled training data.

[0055] Features extracted from both labeled and unlabeled training data through the last hidden layer of the model are input into the constraint seed K-Means clustering, with labeled data used as the indicator for clustering. The similarity of each data point is obtained by calculating the Pearson correlation coefficient between each unlabeled training data point and each cluster center. And find the maximum value among them.

[0056] Finally, compare each unlabeled training data set. and We determine whether r and r′ are equal, and assign pseudo-labels to unlabeled training data with equal values.

[0057] Step 3: Calculate the confidence level of the pseudo-label data;

[0058] Based on the similarity and probability values ​​of the pseudo-label data, calculate the Euclidean distance between these two values, and use the distance value to reflect the non-confidence value of each pseudo-label data.

[0059]

[0060] Among them, C Uj This represents the non-confidence level; the smaller the value, the higher the confidence level.

[0061] Step 4: Use an adaptive method to select data with high confidence in the pseudo-label data;

[0062] An optimal threshold is adaptively and iteratively determined to select unlabeled training data with high confidence. The unconfidence values ​​of all unlabeled training data are iterated over, and in each iteration, a single unconfidence value is selected as the current threshold T. This threshold divides the unconfidence values ​​of the data into two groups, and the inter-class variance between these two groups is calculated separately. The specific formula for calculating the inter-class variance is as follows:

[0063] g = w 0 ×w 1 ×(u 0 -u 1 ) 2

[0064] Where w0 represents the probability that the number of non-confidence values ​​greater than the threshold T is out of the total number of non-confidence values; w 1 This represents the probability that the number of non-confidence values ​​less than the threshold T is relative to the total number of non-confidence values; u 0 This represents the average of all non-confidence values ​​greater than the threshold T; u 1 This represents the average of all non-confidence values ​​less than the threshold T; after one iteration, the maximum inter-class variance g is selected. max The corresponding threshold T max This is the optimal threshold obtained in this iteration; then, a threshold less than T is selected. max Collect all unlabeled training data corresponding to unconfidence values ​​and count the number of unlabeled training data in each class; if there is a class with more than 10 data points, then collect data in classes less than T. max The inter-class variance is calculated by iterating through all the unconfidence values ​​until the number of unlabeled training data in each class is within the range of [0, 10]. Finally, a new optimal threshold T is obtained. max Less than T′ max All unlabeled training data corresponding to non-confidence values ​​are selected pseudo-labeled data with high confidence. Finally, the pseudo-labeled data with high confidence are stored in D. P In, it is represented as:

[0065]

[0066] Among them, D P This represents a pseudo-label dataset with high confidence; k represents the number of pseudo-label data points of each class with high confidence selected; and the superscript of x represents the label of that data point.

[0067] Step 4 uses an adaptive method to select pseudo-label data with high confidence levels, ensuring that the number of high-confidence pseudo-label data in each category is the same. This is to guarantee that the experimental results are not affected by differences in the number of data points.

[0068] Step 5: Take the data with high confidence in the pseudo-labels from Step 4, along with their pseudo-labels, and use them together with the existing labeled data as input to the next neural network model; take the D selected in Step 4... P The corresponding pseudo-labels are considered as data that can be used for model training and are added to D. L In the middle. D P From unlabeled training data D U Remove from the middle;

[0069] Step 6: Iterate through steps 1 to 5. If the optimal threshold T′ is obtained in two consecutive iterations... max If the difference is less than the set threshold, the iteration continues until the convergence condition is met, then the iteration loop stops.

[0070] Step 7: Based on the trained model, apply the new test data D T ={x T1 x T2 , ..., x Tt Clustering is performed, where t represents the number of new test data; the final D obtained from the above steps is then used for clustering. L Data and D T The data is input into the constructed neural network model, and the features extracted from the last hidden layer of the model are subjected to constrained seed K-Means clustering to obtain the clustering results of the new test data.

[0071] Specifically, this paper analyzes and tests two types of labels on three datasets: FashionMNIST, MNIST, and USPS, and evaluates the performance of the method of this invention using clustering accuracy and three other clustering evaluation metrics. Figure 2 The diagram illustrating the analysis and testing results of the FashionMNIST data shows the number of pseudo-labels with high confidence selected in each iteration of the model for both datasets with labels of 0 and 1, as well as the optimal threshold obtained in each iteration.

[0072] Table 1 Comparison of different clustering methods on the FashionMNIST dataset

[0073]

[0074] As can be seen from the clustering evaluation metrics in Table 1, this invention outperforms other clustering methods under all three evaluation metrics. For the FashionMNIST data, the accuracy of this invention is significantly higher than other clustering methods, improving by approximately 10%.

[0075] In summary, the adaptive semi-supervised deep clustering method of the present invention has been shown to outperform other clustering methods.

[0076] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. An adaptive semi-supervised deep clustering method, characterized in that: Includes the following steps: Step 1: Use labeled data to train the neural network model; Step 2: Use the neural network model trained in Step 1 to classify the unlabeled training data, obtain the probability value of the category to which each data belongs, and obtain the features of each unlabeled training data. Then, the features extracted from the last hidden layer of the neural network model for both labeled and unlabeled training data are subjected to constrained seed K-Means clustering to obtain the Pearson similarity between each unlabeled training data and the cluster center. Then, assign pseudo-labels to the unlabeled training data whose labels match the maximum similarity and maximum probability values ​​in each data point; Step 3: Calculate the confidence level of the pseudo-label data; Step 4: Use an adaptive method to select data with high confidence in the pseudo-label data; Step 5: Take the data with high confidence in the pseudo-labels from Step 4, along with their pseudo-labels, and use them together with the existing label data as input to the next neural network model. Step 6: Iterate through steps 1 to 5. If the difference between the optimal thresholds obtained in two adjacent iterations is less than the set threshold, continue iterating until the convergence condition is met, then stop the iteration loop. Step 7: Input the labeled data and new test data obtained in the above steps into the neural network model, and perform constrained seed K-Means clustering on the features after dimensionality reduction of the model to obtain the clustering results of the new test data. The labeled data consists of image data and its corresponding labels, while the unlabeled training data consists of image data.

2. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: The specific method for step 1, which uses labeled data to train a neural network model, is as follows: A deep learning neural network model is trained using n labeled data points, as shown below: ； Among them, D L This represents a labeled dataset; t represents the number of data points in each class of the labeled data; K represents the number of data categories; x represents the labeled data; and the subscript L represents the labeled data and set. D L The corresponding label is Y={y Li The model applies to data D. L The output is F={ f Li }, i∈{1,2,…,n}, Where f Li Representing data x i After inputting the model, the dimensionality-reduced feature of the last layer of the fully connected layer is obtained, and the size of this feature is 1×K; the formula of the model is: f Li = h W,b (x i ); Where W and b are the parameters of the model, h W,b (·) denotes the model function; the loss function used to fine-tune the network model parameters is the cross-entropy loss function, expressed as: 。 3. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: In step 2, the neural network model trained in step 1 is used to classify the unlabeled training data to obtain the probability value of the category to which each data point belongs. The specific method for obtaining the features of each unlabeled training data point is as follows: Use the neural network model trained in step 1 to process the unlabeled training data. Classify the data to obtain the dimensionality reduction features f for each data point. Uj The dimensionality-reduced features are transformed into probability values ​​of their respective categories using a normalization method. , And find the maximum value among them. Where m represents the number of unlabeled training data.

4. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: In step 2, the specific method for performing constraint seed K-Means clustering on the features extracted from the last hidden layer of the neural network model for labeled and unlabeled training data to obtain the Pearson similarity between each unlabeled training data and the cluster center is as follows: input the features extracted from the last hidden layer of the model for labeled and unlabeled training data into the constraint seed K-Means clustering, and perform clustering with labeled data as the indicator information. The similarity of each data point is obtained by calculating the Pearson correlation coefficient between each unlabeled training data point and each cluster center. , And find the maximum value among them. .

5. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: The specific method for assigning pseudo-labels to unlabeled training data whose labels match the maximum similarity and maximum probability values ​​in each data set in step 2 is as follows: Finally, compare each unlabeled training data set... and In and If the values ​​are equal, assign pseudo-labels to the unlabeled training data with equal values.

6. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: The specific method for calculating the confidence level of the pseudo-label data in step 3 is as follows: based on the similarity and probability values ​​of the pseudo-label data, calculate the Euclidean distance between these two values, and use the distance value to reflect the non-confidence value of each pseudo-label data. ； Among them, C Uj This represents the non-confidence level; the smaller the value, the higher the confidence level. Represents the first in the unlabeled dataset The sample is divided into the first... The probability of a class; Represents the first in the unlabeled dataset The sample and the first Pearson correlation coefficient of cluster centers.

7. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: The specific method for selecting data with high confidence levels in pseudo-labeled data using an adaptive approach in step 4 is as follows: First, iterate through all the non-confidence values ​​of the unlabeled training data to obtain the inter-class variance at the corresponding threshold. After one iteration, obtain the threshold corresponding to the maximum inter-class variance. Next, select less than Collect all unlabeled training data corresponding to non-confidence values ​​and count the number of unlabeled training data in each class; If there exists a data type with more than 10 elements, then follow the above method to further refine the data for data smaller than 10. The inter-class variance is calculated by iterating through all the unconfidence values ​​until the number of unlabeled training data in each class is within the range of [0, 10], at which point the calculation stops; finally, a new optimal threshold is obtained. Those smaller than The unlabeled training data corresponding to all non-confidence values ​​are selected pseudo-label data with high confidence.

8. The adaptive semi-supervised deep clustering method according to claim 7, characterized in that: An optimal threshold is adaptively and iteratively determined to select unlabeled training data with high confidence. The unconfidence values ​​of all unlabeled training data are iterated over, and in each iteration, a single unconfidence value is selected as the current threshold T. This threshold divides the unconfidence values ​​of the data into two groups, and the inter-class variance between these two groups is calculated separately. The specific formula for calculating the inter-class variance is as follows: ； in, This represents the probability that the number of non-confidence values ​​greater than the threshold T is relative to the total number of non-confidence values. This represents the probability that the number of non-confidence values ​​less than the threshold T is relative to the total number of non-confidence values. This represents the average of all non-confidence values ​​greater than the threshold T; This represents the average of all non-confidence values ​​less than the threshold T; after one iteration, the maximum inter-class variance is selected. Corresponding threshold This is the optimal threshold obtained in this iteration; then, a threshold smaller than [the threshold value] is selected. Collect all unlabeled training data corresponding to unconfidence values ​​and count the number of unlabeled training data in each class; if there is a class with more than 10 data points, then collect the data in the class with less than 10 data points. The inter-class variance is calculated by iterating through all the unconfidence values ​​until the number of unlabeled training data in each class is within the range of [0, 10], at which point the calculation stops; finally, a new optimal threshold is obtained. less than All unlabeled training data corresponding to non-confidence values ​​are selected pseudo-labeled data with high confidence. Finally, the pseudo-labeled data with high confidence are stored in D. P In, it is represented as: ； Among them, D P This represents a pseudo-label dataset with high confidence, where k represents the number of pseudo-label data points of each class selected with high confidence, and the superscript of x represents the label of that data point.

9. The adaptive semi-supervised deep clustering method according to claim 1, characterized in that: Step 4 uses an adaptive method to select pseudo-label data with high pseudo-label confidence, requiring that the number of pseudo-label data with high confidence in each category be the same.

Images