A pseudo-supervised clustering method based on meta-features

Abstract

The present invention belongs to the field of deep learning technology, and specifically relates to a pseudo-supervised clustering method based on meta-features, comprising the following steps: inputting a data set into a deep model for pre-training; inputting the original image and the image after data enhancement into the pre-training model for instance-level feature extraction and category prediction, respectively; using a classifier to predict category samples, and selecting high-confidence samples for constructing meta-features containing stable semantic information; pseudo-label construction and global pseudo-label assignment; pseudo-label optimization of the deep model, using cross-entropy loss with label smoothing to guide and optimize the selection of meta-features and the assignment of pseudo-labels. The present invention provides a stable clustering supervision signal while effectively ensuring feature semantics, and completes a direct mapping from stable samples to semantic labels by globally assigning pseudo-labels, effectively improving the classification accuracy of the deep model and realizing semantic self-learning of stable features.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and relates to an implementation of applying meta-knowledge in cognitive knowledge to deep clustering in the field of deep learning, and specifically to a pseudo-supervised clustering method based on meta-features. Background Art

[0002] Clustering is one of the most fundamental and important data analysis methods. Implementing fast and intelligent cluster analysis of massive amounts of data is crucial for organizing, summarizing, and storing them. With the rapid development of big data and artificial intelligence, the demand for intelligent analysis and processing of massive amounts of unlabeled data is increasing. Combining traditional clustering with deep learning has become a current research hotspot, an approach we call deep clustering. Deep clustering aims to leverage the powerful representation learning capabilities of deep neural networks to improve the feature quality of data, thereby achieving effective clustering results on complex samples without human annotation.

[0003] Deep clustering generally involves two steps: extracting features using deep neural networks and clustering using these features. While traditional deep clustering methods utilize deep neural networks to automatically extract data features, these methods tend to focus on low-level features that humans consider non-semantic. This is particularly true in industrial or medical fields, where the targets to be identified are often complex and unevenly distributed, and their patterns vary significantly. Deep clustering lacks stable label information as a supervisory signal. Using inaccurate clustering results as supervisory signals to guide neural network training risks misleading the network. Using misleading network-extracted features for clustering further deteriorates the results, making it difficult to achieve ideal clustering results.

[0004] Current deep clustering methods are primarily based on cluster assignment that maximizes mutual information. However, these methods rely heavily on network initialization and are likely to lock onto non-semantic, low-level features. These methods fail to fully consider the model's stability during cognitive decision-making, making them unsuitable for practical applications. Deep clustering is crucial for discovering features with stable semantics, so selecting these features as supervisory signals for clustering becomes a core issue in deep clustering. Summary of the Invention

[0005] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art. A pseudo-supervised clustering method based on meta-features is proposed. The deep clustering model is trained by comprehensively considering the stable semantic features extracted by the model and the stable supervision signals in the clustering process. This can effectively improve the clustering semantics and greatly improve the classification accuracy. This method introduces the stable semantics in human cognition of things into the deep learning model, which is of great significance.

[0006] The technical solution of the present invention is:

[0007] A pseudo-supervised clustering method based on meta-features, comprising the following steps:

[0008] S1: Input the dataset into the deep model for pre-training;

[0009] S2: The original image and the data-augmented image are input into the pre-trained model for instance-level feature extraction and category prediction respectively. The classifier is used to predict the category samples, and the sample features with higher confidence are selected. The high-confidence samples are selected to construct the meta-feature f containing stable semantic information. Meta ;

[0010] Select the K samples with the highest probability in each category as the confident samples C k , the weighted average of the confident samples is used as the meta-feature f Meta ,

[0011] S3: Pseudo-label construction and global pseudo-label assignment; that is, after obtaining the meta-feature, find the semantic nearest neighbor sample for the meta-feature in the global scope, and assign the same pseudo-label to the meta-feature and the corresponding semantic nearest neighbor. k ={N(f Meta )|k∈[1,2…,S]}, where N(·) represents the global nearest neighbor of the sample, l k Pseudo labels assigned to the meta-feature neighbors;

[0012] S4: Pseudo-label optimization deep model, using cross-entropy loss with label smoothing to guide and optimize the selection of meta-features and the assignment of pseudo-labels;

[0013] The label smoothing method improves the category pseudo-label y by adding uniform noise, given the corresponding labels, injecting noise into all categories as follows:

[0014]

[0015] Among them, C is the number of categories, ε~Uniform(0,1) is the noise, y is the category pseudo label, is the category pseudo label after adding uniform noise;

[0016] Use the category pseudo-label after adding uniform noise And the predicted labels of the randomly enhanced samples are used to calculate the cross entropy, which is Among them, χ is the data set, p i It is obtained by applying the logits vector z output by the penultimate layer of the model to the function calculation. Z jThe purpose of using cross entropy loss is to shorten the distance between pseudo labels and label prediction probabilities, so that the model can learn good category semantic information.

[0017] The above-mentioned pseudo-supervised clustering method based on meta-features solves three problems: first, it solves the error accumulation problem of misclassified labels caused by finding the nearest neighbor to train the classifier for each instance in the traditional instance-based image clustering algorithm; second, it uses direct label mapping to solve the suboptimal solution problem caused by multi-layer feature optimization; finally, it enhances the semantic robustness of the assigned labels and improves the clustering accuracy by globally assigning labels through meta-features.

[0018] Furthermore, the deep model in step S1 is a contrastive learning model MoCo, and the data set is input into the contrastive learning model MoCo for pre-training, and the training data set is an image data set.

[0019] Furthermore, in step S1, after the pre-trained deep model converges, the training is stopped and the parameters are frozen for use in subsequent deep models for instance-level feature extraction.

[0020] Furthermore, the meta-feature is formed by weighted averaging of the samples with the highest confidence. The meta-feature is the feature with the highest probability and the most representative category in the corresponding category, which effectively ensures the stable semantic attributes of the category and can effectively represent the semantic information of the category.

[0021] Furthermore, in step S2, the original image is input into the pre-trained deep model to obtain instance-level features; after the enhanced image passes through the pre-trained deep model, the features are input into the classifier to predict the sample probability, and the predicted probability of the enhanced image is output. The sample features with high confidence are selected and weighted averaged as meta-features.

[0022] Furthermore, the enhanced image R(x) is input into the pre-trained deep model to obtain f(R(x)), which is then passed through a classifier Φ consisting of a two-layer multi-layer perceptron. c , the classifier outputs the enhanced image prediction probability P k ,in,

[0023] P k =Φ c (f(R(x))),

[0024] C k =topK(P ki , f(x)); where P kiIt refers to the predicted probability of the i-th category, which is obtained by passing the features obtained by the pre-trained model through a softmax (i.e., normalized exponential function) function; f(x) is the feature obtained after the original image is input into the pre-trained model; topK() is to select the K features with the highest probability in the corresponding category; C k To be a confident sample, we sort the feature probabilities of the corresponding categories from high to low and select the top K with the highest probability as C k .

[0025] Furthermore, in step S4, a cross entropy loss function with label smoothing is used to effectively prevent the model from being overconfident in its predictions. Through continuous iterative operation, the classifier predictions become more and more accurate, ultimately achieving a good clustering effect.

[0026] Beneficial effects of the present invention:

[0027] (1) The present invention proposes to construct stable features, namely meta-features, in the clustering process, and uses the meta-features as supervisory signals to assign pseudo-labels to train classifiers to continuously improve the recognition accuracy of the classification model, so that the model can gradually adapt to the pattern changes caused by complex targets and environmental changes; and according to the characteristics of meta-features, a new pseudo-supervised clustering method based on meta-features is proposed, thereby providing stable clustering supervision signals while effectively ensuring the semantics of features, and completing the direct mapping from stable samples to semantic labels by assigning pseudo-labels on a global scale, effectively improving the classification accuracy of the deep model, and realizing semantic self-learning of stable features.

[0028] (2) The present invention fully considers the inspiration of the human cognitive process using stable experience to guide practice. By comprehensively considering the stable discriminant features in the classification process and applying stable features as supervisory signals, a pseudo-supervised clustering method based on meta-features is proposed, which combines the stable information in human cognition with the deep learning network, which has profound significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the principle of the pseudo-supervised clustering method based on meta-features provided by the present invention;

[0030] Figure 2 Confusion matrices for the CIFAR-10, CIFAR-100, and STL-10 datasets used in the embodiments of the present invention;

[0031] Figure 3 This is the top-3 category map obtained in the STL-10 experiment of the embodiment of the present invention;

[0032] Figure 4 This is a heat map visualization of the main objects on STL-10 according to an embodiment of the present invention;

[0033] Figure 5 A comparison chart of experimental results using different loss methods on three image datasets: CIFAR-10, CIFAR-100, and STL-10.

[0034] Figure 6 This is a comparison chart of the experimental results of three methods in the CIFAR-10 database according to an embodiment of the present invention;

[0035] Figure 7 This is a comparison chart of the experimental results of three methods in the CIFAR-100 database according to an embodiment of the present invention;

[0036] Figure 8 1 is a comparison chart of the experimental results of the three methods in the STL-10 database according to the embodiment of the present invention. DETAILED DESCRIPTION

[0037] To further understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0038] The present invention provides a pseudo-supervised clustering method based on meta-features, comprising:

[0039] A pre-trained deep model is used to extract instance-level features. The input consists of two parts: the original image and the image after data augmentation. The original image is input into the pre-trained model to obtain instance features. After the enhanced image passes through the pre-trained deep model, the features are input into the classifier for predicting sample probabilities. In the clustering stage, samples with higher confidence in the first stage are selected and processed to construct meta-features with stable semantic information. Meta-features are formed by weighted averaging of high-confidence samples and can effectively represent the semantic information of the category. After selecting the meta-features, the semantic nearest neighbors are searched for the meta-features globally, and the same pseudo-labels are assigned to the meta-features and the corresponding semantic nearest neighbors. The cross-entropy loss with label smoothing is used to approximate the features and pseudo-labels to complete the mapping from features to labels of the deep model. This process is repeated to gradually improve the semantic properties and clustering accuracy of the meta-features and pseudo-labels, thereby adapting to the pattern changes brought about by complex targets and environments. This method introduces stable semantics in human cognition of things into deep learning models, which is of great significance.

[0040] Example 1

[0041] like Figure 1As shown in FIG, this embodiment provides a pseudo-supervised clustering method based on meta-features. First, the instance-level contrastive learning model MoCo is used for pre-training. After the parameters are frozen, the instance-level features are extracted by the pre-trained instance-level contrastive learning model. The classifier is used to predict the category samples, and the features of the high-confidence samples are selected and weighted averaged as the meta-features. Figure 3 The top three high-confidence samples of each category involved in constructing meta-features are given in . These samples are semantically robust, which provides a benchmark for the correct assignment of semantic pseudo-labels. At the same time, in order to verify whether the features extracted by the pre-trained deep model pay attention to the correct semantics, such as Figure 4 As shown in the figure, this embodiment visualizes the class activation map of the top three high-confidence samples involved in constructing the meta-feature. As can be seen from the image, the heat of the class activation map focuses on the semantic category, which also ensures the stability of the classification semantics. Then, pseudo labels are assigned to the meta-features in a global range of a batch, and the meta-features and pseudo labels are further optimized through a cross-entropy loss with label smoothing, continuously improving the semantic accuracy of the assigned labels and effectively avoiding the model from being overconfident in predictions, such as Figure 5 As shown in the experimental comparison between the cross entropy loss function with label smoothing and the cross entropy loss function alone, on the three data sets, the experimental results of the cross entropy loss function with label smoothing proposed in the present invention are better than those of the direct use of cross entropy loss. Finally, through continuous iterative operation, the classifier prediction becomes more and more accurate, and finally achieves a good clustering effect. The confusion matrix diagram of the clustering effect of the present invention on CIFAR-10, CIFAR-100 and STL-10 is shown in Figure 2 As shown in the confusion matrix, the diagonal color of the confusion matrix is ​​dark, which proves that the method has accurately classified. Figure 6 、 Figure 7 and Figure 8 The comparison between the proposed method and the category semantics-based method Contrasive Clustering (hereinafter referred to as CC) and the instance feature-based pseudo-label assignment method (hereinafter referred to as SCAN) is given in the paper. These experiments all prove that the pseudo-supervised clustering method based on meta-features proposed in this paper has significant advantages over the above-mentioned CC and SCAN.

[0042] Specifically, the steps of the pseudo-supervised clustering method based on meta-features are as follows:

[0043] A pseudo-supervised clustering method based on meta-features, comprising the following steps:

[0044] S1: The dataset is input into the contrastive learning model MoCo for pre-training; the training datasets are all image datasets; when the pre-trained deep model converges, the training is stopped and the parameters are frozen for instance-level feature extraction by subsequent deep models.

[0045] This embodiment addresses the issue of some existing end-to-end deep clustering methods being sensitive to model initialization. By pre-training an unsupervised learning model to acquire instance-level semantic information, the pre-trained model is used as the feature extraction model. Considering the significant impact of the number of sample pairs on contrastive learning, the work of MoCo is adopted to ensure sufficient negative examples to ensure that the model learns instance-level semantic features. MoCo proposes replacing the memory with a momentum encoder, which maintains a dynamic update sequence q to store features from previous iterations.

[0046] Assume there is a sample s and its augmented version features u and u', for MoCo work, it has the following form:

[0047]

[0048] Here, sim() is a similarity function similar to cosine similarity, τ is an adjustable temperature parameter, and M is the queue size. Clearly, this loss has one positive sample and M negative samples. Since the queue q does not require a gradient, the size of M can be set to be large, which is beneficial for learning features from the dataset.

[0049] S2: The goal is to divide the dataset χ into k predefined categories without using any manual annotation labels, k∈[1, 2…, S]. Specifically, the pre-trained contrastive learning model is used as the feature extraction model f(·). The input part consists of two parts. One part is the original data x (i.e., the original image). After this part of the data is input into the contrastive learning model MoCo, the pre-trained model, when the model converges, freezes the parameters and extracts the instance-level features e=f(x) of the model; in the work, the original image is input into the contrastive learning model MoCo and then randomly enhanced. Then, the convolutional neural model and contrast loss are used to force the two enhancements to produce consistent predictions, ensuring that the model learns good feature representations. The other part is the randomly enhanced image R(x). After this part of the image is input into the model, f(R(x)) is obtained. After passing through the classifier Φ composed of two layers of MLP c , the classifier outputs the enhanced image prediction probability P k ,

[0050] P k =Φ c (f(R(x))),

[0051] C k =topK(Pki , f(x)),

[0052] Among them, P ki Refers to the predicted probability of the i-th category; f(x) is the feature obtained after the original image is input into the pre-training model; topK() is to select the K features with the highest probability in the corresponding category; C k The feature probabilities of the corresponding categories are arranged from high to low, and the K samples with the highest probability in each category are selected as the confident samples C according to the cosine similarity metric. k , the weighted average of the confident samples is used as the meta-feature f Meta ,

[0053]

[0054] The meta-feature is formed by weighted averaging of the samples with the highest confidence. It is the feature with the highest probability and the most representative category in the corresponding category. It effectively ensures the stable semantic properties of the category and can effectively represent the semantic information of the category.

[0055] S3: Pseudo-label construction and global pseudo-label assignment; that is, after obtaining the meta-feature, search for neighboring samples for the meta-feature globally within each batch and assign the same pseudo-label to its neighboring samples. k ={N(f Meta )|k∈[1,2…,s]}, where N(·) represents the global nearest neighbor of the sample, l k Pseudo labels assigned to the meta-feature neighbors.

[0056] S4: Pseudo-label optimization deep model uses cross-entropy loss with label smoothing to guide and optimize the selection of meta-features and the assignment of pseudo-labels. The cross-entropy loss function with label smoothing is used to effectively prevent the model from being overconfident in its predictions. Through continuous iterative operation, the classifier predictions become more and more accurate, ultimately achieving good clustering effects.

[0057] Sample classification is essentially a process of distinguishing samples by assigning labels. Therefore, in this embodiment, a label optimization solution is adopted. Specifically, the label optimization solution mainly consists of the following two parts: a pseudo-label optimization model and a label smoothing loss.

[0058] 1) Pseudo-label optimization model

[0059] The idea of ​​the pseudo-label optimization model is to train the classifier by establishing a mapping relationship between embedded features and pseudo-labels. Unlike existing methods that only mine instance-level semantic similarities or only focus on clustering, the present invention proposes pseudo-supervised classification based on meta-features, which not only retains the discriminant information at the instance level but also takes into account the semantic inconsistency problem that may be caused by the distance metric at the classification boundary. Meta-feature selection, pseudo-label construction and classifier optimization is a dynamic optimization process. Specifically, the obtained pseudo-labels and classifiers are optimized through soft cross-entropy loss. The optimized classifier continues to participate in the selection of confident samples and meta-feature construction in the next round, minimizing the cross-entropy loss between pseudo-labels and classifier predictions. Through multiple iterative optimizations, the classifier learns a good mapping relationship between features and pseudo-labels.

[0060] 2) Label smoothing loss

[0061] The classification model is optimized by minimizing the cross-entropy loss. Considering the inevitable introduction of noise during pseudo-label construction, this paper uses a label smoothing regularization method to adjust the model from overconfident to noisy predictions. Label smoothing methods impose uniform noise on the class pseudo-labels y and improve the calibration of predictions. Given a class with corresponding labels, noise is injected into all classes as follows:

[0062]

[0063] Among them, C is the number of categories, ε-Uniform (0, 1) is noise, y is the category pseudo label, is the category pseudo label after adding uniform noise. and the predicted labels of the randomly augmented samples to calculate the cross entropy.

[0064]

[0065] Among them, p i It is obtained by applying the logits vector z output by the penultimate layer of the model to the function calculation.

[0066]

[0067] Here we mainly focus on the traditional one-hot encoding label model learning process, encouraging the model to predict the probability of the target category close to 1 and the probability of the non-target category close to 0, that is, the target category z in the final predicted logits vector iThe value of will tend to infinity, causing the model to learn in a direction that infinitely increases the difference between the logit values ​​of the predicted correct and incorrect labels. However, excessively large logit differences can make the model lack adaptability and overconfident in its predictions. In label distribution strategies, measurement errors inevitably introduce noisy labels. Label smoothing strategies can reduce the difference between the output values ​​of predicted positive and negative samples, improving model robustness.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, and modifications made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A pseudo-supervised clustering method based on meta-features, characterized in that The following steps are involved: S1: Input the dataset into the deep model for pre-training; S2: The original image and the data-augmented image are input into the pre-trained model for instance-level feature extraction and category prediction respectively. The classifier is used to predict the category samples, and the sample features with higher confidence are selected. The high-confidence samples are selected to construct the meta-feature f containing stable semantic information. Meta ; Select the K samples with the highest probability in each category as the confident samples C k , the weighted average of the confident samples is used as the meta-feature f Meta , S3: Pseudo-label construction and global pseudo-label assignment; that is, after obtaining the meta-feature, find the semantic nearest neighbor sample for the meta-feature in the global scope, and assign the same pseudo-label to the meta-feature and the corresponding semantic nearest neighbor. k ={N(f Meta )|k∈[1,2…,S]}, where N(·) represents the global nearest neighbor of the sample, l k Pseudo labels assigned to the meta-feature neighbors; S4: Pseudo-label optimization deep model, using cross-entropy loss with label smoothing to guide and optimize the selection of meta-features and the assignment of pseudo-labels; The label smoothing method improves the category pseudo-label y by adding uniform noise, given the corresponding labels, injecting noise into all categories as follows: Among them, C is the number of categories, ε~Uniform(0,1) is the noise, y is the category pseudo label, is the category pseudo label after adding uniform noise; Use the category pseudo-label after adding uniform noise And the predicted labels of the randomly enhanced samples are used to calculate the cross entropy, which is Among them, χ is the data set, p i It is obtained by applying the logits vector z output by the penultimate layer of the model to the function calculation. Z j is the target category.

2. A pseudo-supervised clustering method based on meta-features according to claim 1, characterized in that: The deep model in step S1 is the contrastive learning model MoCo. The data set is input into the contrastive learning model MoCo for pre-training. The training data sets are all image data sets.

3. The pseudo-supervised clustering method based on meta-features according to claim 1, characterized in that: In step S1, after the pre-trained deep model converges, the training is stopped and the parameters are frozen for instance-level feature extraction in subsequent deep models.

4. The pseudo-supervised clustering method based on meta-features according to claim 1, characterized in that: The meta-feature is formed by weighted averaging of the samples with the highest confidence. The meta-feature is the feature with the highest probability and the most representative category in the corresponding category, which effectively ensures the stable semantic attributes of the category and can effectively represent the semantic information of the category.

5. The pseudo-supervised clustering method based on meta-features according to claim 1, characterized in that: In step S2, the original image is input into the pre-trained deep model to obtain instance-level features; After the enhanced image passes through the pre-trained deep model, the features are input into the classifier to predict the sample probability, output the predicted probability of the enhanced image, and select the sample features with high confidence and take the weighted average as the meta-feature.

6. A pseudo-supervised clustering method based on meta-features according to claim 5, characterized in that: The enhanced image R(x) is input into the pre-trained deep model to obtain f(R(x)), which is then passed through a classifier Φ composed of two layers of multi-layer perceptrons. c , the classifier outputs the enhanced image prediction probability P k ,in, P k =Φ c (f(R(x))), C k =topK(P ki , f(x)); where P ki Refers to the predicted probability of the i-th category; f(x) is the feature obtained after the original image is input into the pre-training model; topK() is to select the K features with the highest probability in the corresponding category; C k It is a confident sample. By arranging the feature probabilities of the corresponding categories from high to low, the top K with the highest probability are selected as C k .

7. The pseudo-supervised clustering method based on meta-features according to claim 1, characterized in that: In step S4, a cross-entropy loss function with label smoothing is used to effectively prevent the model from being overconfident in its predictions. Through continuous iterative operation, the classifier predictions become more and more accurate, ultimately achieving a good clustering effect.

Images