Image contrast clustering method based on sample layering
By layering the positive samples and building pseudo-labels in the image comparison clustering method, the pseudo-positive samples problem is solved, and the accuracy and semantic extraction ability of the clustering model are improved.
Patent Information
- Application Number
- CN202411672487.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-03-18
AI Technical Summary
The existing image comparison clustering methods are prone to pseudo-positive sample problems during the clustering process, which leads to the accumulation of errors and affects the model accuracy.
A comparison clustering method based on sample hierarchy is proposed. By dividing positive sample pairs into three layers: enhancement level, neighborhood level and prototype level, corresponding positive sample pairs are constructed separately, and pseudo-labeled them. Finally, these positive sample pairs are placed in the loss function to train the clustering model.
By layering the positive sample pairs, the accuracy of positive sample construction is improved, the occurrence of pseudo-positive samples is reduced, the accuracy of clustering is improved, and the semantic extraction ability of deep neural networks is improved.
Smart Images
Figure CN119649075A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image clustering based on deep learning, and in particular to an image contrast clustering method based on sample stratification. Background Art
[0002] As an unsupervised deep learning algorithm, the image clustering algorithm divides unlabeled image data into several clusters. According to various similarity metrics, the data within a cluster is highly similar, while the data between clusters are different.
[0003] Image clustering has important research value. Although thanks to a large number of open source data sets, current deep neural networks already have high recognition capabilities, but the networks trained by these fully supervised data sets can only recognize the categories of specific fields that exist in the data sets. In real applications, there are more than just these categories. A natural way to move forward is to build a larger and more diverse data set, which may contain billions of images. If you want to continue training in a fully supervised way, too much labeled data is required and the cost is expensive. This requires a method that can perform unsupervised training on Internet-scale data sets, and deep clustering algorithms are commonly used unsupervised training methods.
[0004] Image clustering also has a wide range of applications. Image clustering can help understand the underlying patterns and structures in large-scale image datasets. By grouping similar images together, researchers can better understand the relationship between images, identify important image features, and infer potential implicit information in a collection of images. Image clustering also plays a key role in image retrieval. By grouping images into different clusters, users can more easily retrieve images that are similar to the image of interest.
[0005] Recently, contrastive learning has become increasingly popular in the field of unsupervised representation learning. Contrastive learning aims to maximize the similarity between positive sample pairs and minimize the similarity between positive and negative sample pairs. Due to the success of contrastive learning, some clustering methods based on contrastive learning have also been proposed, and contrastive clustering (CC) is the most typical algorithm. CC consists of two parts. The first part is the instance-level contrast process. The instance-level contrast module regards its own two enhanced views as positive sample pairs and the enhanced views of all other samples as negative samples. It performs contrastive learning in the instance-level subspace to learn sufficiently discriminative representations. The other part is the allocation-level contrast process. The allocation-level contrast module learns the most semantically reasonable clustering solution by maximizing the partition confidence in the allocation-level subspace, thereby obtaining the most semantically reasonable allocation. However, for clustering tasks, positive sample pairs should not only contain their own enhanced views, but also samples of the same type as themselves. CC regards the enhanced views of samples of the same type and the enhanced views of samples of different classes as negative samples, which brings the problem of pseudo-negative samples.
[0006] To address this problem, Graph Contrastive Clustering (GCC) essentially introduces the assumption of "similar is the same type" on this basis, constructs a K-nearest neighbor (KNN) graph using the similarity between samples, and regards the interconnected samples in the graph as neighbor sample pairs, and the neighbor sample pairs with high similarity and enhanced view pairs of a certain type as positive sample pairs. This alleviates the problem of pseudo-negative samples to a certain extent, while also increasing the number of positive samples and more effectively utilizing the semantic information in the positive sample pairs.
[0007] However, this assumption is too restrictive. Since the semantic extraction ability of deep neural networks is not strong during training, similar samples in the clustering process are not necessarily true samples. In the clustering process, positive sample pairs are often not similar sample pairs, which brings the problem of false positive samples and causes error accumulation. Therefore, the false positive sample problem has become one of the important issues restricting the accuracy of image contrast clustering models. Summary of the invention
[0008] In order to solve the problem of false positive samples in the clustering process and make better use of positive sample information, the present invention proposes an image contrast clustering method based on sample stratification. First, the positive sample pairs are divided into three layers: the first layer of positive sample pairs are positive sample pairs containing enhanced sample pairs, the second layer of positive sample pairs are positive sample pairs containing neighborhood sample pairs, and the third layer is positive sample pairs containing prototype sample pairs. Then, pseudo labels are given to the trustworthy positive sample pairs in the second and third layers of positive sample pairs. Finally, these positive sample pairs are put into the loss function to train the clustering model. The image contrast clustering method based on sample stratification is summarized as follows: data preprocessing and enhancement, image contrast representation learning model construction, pre-training, image contrast clustering model construction, image contrast clustering model training, and image contrast clustering model testing.
[0009] The image contrast clustering method based on sample stratification specifically includes the following steps:
[0010] Step 1: Data preprocessing and enhancement operations. Use strong data enhancement and weak data enhancement strategies to enhance the image dataset to obtain the training dataset, and use the common data enhancement strategy to enhance the image dataset to obtain the test dataset. The specific enhancement methods are as follows:
[0011] 1.1 A strong data enhancement strategy is applied to the image dataset ImageSet, including contrast cropping, random changes in image brightness, contrast, saturation and hue, grayscale, horizontal flipping, random data enhancement and standardization, to obtain ImageSet s Random data augmentation includes adjusting image color balance, equalization, identity transformation, color scale, tilt, sharpness, and brightness.
[0012] 1.2 Perform two weak data enhancement strategies on the image dataset ImageSet, including contrast cropping, random changes in image brightness, contrast, saturation and hue, grayscale and standardization, and obtain and
[0013] 1.3 Perform common data enhancement strategies on the image dataset ImageSet, including scaling, center cropping and standardization, to obtain ImageSet test .
[0014] Step 2: Construct an image contrast representation learning model. Using ResNet18, ResNet32 or ResNet50 as the basic network architecture, add two multi-layer perceptrons (MLPs) at the output in parallel to output instance-level features and allocation-level features, and set MemoryBank to store the features output during training.
[0015] Step 3: Pre-training: Use the InfoNCE loss function to train for a fixed number of rounds, and update the MemoryBank with each output feature sliding.
[0016] Step 4: image contrast clustering model construction, the specific process is as follows:
[0017] 4.1 Expanding the positive sample pairs into enhanced positive sample pairs, neighborhood positive sample pairs and prototype positive sample pairs includes the following key steps:
[0018] 4.1.1 ImageSet s , and Samples in and Combined into enhanced positive samples, respectively and
[0019] 4.1.2 Use all the features in the MemoryBank constructed in pre-training to find K nearest neighbors for all samples in ImageSet, and then construct neighborhood-level positive sample pairs. The specific process is as follows:
[0020] 4.1.2.1 Calculate the similarity s between all features i，j = cosine(features i , features j ) get the set Similarity = {S i，j |i<N,j<N}, N represents the total number of samples;
[0021] 4.1.2.2 Through Similarity set Find the K nearest neighbors
[0022] 4.1.2.3 Constructing the corresponding neighborhood-level positive sample pairs based on the nearest neighbor set
[0023] 4.1.3 Use all the features in MemoryBank to find prototype samples for all samples in ImageSet, and then construct prototype-level positive sample pairs. The specific process is as follows:
[0024] 4.1.3.1 Perform M clustering on the features in MemoryBank, and the number of clustering centers are {m1, m2, m3, ..., m M}KMeans clustering, obtain M cluster center number sets {CenterID1, CenterID2, CenterID3, ..., CenterID M}, each cluster center sequence number set will record the cluster centers of all corresponding samples in ImageSet;
[0025] 4.1.3.2 Construct M prototype-level positive sample pairs for each sample in the data set based on each cluster center sequence number set
[0026] 4.2 Introduce classifier information into the prototype sample positive sample pairs and the neighborhood sample positive sample pairs to construct pseudo labels for the qualified positive sample pairs. The specific process is as follows:
[0027] 4.2.1 Construct pseudo labels for prototype-level positive sample pairs. The specific process is as follows:
[0028] 4.2.1.1 The m-th prototype positive sample pair Put it into the constructed neural network model and output the probability vector (Prob i , Prob j );
[0029] 4.2.1.2 Calculate whether the m-th prototype-level positive sample pair should be labeled with a pseudo label PS i，j =(Prob i ≥threshold)*(Prob j ≥threshold).
[0030] 4.2.2.1 Construct pseudo labels for neighborhood-level positive sample pairs. The specific process is as follows:
[0031] 4.2.2.2 Neighborhood-level positive samples Put it into the constructed neural network model and output the probability vector (Prob i , Prob j );
[0032] 4.2.2.3 Calculate whether this pair of neighborhood-level positive samples should be labeled with a pseudo label NS i，j =(Prob i ≥threshold)*(Prob j ≥threshold).
[0033] Step 5: Image contrast clustering model training. The specific process is as follows:
[0034] 5.1 Construct the loss function for the enhanced positive sample pair. The specific process is as follows:
[0035] 5.1.1 Enhanced positive samples Hehe Put it into the constructed neural network model and output the instance features respectively and And the probability characteristics and Combining the distribution features by row, we get and Prob s , renumber the columns as and There are C columns in total, which is also the number of clusters;
[0036] 5.1.2 Calculating instance-level loss function and
[0037] 5.1.3 Calculating the Allocation-Level Loss Function
[0038] 5.2 Let NN k Represents the mapping relationship between an image and its neighbor images, and constructs a loss function for the neighborhood-level positive sample pairs. The specific process is as follows:
[0039] 5.2.1 Neighborhood-level positive samples Put it into the constructed neural network model and output the instance features respectively and probability characteristics Randomly select a set of probability features All samples are combined by rows and then divided by columns as in the enhanced stage;
[0040] 5.2.2 Calculating instance-level loss in,
[0041] 5.2.3 Calculating Allocation Level Loss
[0042] 5.3 Proto m It is the mapping relationship between an image and its prototype image. The loss function is constructed for the prototype-level positive sample pair. The specific process is as follows:
[0043] 5.3.1 Prototype-level positive samples Put it into the constructed neural network model and output the instance features respectively and probability characteristics Randomly select a set of probability features All samples are combined by rows and then divided by columns as in the enhanced stage;
[0044] 5.3.2 Calculating instance-level loss
[0045] Among them, φ represents the concentration within the cluster, which is calculated by the distance between each point in the cluster and the center point. The calculation process is:
[0046] 5.3.3 Calculating Allocation Level Loss
[0047] 5.4 Use the previously used Update the MemoryBank by sliding update.
[0048] Step 6: Image contrast clustering model test. To test the model trained in step 5, use the test dataset ImageSet test Input the model, test the clustering ability of the model, and use NMI, ACC and ARI as performance evaluation indicators of the test results. The value range of the performance evaluation index is [0, 1]. The higher the value, the better the clustering model effect. The present invention also uses tSNE graph to intuitively measure the magic clustering performance. The farther the samples of different categories are in the graph, the closer the samples of the same category are in the graph, and the better the clustering effect.
[0049] Beneficial Effects
[0050] A mitigation strategy is proposed for the pseudo-positive sample problem in the image comparison clustering method, which increases the correctness of positive sample construction and improves clustering accuracy. On the one hand, this shows that the semantic extraction ability of the trained deep neural network has been improved, and only a small amount of labeled data needs to be fine-tuned to obtain a high-precision classification model; on the other hand, in the field of image retrieval, higher clustering accuracy means that it can better help users retrieve images similar to the images of interest. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 Flow chart of the image contrast clustering method based on sample stratification proposed by the present invention
[0052] Figure 2 Model structure diagram of image contrast clustering method based on sample stratification
[0053] Figure 3 Loss curves of the sample-based stratified image contrast clustering method on the CIFAR10 dataset. Figure (a) is the instance-level loss curve, and Figure (b) is the allocation-level loss curve.
[0054] Figure 4 Accuracy curves of the sample-stratified image contrast clustering method during training on the CIFAR10 dataset, where (a) is NMI, (b) is ACC, and (c) is ARI.
[0055] Figure 5The final tSNE image of the sample-based image contrast clustering method on the CIFAR10 dataset DETAILED DESCRIPTION
[0056] The following will be combined with the contents of the drawings in the specification to explain the technical solutions in the implementation of the present invention in more detail and completeness. Obviously, the following embodiments are only part of the present invention, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific implementation method Take CIFAR10 as an example:
[0057] Step 1: Use strong data enhancement and weak data enhancement strategies to enhance and preprocess the image dataset to obtain a training dataset, and use ordinary data enhancement and preprocessing to obtain a test dataset, including the following key steps:
[0058] 1.1 Perform a strong data enhancement strategy on the image dataset ImageSet to obtain ImageSet s . From the beginning to the end, a series of atomic linear change composites are called respectively, including the following parameter values: the alpha value of ContrastiveCrop is set to 0.6, the scale is set to [0.2, 1.0], the size is 32, the RandomHorizontalFlip is set to 0.5, the ColorJitter is set to (0.4, 0.4, 0.4, 0.1), the RandomGrayscale is set to 0.2, and the RandAugment. RandAugment also includes a series of atomic changes. The main parameter values are set to: AutoContrast, Brightness (0.9, 0.05), ColorContrast (0.9, 0.05), Equalize, Identity, Posterize (4, 4), Rotate (0.9, 0.05), Sharpness (0.9, 0.05), ShearX (0.3, 0), ShearY (0.3, 0), Solarize (256, 0), TranslateX (0.3, 0), TranslateY (0.3, 0). RandAugment is set to randomly call 5 of the atomic changes to perform data augmentation on the image dataset.
[0059] 1.2 Perform two weak data enhancement strategies on the image dataset ImageSet to obtain and From the beginning to the end, a series of atomic linear changes are called respectively, and the main parameter values are set as follows: the alpha value of ContrastiveCrop is set to 0.6, the scale is set to [0.2, 1.0], the size is 32, the RandomHorizontalFlip is set to 0.5, the ColorJitter is set to (0.4, 0.4, 0.4, 0.1), and the RandomGrayscale is set to 0.2;
[0060] 1.3 Perform common data enhancement on the image dataset ImageSet, including Resize (40) and CenterCrop (32), to obtain the test dataset ImageSet test ;
[0061] 1.4 Regardless of strong data augmentation, weak data augmentation or normal data augmentation, the standardized means are [0.4914, 0.4822, 0.4465] and the variances are [0.2023, 0.1994, 0.2010].
[0062] Step 2: Image contrast representation learning model construction. As described in step 2 of the invention, ResNet18 is selected as the backbone network, where the parameters of the instance-level MLP are [512, 512, 128] and the parameters of the allocation-level MLP are [512, 10].
[0063] Step 3, pre-training. As described in step 3 of the invention, the InfoNCE loss function is used for training for 50 epochs, the initial learning rate is set to 0.4, the batch size is 256, the trainer is SGD, the decay coefficient weight_decay is set to 0.0001, and the cosine annealing algorithm is used to update the learning rate during the training process, and the MemoryBank is updated with the feature sliding each time, and the sliding coefficient is set to 0.5.
[0064] Step 4: image comparison clustering model construction. According to the invention content 4, a positive sample pair training image clustering model is constructed. The specific process is as follows:
[0065] 4.1 According to the invention content 4.1.1, construct and As enhanced positive sample pairs;
[0066] 4.2 According to 4.1.2 of the invention, the parameter K is set to 5, which means that for each sample, the five samples with the highest cosine similarity are found as the neighborhood level positive sample pairs, and the obtained
[0067] 4.3 According to the invention content 4.1.3, set the parameter M to 3, and the number of cluster centers to {2500, 5000, 10000}. This means that all samples are clustered three times with 2500, 5000, and 10000 as the number of cluster centers, and three cluster center number sets {CenterID1, CenterID2, CenterID3} are obtained respectively. Each set will record the cluster centers of the three clustering of 60,000 samples. According to these three cluster center number sets, select three cluster centers for each sample as positive sample pairs, and get
[0068] 4.4 According to the invention content 4.2, pseudo labels are added to the positive sample pairs that meet the conditions in the neighborhood level and prototype level positive sample pairs, where the threshold is selected as 0.6. Output (Prob i , Prob j )Substitute into NS i，j =(Prob i ≥threshold)*(Prob j ≥threshold) and record the calculation results. i，j =1 indicates that the positive sample pair meets the conditions and has been added with a pseudo label, NS i，j =0 indicates that the positive sample pair does not meet the conditions and cannot be added with a pseudo label.
[0069] Step 5: Image contrast clustering model training. For the enhanced positive sample pairs, neighborhood positive sample pairs and prototype positive sample pairs that have been constructed, the corresponding loss functions are constructed according to the invention contents 5.1, 5.2 and 5.3 respectively, and τ in each instance-level loss is set to ia , τ in and τ im are all 0.1, and τ in the distribution level loss ca , τ cn and τ cm is 2.0, and the weight of each loss term is set to 1, in the form of Substitute the instance-level and allocation-level features output by the neural network into the loss function to calculate the loss, and record the loss value of each epoch. The training record results are as follows: Figure 3 As shown in the figure, this step jumps to step 4 every 5 epochs until 950 epochs are trained.
[0070] Step 6: Image comparison clustering model test. The specific process is as follows:
[0071] 6.1 When performing step 5, after every 5 rounds of training, the neural network switches to the test mode, the entire data set is input into the neural network, the obtained distribution results are substituted into the NMI, ACC and ARI models, and recorded, and the accuracy change trend is recorded as follows Figure 4 As shown;
[0072] 6.2 After the training process is completed, the test dataset ImageSet test All of them were input into the neural network, and the obtained distribution results were substituted into the NMI, ACC and ARI models. The results were 0.801, 0.880 and 0.769 respectively, which were better than other current image contrast clustering methods;
[0073] 6.3 After the training process is completed, all data sets are input into the neural network, the representation of the backbone network output is reduced in dimension using the TSNE method in the sklearn package, and the tSNE diagram is drawn using the matplotlib package. The drawing results are shown in Figure 5 shown.
[0074] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary researchers in this technical field, several improvements and variations can be made without departing from the technical principles of the present invention. For example, in the improved method, the selection of the feature network is not limited to the mentioned network, and other networks suitable for the required processing of image data can be selected according to the method proposed by the present invention; the model can not only process common image data sets, but also be applied to other similar types of image data; these improvements and variations should also be regarded as the scope of protection of the present invention.
Claims
1. A sample-stratified image contrast clustering method, which improves the contrast clustering model algorithmically to alleviate the problem of false positive sample pairs in the clustering process. It is characterized by: It mainly includes the following key steps: data preprocessing and enhancement, image contrast representation learning model construction, pre-training, image contrast clustering model construction, image contrast clustering model training, and image contrast clustering model testing.
2. The image contrast clustering method based on sample stratification according to claim 1, characterized in that: The image contrast clustering model was constructed, and the contrast clustering model was modified from two aspects: expanding the positive sample pairs into multi-layer positive sample pairs and constructing pseudo labels for the positive sample pairs that meet the conditions.
3. The image contrast clustering model construction according to claim 2 is characterized in that the positive sample pairs are expanded into multi-level positive sample pairs, mainly three levels of positive sample pairs, and the positive sample pairs are expanded into enhanced-level positive sample pairs, neighborhood-level positive sample pairs and prototype-level positive sample pairs, including the following key steps: 3.1 ImageSet s , and Samples in and Combined into enhanced positive samples, respectively and 3.2 Use all the features in the MemoryBank constructed in pre-training to find K nearest neighbors for all samples in ImageSet, and then construct neighborhood-level positive sample pairs, including the following key steps: 3.2.1 Calculate the similarity s between all features i，j = cosine(features i , features j ) get the set Similarity = {S i，j |i<N,j<N}, where N represents the total number of samples; 3.2.2 Through Similarity set Find the K nearest neighbors 3.2.3 Constructing the corresponding neighborhood-level positive sample pairs based on the nearest neighbor set 3.3 Use all the features in MemoryBank to find prototype samples for all samples in ImageSet, and then construct prototype-level positive sample pairs, including the following key steps: 3.3.1 Perform M clustering on the features in MemoryBank, and the number of clustering centers are {m1,m2,m3,...,m M }KMeans clustering, obtain M cluster center number sets {CenterID1, CenterID2, CenterID3, ..., CenterID M }, each cluster center sequence number set will record the cluster centers of all corresponding samples in ImageSet; 3.3.2 Construct M prototype-level positive sample pairs for each sample in the data set based on each cluster center sequence number set 4. The image contrast clustering model construction according to claim 2, characterized in that: Constructing pseudo labels for qualified positive sample pairs includes the following key steps: 4.1 Construct pseudo labels for prototype-level positive sample pairs, including the following key steps: 4.1.1 The m-th prototype positive sample pair Put it into the constructed neural network model and output the probability vector (Prob i , Prob j ); 4.1.2 Calculate whether the m-th prototype-level positive sample pair should be labeled with a pseudo label PS i，j =(Prob i ≥threshold)*(Prob j ≥threshold). 4.2 Constructing pseudo labels for neighborhood-level positive sample pairs includes the following key steps: 4.2.1 Neighborhood-level positive samples Put it into the constructed neural network model and output the probability vector (Prob i , Prob j ); 4.2.2 Calculate whether this pair of neighborhood-level positive samples should be labeled with a pseudo label NS i，j =(Prob i ≥threshold)*(Prob j ≥threshold).
5. The image contrast clustering method based on sample stratification according to claim 1, characterized in that: Image contrast clustering model training includes the following key steps: 5.1 Constructing the loss function for the enhanced positive sample pair includes the following key steps: 5.1.1 Enhanced positive samples Hehe Put it into the constructed neural network model and output the instance features respectively and And the probability characteristics and Combining the distribution features by row, we get and Prob s , renumber the columns as and There are C columns in total, which is also the number of clusters; 5.1.2 Calculating instance-level loss function and 5.1.3 Calculating the Allocation-Level Loss Function 5.2 Let NN k Represents the mapping relationship between an image and its neighbor images, and constructs a loss function for the neighborhood-level positive sample pairs, including the following key steps: 5.2.1 Neighborhood-level positive samples Put it into the constructed neural network model and output the instance features respectively and probability characteristics Randomly select a set of probability features All samples are combined by rows and then divided by columns as in the enhanced stage; 5.2.2 Calculating instance-level loss in, 5.2.3 Calculating Allocation Level Loss 5.3 Proto m It is the mapping relationship between an image and its prototype image. Constructing the loss function for the prototype-level positive sample pair includes the following key steps: 5.3.1 Prototype-level positive samples Put it into the constructed neural network model and output the instance features respectively and probability characteristics Randomly select a set of probability features All samples are combined by rows and then divided by columns as in the enhanced stage; 5.3.2 Calculating instance-level loss in, φ represents the concentration within the cluster, which is calculated by the distance between each point in the cluster and the center point. The calculation process is: 5.3.3 Calculating Allocation Level Loss 5.4 Use the previously used Update the MemoryBank by sliding update.