A relational deep clustering method based on contrastive learning

Abstract

The present invention discloses a relational deep clustering method based on contrastive learning, comprising: enhancing the original image data to obtain corresponding enhanced image data; importing the original image data and the enhanced image data into a neural network, subjecting the input image data to hierarchical nonlinear mapping, and obtaining distribution probability matrices respectively; calculating the structural relationship matrix of the original image data and the enhanced image data according to the distribution probability matrix of the original image data and the enhanced image data; and constructing a total loss function. The present invention can obtain more positive discriminant features, reduce the intra-class variance of cluster embedding, and thus obtain better clustering results.

Description

Technical Field

[0001] The present invention relates to the technical field of unsupervised image clustering, and in particular to a relational deep clustering method based on contrastive learning. Background Art

[0002] In recent years, a large amount of visual content has been generated in areas such as social media platforms and medical images. People can easily collect thousands of image data every day, most of which are unlabeled, which severely limits their applicability in many scenarios. Manually labeling this data is very expensive and time-consuming. For example, the famous IMAGENET dataset took three years to label. The extremely high cost will inevitably bring huge challenges to the sharing and use of this data, and also lead to the need for people to effectively manage and use such a large amount of data in an unsupervised manner.

[0003] Clustering is a basic unsupervised learning method, and traditional clustering methods, such as K-Means, spectral clustering, non-negative matrix factorization clustering, etc., are widely used in various tasks. However, these methods only focus on or pay too much attention to local, pixel-level information, ignoring the higher-level semantic information of the image, which greatly limits their performance.

[0004] Deep learning has developed rapidly in recent years and has achieved remarkable results in various computer vision tasks. Although the excellent results of most deep learning models benefit from the supervision of expensive and high-quality labels, the amazing potential they show in feature extraction and representation cannot be ignored. Therefore, more and more researchers are applying deep learning to clustering. DCN applies K-Means to the potential representation generated by the autoencoder, while minimizing the reconstruction loss and K-Means clustering loss. DEC chooses to abandon the decoder and only uses the encoder to calculate the potential representation, and learns the encoder and cluster centers at the same time through the clustering loss proposed by the author. However, these clustering methods based on autoencoders will learn too much underlying information, including background or texture, making them only effective on simple datasets such as MNIST, and unable to calculate the appropriate similarity between samples at the cluster level. On the other hand, feature extraction and cluster separation will cause errors to accumulate and propagate in alternating training, resulting in suboptimal results.

[0005] Recent research has focused on how to learn representations and perform clustering in an end-to-end manner. IIC uses images and randomly enhanced images to form data pairs to train the model to learn the consistency of clustering results; PICA learns the most semantically credible clustering solution by maximizing the partition confidence; CC creatively proposes the idea of ​​"label as representation" and explicitly performs instance-level and cluster-level contrastive learning. This type of method regards image data as instances, each instance corresponds to a class, uses data enhancement to construct data pairs, and uses the method of maximizing mutual information to learn instance representation consistency and cluster representation consistency from them. Its clustering results often rely on more separated negative data pairs and less clustered positive data pairs, focusing on distinguishing different instances and expanding inter-class variance. Although learning the difference between different images helps the model distinguish pictures from different semantic classes, in most methods, different images belonging to the same class are regarded as "negative pairs" and are far away from each other, resulting in a large intra-class variance of cluster embedding, which violates the original intention that "a good cluster embedding should have a small intra-class variance and a large inter-class variance". In addition, the increase in intra-class variance will bring instability to the model and cause the accumulation of errors. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides a relational deep clustering method based on contrastive learning. In addition to the contrastive losses at the sample level and cluster level, a new loss function is added to penalize the differences between the structural relationships of multiple samples. This loss focuses on the consistency of the structural relationships between multiple output data rather than a single data pair itself. Combining it with the double contrastive loss can obtain more positive discriminative features, reduce the intra-class variance of cluster embedding, and thus obtain better clustering results.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A relational deep clustering method based on contrastive learning, the relational deep clustering method comprising the following steps:

[0009] S1, for the original image data x i Perform enhancement processing to obtain the corresponding enhanced image data x i ′=T(x i ); T() is the adopted data enhancement function;

[0010] S2, the original image data x i and the enhanced image data x i ′Introduce the neural network, and the input image data is subjected to hierarchical nonlinear mapping to obtain the output original image data x i and the enhanced image data x i′’s distribution probability matrix; where the rows of the distribution probability matrix represent the feature representation of the image; the columns of the distribution probability matrix are the data distribution, representing the cluster representation of different semantic classes;

[0011] S3, based on the original image data x i and the enhanced image data x i ′ is used to calculate the probability matrix of the original image data x i and the enhanced image data x i ′’s structural relationship matrix;

[0012] S4, construct the total loss function α is the weight parameter, is an instance representation loss function, which is used to characterize the consistency of the feature representation of the original image and its enhanced image by minimizing the similarity between the original image and the enhanced image at the sample level; A loss function is represented as a clustering function, which is used to characterize the consistency of the assignment of the original image and its enhanced image by minimizing the similarity between the original class and the enhanced class at the cluster level; is a relation representation loss function, which is used to characterize the consistency of the relation representation of the original image and its enhanced image by minimizing the similarity between the original structural relation and the enhanced structural relation at the relation level.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Furthermore, in step S2, the data enhancement function includes multiple or all of four types of data enhancement functions: random cropping, horizontal flipping, color jittering, and grayscale conversion.

[0015] Furthermore, in step S2, ResNet34 is used as the backbone network to transform the input original image data x i and the enhanced image data x i ′Perform hierarchical nonlinear mapping processing and output the assignment probability matrix as the feature matrix of the corresponding image.

[0016] Furthermore, in step S3, the Euclidean distance and angle in space are used as the original image data x i and the enhanced image data x i The structural relationship between ′ is expressed.

[0017] Further, in step S4, in step S4, the instance represents the loss function The construction process consists of the following sub-steps:

[0018] A41, given a batch of original sample sets x = {x1, x2, ..., x N}, and the corresponding N enhanced sample sets are x′={x1′,x2′,...,x N '};

[0019] A42, for any original image sample x i , x i The original image sample and the corresponding enhanced image sample form a data pair (x i , x i ′) is regarded as a positive pair, and the data pair (x i , x j ′) is regarded as a negative pair; i, j = 1, 2, ..., N, j ≠ i;

[0020] A43, nonlinear MLPg(·) is used to map the original sample set x and the enhanced sample set x′ into the probability distribution space, and the obtained probability distribution is regarded as the feature representation u=g(f θ (x)), u′=g(f θ (x′));

[0021] A44, the cosine similarity is calculated according to the following formula, and the cosine similarity is used as an indicator to evaluate whether the distribution probability of the positive sample pair is consistent:

[0022]

[0023] Among them, ||·||2 represents L2 normalization;

[0024] A45, according to InfoNCE, the instance representation loss function Defined as:

[0025]

[0026] Among them, τ>0 is the temperature parameter, u i is the original image x i The feature representation of u i ′ is the enhanced image x i ′’s feature representation, u j ′ is the enhanced image x j ′’s feature representation; is the mathematical expectation.

[0027] Further, in step S4, in step S4, the clustering represents the loss function The construction process consists of the following sub-steps:

[0028] B41, assuming that the output probability distribution matrices corresponding to the original image and its enhanced image are V = [v1, v2, ..., vC ] N×C and V′=[v1′, v2′,...,v C ′] N×C ; The column space v of V and V′ k With v k ′ is used to indicate which images are assigned to cluster k, and the k-th column of V and V′ is regarded as the representation of the k-th cluster;

[0029] B42, clusters classified into the same category are considered positive pairs, and other clusters are considered negative pairs;

[0030] B43, uses cosine distance to measure the similarity between pairs of cluster representations:

[0031]

[0032] B44, for the temperature parameter τ, the clustering represents the loss function The loss is defined as:

[0033]

[0034] In the formula, τ>0, k, m=1, 2, ..., C, k≠m.

[0035] Further, in step S4, in step S4, the relationship represents the loss function The construction process consists of the following sub-steps:

[0036] C41, using the Euclidean distance in space as the original image x i and the original image x j The relationship between

[0037]

[0038] Where μ is the normalization factor of the distance, u i is the original image x i The feature representation of u j is the original image x j The feature representation of

[0039] C42, set μ to the average distance between all data pairs in the data pair set B of each batch, and use the following formula to calculate the value of μ:

[0040]

[0041] Where |B| is the total number of data pairs in the data pair set B;

[0042] C43, relation representation loss function It is expressed as:

[0043]

[0044] Among them, l σ is the L2 distance, is the enhanced image x i ′ and enhanced image x j The relationship between ′ is expressed.

[0045] The beneficial effects of the present invention are:

[0046] In the relational deep clustering method based on contrastive learning of the present invention, the distance between data samples can be regarded as the structural relationship between them (hereinafter referred to as relationship), thereby respectively obtaining the structural relationship between the original sample and the enhanced sample; by constraining the difference between the original sample relationship and the enhanced sample relationship, the ability of the model to output similar relationship matrices for the same batch of samples is improved, and the robustness of the model can be improved. In addition, the relational representation loss brings more positive data pair information and constraints to the model, and combining it with the double contrast loss can obtain more positive discriminative features, reduce the intra-class variance of cluster embedding, and thus obtain better clustering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a relational deep clustering method based on contrastive learning in an embodiment of the present invention.

[0048] Figure 2 It is a schematic diagram of the differences between sample pairs, category pairs, and structural relationship pairs. Figure 2 (a) is a schematic diagram of the differences between sample pairs. Figure 2 (b) is a schematic diagram of the differences between category pairs. Figure 2 (c) is a schematic diagram of the differences between pairs of structural relationships.

[0049] Figure 3 It is a comparative learning framework diagram consistent with the relationship structure of the embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will now be described in further detail with reference to the accompanying drawings.

[0051] It should be noted that the terms such as "upper", "lower", "left", "right", "front", "back", etc. cited in the invention are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments in their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0052] Figure 1 Flowchart of the relational deep clustering method based on contrastive learning in an embodiment of the present invention. Figure 1, the relational deep clustering method includes the following steps:

[0053] S1, for the original image data x i Perform enhancement processing to obtain the corresponding enhanced image data x i ′=T(x i ); T() is the data enhancement function used.

[0054] S2, the original image data x i and the enhanced image data x i ′Introduce the neural network, and the input image data is subjected to hierarchical nonlinear mapping to obtain the output original image data x i and the enhanced image data x i ′ is an assignment probability matrix; wherein the rows of the assignment probability matrix represent the feature representation of the image; the columns of the assignment probability matrix are the data distribution, representing the clustering representation of different semantic classes.

[0055] S3, based on the original image data x i and the enhanced image data x i ′ is used to calculate the probability matrix of the original image data x i and the enhanced image data x i ′ is the structural relationship matrix.

[0056] S4, construct the total loss function α is the weight parameter, is an instance representation loss function, which is used to characterize the consistency of the feature representation of the original image and its enhanced image by minimizing the similarity between the original image and the enhanced image at the sample level; A loss function is represented as a clustering function, which is used to characterize the consistency of the assignment of the original image and its enhanced image by minimizing the similarity between the original class and the enhanced class at the cluster level; is a relation representation loss function, which is used to characterize the consistency of the relation representation of the original image and its enhanced image by minimizing the similarity between the original structural relation and the enhanced structural relation at the relation level.

[0057] The goal of contrastive learning is to maximize the similarity between positive pairs and minimize the similarity between negative pairs. One of the most important tasks is how to design positive and negative data pairs to meet the requirements of the clustering task, that is, similar samples are close to each other and different samples are far away from each other.

[0058] Aiming at the problem that contrastive learning pays more attention to distinguishing different instances and ignores the intra-class performance, a relational deep clustering method based on contrastive learning is proposed, which uses instance feature representation, category representation and relation representation for clustering. Figure 2 It is a schematic diagram of the differences between sample pairs, category pairs, and structural relationship pairs.

[0059] See also Figure 3 , inspired by SimCLR, uses data augmentation to construct data pairs as input. The SimCLR method comprehensively demonstrates the impact of different augmentation strategies on the performance of downstream tasks. This paper selects four types of data augmentation methods: random cropping, horizontal flipping, color jittering, and grayscale. Specifically, given an original data x i , under the action of data enhancement method T, the corresponding enhanced data x is obtained i ′=T(x i ). As the backbone of the deep clustering model, the neural network mainly maps the input image data through a hierarchical nonlinear mapping to obtain a new low-dimensional feature representation. In order to facilitate comparison with other existing works, ResNet34 is used as the backbone network, and the assignment probability matrix of the neural network output image is regarded as a feature matrix. Naturally, the rows of the assignment probability matrix (that is, the assignment probability vector of each image) are regarded as the feature representation of the image, and according to the idea of ​​"label as representation", the columns of the matrix are regarded as data distribution and cluster representation representing different semantic classes. The structural relationship matrix of the original image and the enhanced image can be obtained through the corresponding probability assignment matrix.

[0060] Figure 3 The middle line frame shows the three losses used. At the sample level, the similarity between the original image and the enhanced image is minimized to ensure the consistency of the feature representation of the original image and its enhancement; at the clustering level, the similarity between the original class and the enhanced class is minimized to ensure the distribution consistency of the original image and its enhancement (that is, the clustering representation consistency between the original class and the enhanced class); at the relationship level, the similarity between the original structural relationship and the enhanced structural relationship is minimized to ensure the consistency of the relationship representation of the original image and its enhancement. The three losses are trained together to help form good and more robust clustering. The following will introduce several loss functions involved in the model in detail.

[0061] (1) Instance representation loss

[0062] Based on the idea of ​​contrastive learning, we regard the original image and its enhancement as positive pairs, and the original image and the enhancement of other images as negative pairs. Formally speaking, given a batch of original samples x = {x1, x2, ..., x N}, and the corresponding N enhanced samples are x′={x1′, x2′, ..., x N ′}, then for any sample x i , x i The method converts the sample into a data pair (x i , xi ′) is considered as a positive pair, and the data pair (x i , x j ′) is considered as a negative pair.

[0063] In order to reduce the information loss caused by contrastive learning, the neural network f is not used directly. θ Instead of outputting the features of the instance, we use a nonlinear MLP g(·) to map them into a probability distribution space, and the resulting probability distribution is regarded as the feature representation u = g(f θ (x)), u′=g(f θ (x′)). The original sample and the enhanced sample are essentially the same instance and should have the same class assignment probability. For convenience, cosine similarity is selected as an indicator to evaluate whether the assignment probability of the positive sample pair remains consistent. The formula is defined as:

[0064]

[0065] Here, ||·||2 represents L2 normalization.

[0066] According to InfoNCE, the instance-level loss can be defined as:

[0067]

[0068] Among them, τ>0 is the temperature parameter.

[0069] (2) Clustering Representation Loss

[0070] When a data sample is projected into a space with a dimension equal to the number of clusters C, the jth element of its feature can be interpreted as the probability that the sample belongs to the jth category, and the feature vector represents its soft label accordingly. Formally, similar to the sample level, assume that the output probability assignment matrices corresponding to the original image and its enhanced image are V = [v1, v2, ..., v C ] N×C and V′=[v1′, v2′,...,v C ′] N×C Ideally, soft labels are often one-hot encoded, so the column space of V and V′ is v k With v k ′ can indicate which images are assigned to cluster k, that is, the kth column of V and V′ can be regarded as the representation of the kth cluster. Therefore, clusters classified into the same category can be regarded as positive pairs, for example, v k ′ is actually v kThe enhancements should belong to the same category and can be regarded as positive pairs, and their cluster representations should be consistent. Similarly, the cosine distance is used here to measure the similarity between cluster representation pairs, that is:

[0071]

[0072] Correspondingly, for the temperature parameter τ, the cluster-level loss can be defined as:

[0073]

[0074] (3) Relationship Representation Loss

[0075] The relationship mentioned here refers to the structural relationship between different samples. When high-dimensional data is projected into different low-dimensional spaces, the structural relationship between samples should remain consistent. There are many ways to express structural relationships, such as distance, angle, etc. For convenience, the Euclidean distance in space is used as the relationship between two images:

[0076]

[0077] Here, μ is the normalization factor of the distance.

[0078] In order to focus on the relative distances between other sample pairs, μ is set to the average distance between all data pairs in the data pair set B of each batch, and the value of μ is calculated using the following formula:

[0079]

[0080] The distance structure relationship of data points in the same batch of images under different enhancements should be consistent. Based on this, a new loss function is designed:

[0081]

[0082] Among them, l σ is the L2 distance.

[0083] Then, the total loss function combining the above three losses can be written as:

[0084]

[0085] Among them, α is the weight parameter.

[0086] Examples

[0087] This example uses PyTorch 1.4 to complete all experiments, and uses Adam for optimization, setting the learning rate to a fixed value of 0.003. For the hyperparameters involved in the model, they are set to fixed values, namely, temperature parameter τ = 0.5 and weight parameter α = 0.004. In the experiment, each batch of samples is repeated 3 times, and the same data augmentation method is used. The experiment uses Nvidia TITAN RTX 24G to train the model from scratch for 200 epochs.

[0088] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A relational deep clustering method based on contrastive learning, characterized in that: The relational deep clustering method comprises the following steps: S1, for the original image data x i Perform enhancement processing to obtain the corresponding enhanced image data x i '=T(x i ); T() is the adopted data enhancement function; S2, the original image data x i and the enhanced image data x i 'Import the neural network, and transform the input image data through hierarchical nonlinear mapping to obtain the output original image data x i and the enhanced image data x i '; wherein the rows of the probability matrix represent the feature representation of the image; the columns of the probability matrix represent the data distribution and the cluster representation of different semantic classes; S3, based on the original image data x i and the enhanced image data x i 'The distribution probability matrix is ​​calculated to obtain the original image data x i and the enhanced image data x i 'Structural relationship matrix; S4, construct the total loss function α is the weight parameter, is an instance representation loss function, which is used to characterize the consistency of feature representation of the original image and its enhanced image at the sample level by minimizing the loss function to maximize the similarity between the original image and the enhanced image; A loss function is represented for clustering, which is used to characterize the consistency of the assignment of the original image and its enhanced image at the cluster level by minimizing the loss function to maximize the similarity between the original class and the enhanced class; is a relation representation loss function, which is used to characterize the consistency of the relation representation of the original image and its enhanced image at the relation level by minimizing the loss function to maximize the similarity between the original structural relation and the enhanced structural relation; In step S4, the relationship represents the loss function The construction process consists of the following sub-steps: C41, using the Euclidean distance in space as the original image x i and the original image x j The relationship between Where μ is the normalization factor of the distance, u i is the original image x i The feature representation of u j is the original image x j The feature representation of C42, set μ to the average distance between all data pairs in the data pair set B of each batch, and use the following formula to calculate the value of μ: Where |B| is the total number of data pairs in the data pair set B; C43, relation representation loss function It is expressed as: Among them, l σ is the L2 distance, is the enhanced image x i ' and enhanced image x j 'The relationship between them is expressed.

2. The relational deep clustering method based on contrastive learning according to claim 1, characterized in that: In step S2, the data enhancement function includes multiple or all of four types of data enhancement functions: random cropping, horizontal flipping, color jittering, and grayscale.

3. The relational deep clustering method based on contrastive learning according to claim 1, characterized in that: In step S2, ResNet34 is used as the backbone network to input the original image data x i and the enhanced image data x i 'Perform hierarchical nonlinear mapping processing and output the assignment probability matrix as the feature matrix of the corresponding image.

4. The relational deep clustering method based on contrastive learning according to claim 1, characterized in that: In step S3, the Euclidean distance and angle in space are used as the original image data x i and the enhanced image data x i 'The structural relationship between them is expressed.

5. The relational deep clustering method based on contrastive learning according to claim 1, characterized in that: In step S4, in step S4, the instance represents the loss function The construction process consists of the following sub-steps: A41, given a batch of original sample sets x={x1,x2,...,x N }, and the corresponding N enhanced sample sets are x'={x1',x2',...,x N '}; A42, for any original image sample x i , x i The original image sample and the corresponding enhanced image sample form a data pair (x i ,x i ') is regarded as a positive pair, and the data pair (x i ,x j ') is regarded as a negative pair; i,j=1,2,…,N,j≠i; A43, nonlinear MLPg(·) is used to map the original sample set x and the enhanced sample set x' into the probability distribution space, and the obtained probability distribution is regarded as the feature representation u = g(f θ (x)), u'=g(f θ (x')); A44, the cosine similarity is calculated according to the following formula, and the cosine similarity is used as an indicator to evaluate whether the distribution probability of the positive sample pair is consistent: Among them, ||·||2 represents L2 normalization; A45, according to InfoNCE, the instance representation loss function Defined as: Among them, τ>0 is the temperature parameter, u i is the original image x i The feature representation of u i ' is the enhanced image x i 'The characteristic representation, u j ' is the enhanced image x j 'Feature representation; is the mathematical expectation.

6. The relational deep clustering method based on contrastive learning according to claim 1, characterized in that: In step S4, the clustering represents the loss function The construction process consists of the following sub-steps: B41, assuming that the output probability distribution matrices corresponding to the original image and its enhanced image are V = [v1, v2, ..., v C ] N×C and V'=[v1',v2',...,v C '] N×C ; The column space v of V and V' k With v k 'The k-th column of V, V and V' is considered as the representation of the k-th cluster; B42, clusters classified into the same category are considered positive pairs, and other clusters are considered negative pairs; B43, uses cosine distance to measure the similarity between pairs of cluster representations: B44, for the temperature parameter τ, the clustering represents the loss function The loss is defined as: In the formula, τ>0, k,m=1,2,…,C, k≠m.

Images