An unsupervised image representation learning method based on clustering space decorrelation

Abstract

The present application relates to a kind of unsupervised image representation learning method based on clustering space decorrelation, and the input of one batch image is twice random enhancement and obtains two batch data input twin network;Similarity input to each cluster is calculated based on the output and cluster center;Based on the intercorrelation matrix between the input in a batch based on clustering space feature, the unitization loss is calculated;The membership of input to each cluster is obtained by normalizing clustering space feature, the membership of one image is used as the target distribution of another relevant image, and the alignment loss between the prediction of model to input and its target distribution is calculated;Unitization loss and alignment loss are added as total loss for model training;The minimization of unitization loss realizes the consistency and decorrelation of cluster level, and each other target loss realizes the consistency of sample level to input disturbance, finally effectively unsupervised image representation learning is carried out under the condition of no negative sample pair.

Description

Technical Field

[0001] This invention relates to the technical field of computation, estimation, or counting, and particularly to an unsupervised image representation learning method based on decorrelation of cluster space. Background Technology

[0002] Image representation learning methods are mainly divided into two categories: supervised learning and unsupervised learning. Supervised learning methods rely on large datasets with manually labeled information. However, in real-world applications, the inability to collect large amounts of labeled data leads to insufficient label information during supervised learning, causing training models to encounter bottlenecks. For business development needs, collecting large amounts of labeled data often incurs very high costs. Therefore, unsupervised learning methods have emerged to address this technical problem. Unsupervised learning utilizes the information inherent in the data itself for model learning, avoiding the training bottleneck caused by a lack of labeled data and compensating for the shortcomings of supervised learning methods.

[0003] In recent years, unsupervised feature learning has made great strides, especially in contrastive and self-constructed supervised information. These methods mainly utilize constructing auxiliary tasks (pretext) to obtain supervised information from the data itself for network training, thereby learning representations with good generalization and transferability to downstream tasks. Compared to earlier methods that relied on pixel-level reconstruction, recent examples-based methods are simpler and more universal.

[0004] Since there are no external task supervision signals, the biggest challenge of unsupervised learning is that generating the same and constant features leads to so-called model collapse.

[0005] Contrastive learning is a representative method for preventing model collapse. Self-supervised methods based on example contrast directly compare a large number of explicit pairwise features, using two different images as negative example pairs, requiring their feature representations to be far apart, while using two randomly augmented images from the same image as positive example pairs, requiring their features to be close together. This requires sampling and storing a sufficient number of negative example pairs through additional storage space or increasing the data batch size. Another approach is to use intermediate results learned through self-learning as alignment targets or training signals, such as using cluster membership degrees obtained from clustering as pseudo-labels for model training. This method of predicting alignment based on self-derived targets does not require negative example pairs. Its key lies in how to derive sufficiently diverse alignment targets, i.e., features or soft labels, to achieve unsupervised representation learning that avoids model collapse even without using negative example pairs. Existing techniques are based on cluster balancing strategies, such as the paper "Unsupervised Learning of Visual Features by Contrasting ClusterAssignments." These techniques perform clustering simultaneously with deep representation learning, using the membership distribution of the input to each cluster as the alignment target for the probability distribution predicted by the model for that input. By reassigning membership to clusters to form more balanced clusters, they avoid all samples in a batch being grouped into a single cluster, thus improving the diversity of the target distribution. However, when the training dataset has an imbalanced class distribution, the effectiveness of training the model through cluster balancing decreases, or even completely fails, i.e., the collapse problem still occurs. Summary of the Invention

[0006] This invention addresses the problems existing in the prior art by providing an unsupervised image representation learning method based on decorrelation in cluster space. It inputs two batches of data obtained by performing two random augmentations on a batch of input images into a Siamese network. The similarity between the input and each cluster is calculated based on the Siamese network output and cluster centers to represent its feature representation in the cluster space. A cross-correlation matrix between inputs within a batch is obtained based on the cluster space features, and a normalized loss measuring the alignment between the cross-correlation matrix and the identity matrix is ​​calculated. The cluster space features are then softmax normalized to obtain the membership degree of the input to each cluster. The membership degree of one image is used as the target distribution of another related image (two augmentations from the same original image), thereby calculating the alignment loss between the model's prediction of the input and its target distribution. The normalized loss and the alignment loss of the mutual target are added to obtain the total loss used for model training. Minimizing the normalized loss achieves cluster-level consistency and decorrelation, while the mutual target loss achieves sample-level consistency with input perturbations (random augmentations), ultimately achieving effective unsupervised image representation learning under the condition of no negative sample pairs. Compared to existing cluster balancing methods, this invention has stronger performance in learning class-imbalanced data.

[0007] The technical solution adopted in this invention is an unsupervised image representation learning method based on cluster space decorrelation, the method comprising the following steps:

[0008] Step 1: Randomly select two different data augmentation methods and apply them to the B of the current input batch. n Two different enhancement methods were applied to the image to obtain two sets of enhanced images. and

[0009] Step 2: Based on the Siamese network model structure, X a and X b Input the encoder to obtain the corresponding feature representation. and

[0010] Step 3: Based on the K cluster centers, assign each Mapping to the clustering space to obtain the feature representation of the K-dimensional clustering space Using softmax Normalization yields the membership distribution Among them, m={a,b}, i={1,…B n} represents the sequence number of the corresponding input image;

[0011] Step 4: Based on the features of the current batch in the clustering space Calculate the cross-correlation matrix C to measure the correlation between features in the cluster space;

[0012] Step 5: Based on the Siamese network model and cluster centers, obtain the membership distribution of two enhanced images from the same original image. Use the membership distribution of one image as the target distribution of the other image to calculate the alignment loss between the prediction and the target.

[0013] Step 6: Construct a normalized loss to measure the alignment between the cross-correlation matrix C and the identity matrix I in order to control redundancy among variables in the cluster space;

[0014] Step 7: Add the mutual target loss and the unitization loss together to obtain the total loss. Train the network parameters by minimizing the total loss to complete unsupervised image representation learning, giving it good generalization performance for downstream task applications.

[0015] Preferably, in step 1, data augmentation includes multiple methods such as rotation, cropping, adding noise, blurring, and color change; generally, at least three augmentation methods are used. The augmentation methods of group a and group b can be exactly the same, but the augmented images are different, that is, the range of data augmentation values ​​is different, and the range of image changes is also different.

[0016] Preferably, in step 2, X a and X b After preprocessing, the data is input into the encoder. The preprocessing includes data cleaning, adjusting the images to the same size and format, and setting a specific batch size value.

[0017] Preferably, in step 3, for each K-dimensional clustering space, the features... Where W is the K cluster centers = {w1,…,w K Let K be the number of clusters. Applying softmax normalization, we obtain the membership distribution.

[0018] Preferably, in step 4, the cross-correlation matrix between inputs within a batch is obtained based on the cluster space features to measure the correlation between various features in the cluster space, i.e. Where each H is a B n The ×K matrix represents B n Each sample is mapped through the model to obtain K-dimensional clustering space features, B n The number of samples in a batch, specifically, each c kf ∈C is calculated as H a The kth column and H b The correlation coefficient between the f-th column,

[0019] Preferably, in step 5, for any original image i, the membership distribution of its two enhanced images is obtained based on the Siamese network model and cluster centers. and Using the membership distribution of one image as the target distribution of another image, the alignment loss between the prediction and the target is calculated, resulting in the mutual target loss as shown in equation (1).

[0020]

[0021] in, [q] represents the KL divergence, used to calculate the distance between the predicted distribution and the target distribution. sg This indicates that the gradient flow is truncated during the backpropagation of the model. In this case, q is treated as a constant, representing the target distribution of the current image.

[0022] Preferably, in step 6, to reduce redundancy among variables in the clustering space and maximize the capture of category information in the K-dimensional clustering space, the off-diagonal elements of C should be close to 0, and the diagonal elements should be close to 1; by constructing a normalized loss to measure the alignment between the cross-correlation matrix C and the identity matrix I, redundancy among variables in the clustering space is controlled, achieving cluster-level consistency and decorrelation. c represents the diagonal elements of matrix C. kf Let the off-diagonal elements of matrix C be represented by minimizing the normalization loss expressed in equation (2) to reduce redundancy among features in the cluster space.

[0023]

[0024] To adjust the effect of the two parts of the unitized loss on model training, λ is used to control the weight of the inter-cluster decorrelation term.

[0025] Preferably, in step 7, the mutual target loss and the unitized loss are added together as the total loss. The network parameters are trained by minimizing the total loss, so that the model has good generalization performance for downstream task application. The network parameters are trained and the cluster centers are updated using equation (3).

[0026]

[0027] This invention relates to an unsupervised image representation learning method based on decorrelation in cluster space. The method performs two random augmentations on a batch of input images to obtain two batches of data input to a Siamese network. The similarity between the output of the Siamese network and the cluster centers is calculated to represent its feature representation in the cluster space. A cross-correlation matrix between inputs within a batch is obtained based on the cluster space features, and a normalized loss is calculated to measure the alignment between the cross-correlation matrix and the identity matrix. The cluster space features are then softmax normalized to obtain the membership degree of the input to each cluster. The membership degree of one image is used as the target distribution of another related image (two augmentations from the same original image), thereby calculating the alignment loss between the model's prediction of the input and its target distribution. The normalized loss and the alignment loss of the mutual target are added to obtain the total loss used for model training. Minimizing the normalized loss achieves cluster-level consistency and decorrelation, while the mutual target loss achieves sample-level consistency with input perturbations (random augmentations), ultimately achieving effective unsupervised image representation learning under the condition of no negative sample pairs.

[0028] The beneficial effects of this invention are as follows:

[0029] (1) Make full use of the information contained in the data itself for model learning, avoiding the training bottleneck caused by the lack of labeled data;

[0030] (2) Compared with the self-supervised method based on sample comparison, which requires a large number of negative sample pairs or additional storage space to obtain sufficient diversity by directly performing a large number of explicit pairwise feature comparisons, this method uses two enhanced images from the same image as mutual target distributions, which not only reduces the computational cost but also does not require additional storage space to store a sufficient number of negative sample pairs.

[0031] (3) Compared with existing cluster-based cluster balancing algorithms and using cluster membership as pseudo-labels for model training, this method achieves cluster-level consistency and decorrelation by aligning the cross-correlation matrix of cluster space features with the identity matrix, and can still train an effective model even in the case of class imbalance.

[0032] (4) The effectiveness was verified on the CIFAR10 and CIFAR100 datasets. The accuracy on the CIFAR10 validation set reached 87.22%, and the accuracy on the CIFAR100 validation set reached 63.69%. It has good generalization and transferability for downstream classification tasks. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention;

[0034] Figure 2This is a schematic diagram illustrating the clustering consistency and decorrelation-based principles of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to embodiments, but the scope of protection of the present invention is not limited thereto.

[0036] This invention relates to an unsupervised image representation learning method based on decorrelation in cluster space. The method performs two random augmentations on a batch of input images to obtain two batches of data input to a Siamese network. The similarity between the output of the Siamese network and the cluster centers is calculated to represent its feature representation in the cluster space. A cross-correlation matrix between inputs within a batch is obtained based on the cluster space features, and a normalized loss is calculated to measure the alignment between the cross-correlation matrix and the identity matrix. The cluster space features are then softmax normalized to obtain the membership degree of the input to each cluster. The membership degree of one image is used as the target distribution of another related image (two augmentations from the same original image), thereby calculating the alignment loss between the model's prediction of the input and its target distribution. The normalized loss and the alignment loss of the mutual target are added to obtain the total loss used for model training. Minimizing the normalized loss achieves cluster-level consistency and decorrelation, while the mutual target loss achieves sample-level consistency with input perturbations (random augmentations), ultimately achieving effective unsupervised image representation learning under the condition of no negative sample pairs.

[0037] The method includes the following steps:

[0038] Step 1: Randomly select two different data augmentation methods and apply them to the B of the current input batch. n Two different enhancement methods were applied to the image to obtain two sets of enhanced images. and

[0039] In step 1, data augmentation includes multiple methods such as rotation, cropping, adding noise, blurring, and color changes.

[0040] In this embodiment, two publicly available datasets, CIFAR10 and CIFAR100, are used as the experimental datasets. The two CIFAR datasets are 3-channel RGB images with a size of 32×32. CIFAR10 has 10 categories, with 6,000 images for each category, 50,000 of which are training images and 10,000 are test images. CIFAR100 has 100 categories, with 600 images for each category, 50,000 of which are training images and 10,000 are test images.

[0041] In this invention, data augmentation includes, but is not limited to, rotation, cropping, adding noise, blurring, and color changes. Generally, multiple random methods are selected for data augmentation operations.

[0042] Step 2: Based on the Siamese network model structure, X a and X b Input the encoder to obtain the corresponding feature representation. and

[0043] In step 2, X a and X b After preprocessing, the data is input into the encoder. The preprocessing includes data cleaning, adjusting the images to the same size and format, and setting specific batch size values.

[0044] In this embodiment, the CIFAR dataset is preprocessed and input into the network. The preprocessing includes, but is not limited to, data cleaning, adjusting images to the same size and format (i.e., deleting obviously erroneous data due to garbled characters during collection, deleting data with missing or incorrect formats, ensuring that the length and width of each image are equal, uniformly converting image formats to PNG or JPEG, and setting specific batch sizes). The enhanced data is then input into the network to obtain its output Z. a and Z b .

[0045] In this embodiment, after image preprocessing, the images in the CIFAR dataset are uniformly adjusted to 32×32 pixel color images, or all images are cropped to the same size as needed for pre-training the model. These operations can be performed on the dataset using deep learning frameworks such as PyTorch. The torchvision package in PyTorch provides operations such as image cropping and data augmentation (image normalization). After completion, the images are loaded into the DataLoader through the provided framework interface for subsequent training, with a batch size of 128. The ResNet18 network model is selected as the encoder network, and the data in the DataLoader is traversed and input into the encoder to obtain their feature representations.

[0046] Step 3: Based on the K cluster centers, assign each Mapping to the clustering space to obtain the feature representation of the K-dimensional clustering space Using softmax Normalization yields the membership distribution Among them, m={a,b}, i={1,…B n} represents the sequence number of the corresponding input image;

[0047] In step 3, for each K-dimensional clustering space feature Where W is the K cluster centers = {w1,…,wK}, where K is the number of clusters; to leverage the representation learning capabilities of deep neural networks in images, the output of a Siamese network is used. Spatial clustering is performed, and the samples are mapped to each cluster spatial feature based on their distance from the cluster center, B. n The size of the training data in the dataset.

[0048] By z i and each w i Normalizing unit vectors and minimizing their Euclidean distance is equivalent to maximizing their cosine similarity or inner product. Specifically, this is achieved by setting the distance between vectors in ||z|. i ||=1 and ||w i When ||=1, the clustering spatial features can be expressed as: Then, softmax normalization is applied to each row of H to obtain the membership distribution.

[0049] In this embodiment, the dimension of K is set to 1024.

[0050] Step 4: Based on the features of the current batch in the clustering space Calculate the cross-correlation matrix C to measure the correlation between features in the cluster space;

[0051] In step 4, the cross-correlation matrix between inputs within a batch is obtained based on the cluster space features to measure the correlation between various features in the cluster space. Where each H is a B n The ×K matrix represents B n Each sample is mapped through a model to obtain features in a K-dimensional clustering space, B n The number of samples in a batch, specifically, each c kf ∈C is calculated as H a The kth column and H b The correlation coefficient between the f-th column, i.e.

[0052] In this embodiment, the cross-correlation matrix has a dimension of 1024×1024, and each row represents the membership distribution of the current cluster space features in the 1024 cluster centers.

[0053] Step 5: Based on the Siamese network model and cluster centers, obtain the membership distribution of two enhanced images from the same original image. Use the membership distribution of one image as the target distribution of the other image to calculate the alignment loss between the prediction and the target.

[0054] In step 5, for any original image i, the membership distribution of its two enhanced images is obtained based on the Siamese network model and cluster centers. and Using the membership distribution of one image as the target distribution of another image, the alignment loss between the prediction and the target is calculated, resulting in the mutual target loss as shown in equation (1).

[0055]

[0056] in, Let [q] represent the KL divergence. sg This indicates that the gradient flow is truncated during the backpropagation of the model. At this time, q is regarded as a constant, representing the target distribution of the current image. For data distortion caused by random augmentation, the above mutually target loss achieves the consistency of the input perturbation at the sample level.

[0057] In this embodiment, the membership distribution is obtained by normalizing the clustering spatial features using softmax. The dimensions are 128×1024, where 128 is the batch size and 1024 represents the membership dimension. Each data set is used as a target distribution and aligned with each other.

[0058] Step 6: Construct a normalized loss to measure the alignment between the cross-correlation matrix C and the identity matrix I in order to control redundancy among variables in the cluster space;

[0059] In step 6, to reduce redundancy among variables in the clustering space and maximize the capture of category information in the K-dimensional clustering space, the off-diagonal elements of C should be close to 0, and the diagonal elements should be close to 1. A normalized loss is constructed to measure the alignment between the cross-correlation matrix C and the identity matrix I, thereby controlling redundancy among variables in the clustering space and achieving cluster-level consistency and decorrelation. c represents the diagonal elements of matrix C. kf Let the off-diagonal elements of matrix C be represented by minimizing the normalization loss expressed in equation (2) to reduce redundancy among features in the cluster space.

[0060]

[0061] To adjust the effect of the two parts of the unitized loss on model training, λ is used to control the weight of the inter-cluster decorrelation term. In this embodiment, λ is set to 1.

[0062] Step 7: Add the mutual target loss and the unitization loss together to obtain the total loss. Train the network parameters by minimizing the total loss to complete unsupervised image representation learning, giving it good generalization performance for downstream task applications.

[0063] In step 7, the mutual target loss and the unitized loss are added together as the total loss, and the network parameters are trained and the cluster centers are updated using equation (3).

[0064]

[0065] In this invention, the mutual target loss and the unitized loss are added together as the total loss. The network parameters are trained and the cluster centers are updated by minimizing the total loss. The backbone network model is pre-trained using the stochastic gradient descent optimizer provided by PyTorch. The model is trained through 300 iterations, and the model parameters are retained for 100, 200 and 300 iterations respectively for subsequent testing of the pre-trained model performance.

[0066] A ResNet18 neural network was trained on the CIFAR10 dataset as a pre-trained model. A three-layer 1024×1024-dimensional multilayer perceptron with randomized parameters was used as the prediction network of the model. Layer-wise Adaptive Rate Scaling (LARS) was used for optimization, with weight decay set to 0.00001. The training lasted for 300 epochs. The initial learning rate was 0.2, which decayed to 0 with cosine decay over epochs.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0072] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An unsupervised image representation learning method based on cluster space decorrelation, characterized in that: The method comprises the following steps: Step 1: Randomly select two different data augmentation methods, and perform two different augmentations on the current input batch of B n images to obtain two sets of augmented images and Step 2: Based on the twin network model structure, X a and X b are input into the encoder to obtain the corresponding feature representation and Step 3: Based on K cluster centers, each is mapped to the cluster space to obtain a feature representation of K-dimensional cluster space is obtained by using softmax is normalized to obtain a membership distribution where m = {a, b}, i = {1, …B n is the serial number corresponding to the input image; Step 4: Features based on current batch in cluster space Compute cross-correlation matrix C, which measures the correlation between the features in the cluster space. Step 5: obtaining the membership distribution of two enhanced images from the same original image according to the twin network model and the cluster center, taking the membership distribution of one image as the target distribution of the other image to calculate the alignment loss between the prediction and the target; Step 6: constructing a unitization loss to measure the alignment degree between the mutual correlation matrix C and the unit matrix I to control the redundancy between variables in the clustering space; Step 7: adding the mutual target loss and the unitization loss as the total loss, training the network parameters by minimizing the total loss, and completing the unsupervised image representation learning. 2.The unsupervised image representation learning method based on cluster space decorrelation according to claim 1, characterized in that: In the step 1, the data enhancement includes multiple kinds of rotation, clipping, adding noise, blurring and color change. 3.The unsupervised image representation learning method based on cluster space decorrelation according to claim 1, characterized in that: In step 2, X a and X b The pre-processing includes data cleaning, adjusting images to the same size and uniform format, and setting a specific batch size value.

4. The unsupervised image representation learning method based on cluster space decorrelation according to claim 1, characterized in that: In step 3, for each K-dimensional cluster space feature where W is the K cluster centers = {w1,..., wK}, K is the number of clusters, and the membership distribution is obtained by applying softmax normalization K}, K is the number of clusters, and the membership distribution is obtained by applying softmax normalization 5. The unsupervised image representation learning method based on cluster space decorrelation according to claim 1, characterized in that: In step 4, the correlation between each feature in the cluster space is measured based on the inter-correlation matrix between inputs in a batch, i.e. where each H is a B n x K matrix, representing the K-dimensional cluster space features of B n samples mapped by the model, B n is the number of samples in a batch, and each c kf ∈ C is calculated as the correlation coefficient between the k-th column of H a and the f-th column of H b .

6. The unsupervised image representation learning method based on cluster space decorrelation according to claim 1, characterized in that: In step 5, for any one original image i, the membership distribution of its two enhanced images is obtained according to the twin network model and the cluster center and The alignment loss between the prediction and the target is calculated by taking the membership distribution of one image as the target distribution of the other image, and the mutual target loss is formula (1) where, denotes the KL divergence, used to compute the distance between the predictive distribution and the target distribution, [q] sg denotes the truncation of the gradient flow during backpropagation of the model, where q is treated as a constant, denotes the target distribution of the current image.

7. The unsupervised image representation learning method based on cluster space decorrelation according to claim 6, characterized in that: In step 6, denotes the diagonal elements of the matrix C, c kf denotes the off-diagonal elements of the matrix C, to reduce the redundancy between the features of the clustering space by minimizing the unitization loss expressed by equation (2), In order to adjust the effect of the two parts of the unitization loss on the model training, the weight of the inter-cluster decorrelation term is controlled by λ.

8. The unsupervised image representation learning method based on cluster space decorrelation according to claim 7, characterized in that: In the step 7, the mutual target loss and the unitization loss are combined to train the network parameters and update the cluster center according to the formula (3),

Images