A deep image clustering method, system, device, medium and terminal

Abstract

The present invention belongs to the technical field of unsupervised image recognition, and discloses a deep image clustering method, system, device, medium and terminal, wherein the deep image clustering method comprises: inputting a group of pictures, obtaining multiple sets of codes for each picture through an encoder in a SIMO autoencoder; inputting the multiple sets of codes for each picture and the codes after information fusion of these codes into the same decoder to obtain reconstructed pictures; calculating the reconstruction loss of the reconstructed pictures and the pictures obtained after data enhancement of the input pictures, and performing pre-training of the network; after the pre-training is completed, clustering the fused codes to obtain the comparative clustering loss, adding it into the training process for joint training, and realizing parameter optimization; after the joint training is completed, feature extraction is performed on the entire data set, and the fused codes are obtained and input into the traditional clustering algorithm to obtain the clustering result. The present invention can simultaneously extract semantic information from multiple angles of a single image, thereby increasing the richness of information.

Description

Technical Field

[0001] The present invention belongs to the technical field of unsupervised image recognition, and in particular relates to a deep image clustering method, system, device, medium and terminal. Background Art

[0002] At present, with the rapid advent of the big data era, hundreds of millions of image data are generated on the Internet every day. If we can fully utilize and mine the information of massive image data, it will be able to generate extremely high commercial value. For example, image recognition is a specific application scenario for mining the value of images, but most traditional data mining algorithms cannot directly process image data, so people often manually calculate various features of the image and then use these algorithms for further recognition.

[0003] In recent years, with the development of computer vision technology, image recognition technology based on deep learning has become extremely mature, especially in the case of image annotation. In 2015, the recognition accuracy of deep learning methods for images has exceeded that of humans. In recent years, it has been widely used in various fields, such as face recognition access control systems in schools, companies, hospitals and other institutions, and vehicle body recognition and lane line recognition systems in traffic scenes, and has achieved relatively good results. In addition to image annotation, there are also some methods that can recognize images without image annotation. Generally, deep learning is used to extract low-dimensional features of images and then traditional unsupervised machine learning methods are used for recognition. Specifically, the autoencoder structure in the deep neural network is used to extract features under the optimization of reconstruction loss and clustering loss, and then KMeans or GMM algorithm is used for clustering. The network structures involved in the former include denoising autoencoder, variational autoencoder, adversarial autoencoder, etc., and the loss functions include KMeansLoss, Cluster assignment hardening loss, Locality-preserving loss, etc.

[0004] At present, the general steps of the technology based on unsupervised image recognition are to first use a certain autoencoder structure to extract image features, such as denoising autoencoders, variational autoencoders, etc. In addition, a certain clustering loss based on Euclidean distance will be added to guide the network to output codes that are conducive to clustering. However, on the one hand, these structures at this stage only compress the image into a set of codes. From a physical point of view, this set of codes can only represent the information of a certain angle of the image, which means that a lot of information is lost in the process of feature extraction. On the other hand, the current significant assumption of clustering loss based on Euclidean distance is that the output of the code needs to be separable in Euclidean space, but neural networks are often nonlinear, so this assumption is difficult to meet, and it is difficult to guarantee whether the clustering effect can be significantly improved after adding these losses. Therefore, a new deep image clustering method and system is urgently needed.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) In the existing technologies based on unsupervised image recognition, the current structures simply compress the image into a set of codes. From a physical point of view, this set of codes can only represent the information of a certain angle of the image, which means that a large amount of information is lost in the process of feature extraction.

[0007] (2) The current clustering loss based on Euclidean distance has a significant assumption that the output of the code needs to be separable in Euclidean space. However, neural networks are often nonlinear, so this assumption is difficult to meet. It is also difficult to guarantee whether the addition of these losses can significantly improve the clustering effect.

[0008] The difficulty of solving the above problems and defects is as follows: First, for (1), the difficulty lies in how to fully mine and extract multiple sets of semantic information of the image to help clustering without significantly increasing the number of model parameters and training time. For (2), the difficulty lies in how to use a clustering loss to ensure the improvement of clustering effect when the encoding does not meet the assumption of Euclidean space distribution.

[0009] The significance of solving the above problems and defects is: if (1) can be solved, in addition to the clustering problem, the greater significance lies in providing a new paradigm for autoencoder to compress image encoding and retain and highlight the extremely valuable semantic information of the image. In addition to clustering, it can also be used for other tasks, and in addition to images, it can also be used for other types of data (such as text, speech, etc.). If (2) can be solved, in fact, the encoding can still be separable in non-Euclidean space, then it can be applied to more data sets. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides a deep image clustering method, system, device, medium and terminal, and more particularly, relates to a deep image clustering method, system, device, medium and terminal based on a single-input multiple-output (SIMO) autoencoder and contrastive clustering loss.

[0011] The present invention is implemented in this way: a deep image clustering method, the deep image clustering method comprising the following steps:

[0012] Step 1: Input a group of pictures and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder. The purpose of this step is to extract multiple sets of codes for a picture at one time in a faster way while adding very few parameters, which increases the richness of the codes and reduces the time cost of actual training and inference.

[0013] In step 2, multiple sets of codes for each image and the codes after information fusion of each set of codes are input into the same decoder to obtain the reconstructed image; the purpose of this step is to make each set of codes obtained in step 1 have interpretable physical information, that is, although the semantic information of these codes is different, the clustering information is similar.

[0014] In step three, the reconstruction loss is calculated for the reconstructed image and the image obtained after data enhancement of the input image, and the network is pre-trained. In order to make full use of the existing autoencoder theoretical assumptions, this step uses the reconstruction loss of the image and the reconstructed image to optimize the network parameters. The data enhancement of the image is used to ensure that the clustering information of these codes mentioned in step two is the same.

[0015] Step 4: After the pre-training is completed, cluster the fusion code to obtain the contrast clustering loss and add it to the training process for joint training to achieve parameter optimization; the purpose of this step is to make the fusion code extracted from the network have richer clustering information and make the code cluster-friendly.

[0016] Step 5: After the joint training is completed, feature extraction is performed on the entire data set to obtain a fusion code that is input into the traditional clustering algorithm to obtain the clustering result. The purpose of this step is to obtain an encoder with optimal parameters suitable for the original data distribution after the joint training is completed, so that information extraction and clustering of unknown data can be performed.

[0017] Further, in step 1, a group of pictures are input, and multiple groups of codes for each picture are obtained by the encoder in the SIMO autoencoder, including:

[0018] Assume that a picture is x, and its size is W×H×C, where W, H, and C refer to the width, height, and channel of the picture respectively. For example, for grayscale pictures, C=1, and for color pictures, C=3. The autoencoder network structure is generally divided into two parts. Assuming that the first part is f(·), a picture is encoded to get z i =f(x i ); For SIMO autoencoder, each picture generates multiple codes, namely:

[0019] z1,z2,…,z m =f(x);

[0020] Among them, m is an adjustable parameter, which means that m groups of codes are generated for each picture.

[0021] Convolutional autoencoders are constructed using convolutional blocks, where each convolutional block consists of a continuous convolution kernel of size 3×3×c in ×c out , a convolutional layer with a step size of 1, a BN layer, and a leakyReLU activation function. After two consecutive convolutional blocks, a maximum pooling layer with a step size of 2 is used to reduce the dimension of the feature map. After continuous stacking, assuming that the size of the feature map is W l ×H l ×C l , the feature map is flattened, that is, the feature map is changed from a 3-dimensional tensor to a 1-dimensional vector. At this time, the vector is passed through the fully connected layers of different branches to obtain the final multiple groups of codes.

[0022] Furthermore, in step 2, the multiple sets of codes for each picture and the codes after information fusion of each set of codes are input into the same decoder to obtain the reconstructed picture, including:

[0023] Let each group of codes complete their own reconstruction tasks, so that these codes have different physical meanings; perform weighted average information fusion on these codes to obtain Δz, that is, to obtain m+1 codes; input the m+1 codes into the same decoder, where the decoder is basically symmetrical with the encoder, and first maps the codes back to a dimension of W by the fully connected layer. l ×H l ×C l The vector is then reshaped to obtain the three-dimensional feature map, and then upsampled and convolved to obtain a reconstructed image of the same size as the encoder input image.

[0024] Furthermore, in step 3, the reconstructed image and the image obtained after data enhancement of the input image are used to calculate the reconstruction loss for network pre-training, including:

[0025] After obtaining the decoded reconstructed image, use the data enhancement of the input image, including rotation, flipping and random cropping operations, to obtain m+1 enhanced images that correspond one to one with the encoding:

[0026] aug1, aug2, ..., aug m =random_aug(x);

[0027] Calculate the reconstruction loss of the decoded reconstructed image and the enhanced image:

[0028]

[0029] Where l(·,·) is a similarity metric function; the m+1th reconstructed image It is obtained by fusion coding Δz, so the reconstruction loss of x is calculated and added to the final loss function for training. The Adam optimizer is used during training, the learning rate is 0.001, the batch_size is 512, and there is no learning rate decay strategy.

[0030] Further, in step 4, after the pre-training is completed, the fusion code is clustered to obtain the comparison clustering loss and added to the training process for joint training to optimize the parameters, including:

[0031] After the pre-training is completed, a valuable code Δz is obtained. By referring to the idea of ​​contrastive learning, the positive and negative sample pairs are replaced with the cluster center of the code and the class to which it belongs, and the cluster center of the code and the class to which it does not belong is used to obtain the contrastive clustering loss function, including:

[0032] For a given batch, get its fusion code z batch , for z batch Clustering is performed to obtain cluster centers c1, c2, ..., c k , we get z batch Each z i The corresponding cluster label y i , then the contrast clustering loss is:

[0033]

[0034] Among them, cos(·,·) is the cosine similarity; therefore, during joint training, the total loss function is:

[0035] L=L r +αL c ;

[0036] Among them, the hyperparameter α is a balance parameter and can be adjusted independently.

[0037] Furthermore, in step 5, after the joint training is completed, feature extraction is performed on the entire data set to obtain a fusion code which is input into a traditional clustering algorithm to obtain a clustering result, including:

[0038] After the joint training is completed, for the entire data set X, the information fused Δz is obtained through the encoder. At this time, Δz is a two-dimensional matrix similar to traditional table data, so the traditional KMeans or GMM algorithm is used for clustering to obtain the final clustering label matrix Y; for new samples, the trained KMeans or GMM algorithm can continue to be used for label discrimination, or supervised learning can be performed on X and Y to obtain a classifier for label discrimination of test samples.

[0039] Another object of the present invention is to provide a deep image clustering system applying the deep image clustering method, the deep image clustering system comprising:

[0040] The picture encoding acquisition module is used to input a group of pictures and obtain multiple groups of codes for each picture through the encoder in the SIMO autoencoder;

[0041] An information fusion and reconstruction module is used to input multiple sets of codes of each picture and the codes after information fusion of each set of codes into the same decoder to obtain a reconstructed picture;

[0042] The network pre-training module is used to pre-train the network by calculating the reconstruction loss of the reconstructed image and the image obtained after data enhancement of the input image;

[0043] The joint training module is used to cluster the fusion code after the pre-training is completed, obtain the comparative clustering loss, and add it into the training process for joint training to achieve parameter optimization;

[0044] The feature extraction and clustering module is used to extract features from the entire data set after the joint training is completed, and the fusion code is input into the traditional clustering algorithm to obtain the clustering result.

[0045] Another object of the present invention is to provide a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0046] Input a group of pictures, and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder; input the multiple sets of codes for each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed pictures; calculate the reconstruction loss of the reconstructed pictures and the pictures obtained after data enhancement of the input pictures, and perform pre-training of the network;

[0047] After the pre-training is completed, the fusion code is clustered to obtain the contrast clustering loss and added to the training process for joint training to achieve parameter optimization. After the joint training is completed, feature extraction is performed on the entire data set to obtain the fusion code, which is input into the traditional clustering algorithm to obtain the clustering result.

[0048] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0049] Input a group of pictures, and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder; input the multiple sets of codes for each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed pictures; calculate the reconstruction loss of the reconstructed pictures and the pictures obtained after data enhancement of the input pictures, and perform pre-training of the network;

[0050] After the pre-training is completed, the fusion code is clustered to obtain the contrast clustering loss and added to the training process for joint training to achieve parameter optimization. After the joint training is completed, feature extraction is performed on the entire data set to obtain the fusion code, which is input into the traditional clustering algorithm to obtain the clustering result.

[0051] Another object of the present invention is to provide an information data processing terminal, which is used to implement the deep image clustering system.

[0052] Combined with all the above technical solutions, the advantages and positive effects of the present invention are as follows: the deep image clustering method provided by the present invention can extract semantic information from multiple angles of a single image at the same time without significantly increasing the amount of calculation, thereby increasing the richness of information. Due to limited knowledge, there is no precedent for such a practice in the industry. Therefore, in addition to being used for clustering, the present invention actually provides a new paradigm for an autoencoder to extract image semantic information in theory, that is, it is not limited to the reconstruction mapping of a single image to a single image of a traditional autoencoder, but can completely construct a reconstruction mapping of a single image to multiple images, and fully mine the semantic information of a single image in a greedy manner for other tasks. In addition to image data, it is also effective for other types of data (such as text, speech) in theory. In addition, the contrast clustering loss based on cosine similarity does not require encoding linear separable constraints, can be applied to more data sets, and its physical meaning is more obvious.

[0053] The present invention proposes to use a network structure different from the general autoencoder in the process of extracting image features. In particular, the present invention increases the number of branches after the encoder while keeping other parameters shared, so that the network has the characteristics of single input and multiple output. The present invention proposes that in addition to the reconstruction loss, during batch training, the batch data is clustered and combined with contrastive learning to obtain contrastive clustering loss, and joint training is performed to ensure that the features have rich clustering information, further improving the clustering accuracy.

[0054] The present invention proposes to use the proposed SIMO autoencoder in the feature extraction step, and the obtained encoding has richer information and is more conducive to clustering than the encoding obtained by any encoder involved in the third. The present invention proposes not to use the clustering loss based on Euclidean distance in the third, but to use the contrast clustering loss based on cosine similarity, which no longer requires a strong linear separable assumption and has a strong physical interpretable meaning. Combined with the proposed autoencoder, it can further improve and stabilize the clustering effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a flow chart of a deep image clustering method provided by an embodiment of the present invention.

[0057] Figure 2 It is an overall block diagram of the degree image clustering method provided by an embodiment of the present invention.

[0058] Figure 3 It is a principle diagram of a deep image clustering method provided by an embodiment of the present invention.

[0059] Figure 4 is a structural block diagram of a deep image clustering system provided by an embodiment of the present invention;

[0060] In the figure: 1. Image coding acquisition module; 2. Information fusion and reconstruction module; 3. Network pre-training module; 4. Joint training module; 5. Feature extraction and clustering module.

[0061] Figure 5 and Figure 6 This is an example diagram of a network structure when the output code is 3 provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail in conjunction with the following embodiments. It is shown that the input of the present invention is any set of unlabeled picture data sets, and the output after the model is the discriminative information encoding of these pictures, so that the obtained encoding can be flexibly used to support different downstream tasks. For example, if a clustering algorithm is used, clustering labels are obtained. If the original picture data has all or part of the labels, supervised and semi-supervised learning can be performed to obtain the labels of unknown pictures. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0063] In view of the problems existing in the prior art, the present invention provides a deep image clustering method, system, device, medium and terminal. The present invention is described in detail below with reference to the accompanying drawings.

[0064] like Figure 1 As shown, the depth image clustering method provided by the embodiment of the present invention includes the following steps:

[0065] S101, input a group of pictures, normalize all pixels of the pictures to the range of [0,1], and then obtain multiple groups of codes for each picture through the encoder in the SIMO autoencoder;

[0066] S102, inputting the multiple groups of codes of each picture and the codes after information fusion of each group of codes into the same decoder to obtain a reconstructed picture;

[0067] S103, calculating the reconstruction loss of the reconstructed image and the image obtained after data enhancement of the input image, and performing network pre-training;

[0068] S104, after the pre-training is completed, clustering the fusion code to obtain the contrast clustering loss and adding it into the training process for joint training to achieve parameter optimization;

[0069] S105, after the joint training is completed, feature extraction is performed on the entire data set to obtain a fusion code which is input into a traditional clustering algorithm to obtain a clustering result.

[0070] The overall block diagram of the deep image clustering method provided by the embodiment of the present invention is as follows: Figure 2 shown.

[0071] like Figure 4 As shown, the deep image clustering system provided by the embodiment of the present invention includes:

[0072] The picture encoding acquisition module 1 is used to input a group of pictures and obtain multiple groups of codes for each picture through the encoder in the SIMO autoencoder;

[0073] The information fusion and reconstruction module 2 is used to input the multiple sets of codes of each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed picture;

[0074] The network pre-training module 3 is used to calculate the reconstruction loss of the reconstructed image and the image obtained after data enhancement of the input image to perform network pre-training;

[0075] Joint training module 4 is used to cluster the fusion code after the pre-training is completed to obtain the comparison clustering loss and add it into the training process for joint training to achieve parameter optimization;

[0076] The feature extraction and clustering module 5 is used to extract features from the entire data set after the joint training is completed, and obtain a fusion code that is input into a traditional clustering algorithm to obtain a clustering result.

[0077] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0078] refer to Figure 3 The algorithm flow chart and Figure 5 The network structure example diagram is shown in FIG. , and the specific steps of the present invention are as follows.

[0079] S1. Input a set of pictures and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder.

[0080] Assume that a picture is x, and its size is W×H×C, where W, H, and C refer to the width, height, and channel of the picture respectively. For example, for grayscale pictures, C=1, and for color pictures, C=3. Secondly, the autoencoder network structure is generally divided into two parts. Assuming that the first part is f(·), a picture can be encoded to get z i =f(x i ), and for the SIMO autoencoder of the present invention, each picture can generate multiple codes, namely:

[0081] z1,z2,…,z m =f(x)

[0082] Among them, m is an adjustable parameter, which means that m sets of codes are generated for each picture. Because the convolution network can have a better feature extraction effect for images, the present invention also uses convolution blocks to construct convolutional autoencoders, where each convolution block consists of a continuous convolution kernel size of 3×3×c in ×c oxt , a convolutional layer with a step size of 1, a BN layer, and a leakyReLU activation function. Generally, after two consecutive convolutional blocks, a maximum pooling layer with a step size of 2 is used to reduce the dimension of the feature map. After continuous stacking, it is assumed that the size of the feature map is W l ×Hl ×C l Because the code needs to be input into the traditional clustering method, the present invention flattens the feature map here, so that the feature map is converted from a 3-dimensional tensor to a 1-dimensional vector. At this time, the vector is passed through the fully connected layers of different branches to obtain the final multiple groups of codes.

[0083] S2. Input the multiple sets of codes for each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed pictures.

[0084] At this time, although the present invention obtains different codes, the present invention needs to let each group of codes complete their own reconstruction tasks to make these codes have different physical meanings. Secondly, in order not to increase the amount of calculation during clustering, the present invention performs weighted average information fusion on these codes to obtain Δz. Therefore, in fact, the present invention obtains m+1 codes at this time. Then, in order not to increase the number of parameters of the network structure, these m+1 codes are input into the same decoder. The decoder is basically symmetrical with the encoder. The fully connected layer first maps the code back to a dimension size of W. l ×H l ×C l The vector is then reshaped to obtain the three-dimensional feature map, and then upsampled and convolved to obtain a reconstructed image of the same size as the encoder input image.

[0085] S3. Calculate the reconstruction loss of the reconstructed image and the image obtained after data enhancement of the input image to pre-train the network.

[0086] After obtaining the decoded reconstructed picture, in order to ensure that different codes have different physical meanings, the present invention uses data enhancement of the input picture, including rotation, flipping, random cropping and other operations, to obtain m+1 enhanced pictures corresponding to the codes one by one:

[0087] aug1, aug2, ..., aug m =random_aug(x)

[0088] At this time, the reconstruction loss of the decoded reconstructed image and the enhanced image is calculated:

[0089]

[0090] Among them, l(·,·) is a similarity metric function. It is worth noting that the m+1th reconstructed image It is obtained by fusion coding Δz, so the reconstruction loss of it and x is calculated and added to the final loss function for training. The Adam optimizer is used during training, the learning rate is 0.001, the batch_size is 512, and there is no learning rate decay strategy.

[0091] S4. After the pre-training is completed, cluster the fusion code to obtain the contrast clustering loss and add it into the training process for joint training and parameter optimization.

[0092] After the pre-training is completed, the present invention obtains a valuable code Δz. At this time, the accuracy of clustering on some data sets is comparable to and exceeds the accuracy that can be achieved by most clustering loss technologies, but the accuracy curve is not stable enough. Therefore, the present invention draws on the idea of ​​contrastive learning and replaces the positive and negative sample pairs with the code and the cluster center of the class to which it belongs, and the code and the cluster center of the class to which it does not belong to obtain a contrastive clustering loss function. The specific steps are as follows: for a given batch, the present invention obtains its fusion code z batch , for z batch Clustering is performed to obtain cluster centers c1, c2, ..., c k , and get z batch Each z i The corresponding cluster label y i , then the contrast clustering loss is:

[0093]

[0094] Where cos(·,·) is the cosine similarity. So during joint training, the total loss function is:

[0095] L=L r +αL c

[0096] The hyperparameter α is a balance parameter and can be adjusted independently.

[0097] S5. After the joint training is completed, feature extraction is performed on the entire data set to obtain a fusion code which is input into the traditional clustering algorithm to obtain the clustering result.

[0098] After the joint training is completed, for the entire data set X, the information fused Δz is obtained through the encoder. At this time, Δz is a two-dimensional matrix similar to traditional table data. Therefore, the traditional KMeans or GMM algorithm can be used for clustering to obtain the final clustering label matrix Y. Then, for new samples, the trained KMeans or GMM algorithm can continue to be used for label discrimination, or supervised learning can be performed on X and Y to obtain a classifier for label discrimination of test samples.

[0099] The present invention proposes to use a network structure different from the general autoencoder in the process of extracting image features. In particular, the present invention increases the number of branches after the encoder while keeping other parameters shared, so that the network has the characteristics of single input and multiple output. The present invention proposes that in addition to the reconstruction loss, during batch training, the batch data is clustered and combined with contrastive learning to obtain contrastive clustering loss, and joint training is performed to ensure that the features have rich clustering information, further improving the clustering accuracy.

[0100] The present invention proposes to use the proposed SIMO autoencoder in the feature extraction step, and the obtained encoding has richer information and is more conducive to clustering than the encoding obtained by any encoder involved in the third. The present invention proposes not to use the clustering loss based on Euclidean distance in the third, but to use the contrast clustering loss based on cosine similarity, which no longer requires a strong linear separable assumption and has a strong physical interpretable meaning. Combined with the proposed autoencoder, it can further improve and stabilize the clustering effect.

[0101] The actual effects of the present invention compared with the existing technical solutions have been compared on several data sets, see Table 1.

[0102] Table 1. Comparison of experimental effects of the present invention and other solutions

[0103]

[0104] The present invention uses the common indicators ACC and NMI for measuring clustering effects to demonstrate the advantages of the proposed technical solution. First, it can be seen that when only the proposed SIMO encoder is used without clustering loss, the method of the present invention has surpassed most of the solutions using clustering loss. Secondly, after adding the proposed clustering loss, it obtains the best performance comparable to the comparison schemes on the USPS and MNIST datasets, while on the Fashin-Mnist and Kannada-Mnist datasets, the present invention achieves comprehensive leading advantages, especially in the ACC indicator, which is more than 10% ahead.

[0105] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0106] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A deep image clustering method, characterized in that: The deep image clustering method comprises the following steps: Step 1: Input a group of pictures, and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder; Step 2: Input the multiple sets of codes of each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed picture; Step 3: Calculate the reconstruction loss of the reconstructed image and the image obtained after data enhancement of the input image, and perform network pre-training; Step 4: After the pre-training is completed, cluster the fusion code to obtain the contrast clustering loss and add it into the training process for joint training to achieve parameter optimization; Step 5: After the joint training is completed, feature extraction is performed on the entire data set to obtain a fusion code that is input into a traditional clustering algorithm to obtain a clustering result. In step 1, a group of pictures are input, and multiple groups of codes for each picture are obtained through the encoder in the SIMO autoencoder, including: Assume that a picture is x, and its size is W×H×C, where W, H, and C refer to the width, height, and channel of the picture respectively. For example, for grayscale pictures, C=1, and for color pictures, C=3. The autoencoder network structure is generally divided into two parts. Assuming that the first part is f(·), a picture is encoded to obtain z=f(x). For SIMO autoencoders, each picture generates multiple codes, namely: z1,z2,...,z m =f(x); Among them, m is an adjustable parameter, which means that m sets of codes are generated for each picture; Convolutional autoencoders are constructed using convolutional blocks, where each convolutional block consists of a continuous convolution kernel of size 3×3×c in ×c out , a convolutional layer with a step size of 1, a BN layer, and a leakyReLU activation function. After two consecutive convolutional blocks, a maximum pooling layer with a step size of 2 is used to reduce the dimension of the feature map. After continuous stacking, assuming that the size of the feature map is W l ×H l ×C l , the feature map is flattened, that is, the feature map is changed from a 3-dimensional tensor to a 1-dimensional vector. At this time, the vector is passed through the fully connected layers of different branches to obtain the final multiple groups of codes.

2. The deep image clustering method according to claim 1, characterized in that: In step 2, the multiple sets of codes for each picture and the codes after information fusion of each set of codes are input into the same decoder to obtain the reconstructed picture, including: Let each group of codes complete their own reconstruction tasks, so that these codes have different physical meanings; perform weighted average information fusion on these m groups of codes to obtain Δz, that is, to obtain m+1 codes; the m+1 codes are input into the same decoder, where the decoder is basically symmetrical with the encoder, and the fully connected layer first maps the codes back to a dimension size of W l ×H l ×C l The vector is then reshaped to obtain the three-dimensional feature map, and then upsampled and convolved to obtain a reconstructed image of the same size as the encoder input image.

3. The deep image clustering method according to claim 1, characterized in that: In step 3, the reconstruction loss is calculated for the reconstructed image and the image obtained after data augmentation of the input image to pre-train the network, including: After obtaining the decoded reconstructed image, use the data enhancement of the input image, including rotation, flipping and random cropping image data enhancement operations, to obtain m+1 enhanced images corresponding to the encoding one by one: aug1,aug2,...,aug m =random_aug(x); Calculate the reconstruction loss of the decoded reconstructed image and the enhanced image: Where l(·,·) is a similarity metric function; the m+1th reconstructed image It is obtained by fusion coding Δz, so the reconstruction loss of x is calculated and added to the loss function for training. The Adam optimizer is used during training, the learning rate is 0.001, the batch_size is 512, and there is no learning rate decay strategy.

4. The deep image clustering method according to claim 1, characterized in that: In step 4, after the pre-training is completed, the fusion code is clustered to obtain the contrast clustering loss and added to the training process for joint training and parameter optimization, including: After the pre-training is completed, a valuable code Δx is obtained. By referring to the idea of ​​contrastive learning, the positive and negative sample pairs are replaced with the cluster center of the code and the class to which it belongs, and the cluster center of the code and the class to which it does not belong is used to obtain the contrastive clustering loss function, including: For a given batch, get its fusion code z batch , for z batch Clustering is performed to obtain cluster centers c1, c2, ..., c k , we get z batch Each z i The corresponding cluster label y i , then the contrast clustering loss is: Among them, cos(·,·) is the cosine similarity; therefore, during joint training, the total loss function is: L=L r +αL c ; Among them, the hyperparameter α is a balance parameter and can be adjusted independently.

5. The deep image clustering method according to claim 1, characterized in that: In step 5, after the joint training is completed, feature extraction is performed on the entire data set to obtain a fusion code which is input into a traditional clustering algorithm to obtain a clustering result, including: After the joint training is completed, for the entire data set X, the information fused Δz is obtained through the encoder. At this time, Δz is a two-dimensional matrix similar to traditional table data, so the traditional KMeans or GMM algorithm is used for clustering to obtain the final clustering label matrix Y; for new samples, the trained KMeans or GMM algorithm can continue to be used for label discrimination, or supervised learning can be performed on X and Y to obtain a classifier for label discrimination of test samples.

6. A deep image clustering system implementing the deep image clustering method according to any one of claims 1 to 5, characterized in that: The deep image clustering system comprises: The picture encoding acquisition module is used to input a group of pictures and obtain multiple groups of codes for each picture through the encoder in the SIMO autoencoder; An information fusion and reconstruction module is used to input multiple sets of codes of each picture and the codes after information fusion of each set of codes into the same decoder to obtain a reconstructed picture; The network pre-training module is used to pre-train the network by calculating the reconstruction loss of the reconstructed image and the image obtained after data enhancement of the input image; The joint training module is used to cluster the fusion code after the pre-training is completed to obtain the comparative clustering loss and add it into the training process for joint training to achieve parameter optimization; The feature extraction and clustering module is used to extract features from the entire data set after the joint training is completed, and the fusion code is input into the traditional clustering algorithm to obtain the clustering result.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the deep image clustering method according to any one of claims 1 to 5, including the following steps: Input a group of pictures, and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder; input the multiple sets of codes for each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed pictures; calculate the reconstruction loss of the reconstructed pictures and the pictures obtained after data enhancement of the input pictures, and perform pre-training of the network; After the pre-training is completed, the fusion code is clustered to obtain the contrast clustering loss and added to the training process for joint training to achieve parameter optimization. After the joint training is completed, feature extraction is performed on the entire data set to obtain the fusion code, which is input into the traditional clustering algorithm to obtain the clustering result.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the deep image clustering method according to any one of claims 1 to 5, comprising the following steps: Input a group of pictures, and obtain multiple sets of codes for each picture through the encoder in the SIMO autoencoder; input the multiple sets of codes for each picture and the codes after information fusion of each set of codes into the same decoder to obtain the reconstructed pictures; calculate the reconstruction loss of the reconstructed pictures and the pictures obtained after data enhancement of the input pictures, and perform pre-training of the network; After the pre-training is completed, the fusion code is clustered to obtain the contrast clustering loss and added to the training process for joint training to achieve parameter optimization. After the joint training is completed, feature extraction is performed on the entire data set to obtain the fusion code, which is input into the traditional clustering algorithm to obtain the clustering result.

9. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the deep image clustering system as described in claim 6.

Images