An object view distillation method for general multi-view object clustering

Abstract

An object view distillation method for general multi-view object clustering, collecting initial multi-view data of several samples; constructing an autoencoder, a teacher network, a student network and knowledge distillation; the encoder projects the views of the samples into potential representations and constructs a low-dimensional latent space; training the teacher network using multi-view data; training the student network using multi-view data, and using the dark knowledge generated by knowledge distillation as a new self-supervisory signal to guide the student network to perform fine-tuning; feeding the entire original multi-view data set to the overall network, obtaining the probability distribution of all view clustering through the student network, weighting and summing the probability of each view, and obtaining the final clustering result; the present invention uses the knowledge distillation method to solve the problem of pseudo-labels in multi-view clustering algorithms misleading model training and correcting inaccurate features, thereby significantly improving clustering performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and image clustering, and in particular relates to an object view distillation method for general multi-view object clustering. Background Art

[0002] Most of the data in the real world are collected from different sensors or obtained from different feature extractors. If we make full use of the different modalities of the data or the different viewpoints of the image, we can better build a visual model for the data to achieve the purpose of analysis or clustering. Multi-view clustering is a multi-stage clustering method that aims to classify visual objects into different clusters to improve the effectiveness of the model and benefit subsequent tasks such as object detection and action recognition. To achieve this goal, it is crucial to explore the common semantics between different views and make full use of the pseudo-labels obtained through self-supervised learning. However, multi-view clustering has some shortcomings and limitations when applied to multi-modal or multi-view. In fact, samples from different views contain more features, and their distributions are perturbed by noise points and missing data. If the linear separation of traditional representations is used as pseudo-labels, it will lead to overconfidence (i.e., low entropy prediction) in creating pseudo-labels, which in turn misleads model training and ultimately leads to inaccurate clustering. Therefore, avoiding the destructive influence of pseudo-labels and correcting inaccurate features during feature learning is a key task in multi-view clustering.

[0003] In order to solve the pseudo-label problem in clustering, multi-stage deep multi-view clustering algorithms have attracted more and more attention, but their performance is limited due to the following shortcomings: 1) Although pseudo-labels can provide clear indicators for self-supervised learning, when multi-view instances are represented by pseudo-labels, intra-cluster and inter-cluster associations are ignored, which reduces the performance of feature representation and harms the multi-view clustering results. 2) Multi-view data samples contain different features, and their distribution is affected by the integrity of multi-view data, so inaccurate features will affect the accuracy of their learning during feature learning. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an object view distillation method for general multi-view object clustering, so as to solve the problem of false guidance of model training and inaccurate correction of pseudo labels in multi-view clustering algorithms, and improve the multi-view clustering effect.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] An object view distillation method for general multi-view object clustering includes the following steps:

[0007] Step 1: The number of view data for each sample is V, and the initial multi-view object data of N samples are collected;

[0008] Step 2: construct an autoencoder, a teacher network, a student network, and knowledge distillation; the autoencoder consists of an encoder and a decoder, which are shared by all views. The encoder converts the view X of each sample into 1 ,X 2 ,...,X V Projected into potential representation Z 1 ,Z 2 ,...,Z V , and construct a low-dimensional latent space; the decoder remaps the latent representation into a view;

[0009] Step 3: Train the teacher network using the multi-view data output by the autoencoder; the teacher network linearly separates the learned high-dimensional features into pseudo labels based on contrastive learning;

[0010] Step 4: Train the student network using the multi-view data output by the autoencoder; the student network is used to extract multi-view features and project the original features into feature spaces at different levels, and learn common semantics by maximizing the mutual information with feature spaces at different levels;

[0011] Step 5: Use knowledge distillation to convert the pseudo labels generated by the teacher network into dark knowledge (K-dimensional). Dark knowledge will serve as a new self-supervisory signal to provide optimization direction for the student network and guide fine-tuning until learning is completed.

[0012] Step 6, feeding the original multi-view dataset to the overall network, the predictor in the student network branch will obtain the probability distribution of all view clusters, weight and sum the probability of each view, and obtain the final clustering result; the overall network consists of the autoencoder, knowledge distillation, teacher network and student network.

[0013] Compared with the prior art, the present invention has the following beneficial effects:

[0014] In the current multi-view clustering, multi-view data introduces more features, so it is difficult for overconfident pseudo-labels to represent these features, accompanied by more noise, which makes it difficult for existing multi-stage clustering methods to adapt to this multi-view clustering scenario.

[0015] This paper explores the application of knowledge distillation in multi-view clustering and proposes a multi-view knowledge distillation technique to transform overconfident pseudo-labels into dark knowledge, reducing the impact of false pseudo-labels on multi-view feature learning. In addition, a contrast method is used to learn multi-view semantics in feature spaces at different levels. In low-dimensional latent space, invariant information clustering is used to directly maximize mutual information, and in high-dimensional subspace, the lower bound of mutual information is improved according to fixed points related to the size of negative samples. This can correspondingly improve the performance of self-supervised learning multi-view representation for multi-view clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process of the present invention.

[0017] Figure 2 It is a schematic diagram of the model structure of the present invention.

[0018] Figure 3 It is the teacher network structure diagram of the present invention.

[0019] Figure 4 It is the student network structure diagram of the present invention.

[0020] Figure 5 It is a schematic diagram of the clustering process of the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0022] The concepts and parameters that may be involved in the present invention are introduced as follows:

[0023] A sample usually has multiple views, which may be complete or incomplete. Given a dataset, it contains multiple view data of multiple samples, where multiple view data refers to different angle views of the same sample, or different modal views of the same sample at the same angle (such as RGB images or depth maps). In the multiple view data, there is likely to be interference from noise points and missing data. To address this phenomenon, the object view distillation method for general multi-view object clustering of the present invention can still maintain good clustering performance when the pseudo-label guided training is overconfident.

[0024] refer to Figure 1 , the complete process of the present invention is as follows:

[0025] Step 1: Collect an initial multi-view dataset of several samples. Each view takes N samples, that is, the initial multi-view object data includes N samples. The number of view data of each sample is V, v∈{1,...V}. D v represents the vth view sample Xv The dimension is , and k is the number of categories to be clustered.

[0026] The goal of the present invention is to cluster all N samples into k clusters. Define a set of N samples {X 1 ,X 2 ,...,X V}, where X 1 indicates that the number of views in the sample is 1, X 2 indicates that the number of views in the sample is 2, X v It indicates that the number of views in the sample is v. In this step, the views of the sample can be views of the same sample at different angles, or views of the same sample at the same angle but in different modes.

[0027] Step 2: Construct the autoencoder, teacher network, student network and knowledge distillation module.

[0028] The schematic diagram of the model structure is as follows: Figure 2 As shown in Figure 1, it consists of an autoencoder, a teacher network, a student network, and knowledge distillation. The autoencoder consists of an encoder and a decoder, which are shared by all views. The encoder converts the view X of each sample into 1 ,X 2 ,...,X V Projected into potential representation Z 1 ,Z 2 ,...,Z V , and construct a low-dimensional latent space; the decoder remaps the latent representation to a view. The encoder and decoder of the view v use f v and g v Indicates. Encoder f1 converts X 1 Projected into the latent representation Z 1 , the second encoder f2 converts X 2 Projected into the latent representation Z 2 , the structure of the autoencoder for the vth view is defined as Among them, Fc 512 Represents a fully connected neural network with 512 neurons and each layer is followed by a ReLU layer.

[0029] like Figure 3 As shown in Figure 1, the teacher network constructs an independent high-dimensional subspace by the contrast module, indirectly improves the lower boundary of the mutual information through contrast learning in this high-dimensional subspace, and linearly separates the learned high-dimensional features into pseudo labels. The teacher network structure has two linear layers, and a ReLU activation function is added in the middle.

[0030] like Figure 4As shown in Fig. 1, the student network consists of a student network wp, a student network ws and a contrast module. The student network wp converts the features of the student network ws into a probability distribution and uses it as a soft label for distillation. The contrast module constructs an independent subspace and captures the high-dimensional hierarchical public representation and the potential representation X in the original low-dimensional feature space through contrast learning in the subspace. 1 ,X 2 ,…,X V Maximize mutual information in pairs. Since maximizing mutual information is achieved between two networks, we need to explore the same information between the two networks as much as possible, so we maximize mutual information in pairs. Compare the probability distribution of the output features of the student network with the dark knowledge output of the teacher network to calculate the KL divergence.

[0031] The knowledge distillation module is designed to overcome the overconfidence of the pseudo-labels output by the teacher network. Knowledge distillation uses the k-dimensional features output by the teacher network and converts the one-dimensional pseudo-labels into k-dimensional dark knowledge by adjusting the temperature and adding the Softmax activation function. Because dark knowledge contains basic hierarchical information that is not contained in the pseudo-labels, the dark knowledge obtained by the final distillation is used as the ground truth as a new self-supervisory signal to guide feature learning.

[0032] In this embodiment, knowledge distillation is introduced for the teacher network to extract the dark knowledge of the pseudo-label distribution, and the dark knowledge acts as a self-supervisory signal to provide more accurate guidance to the student network. The present invention evaluates the superiority of the present invention on eight public datasets: 1) Scene contains 4485 images from 15 different indoor and outdoor scene categories, as well as PHOG and GIST features. 2) MNIST-USPS contains 5000 digital image samples with two different styles, USPS images are 256 dark, and MNIST images are 784 dark. 3) BDGP consists of 5 categories and 2500 samples, each of which has 500 samples, and each sample is represented by visual and text features. 4) Fashion has a total of 10 categories, providing 60,000 28*28 pixel clothes and pants pictures and labels for training, and 10,000 28*28 pixel clothes and pants pictures and labels for testing. 5) Caltech consists of 9144 images distributed in 102 categories. In order to evaluate the robustness of the present invention on the number of views, Caltech, a multi-view RGB image dataset, is decomposed into Caltech-2V, Caltech-3V, Caltech-4V, and Caltech-5V. The detailed statistical information of the dataset is summarized in Table 3-1.

[0033] Table 3-1 Dataset summary

[0034]

[0035]

[0036] Step 3: Train the teacher network using the multi-view data output by the autoencoder.

[0037] Multi-view data is represented as X 1 The potential representation Z is obtained through the encoder f1 1 , X 2 The potential representation Z is obtained through the encoder f2 2 , X v Through the encoder f in network 1 v Get the potential representation Z v . Z 1 , Z 2 and Z v are the potential representations of the first view, the second view, and the vth view, respectively. For example, training for 300 epochs can provide better initialization parameters for the autoencoder in the training network 2 in step 4.

[0038] Based on Z 1 , Z 2 and Z v , construct three objective functions that need to be further optimized:

[0039] i) Build a deep autoencoder to capture the salient features of the data by minimizing Enables an autoencoder to transform heterogeneous multi-view data into cluster-friendly latent representations where:

[0040]

[0041] is an evaluation metric for the autoencoder to convert heterogeneous multi-view data into latent representations, so The formula is to make a difference, the smaller the value, the better. For the vth view, f v () is the encoder, g v () is the decoder, represents the nth feature vector, and the learned latent representation is defined as Z v , represents the nth potential representation, It's Z v This design enables the autoencoder to maintain the diversity of their respective views, avoid trivial solutions, and prevent model collapse, which is the basis for improving the performance of multi-view clustering.

[0042] In order for the model to perform feature learning effectively, the teacher network and the student network project a low-dimensional representation {Z 1 ,Z 2,...Z v}Entering high-dimensional space {t 1 ,t 2 ,...t v} and {y 1 ,y 2 ,...y v} are in different hierarchical structures. In order to achieve effective feature learning at different levels, an objective function for learning common semantics is proposed:

[0043]

[0044] ii) Minimizing InfoNCE at a high-dimensional level can be regarded as indirectly maximizing the lower bound of mutual information, while the present invention directly maximizes the mutual information between different views of the low-dimensional hierarchy, which is called invariant information clustering. The invariant information clustering can be expressed as:

[0045]

[0046] Where I represents mutual information, represents the maximization of mutual information, represents the learned n-th potential representation of the v′-th view. Figure 3 As shown, according to the invariant information clustering, and Approximately two independent discrete distributions, further get and The joint probability distribution of . Therefore, I directly calculates:

[0047]

[0048] iii) If Figure 3 As shown in the figure, the teacher network uses a feature learning method, the goal is to provide a supervisory signal for the optimization of the student network and provide high-dimensional features {t 1 ,t 2 ,...t v} is used for linear separation. The use of contrastive learning helps to better fit the probability distribution and learn the mutual information at different levels. Given a sample pair and To optimize the symmetric cross entropy loss as follows:

[0049]

[0050] where τ t is the temperature parameter of the teacher network that controls the softness of the distribution. Considering all views on the dataset, the optimization objective of the teacher network is given as:

[0051]

[0052] Among them, {t 1 ,t 2 ,…t v} is the high-dimensional feature obtained by projecting the potential representation in the low-dimensional feature space by the teacher network, {y 1 ,y 2 ,...y v} is the high-dimensional feature obtained by projecting the potential representation in the low-dimensional feature space by the student network. is a sample pair, is the representation of the optimized teacher network.

[0053] Step 4: Use the multi-view data output by the autoencoder to train the student network.

[0054] Through this step, the student network is trained to predict the n×n possible To this end, the student network wp maximizes the n positive sample pairs on the diagonal and cosine similarity, while minimizing (n 2 -n) negative sample pairs to learn the multi-view embedding space feature matrix Feature Matrix The pairwise similarity in is measured by cosine similarity:

[0055]

[0056] where n,m∈[1,N],v,v∈[1,V], and represents a sample pair, represents transpose, and Is the student network wp will be embedded and The probability distribution obtained by conversion is and It is a potential representation and The resulting embedding is fed to the student network ws.

[0057] like Figure 4 As shown in , the student network and the teacher network use the same feature learning method, especially the student network needs a regularization term to prevent the model from collapsing. Similarly, in order to optimize the pairwise similarity without losing generality, given a sample pair and Optimizing symmetric cross entropy loss:

[0058]

[0059] in, is the representation of the optimized student network, τ s The student network temperature parameter to control the softness of the distribution, The present invention hopes to identify all positive pairs of the entire data set, so it is necessary to calculate the contrast loss of the sample pairs and on all views and extend it to Vv2 as follows:

[0060]

[0061] In the above equation, an additional entropy balance term can be added

[0062]

[0063] This regularization term avoids trivial solutions and prevents all sample points from clustering into the same class.

[0064] In step 5, knowledge distillation is used to convert the pseudo labels generated by the teacher network into dark knowledge, so that the dark knowledge can be used as a new self-supervisory signal to provide optimization direction for the student network and guide fine-tuning until learning is completed.

[0065] In order to better utilize the learned common semantics for clustering, it is necessary to add a knowledge distillation module for the two independent student and teacher subspaces for fine-tuning. The present invention does not directly use the soft labels output by the teacher network as the distribution required for extraction, because such a probability distribution does not contain obvious clustering information. First, the clustering information contained in the high-dimensional features is used to improve the clustering effect of the semantic labels, and the new cluster center C is obtained by optimizing the following objectives:

[0066]

[0067] where θ is the parameter of the teacher network, d v Yes n The dimension of . is a symbolic representation of the new cluster center C, is the cluster center of each view, is the data set, and is the nth potential representation. This step is more efficient using the K-means algorithm, so it can be linearly separated according to the cluster center C. n , to obtain V sets of pseudo labels The Softmax activation function will be added to the last layer of the predictor. is defined as the probability that the nth sample is clustered into the mth cluster, so there are V groups of probability distributions exist In the case of dark knowledge obtained, the KL divergence distillation model is used:

[0068]

[0069] where y v is a sample of the student network, τ d is the distillation factor, u is the introduced distribution, here is the Gaussian distribution, is the dark knowledge obtained. The softmax function makes y v It becomes a relatively sharp distribution, while dark knowledge is a relatively smooth distribution. KL divergence can make the two confront each other, thus effectively preventing the model from collapsing.

[0070] Step 6, such as Figure 5 As shown, the original multi-view dataset is fed to the overall network, and the predictor in the student network branch will obtain the probability distribution of all view clusters, weight and sum the probability of each view, and get the final clustering result.

[0071] The objective function of the overall network is:

[0072]

[0073] The entire original multi-view dataset is fed to the overall network, and the student network wp is used as the predictor in the student network branch to obtain the probability distribution of all view clusters, and the probability of each view is weighted and summed to obtain the final clustering result. The present invention uses the knowledge distillation method to solve the misguidance of pseudo-labels on model training in the multi-view clustering algorithm and correct inaccurate features, significantly improving the clustering performance.

[0074] Regarding the selection of evaluation indicators of the present invention, three indicators, namely, accuracy (ACC), normalized mutual information (NMI) and purity (PUR), are used to evaluate clustering performance. The higher the values ​​of these evaluation indicators, the better the clustering performance.

[0075] Table 3-2 Ablation experiments on the Caltech-2V dataset

[0076]

[0077] Table 3-2 shows the loss components and experimental results corresponding to the five variables. The present invention designs six sets of schemes on four datasets with different numbers of views, and observes the following results: a) All losses play an indispensable role in the object view distillation method for general multi-view object clustering; b) After introducing the knowledge distillation method on (1)(3)(5)(6), significant improvements have been achieved, which further proves that our method can effectively alleviate the overconfidence problem in pseudo-labeling, thereby improving clustering performance; c) Adding self-distillation in (2)(4) leads to model degradation; d) Comparing (1) and (6), it can be seen that the optimization loss It can bring huge improvements, which proves the effectiveness of our proposed method in maximizing mutual information at different levels; e) The above four observations are applicable to all datasets, which also proves the robustness of the method used in the present invention.

[0078] Table 3-3 Clustering performance on different data sets

[0079] Dataset ACC NMI PUR Scene 0.428 0.432 0.448 MNIST-USPS 0.996 0.987 0.996 BDGP 0.971 0.971 0.991 Fashion 0.993 0.982 0.993 Caltech-2V 0.619 0.533 0.619 Caltech-3V 0.650 0.575 0.663 Caltech-4V 0.809 0.695 0.809 Caltech-5V 0.824 0.709 0.824

[0080] Table 3-3 describes the clustering performance of the present invention under different scale datasets, and lists the clustering performance of all methods on eight datasets, from which we obtain the following observations: Our object view distillation method for general multi-view object clustering achieves the best performance on all datasets. On the other hand, the present invention uses dark knowledge instead of pseudo labels to provide more precise guidance for self-supervised clustering, thereby obtaining excellent clustering results.

Claims

1. An object view distillation method for general multi-view object clustering, characterized in that: The steps include: Step 1: The number of view data for each sample is V, and the initial multi-view object data of N samples are collected; Step 2: construct the autoencoder, teacher network, student network and knowledge distillation; The autoencoder consists of an encoder and a decoder, which are shared by all views. The encoder converts the view X of each sample 1 ,X 2 ,...,X V Projected into potential representation Z 1 ,Z 2 ,…,Z V , and construct a low-dimensional latent space; the decoder remaps the latent representation into a view; The teacher network constructs an independent high-dimensional subspace by the contrast module, indirectly improves the lower boundary of the mutual information through contrast learning in the high-dimensional subspace, and linearly separates the learned high-dimensional features into pseudo labels; The student network consists of a student network wp, a student network ws and a contrast module. The student network wp converts the features of the student network ws into probability distributions and uses them as soft labels for distillation. The contrast module constructs an independent subspace and captures high-dimensional hierarchical public representations and potential representations X in the original low-dimensional feature space through contrast learning in the subspace. 1 ,X 2 ,…,X V Maximize mutual information pairwise; The knowledge distillation utilizes the k-dimensional features output by the teacher network, and converts the one-dimensional pseudo-label into k-dimensional dark knowledge by adjusting the temperature and adding the Softmax activation function. The dark knowledge contains basic hierarchical information not included in the pseudo-label, and the dark knowledge finally obtained by distillation is used as the basic truth as a self-supervisory signal to guide feature learning; Step 3: Use the multi-view data output by the autoencoder to train the teacher network as follows: Construct a deep autoencoder by minimizing Enables an autoencoder to transform heterogeneous multi-view data into cluster-friendly latent representations where: is an evaluation metric for the autoencoder to convert heterogeneous multi-view data into latent representations. For the vth view, f v () is the encoder, g v () is the decoder, represents the nth feature vector, and the learned latent representation is defined as Z v , represents the nth potential representation, It's Z v Reconstructed view of Maximizing mutual information between different views of a low-dimensional hierarchy is called invariant information clustering, which is expressed as: represents maximizing mutual information, I represents mutual information, represents the learned n-th latent representation of the v′-th view Step 4: Train the student network using the multi-view data output by the autoencoder. The student network is trained to predict the n×n possible Which of the pairs actually occurs? The student network wp maximizes the n positive sample pairs on the diagonal and cosine similarity, while minimizing (n 2 -n) cosine similarity of the embeddings of negative sample pairs to learn the multi-view embedding space feature matrix Feature Matrix The pairwise similarity in is measured by cosine similarity: in and represents a sample pair, () represents transposition, and Is the student network wp will be embedded and The probability distribution obtained by conversion is and It is a potential representation and Feed the embedding obtained by the student network ws; The student network and the teacher network use the same feature learning method. and Optimizing symmetric cross entropy loss: The contrastive loss for sample pairs and is calculated over all views and extended to V ≥ 2 as follows: is the representation of the optimized student network, τ s To control the distribution softness, the student network temperature parameter, is a sample pair; Step 5: Use knowledge distillation to convert the pseudo labels generated by the teacher network into dark knowledge, provide optimization direction for the student network and guide fine-tuning until learning is completed; In step 6, the original multi-view dataset is fed to the overall network, and the predictor in the student network branch will obtain the probability distribution of all view clusters, weight and sum the probability of each view, and get the final clustering result.

2. The object view distillation method for general multi-view object clustering according to claim 1, characterized in that: In step 1, the views of the sample are views of the same sample at different angles, or views of the same sample at the same angle in different modes.

3. The object view distillation method for general multi-view object clustering according to claim 1, characterized in that: The teacher network uses a feature learning method, the goal is to provide a supervisory signal for the optimization of the student network and provide high-dimensional features {t 1 ,t 2 ,...t v } for linear separation; sample pairs and To optimize the symmetric cross entropy loss as follows: where τ t is the temperature parameter of the teacher network that controls the softness of the distribution. Considering all views on the dataset, the optimization objective of the teacher network is given as: Among them, {t 1 ,t 2 ,...t v } is the high-dimensional feature obtained by projecting the potential representation in the low-dimensional feature space by the teacher network, {y 1 ,y 2 ,...y v } is the high-dimensional feature obtained by projecting the potential representation in the low-dimensional feature space by the student network. is a sample pair, is the representation of the optimized teacher network.

4. The object view distillation method for general multi-view object clustering according to claim 1, characterized in that: In step 5, the clustering information contained in the high-dimensional features is first used to improve the clustering effect of the semantic labels, and a new clustering center C is obtained by optimizing the following objectives: where θ is the parameter of the teacher network, is a symbolic representation of the new cluster center C, is the cluster center of each view, is the data set, is the nth potential representation, d v Yes n The dimension of Linear separation t according to cluster center C n , and get the V group of pseudo labels exist In the case of the obtained dark knowledge, the KL divergence distillation model is used: where y v is a sample of the student network, τ d is the distillation factor, u is the introduced distribution, P *v To obtain dark knowledge.

5. The object view distillation method for general multi-view object clustering according to claim 1, characterized in that: In step 6, the objective function of the overall network is: The entire original multi-view dataset is fed to the overall network, and the student network wp is used as the predictor in the student network branch to obtain the probability distribution of all view clustering. The probability of each view is weighted and summed to get the final clustering result.

Images