A unified clustering method for unpaired multi-view data

Abstract

The application discloses a unified clustering method for unpaired multi-view data, belongs to the field of artificial intelligence and data mining, and comprises the following steps: independently configuring an encoder, a projection head and a decoder for each view, mapping original samples to a shared latent space and reconstructing input; performing K-means clustering on each view in the shared space respectively, establishing a cross-view cluster center matching relationship through a nearest neighbor strategy after obtaining the cluster center; then, using the matched cluster center as a teacher distribution, performing cross-view distillation on the sample class distribution, realizing alignment of the clustering structure; meanwhile, introducing a view discriminator and utilizing an adversarial training mechanism to suppress view specificity, learning view-invariant features; and finally outputting consistent cross-view clustering results based on the view-invariant features. The application effectively realizes cross-view structure alignment and consistent clustering without sample pairing and supervised labels.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and data mining, and particularly relates to a unified clustering method for unpaired multi-view data. BACKGROUND

[0002] In the real world, data often has heterogeneous characteristics, comes from different sources or sensors, and exists in various forms, constituting multi-view data. However, in actual applications, due to various unpredictable factors in the data collection process, such as equipment failure, data transmission error or privacy protection demand, data in part of the views may be missing, resulting in incomplete multi-view data. In a more extreme case, the observed samples in different views are completely inconsistent, that is, there is no common sample between views, which is called unpaired multi-view data.

[0003] Unpaired multi-view data is ubiquitous in real-world applications. For example, in a cross-language news recommendation system, the same event may be reported in different languages in different media, but there is no strict one-to-one correspondence between the contents. For another example, in cross-regional traffic monitoring, different cities deploy sensors of different types, different control densities and different data formats, making it difficult to directly align the data records left by the same vehicle in multiple cities.

[0004] Clustering is an unsupervised learning method commonly used to discover the potential structure in data, and its core goal is to divide samples into several groups with similarity. For multi-view data, according to whether the data is complete, the related clustering tasks can be divided into multi-view clustering and incomplete multi-view clustering, and these two directions have been explored by many existing studies. However, most existing multi-view clustering methods assume that there is a clear one-to-one pairing relationship between views, that is, the samples in different views can be accurately matched. This assumption is often difficult to meet in actual applications, especially in cross-modal, cross-device or cross-domain data scenarios, where different views are often independently collected, resulting in no pairing relationship between samples. In this unpaired scenario, traditional methods cannot directly establish alignment or fusion mechanism between samples, so they cannot effectively utilize the complementary information between multi-views.

[0005] Furthermore, unpaired multi-view clustering, as a more challenging task, requires that under the premise of lacking paired samples between views, the potential association between views can still be mined and consistent clustering division can be achieved. However, existing multi-view clustering methods often lack effective structure alignment strategies when dealing with such problems, making it difficult to build high-quality association relationships between cross-views, and thus have weak adaptability to unpaired data. SUMMARY

[0006] To solve the above technical problems, the application provides a unified clustering method for unpaired multi-view data, comprising:

[0007] At least two views without sample pairing are obtained.

[0008] For each view, all samples of each view are encoded to obtain latent features of the view.

[0009] According to the latent features of the view, a shared representation of the view in a shared latent space is obtained through mapping.

[0010] According to the shared representation of the view, clustering is performed on the view to obtain cluster centers of the view.

[0011] According to the cluster centers of different views, cross-view matched cluster pairs are generated through mutual nearest neighbor criteria.

[0012] According to the cross-view matched cluster pairs, the shared representation of the view is distilled and constrained in each view with the matched cluster center as the teacher to obtain a structure-aligned shared representation.

[0013] According to the structure-aligned shared representation, the view discriminator performs adversarial training to suppress view specificity and obtain a view-invariant shared representation.

[0014] According to the view-invariant shared representation, final clustering is performed on all samples to output cross-view clustering results. Figure One

[0015] Optionally, the process of obtaining the latent features of the view comprises:

[0016] An independent encoder is configured for each view, and the sample is input into the encoder to obtain an intermediate representation.

[0017] The intermediate representation is reconstructed by a decoder, and the encoder parameters are updated according to the reconstruction error, so that the latent features retain the original input information, and the latent features of the view are generated.

[0018] Optionally, the process of obtaining the shared representation of the view in the shared latent space through mapping comprises:

[0019] An independent projection head is configured for each view, and the projection head takes the intermediate representation as input and outputs the shared representation in the shared latent space.

[0020] The projection head adopts a multi-layer perception structure to complete the mapping of view-specific distribution to shared distribution and obtain the shared representation of the view in the shared latent space.

[0021] Optionally, the process of generating cross-view matched cluster pairs through mutual nearest neighbor criteria comprises:

[0022] ​After obtaining the cluster centers of the first view, distances between the cluster centers of the first view and the cluster centers of the second view are calculated.

[0023] According to the distances, the nearest neighbor cluster center of the cluster center of the first view in the second view is determined;

[0024] According to the nearest neighbor criterion, the cluster center pair is marked as a cross-view matched cluster pair only when the corresponding cluster centers of the first view and the second view are the nearest neighbors of each other.

[0025] According to the cluster center pairs that satisfy all the conditions, a set of cross-view matched cluster pairs is generated.

[0026] Optionally, the process of obtaining the structure-aligned shared representation comprises:

[0027] After obtaining the cross-view matched cluster pairs, for a sample in any view, a student soft class distribution is calculated according to the shared representation of the sample;

[0028] According to the shared representation of the matched cluster center corresponding to the sample in the other view, a teacher soft class distribution is calculated;

[0029] According to the student soft class distribution and the teacher soft class distribution, a KL divergence under a temperature coefficient is calculated as a distillation loss;

[0030] According to the distillation loss, the encoder and the projection head are updated so that the shared representations of different views have a structure-aligned class distribution, and the structure-aligned shared representation is generated.

[0031] Optionally, the process of suppressing view specificity and obtaining a view-invariant shared representation comprises:

[0032] After obtaining the structure-aligned shared representation, the shared representation is input into a view discriminator;

[0033] According to the output of the view discriminator, a probability distribution of the sample belonging to each view is obtained;

[0034] According to the probability distribution and the real view label, a cross-entropy loss is calculated to train the view discriminator;

[0035] According to the gradient reversal layer, the cross-entropy loss is backpropagated, and the encoder and the projection head are updated so that the shared representation is indistinguishable to the view discriminator, and a view-invariant shared representation is obtained.

[0036] Optionally, the process of outputting the clustering result across the views comprises: Figure One inputting the latent features of each view into a K-means clustering algorithm respectively to obtain a final consistent cross-view clustering division result.

[0037] Optionally, the views include but are not limited to image views, text views, sensor data views, and cross-language text views.

[0038] In another aspect, the present application also provides an electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0039] In another aspect, the present application also provides a computer readable storage medium storing a computer program, wherein the computer program is executable on a processor to implement the method.

[0040] Compared with the prior art, the present application has the following advantages and technical effects:

[0041] The present application provides a unified clustering method for unpaired multi-view data, aiming to learn a discriminative and structure-consistent clustering representation in a multi-view scene without labels and sample pairing. The method combines cross-view distillation and adversarial discrimination mechanism, which can effectively handle the challenges of structure alignment and view invariance modeling in unpaired multi-view heterogeneous data, has high clustering accuracy, and significantly improves the clustering effect for unpaired multi-view heterogeneous data. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety, and the illustrative embodiments thereof and their description serve the purpose of explanations and are not intended to limit the present application. In the drawings:

[0043] Figure 1 The method flowchart of the embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0045] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0046] Embodiment one

[0047] As shown in the drawings, Figure 1 The present embodiment provides a unified clustering method for unpaired multi-view data, comprising:

[0048] Obtaining at least two views without sample pairing;

[0049] For each view, encoding all samples of each view to obtain latent features of the view;

[0050] According to the potential characteristics of the views, a shared representation of the views is obtained by mapping;

[0051] According to the shared representation of the views, clustering is performed on the views to obtain cluster centers of the views;

[0052] According to the cluster centers of different views, cross-view matched cluster pairs are generated by the nearest neighbor criterion;

[0053] According to the cross-view matched cluster pairs, the shared representation of the views is distilled in each view with the matched cluster centers as teachers, to obtain a shared representation after structure alignment;

[0054] According to the shared representation after structure alignment, the view discriminator is executed to perform adversarial training, to suppress view specificity and obtain a view-invariant shared representation;

[0055] According to the view-invariant shared representation, final clustering is performed on all samples, and a cross-view clustering result is output. Figure One

[0056] Specifically:

[0057] S1. Problem definition and input representation;

[0058] Given unpaired views , where the th view contains samples, each sample belongs to a feature space :

[0059] ;

[0060] where is an unpaired view, represents the samples of the view, is the feature space, is the number of samples.

[0061] There is no sample pairing relationship between these views, i.e. and ( ) do not have a corresponding relationship.

[0062] The goal is to learn consistent clustering structure in multiple views, so that samples under different views can share consistent class space division.

[0063] S2. Feature encoding and shared projection space;

[0064] In order to extract representations with discriminative and alignment capabilities from each view, the embodiment encodes each view​ An independent encoder is designed , a projection head and a decoder . First, the sample is encoded by the encoder to get the intermediate representation, and then mapped to the shared latent space by the projection head , which is denoted as follows:

[0065] ;

[0066] where is the mapped representation, and is a convolutional neural network or an auto-encoder network selected according to the category of the view data, and is an MLP used to project the view features to the shared representation space.

[0067] To preserve the information of the original input, the decoder is introduced to reconstruct the encoded representation before projection, and output the reconstructed sample :

[0068] ;

[0069] The reconstruction loss function is defined as the mean square error between the input and the reconstructed sample:

[0070] ;

[0071] where is the reconstruction loss.

[0072] S3. Intra-view clustering and cluster center representation

[0073] In the embedding space of each view, the representations are clustered using the K-Means to get the cluster centers of the view:

[0074] ;

[0075] where represents the set of cluster centers of the view, and represents the k-th cluster center, K represents the number of clusters, and k represents the k-th.

[0076] The class membership probability of each sample can be obtained by the distance between it and the cluster center through softmax:

[0077] ; ​​​

[0078] wherein, denotes the class membership probability of each sample , denotes the 1st cluster center in view v, denotes the kth cluster center.

[0079] S4. Cross-view cluster matching;

[0080] To establish the consistency of clustering structure in different views, the embodiment adopts the nearest neighbor strategy to align the respective cluster centers between different views. Specifically, for views and , their cluster center sets are constructed as follows, respectively:

[0081] ;

[0082] wherein, denotes the cluster center set of view , denotes the kth cluster center of view .

[0083] If the cluster centers and are the nearest neighbors of each other, they are considered to constitute a set of cross-view matching cluster pairs. The following conditions must be met simultaneously:

[0084] ;

[0085] wherein, is any cluster center in the view cluster center set, is the jth cluster center in view v.

[0086] The set consisting of all cluster pairs satisfying the above conditions is denoted as , which is used in the subsequent cross-view distillation module as an alignment reference.

[0087] S5. Cross-view distillation alignment;

[0088] For each sample , the embodiment defines its teacher as its corresponding matching cluster , and converts the softmax distribution for structural distillation.

[0089] The student output distribution is defined as:

[0090] ;

[0091] wherein, is the student output distribution, is the temperature coefficient;

[0092] The teacher distribution is defined as:

[0093] ;

[0094] where, is the teacher output distribution.

[0095] The distillation loss is:

[0096] ;

[0097] This loss enables the embeddings under different views to learn consistent class distribution structures.

[0098] S6. Discriminator and adversarial view-invariance learning;

[0099] To remove the view-specific information in different views, this embodiment introduces a view discriminator to identify which view each sample belongs to, so as to learn a view-undiscriminable shared representation through an adversarial training mechanism. For a sample from view , its embedding representation is denoted as , and the discriminator outputs the probability distribution of the sample belonging to each view:

[0100] ;

[0101] where, is the probability distribution of the sample belonging to each view, denotes, denotes a probability simplex of dimension V:

[0102] ;

[0103] where, is the probability vector, is a V-dimensional real space, is the jth component of the probability vector p, and V is the total number of views.

[0104] In training, the discriminator aims to correctly classify the view label of each sample , and the supervised loss is the standard cross-entropy:

[0105] ;

[0106] where, is the discriminator loss function, denotes the discriminator of sample i in view v, denotes the embedding representation of sample i in view v, denotes the number of samples in view v.

[0107] The embodiment uses the loss for the discriminator and the encoder in both directions by Gradient Reversal Layer (GRL). This adversarial mechanism guides the model to learn view-invariant features in the shared latent space, providing a representation basis for cross-view structural alignment and consistent clustering.

[0108] The final training objective of the embodiment is to jointly optimize the following loss functions:

[0109] ;

[0110] where, is the reconstruction loss; is the cross-view distillation alignment loss; is the adversarial view-invariant learning; is the hyper-parameter weight.

[0111] Finally, after the model optimization is completed, the embodiment performs -means clustering on the embedding representation of each view respectively to obtain the final clustering division result.

[0112] On the other hand, the embodiment also provides an electronic device, which includes a memory, a processor, and a computing program stored in the memory and executable on the processor, and the processor implements the method when executing the computing program.

[0113] On the other hand, the embodiment also provides a computer-readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method.

[0114] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A unified clustering method for unpaired multi-view data, characterized in that, include: Obtain at least two views without sample pairings; For each view, all samples of each view are encoded to obtain the latent features of the view; Based on the latent features of the view, a shared representation of the view in the shared latent space is obtained through mapping; Based on the shared representation of views in the shared latent space, clustering is performed on the views to obtain the cluster centers of the views; Based on the cluster centers of different views, cross-view matching cluster pairs are generated using the nearest neighbor criterion. Based on the cross-view matching cluster pairs, the shared representation of the view is distilled with the center of the matching cluster as the teacher in each view to obtain the shared representation after structural alignment. Based on the shared representation after structural alignment, adversarial training is performed through the view discriminator to suppress view specificity and obtain a view-invariant shared representation; Based on the view-invariant shared representation, perform final clustering on all samples and output consistent clustering results across views; The process of obtaining the latent features of the view includes: Configure an independent encoder for each view, input the sample into the encoder, and obtain the intermediate representation; The intermediate representation is reconstructed by the decoder, and the encoder parameters are updated according to the reconstruction error, so that the latent features retain the original input information and generate the latent features of the view. The process of obtaining a shared representation of a view in a shared latent space through mapping includes: Each view is configured with an independent projection head, which takes an intermediate representation as input and outputs a shared representation in a shared latent space. The projection head employs a multilayer perceptron structure to complete the mapping from a view-specific distribution to a shared distribution, thereby obtaining a shared representation of the view in the shared latent space.

2. The method according to claim 1, characterized in that, The process of generating cross-view matching cluster pairs using the nearest neighbor criterion includes: After obtaining the cluster center of the first view, calculate its distance from each cluster center of the second view; Based on the distance, determine the nearest neighbor cluster center of the cluster center in the first view and the cluster center in the second view; According to the nearest neighbor criterion, a cluster center pair is marked as a cross-view matching cluster pair if and only if the corresponding cluster centers of the first view and the second view are each other's nearest neighbors; Generate a set of cross-view matching cluster pairs based on the cross-view matching cluster pairs.

3. The method according to claim 1, characterized in that, The process of obtaining the shared representation after structural alignment includes: After obtaining cross-view matching cluster pairs, the student soft class distribution is calculated for samples in any view based on their shared representation. The distribution of teacher soft classes is calculated based on the shared representation of the matching cluster centers in another view corresponding to the samples. Based on the soft class distribution of students and teachers, the KL divergence under the temperature coefficient is calculated as the distillation loss. Based on the distillation loss, the encoder and projection head are updated to ensure that the shared representations of different views have a structurally consistent class distribution, generating a structurally aligned shared representation.

4. The method according to claim 1, characterized in that, The process of suppressing view specificity and obtaining a view-invariant shared representation includes: After obtaining the shared representation after structural alignment, the shared representation is input into the view discriminator; Based on the output of the view discriminator, the probability distribution of the sample belonging to each view is obtained; The cross-entropy loss is calculated based on the probability distribution and the true view labels to train the view discriminator; Based on the backpropagation cross-entropy loss of the gradient inversion layer, the encoder and projection head are updated to make the shared representation indistinguishable from the view discriminator, thus obtaining a view-invariant shared representation.

5. The method according to claim 1, characterized in that, The process of outputting consistent clustering results across views includes: inputting the latent features of each view into the K-means clustering algorithm to obtain the final consistent cross-view clustering partitioning results.

6. The method according to claim 1, characterized in that, The views include image views, text views, sensor data views, and cross-language text views.

7. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images