An unsupervised multi-view clustering approach for the non-aligned view missing problem

Abstract

An unsupervised multi-view clustering method for the problem of missing non-aligned views is proposed for the situation where consistency learning of multi-view data is insufficient in the case of PVP and PSP. HmiMVC is proposed. First, the PVP problem is solved by maximizing the cross-view mutual information to achieve category-level alignment. For the PSP problem in the PVP problem, the idea of ​​dual prediction is used to predict the missing data, while minimizing the conditional entropy to achieve natural category-level alignment. Data recovery and alignment are integrated in a hierarchical consistency framework. HmiMVC adopts different hierarchical alignment strategies and methods for PSP and PVP respectively, but consistency learning in turn makes PSP and PVP learning reinforce each other. The present invention combines category-level alignment and instance-level recovery alignment, effectively overcoming the problem of insufficient consistency learning and significantly improving clustering performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and image clustering, and in particular relates to an unsupervised multi-view clustering method for the problem of missing non-aligned views. Background Art

[0002] In real life, most data usually appears in the form of multimodal or multi-view. For example, the color information and texture information of an image are two different features. Single-modal or single-view methods cannot effectively utilize the information in multimodal data. If there is an effective multimodal learning method that can comprehensively utilize the information of multiple modalities in the data, it will enable us to understand the data information more comprehensively. However, existing methods explicitly require that multi-view data meet the requirements of cross-view. Figure 1 The consistency assumption is data integrity. However, in practical applications, data loss may occur during data collection and transmission, resulting in the problem of view misalignment. Even view misalignment and sample missing problems exist at the same time. Therefore, complete view data is very scarce, resulting in the incomplete multi-view problem. The key to solving the complete multi-view problem is to infer the missing information from the existing data, or to make correct judgments using the existing data information.

[0003] In order to solve the incomplete multi-view problem in clustering, incomplete multi-view clustering algorithms have attracted widespread attention, but their performance is limited due to the following shortcomings: 1) Most incomplete multi-view clustering methods have difficulty in obtaining high-level semantic features and have poor clustering performance on complex real data. 2) Only complete aligned data can be used for representation learning based on data consistency assumptions, but the complete aligned data is very scarce and insufficient to learn consistent information. The hidden information hidden in incomplete data is particularly important for multi-view clustering. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art and the problem of insufficient consistency learning caused by view misalignment and view missing, the purpose of the present invention is to provide an unsupervised multi-view clustering method for the problem of non-aligned view missing, projecting the original data set into a latent space that ensures information consistency. Since the missing and misaligned data exist in a heterogeneous hierarchical structure, the PVP (misalignment) is first solved by maximizing the mutual information through contrastive learning in a hierarchical manner to achieve category-level alignment. For the PSP (view missing) in PVP, the idea of ​​dual prediction is introduced to predict the missing data while minimizing the conditional entropy, which constitutes a natural instance-level alignment. Finally, the alignment strategies of the two levels are merged into a unified reconstruction process to avoid model collapse and trivial solutions. This method further improves efficiency based on existing data, achieves better prediction effects, and further improves clustering accuracy.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] An unsupervised multi-view clustering method for the non-aligned view missing problem includes the following steps:

[0007] S1: Collect initial multi-view data of several samples, the number of view data of each sample is 2, and divide each sample into complete aligned data, misaligned data and missing data; the complete aligned data means that both views of the sample are not missing and aligned, the misaligned data means that the two views of the sample are not aligned, and the missing data means that one of the two views of the sample is missing. The misaligned data and missing data constitute incomplete data.

[0008] S2: Build network A.

[0009] The network A consists of two groups of autoencoders, a prediction module and a dual prediction module. Each group of autoencoders includes an encoder and a decoder. The encoder converts two views X of a sample into (1) and X (2) Projected into latent representations and The prediction module is based on contrastive learning and can predict the corresponding view based on a sample in the incomplete data. Specifically, it predicts the corresponding category-level aligned view based on the unaligned view, thereby achieving and The mutual prediction between them can achieve category-level alignment; the dual prediction module fills in the missing data through dual prediction and minimizing conditional entropy, while achieving natural instance-level alignment.

[0010] S3: Train network A using the fully aligned data.

[0011] S4: Test the trained network A using complete aligned data, unaligned data, and missing data.

[0012] S5: The original multi-view data is passed to network A for processing and clustering, and the Kmeans clustering algorithm is used on the public representation to obtain the final clustering result.

[0013] Preferably, (S1) specifically means that the two views of the sample are views of the same sample at different viewing angles, or views of the same sample at the same angle but in different modes.

[0014] Preferably, (S2) is specifically as follows: in the network A, a sample and itself are taken as a positive sample pair, and cross-view samples are randomly selected to form a negative sample pair.

[0015] Its prediction module uses a contrastive learning network, whose architecture uses two sets of autoencoders f1 and f2 to transform the view X(1) and X (2) Projected into latent representations and Through contrastive learning, the mutual information between positive samples is minimized and the mutual information between negative samples is maximized.

[0016] based on and Calculate the prediction loss, noise-robust contrast loss, and view reconstruction loss, and back-propagate the updated gradients; maximize via contrastive learning and The mutual information between different views can be used to mine the common semantics between different views. and Category-level alignment of and The mutual prediction between them enables the prediction module to predict the corresponding aligned view based on a sample in the unaligned data. At the same time, the dual prediction module achieves and Fill in the missing data between .

[0017] The dual prediction module uses a dual prediction network and an approximately minimized conditional entropy network. Its main architecture is composed of a group of autoencoders, in which the dual prediction and approximately minimized conditional entropy can fill in the missing data and achieve natural instance-level alignment.

[0018] Preferably, (S3) is specifically as follows: the complete aligned data divided by S1 is trained using network A, and the training cycle is 150 epochs.

[0019] In S3, the positive and negative samples in the complete aligned data are divided, the aligned data is used as the positive samples, the cross samples are randomly selected as the negative samples, and then training is performed.

[0020] Preferably, (S4) is specifically: using network A to test the unaligned data and missing data divided by S1.

[0021] Preferably, (S5) is specifically as follows: the original multi-view data is passed to the trained network A, the unaligned data is aligned at the category level by the prediction module, the missing data is restored and aligned at the instance level by the dual prediction module, and a multi-view high-dimensional public representation is obtained; based on the public representation, the Kmeans clustering algorithm is used to obtain the final clustering result.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] Existing incomplete multi-view clustering methods can only explore consistency and common semantics based on aligned samples in different views of the complete data, while ignoring the information of misaligned data or missing data. In fact, the rational use of information in misaligned data and missing data can greatly improve clustering performance. In addition, most of the learning methods of common semantics are based on traditional machine learning methods, which have the disadvantage of high algorithm complexity and difficulty in scaling on large-scale datasets.

[0024] In order to solve the problem of misaligned views in data, the present invention uses contrastive learning to maximize mutual information to achieve category-level alignment, and restores missing data in the data through dual prediction and minimization of conditional entropy to achieve instance-level alignment, solving the problem of misaligned and missing views in multi-view data. Even when the data is misaligned or has a high missing rate, the best clustering performance can be achieved. All network models are based on deep learning and introduce deep neural networks, which makes the model more generalizable and has good scalability and robustness when facing larger and more complex data sets in real applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the process of the present invention.

[0026] Figure 2 It is a structural diagram of network A of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0028] The present invention is an unsupervised multi-view clustering method for the problem of non-aligned view missing, which is used for the situation where the consistency learning of multi-view data is insufficient in the case of PVP (misaligned) and PSP (view missing), and proposes HmiMVC (multi-view clustering with maximized hierarchical mutual information).

[0029] The concepts and parameters that may be involved in the present invention are introduced as follows:

[0030] The multiple views contained in a sample may be complete or incomplete. In a given data set, multiple view data of multiple samples are included. The multiple view data refers to different angle views of the same sample or different modal views of the same sample at the same angle (such as the color and texture of the image). In the multiple view data, there may be misaligned views or missing views. To address this phenomenon, the unsupervised multi-view clustering method for the problem of missing non-aligned views of the present invention can still maintain good clustering performance when the above situation exists in actual situations.

[0031] refer to Figure 1, the complete process of the present invention is as follows:

[0032] S1, collect initial multi-view data of several samples. For the convenience of explanation and processing, the number of view data of each sample in the present invention is 2. The present invention divides each sample into complete aligned data, unaligned data and missing data, wherein complete aligned data means that both views of the sample are not missing and aligned, unaligned data means that the two views of the sample are not aligned, and missing data means that one of the two views of the sample is missing. Obviously, based on this principle, the expansion of the number of view data in the present invention to other situations has the same processing method.

[0033] The data set of the present invention has two view data, with a total of N samples, wherein each sample in the N samples has complete two view data. The data set is divided into three parts: complete alignment data X N , Unaligned Data S N and missing data W N . N X The view data of samples constitutes the complete alignment data set, N S The view data of samples constitutes the unaligned data set, N W The view data of samples constitute the missing data set, where N = N X +N S +N W In the fully aligned dataset, use and They respectively represent the feature vectors of the Vth view of the Nth sample.

[0034] S2, build network A.

[0035] like Figure 2 As shown, network A consists of two sets of encoders, two sets of decoders and a dual prediction module. The first set of encoders f1 converts the view X (1) Projection into latent representation The second encoder f2 transforms the view X (2) Projection into latent representation Obviously, when the amount of view data is other, the principle is similar. The prediction module is implemented based on contrastive learning and The mutual prediction between them enables it to predict the corresponding aligned view based on a sample in the unaligned data. The dual prediction module is implemented by dual prediction and minimizing conditional entropy. and Fill in the missing data between .

[0036] In this embodiment, the prediction module and the dual prediction module introduced to network A use the information of the original data to complete the category-level data alignment and instance-level data filling alignment. The present invention evaluates the superiority of the present invention on four public datasets. 1) Scene-15 consists of 4485 images distributed in 15 scene categories, using PHOG and GIST features, and two views of 20D and 59D feature vectors. 2) Deep Animal consists of 10158 images of 50 categories, containing two types of features extracted by DECAF and VGG19 as two views, with a feature dimension of 4096. 3) MNIST-USPS handwritten digit dataset, containing 5000 samples of digital images in two different styles. The detailed statistical information of the dataset is summarized in Table 3-1.

[0037] Table 3-1 Dataset summary

[0038] Dataset Sample size Number of categories Dimensions Scene-15 4485 15 20 / 59 Deep Animal 10158 50 64 / 64 MNIST-USPS 5000 10 784 / 784

[0039] S3, train network A using the complete aligned data.

[0040] The complete alignment data is represented as {X (1) ,X (2)}, the aligned view X in the complete data (1) Passed to the encoder f1 of network A to get X (2) Passed to the encoder f2 of network A to get and are the latent representations of the first and second views, respectively.

[0041] based on and Construct three objective functions that need to be further optimized:

[0042] i) Reconstruction loss L obtained by reconstructing different views through the autoencoder rec for:

[0043]

[0044] Among them, N x is the number of samples in the complete aligned data set, N is the total number of sample pairs, represents the vth view in the i-th sample pair in the original view, g v represents the decoder of the vth view, represents the potential representation of the i-th sample pair, which is given by:

[0045]

[0046] Among them, f v The encoder representing the vth view, v∈(1,2);

[0047] ii) Through contrastive learning, maximize and Mutual information between, contrast loss L cl for:

[0048]

[0049] Where N represents the total number of sample pairs, Y = 1 / 0, represents the positive / negative sample pairs, and Represent the loss of the i-th positive sample pair and negative sample pair respectively;

[0050] Loss for positive pairs for:

[0051]

[0052] Among them, f1 and f2 represent the autoencoder, and Represent the two views in the positive sample pair, represents the representation of v views by the i-th sample in the i-th view.

[0053] Loss for negative pairs for:

[0054]

[0055] Among them, max(·) represents the distance between the processed negative samples, and They represent the negative sample pairs formed by the randomly selected i-th view and j-th view in the original view, and the potential representation output after being processed by the autoencoder. τ is the temperature parameter calculated only once in the initial state, and the calculation formula is as follows:

[0056]

[0057] Among them, N p and N n Represent the number of positive sample pairs and negative sample pairs respectively;

[0058] iii) The loss L of the dual prediction module pre for:

[0059]

[0060] Among them, d (1) and d (2) They represent the network that approximately minimizes the conditional entropy, Z(1) and Z (2) They represent the potential representation of mutual prediction between sample pairs;

[0061] Finally, the overall objective function of the training network A is obtained as follows:

[0062] L=L cl +L rec +L pre

[0063] Based on this objective function back propagation, network A is updated by gradient descent. The training of network A with complete aligned data provides a good foundation for subsequent clustering.

[0064] Regarding the selection of evaluation indicators of the present invention, three indicators, namely normalized mutual information (NMI), accuracy (ACC) and adjusted Rand index (ARI), are used to evaluate clustering performance. The higher the values ​​of these evaluation indicators, the better the clustering performance.

[0065] Table 3-2 shows the loss components and experimental results corresponding to each component in the method. In (1), it can be seen that due to L pre It is not optimized, so the low-dimensional potential representation will lose more complementary information. Therefore, the contrast prediction loss L is used alone. pre It may lead to trivial solutions or model collapse. By comparing (4)(5)(6) with (1)(2)(3), it can be seen that the double loss is more effective than the single consistency learning module. By comparing (7) with (2), it can be asserted that the two modules of contrastive learning and mutual prediction have greatly improved the clustering effect. It is worth noting that each module of the present invention improves the clustering performance, which further proves the effectiveness of the method.

[0066] Table 3-2 Ablation experiment on MNIST-USPS dataset

[0067] Loss Item Combination NMI ACC ARI (1) <![CDATA[L pre ]]> 0.169 0.211 0.092 (2) <![CDATA[L rec ]]> 0.329 0.457 0.242 (3) <![CDATA[L cl ]]> 0.485 0.541 0.234 (4) <![CDATA[L rec +L pre ]]> 0.486 0.523 0.347 (5) <![CDATA[L pre +L cl ]]> 0.539 0.607 0.413 (6) <![CDATA[L rec +L cl ]]> 0.693 0.760 0.675 (7) <![CDATA[L rec +L pre +L cl ]]> 0.726 0.782 0.658

[0068] Table 3-3 describes the clustering performance of the present invention under data sets of different sizes. It can be seen that the present invention can effectively process the misaligned data and missing data in the incomplete data, and significantly improve the clustering performance.

[0069] Table 3-3 Clustering performance on different data sets

[0070]

[0071] In summary, the present invention first projects the original data set into a hierarchical latent space, and solves the PVP problem by maximizing the cross-view mutual information through the prediction module to achieve category-level alignment. Secondly, for the PSP problem in the PVP problem, a dual prediction module is used to predict and restore the missing data, thereby achieving natural category-level alignment. Finally, the two-level alignment strategies are merged into a unified reconstruction process. HmiMVC adopts different hierarchical alignment strategies and methods for PSP and PVP, but consistency learning in turn reinforces PSP and PVP learning. The present invention combines category-level alignment and instance-level recovery alignment, effectively overcoming the problem of insufficient consistency learning and significantly improving clustering performance.

Claims

1. An unsupervised multi-view clustering method for the problem of non-aligned view loss, characterized in that: The steps include: Step 1, collecting initial multi-view data of several samples, the number of view data of each sample is 2, and dividing each sample into complete aligned data, misaligned data and missing data; the complete aligned data means that both views of the sample are not missing and aligned, the misaligned data means that the two views of the sample are not aligned, and the missing data means that one of the two views of the sample is missing; the misaligned data and missing data constitute incomplete data; Step 2, build network A; The network A is composed of two groups of autoencoders, a prediction module and a dual prediction module; each group of autoencoders includes an encoder and a decoder, and the encoder converts two views X of a sample (1) and X (2) Projected into latent representations and The prediction module is based on contrastive learning. According to a sample in the incomplete data, it predicts the corresponding category-level aligned view, thereby achieving and The dual prediction module fills in missing data by dual prediction and minimizing conditional entropy, achieving natural instance-level alignment. and Construct three objective functions: i) Reconstruction loss L obtained by reconstructing different views through the autoencoder rec for: Among them, N x is the number of samples in the complete aligned data set, N is the total number of sample pairs, represents the vth view in the i-th sample pair in the original view, g v represents the decoder of the vth view, represents the potential representation of the i-th sample pair, which is given by: Among them, f v The encoder representing the vth view, v∈(1,2); ii) Through contrastive learning, maximize and Mutual information between, contrast loss L cl for: Where N represents the total number of sample pairs, Y = 1 / 0, represents the positive / negative sample pairs, and Represent the loss of the i-th positive sample pair and negative sample pair respectively; Loss for positive pairs for: Among them, f1 and f2 represent the autoencoder, and Represent the two views in the positive sample pair, represents the representation of v views by the i-th sample in the i-th view; Loss for negative pairs for: Among them, max(·) represents the distance between the processed negative samples, and They represent the negative sample pairs formed by the randomly selected i-th view and j-th view in the original view, and the potential representation output after being processed by the autoencoder. τ is the temperature parameter calculated only once in the initial state, and the calculation formula is as follows: Among them, N p and N n Represent the number of positive sample pairs and negative sample pairs respectively; iii) The loss L of the dual prediction module pre for: Among them, d (1) and d (2) They represent the network that approximately minimizes the conditional entropy, Z (1) and Z (2) They represent the potential representation of mutual prediction between sample pairs; Finally, the overall objective function of the training network A is obtained as follows: L=L cl +L rec +L pre Step 3, train network A using the complete aligned data; Step 4, test the trained network A using complete aligned data, unaligned data, and missing data; Step 5: The original multi-view data is passed to network A for processing and clustering, and the Kmeans clustering algorithm is used on the public representation to obtain the final clustering result.

2. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: In step 1, the two views of the sample are views of the same sample at different angles, or views of the same sample at the same angle in different modes.

3. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: In the step 2, in network A, the sample and itself are taken as a positive sample pair, and cross-view samples are randomly selected to form a negative sample pair.

4. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: In the step 2 network A, the prediction module uses a contrastive learning network, whose architecture is to use two sets of autoencoders f1 and f2 to transform the view X (1) and X (2) Projected into latent representations and Through contrastive learning, the mutual information between positive samples is minimized and the mutual information between negative samples is maximized.

5. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: In step 2, network A is based on and Compute prediction loss, noise-robust contrast loss, and view reconstruction loss; maximize via contrastive learning and to achieve the mutual information between and Category-level alignment.

6. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: In the step 2 network A, the dual prediction module adopts a dual prediction network and an approximately minimized conditional entropy network, and its architecture includes a group of autoencoders, in which the dual prediction and the approximately minimized conditional entropy realize the filling of missing data and achieve natural instance-level alignment.

7. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: The step 3 divides the positive and negative samples in the complete aligned data, uses the aligned data as positive samples, randomly selects cross samples as negative samples, and then performs training.

8. The unsupervised multi-view clustering method for the non-aligned view missing problem according to claim 1, characterized in that: In step 5, the original multi-view data is passed to the trained network A, the unaligned data is aligned at the category level by the prediction module, the missing data is restored and aligned at the instance level by the dual prediction module, and a multi-view high-dimensional public representation is obtained; based on the public representation, the Kmeans clustering algorithm is used to obtain the final clustering result.

Images