A multi-view clustering method based on fine-grained contrastive learning and feature fusion

Abstract

The application discloses a multi-view clustering method based on fine-grained contrast learning and feature fusion and belongs to the technical field of computer vision. The application dynamically adjusts the fusion weight by calculating the mutual information score of each view feature and the fusion feature, so that the view mainly containing more information is mainly concerned. The in-view sample adjacent relationship is obtained through the obtained pseudo-label probability matrix, and the strong and weak relationships are introduced by comprehensively considering the sample relationships between all views, stronger contrast learning is performed on the strong sample pairs, and the distance between the strong sample pairs is shortened or lengthened. Furthermore, a new technical solution is brought for further solving the multi-view clustering problem.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a multi-view clustering method based on fine-grained contrastive learning and feature fusion. BACKGROUND

[0002] Multiple views can often provide more comprehensive information for data analysis. In order to more comprehensively and accurately describe data objects, multiple views should be fully used for complementation. At the same time, since multiple views describe the same object, part of the features learned from different views should be semantically shared and associated, and such semantic consistency ensures the complementarity between views. The multi-view clustering goal is to maximize the mining of complementary information in views and maintain semantic consistency to make data more separable under unsupervised conditions.

[0003] Current multi-view clustering methods mainly include traditional machine learning and deep learning-based methods. Traditional multi-view clustering methods can be divided into the following categories, which are respectively based on multi-kernel methods, nonlinear transformation in high-dimensional space is found to enable data in different views to be clustered in the space. Subspace-based methods learn a common latent subspace to enable data in different views to be clustered in the subspace. Non-negative matrix factorization-based methods. The data matrix is decomposed into the product of two non-negative matrices, so as to represent the data in a low-dimensional space and cluster in the low-dimensional space.

[0004] With the increase of computing power and the powerful feature extraction and representation ability of deep networks, many deep multi-view clustering models have emerged in recent years. Although they all have their own model architecture and working methods, they can be divided into a fixed component set. That is, current methods usually follow the first step of stacking view-specific autoencoders for each view to generate view-specific representations. Supervised tasks can be divided into two categories, namely single-view self-supervision and multi-view self-supervision. The difference between the two is that the former only uses a single view, while the latter simultaneously uses all views, allowing the model to simultaneously use information from all views to learn better features. Some models also design a fusion module to combine view-specific representations into shared representations for all views. The fusion operation can be completed using simple weighted averaging and concatenation operations, of course, some methods also use attention mechanisms or design a separate network to learn view fusion weights. Some methods also choose to combine with traditional methods in the final clustering layer, and use K-means to further improve clustering performance.

[0005] As a powerful learning method in unsupervised tasks, contrastive learning has been applied more and more widely in multi-view clustering tasks. Its purpose is to learn a useful feature representation by maximizing the similarity between similar samples and minimizing the similarity between different samples. Therefore, the contrastive learning model can better capture the relationship between samples and samples, improve the tightness and separation of clustering, and thus improve the overall quality of clustering. In terms of consistency and complementarity, two important criteria of multi-view clustering: the contrast-based method can effectively fuse information from different views, learn a fused feature representation, so that the information from different views can be complementary, and by contrasting the same samples under different views, the model can learn the consistency between views, thereby improving the overall quality of clustering.

[0006] The above method has achieved good performance, but there are still some key problems to be solved. For example:

[0007] (1) In the fusion of features of different views, how to determine the weight of each view is a key problem, some methods obtain view weights by using average weighting or cascading operation, and some methods use attention or feedforward neural network to obtain adaptive fusion weight, but for the quality of the view and the richness of the information contained, there is a lack of explainability, and there is no intuitive loss to constrain.

[0008] (2) In addition, the current contrastive learning method mainly emphasizes the similarity of different views of the same sample, and only takes the views between each other as positive samples, and the rest as negative samples. When performing instance-level feature contrastive learning, only M-1 pairs of different views of the same sample are taken as positive pairs, and the remaining M(N-1) pairs are taken as negative samples. Such positive and negative sample selection method moves away the samples belonging to the same class, and this unstable contrastive learning directly affects the clustering quality.

[0009] The above problems need to be solved, therefore, the present application provides a multi-view clustering method based on fine-grained contrastive learning and feature fusion. SUMMARY

[0010] The technical problem to be solved by the present application is how to solve the lack of explainability in the existing method in the multi-view clustering task, and the problem that unstable contrastive learning directly affects the clustering quality, and provides a multi-view clustering method based on fine-grained contrastive learning and feature fusion, which brings a new technical solution for further solving the multi-view clustering problem.

[0011] The present application solves the above technical problems by the following technical solutions, the present application comprises the following steps:

[0012] S1: prepare an original data set, the original data set including multiple views of multiple samples;

[0013] S2: input each view into an encoder E (v) to extract latent features, and then use a decoder D (v) to obtain a reconstructed representation, and introduce a reconstruction loss L rec , wherein the decoder is symmetrical with the encoder structure;

[0014] S3: after extracting view features, the classification probability of the sample is output by superimposing shared linear layers and using a softmax function in the last layer, and a consistency loss L PC is introduced to realize end-to-end clustering assignment;

[0015] S4: obtain fused features by initializing average weight weighting, the fused features including global information of all views, taking global pseudo label probability as a base value, dynamically obtaining an adjustment factor by calculating mutual information score of each view pseudo label and global pseudo label probability matrix to adjust adaptive weight of view fusion; and imposing corresponding loss constraints on feature level and pseudo label probability soft assignment level;

[0016] S5: for each view pseudo label assignment Q V , a broadcast operation is performed on the pseudo label of the sample to obtain a Boolean matrix representing the adjacency relationship within each view, the sum of the Boolean matrix is obtained to obtain a global adjacency relationship matrix G, strong pseudo label contrast learning and weak pseudo label contrast learning are performed based on the global adjacency relationship matrix G, and a feature contrast loss is defined;

[0017] S6: based on the above learning process, a comprehensive loss function L all is constructed;

[0018] S7: the model is trained using the comprehensive loss function L all , until the model converges or reaches a termination condition, and the trained model is saved;

[0019] S8: the saved model is applied to a multi-view clustering task to complete the multi-view clustering work.

[0020] Further, in the step S2, the specific processing process is as follows:

[0021] S21: obtain the latent representation of the vth view through the encoder, i.e. latent feature:

[0022]

[0023] S22: reconstruct the sample by decoding the fused representation

[0024]

[0025] wherein, denotes the feature obtained after decoding;

[0026] S23: Introduce reconstruction loss to reduce the difference between the original feature representation and the reconstructed feature representation:

[0027]

[0028] wherein, and denote the features obtained after encoding and decoding of the ith sample and the vth view, respectively, and V and N denote the number of views and samples.

[0029] Further, in the step S3, the specific processing process is as follows:

[0030] S31: Calculate the pseudo label probability matrix Z of each view v :

[0031] Z v = Softmax(MLP(H v ))

[0032] wherein, MLP is a fully connected layer shared by all views, and Softmax is stacked in the last layer to obtain the label probability distribution of the sample;

[0033] S32: Scale the label probability using the following formula:

[0034]

[0035] wherein, denotes that the ith sample belongs to the jth class in the vth view, and the pseudo label of the sample is identified by the largest element in the cluster assignment;

[0036] S33: Take the same column of each pseudo label probability matrix as (M-1) columns of positive samples, and take all other columns as M(K-1) columns of negative samples. Ideally, each row of P is a one-hot vector. The similarity of the pseudo label probability is calculated using matrix inner product:

[0037]

[0038] wherein, M is the number of views, K is the total number of classes of samples, and P represents the label probability matrix predicted by each view; denotes the probability distribution of belonging to the jth class in the view v1, denotes the probability distribution of belonging to the jth class in the view v2;

[0039] S34: consistency loss L of introduced pseudo label PC The definitions are as follows:

[0040]

[0041]

[0042]

[0043] wherein, α and β respectively represent the hyperparameter coefficients of the corresponding loss.

[0044] Further, in the step S4, the formula for adjusting the adaptive weight of view fusion is as follows:

[0045]

[0046]

[0047] wherein, the fusion feature H F is obtained by w v weighted summation, w v is obtained by calculating the NMI normalized mutual information score of each view label Q v and the fusion view label Q F , that is, the adjustment factor of the adaptive weight.

[0048] Further, in the step S4, at the feature level, the structural relationship of multi-view is integrated by maximizing the mutual information between the fusion feature and the specific feature of each view, which is defined as follows:

[0049]

[0050] wherein, I is used to measure the mutual information score between two features;

[0051] At the level of pseudo label probability soft assignment, the fusion feature obtains the pseudo label through the classifier, according to the consistency of multi-view data, on the one hand, the mean square error is used to keep the consistency of the pseudo label of each specific view and the pseudo label of the fusion feature at the semantic level, on the other hand, the adaptive weight is used to sum the specific view pseudo label probability and the global pseudo label loss:

[0052]

[0053]

[0054] wherein, P f is obtained by w v weighted summation, P FThe pseudo label probability obtained by fusing the features.

[0055] Further, in the step S5, the specific processing process is as follows:

[0056] S51: Pseudo label assignment Q for each view V The broadcast operation is performed on the pseudo labels of the samples to obtain a Boolean matrix representing the adjacency relationship within each view, and the global adjacency relationship matrix G is obtained by summing the Boolean matrices. Each element of the global adjacency relationship matrix G is g ij ;

[0057] S52: In strong pseudo label contrast learning, the cosine similarity between sample features is calculated first:

[0058]

[0059] Where < > represents the inner product of vectors. The strong relationship in the global adjacency relationship matrix G is subjected to a MASK operation, that is, only the cases of 0 and M are calculated in strong pseudo label contrast learning.

[0060] Then, cross-entropy loss is used for contrast learning of the features:

[0061]

[0062] Where g ij The value of sample i and sample j in the corresponding view is consistent, represents the latent feature of sample i in view v1, represents the latent feature of sample j in view v2.

[0063] S53: In weak pseudo label contrast learning, all sample pairs participate in the calculation, and the pseudo label relationship matrix of the fused features is additionally introduced to supplement the internal structure information of the samples. The value of each element in the global adjacency relationship matrix G is between 0 and M+1. In weak pseudo label contrast learning, values greater than or equal to M are still regarded as positive samples, and the rest are regarded as negative samples, which can effectively avoid the case that samples of the same class have inconsistent labels in some views due to noise:

[0064]

[0065] S54: The feature contrast loss is defined as follows:

[0066]

[0067] Where λ and respectively represent the loss hyperparameter coefficients of strong and weak pseudo label contrast learning.

[0068] Further, in the step S6, the comprehensive loss function is defined as follows:

[0069] L all = L rec + L PC + L M + L D + L FC .

[0070] Compared with the prior art, the present application has the following advantages:

[0071] 1. Dynamically adjusting the view fusion weight

[0072] The existing multi-view clustering method usually adopts average weight or splicing to obtain fused features. This average method usually does not consider the richness of the information contained in each view. The present application dynamically adjusts the fusion weight by calculating the mutual information score of each view feature and the fused feature, so that the view mainly containing more information will be mainly concerned.

[0073] 2. Strong and weak label guided feature contrast learning

[0074] The previous feature contrast learning method only regards different views of the same sample as positive samples, and other same view different view sample pairs are regarded as a negative sample relationship. This coarse-grained method will inevitably pull apart some samples belonging to the same class. The present application obtains the view-in sample adjacency relationship through the obtained pseudo-label probability matrix, and introduces the strong and weak relationship by comprehensively considering all sample relationships between views, and performs stronger contrast learning between strong sample pairs to pull or pull apart the distance between them. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is a flowchart of the multi-view clustering method based on fine-grained contrast learning and feature fusion in the second embodiment of the present application;

[0076] Figure 2 is a schematic diagram of the existing contrast learning in the third embodiment of the present application. DETAILED DESCRIPTION

[0077] The embodiments of the present application will be described in detail below. The present embodiments are implemented on the premise of the technical scheme of the present application, and detailed implementation modes and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0078] Embodiment one

[0079] Multi-view clustering aims to utilize multiple views of the same object that can be observed to jointly learn to obtain a better consensus representation than single-view, and then obtain a more separable clustering representation. Deep learning-based methods have shown strong learning ability for this task. However, most existing deep clustering methods adopt a contrastive learning-based method, and undoubtedly, the selection of positive and negative samples will greatly affect the final result, so selecting appropriate positive and negative samples will be the core task of contrastive learning. In addition, in the feature fusion-based method, the design of a separate network to learn the weight lacks factual basis and thus it is difficult to obtain more accurate view weight. In the embodiment, a new multi-view clustering method is proposed to solve the above problems. Specifically, by comparing the similarity of each view and the fused view, the adaptive weight of view fusion is adjusted; and a new selection method of positive and negative samples is proposed, which uses all view and fused view information to select high-confidence samples, and then uses the contrastive learning method to make the clustering more separable.

[0080] In the embodiment, the target is achieved through the following main steps:

[0081] Step one: stack specific view encoding

[0082] The original multi-view data usually contains noise and redundant information, which may affect the clustering effect. The encoder (stacked specific view encoder) can effectively reduce noise and redundancy by learning the potential structure of the data, and improve the robustness of clustering. Each view in the multi-view data set has different feature dimensions, and these features can be very high-dimensional and sparse. The encoder maps the data of different views to a shared latent representation space, so that these representations are aligned in the same space, making it easier to discover the association and similarity between different views, which not only reduces the computational complexity, but also extracts the key features of the data and removes redundant information.

[0083] Step two: clustering-level contrastive learning

[0084] After extracting the view features, the clustering assignment is realized by stacking the shared linear layer and using the softmax function to output the classification probability of the sample at the last layer. At the same time, the pseudo-label of the sample is identified by the largest element in the cluster assignment. Different views only describe the sample more comprehensively, but they should also describe the same object, so the pseudo-label probability of each view should be aligned, that is, each view should represent the same class, which can also constrain each view to have the same predicted semantic label; the consistency goal is achieved by using contrastive learning.

[0085] Step three: feature fusion and calibration

[0086] Firstly, the fusion weight of each view is initialized to obtain a fusion feature and the global pseudo label is obtained by the same classifier, the pseudo label of each view and the global pseudo label are calculated NMI score, and the fusion weight is adjusted in the opposite direction, so that the good view can be paid more attention, in order to ensure the effectiveness of the fusion feature, the mutual information maximization loss is used in the feature level, and the mean square error loss is used in the semantic label level.

[0087] Step four: feature level contrast learning

[0088] According to the above steps, the pseudo label results of each view and the fusion feature have been obtained. The pseudo label of each view is broadcasted to form a two-dimensional sample relationship matrix, and the relationship matrices are added to obtain a global relationship guide matrix, so that when the value in the matrix ranges from zero to the number of views, when it is equal to zero, it can be considered that there is no relationship between the samples in all views belonging to the same class number, which is defined as a strong negative sample pair. Similarly, when the value is equal to the number of views, it is considered to belong to the same class in all views, which is defined as a strong positive sample pair. Of course, this rule setting is very strict, and it is inevitable that there will be noise in some views, causing misjudgment. Therefore, in the second round of discrimination, the pseudo label result of the fusion feature is additionally introduced to reduce the influence of noise and enhance the robustness of the method. The sample relationship in the fusion feature is added to the relationship matrix obtained in the last step to supplement the relationship between the samples. The sample pair with the value of the number of views in the new relationship matrix is still considered as a positive pair, and the rest is still considered as a negative pair, which can avoid the influence of noise to a certain extent. About the setting of strong and weak contrast, firstly, the contrast times of strong sample pairs are twice, and secondly, the loss coefficient is controlled by a hyperparameter, so that important relationships can be compared more strongly.

[0089] In the present application, the following solutions are adopted to solve the problems in the background art:

[0090] (1) Fusion calibration

[0091] In the present application, the fusion features are obtained by initializing the average weight weighting, which contains the global information of all views, and the global pseudo label probability is taken as the base value, and the mutual information score of the pseudo label of each view and the global pseudo label probability matrix is calculated to dynamically obtain the adjustment factor to adjust the adaptive weight; in order to make the fusion features more comprehensive and complementary, loss constraints are imposed on two levels: at the feature level, the structural relationship of multi-view is integrated by maximizing the mutual information between the fusion features and the specific features of each view; on the soft distribution level of the pseudo label probability, the pseudo label is obtained by the fusion features through the classifier, and according to the consistency of multi-view data, on the one hand, the mean square error is used to keep the consistency of the pseudo label of each specific view and the pseudo label of the fusion features at the semantic level, and on the other hand, the adaptive weight is used to sum the pseudo label probability of the specific view and the loss of the global pseudo label probability.

[0092] (2) Fine-grained contrast

[0093] In the present application, a new positive and negative sample selection strategy is proposed, and different confidence levels are given for different intensity contrast learning; the internal structural relationship between samples in the pseudo label map obtained by combining all views including fusion features is mined, and positive and negative samples are selected by comprehensive judgment, and stronger contrast learning is performed on samples with higher confidence; specifically, the pseudo label of each sample in each view is broadcasted to obtain a Boolean matrix representing the adjacency relationship in each view, and the global adjacency relationship can be obtained by summing these matrices, based on which the relationship guide matrix is proposed, and strong and weak instance feature level contrast learning is proposed.

[0094] Embodiment two

[0095] As shown in Figure 1 , the multi-view clustering method based on fine-grained contrast learning and feature fusion provided in the present embodiment includes the following steps:

[0096] Step 1: Prepare the original data set (Scene-15, BDGP, Fashion, MSRC-v1, MNIST-USPS, Caltech-5V);

[0097] Step 2: Equip each view with an independent encoder E (v) and a decoder D (v) , wherein the encoder is used to extract feature representation, the decoder is symmetrical to the encoder and is used to obtain reconstruction representation, in order to ensure the effectiveness of the encoder and the decoder, a reconstruction loss function L rec is defined, so that the reconstructed data is as close to the original data as possible. The mean square error (MSE) is used as the reconstruction loss function.

[0098] In this step, the specific processing process is as follows:

[0099] 1. Obtain the latent representation of the v-th view, i.e., the latent feature, by encoder:

[0100]

[0101] where H v represents the latent feature obtained by encoding the v-th view;

[0102] 2. For subsequent feature fusion and contrastive learning, the decoder symmetrical to the encoder reconstructs the sample into

[0103]

[0104] where, represents the feature obtained after decoding.

[0105] 3. Introduce reconstruction loss to reduce the difference between the original feature representation and the reconstructed feature representation:

[0106]

[0107] where, and represent the features obtained by encoding and decoding the i-th sample and the v-th view, respectively, and V and N represent the number of views and samples.

[0108] Step 3: Cluster-level contrastive learning. After extracting the view features, implement end-to-end clustering assignment by stacking shared linear layers and using softmax function in the last layer to output the classification probability of the sample.

[0109] In this step, the specific process is as follows:

[0110] 1. Obtain the pseudo-label probability matrix Z of each view using the following formula: v :

[0111] Z v = Softmax(MLP(H v ))

[0112] where MLP is a fully connected layer shared by all views, and the label probability distribution of the sample is obtained by stacking Softmax in the last layer;

[0113] 2. In order to increase the difference of class probability, use the following formula to scale the label probability:

[0114]

[0115] where, represents the ith sample belongs to the jth class in the vth view, and the pseudo label of the sample is identified by the largest element in the cluster assignment;

[0116] 3、Different views are only more comprehensive descriptions of samples, but they should also describe the same object, so the pseudo label probabilities of each view should be aligned, that is, each view should obtain a column of the pseudo label probability matrix, which represents the same class, which can also constrain each view to have the same predicted semantic label. The consistency goal is achieved by using contrastive learning. Specifically, the same column of each pseudo label probability matrix is taken as a positive sample, and all other columns are taken as negative samples. Ideally, each row of P is a one-hot vector, and the similarity of pseudo label probabilities is calculated using matrix inner product:

[0117]

[0118] where M is the number of views, K is the total number of classes of samples, and P represents the label probability matrix predicted by each view; represents the probability distribution of the jth class in view v1, represents the probability distribution of the jth class in view v2;

[0119] Therefore, the consistency loss of the pseudo label is defined as:

[0120]

[0121]

[0122]

[0123] where α and β represent the hyperparameter coefficients of the corresponding loss.

[0124] The first part of the loss aims to learn the clustering consistency of all views, so that different views of each sample have the same semantic label, and the second part is a regularization term to avoid assigning all samples to a single cluster.

[0125] Step 4: In this step, in order to obtain a good fusion feature, the fusion feature is obtained by weighting with the initialized average weight, which includes the global information of all views. The global pseudo label probability is taken as the base value, and the mutual information score of each view pseudo label and the global pseudo label probability matrix is calculated to dynamically obtain the adjustment factor, which is used to adjust the adaptive weight of view fusion:

[0126]

[0127]

[0128] where H is the fusion featureF by each view feature by w v weighted summation, w v by calculating the NMI normalized mutual information score of each view label Q v and the fusion view label Q F .

[0129] To make the fusion feature representation more comprehensive and complementary, loss constraints are imposed on two levels: at the feature level, the structural relationship of multi-view is integrated by maximizing the mutual information between the fusion feature and the specific feature of each view:

[0130]

[0131] where I is used to measure the mutual information score between two features.

[0132] At the level of soft assignment of pseudo-label probability, the fusion feature goes through the classifier to get the pseudo-label, according to the consistency of multi-view data, on the one hand, the mean square error is used to keep the consistency of the pseudo-label of each specific view and the pseudo-label of the fusion feature at the semantic level, on the other hand, the adaptive weight is used to sum the specific view pseudo-label probability and the global pseudo-label loss:

[0133]

[0134]

[0135] where P f is obtained according to the view fusion factor (adjusting factor) and the label assignment probability P v of each view, P F represents the pseudo-label probability obtained by the fusion feature.

[0136] Step 5: instance feature-level contrastive learning. For each view, the pseudo-label assignment Q V , the pseudo-label of the sample is broadcasted to get a Boolean matrix representing the adjacency relationship within each view, and the global adjacency relationship matrix G can be obtained by summing the Boolean matrix, where the element g ij represents how many views the sample i and sample j have consistent predictions, for example, when g ij = 0, it means that the two samples do not belong to the same class in any view, when g ij = M, it means that the two samples have the same semantic label in all views. When the value is M and 0, it can be naturally considered that these sample pairs have the strongest positive and negative relationship, in strong pseudo-label contrast, we only calculate this strongest positive and negative relationship, first calculate the cosine similarity between sample features:

[0137]

[0138] Where, < > represents the vector inner product, and the strong relationship in the above global relationship matrix (global adjacency relationship matrix G) is subjected to the MASK operation, that is, only the cases of 0 and M are calculated in the strong pseudo label comparison.

[0139] MASK = (G = 0) & (G = M)

[0140] Use cross-entropy loss for feature contrast learning:

[0141]

[0142] Where:

[0143]

[0144] When g ij =M, the samples are more similar, and when g ij =0, the sample distance is pulled away, and the similarity is reduced. Strong pseudo label comparison learning is also performed on the intra-view and fusion features to explore more rich consistency information.

[0145] The next step is to perform weak pseudo label comparison. The difference between strong pseudo label comparison and weak pseudo label comparison is that all sample pairs participate in the calculation, and the pseudo label relationship matrix of the fusion feature is additionally introduced to supplement the internal structure information of the sample. The value of the matrix will be between 0 and M+1. In weak pseudo label comparison, values greater than or equal to M are still considered as positive samples, and the rest are considered as negative samples (such selection can be considered as 80% confidence). This can effectively avoid the case that the same class samples have inconsistent labels in some views due to noise:

[0146]

[0147] Overall, the feature contrast loss is defined as:

[0148]

[0149] Where, λ and respectively represent the loss hyperparameter coefficients of strong and weak contrast. L sp and L wp represent the strong and weak feature contrast losses, respectively.

[0150] Step 6: To better optimize the model, based on the above learning method, a comprehensive loss is constructed:

[0151] L all =L rec +L PC +L M +L D +LFC

[0152] Step 7: Use the following formula to obtain the label of the sample:

[0153]

[0154] Step 8: Repeat steps 2-7 above until the model converges or the termination condition is reached;

[0155] Step 9: Set the number of training rounds until the model converges or reaches the termination condition and save the model;

[0156] Step 10: Apply the saved model to the multi-view clustering task to complete the multi-view clustering work.

[0157] Example 3

[0158] like Figure 2 As shown in the figure, in the multi-view clustering task, this paper proposes a new method for constructing sample relationship pairs. Specifically, for samples 1 and 2, they obtain consistent prediction results in all views, so they are subjected to strong feature contrast learning. Similarly, for samples 1 and 5, they do not obtain consistent prediction results in all views, so they are subjected to strong distance learning. Subsequently, pseudo-labels based on fused features are introduced to explore more structural relationships between samples in different views.

[0159] To address the above issues, this embodiment uses the multi-view clustering method based on fine-grained contrastive learning and feature fusion described in Example 1 for contrastive learning. Experimental results on five multi-view datasets show that the method of the present invention demonstrates significant effectiveness and superiority compared to existing methods. The experimental results for some datasets are shown in Table 1:

[0160] Table 1 Experimental results of some datasets

[0161]

[0162] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A multi-view clustering method based on fine-grained contrastive learning and feature fusion, characterized by: The following steps are involved: S1: Prepare the original dataset, which includes multiple views of multiple samples; S2: Feed each input view into the encoder E (v) Extract the latent features and then use the decoder D (v) Get the reconstructed representation and introduce the reconstruction loss L rec , where the decoder and encoder have symmetrical structures; S3: After extracting the view features, the classification probability of the sample is output by stacking shared linear layers and using the softmax function in the last layer, and the consistency loss L is introduced PC , to achieve end-to-end cluster allocation; S4: Obtain fusion features by initializing average weights. The fusion features include global information from all views. The global pseudo-label probability is used as the base value. The adjustment factor is dynamically obtained by calculating and evaluating the mutual information score between the pseudo-label of each view and the global pseudo-label probability matrix to adjust the adaptive weight of view fusion. And impose corresponding loss constraints at the feature level and the pseudo-label probability soft assignment level; S5: Pseudo label assignment Q for each view V , the pseudo labels of the samples are broadcasted to obtain a Boolean matrix within each view to represent the adjacency relationship. The global adjacency matrix G is obtained by summing the Boolean matrices. Strong pseudo-label contrast learning and weak pseudo-label contrast learning are performed based on the global adjacency matrix G, and the feature contrast loss is defined; S6: Based on the above learning process, construct a comprehensive loss function L all ; S7: Using the comprehensive loss function L all Train the model until it converges or reaches the termination condition and save the trained model; S8: Apply the saved model to the multi-view clustering task to complete the multi-view clustering work.

2. A multi-view clustering method based on fine-grained contrastive learning and feature fusion according to claim 1, characterized in that: In step S2, the specific processing process is as follows: S21: Obtain the potential representation of the v-th view through the encoder, that is, the potential features: S22: Reconstruct the sample by decoding the fusion representation in, Represents the features obtained after decoding; S23: Reconstruction loss is introduced to reduce the difference between the original feature representation and the reconstructed feature representation: in, and They represent the features obtained by encoding and decoding the i-th sample and the v-th view respectively, and V and N represent the number of views and samples.

3. The multi-view clustering method based on fine-grained contrastive learning and feature fusion according to claim 2, characterized in that: In step S3, the specific processing process is as follows: S31: Calculate the pseudo label probability matrix Z for each view v : Z v =Softmax(MLP(H v )) Among them, MLP is a fully connected layer shared by all views, and Softmax is stacked on the last layer to obtain the label probability distribution of the sample; S32: Use the following formula to scale the label probability: in, Indicates that the i-th sample belongs to the j-th class in the v-th view, and the pseudo label of the sample is identified by the largest element in the cluster assignment; S33: Take the same column (M-1) of each pseudo-label probability matrix as positive samples, and all other columns (M(K-1)) as negative samples. Ideally, each row of P is a one-hot vector, and use the matrix inner product to calculate the similarity of the pseudo-label probability: Where M is the number of views, K is the total number of sample categories, and P represents the label probability matrix predicted for each view; represents the probability distribution of belonging to class j in view v1, represents the probability distribution of belonging to the jth class in view v2; S34: Introduced pseudo-label consistency loss L PC The definition is as follows: Among them, α and β represent the hyperparameter coefficients of the corresponding loss respectively.

4. The multi-view clustering method based on fine-grained contrastive learning and feature fusion according to claim 3, characterized in that: In step S4, the formula for adjusting the adaptive weight of view fusion is as follows: Among them, the fusion feature H F Each view feature is passed through w v The weighted summation is w v By calculating each view label Q v With Fusion View Label Q F The NMI normalized mutual information score is obtained, which is the adjustment factor of the adaptive weight.

5. The multi-view clustering method based on fine-grained contrastive learning and feature fusion according to claim 4, characterized in that: In step S4, at the feature level, the structural relationship of multiple views is integrated by maximizing the mutual information between the fused features and the specific features of each view, which is defined as follows: Among them, I is used to measure the mutual information score between two features; At the level of soft assignment of pseudo-label probabilities, the fused features are passed through a classifier to obtain pseudo-labels. Based on the consistency of multi-view data, the mean squared error is used to maintain semantic consistency between the pseudo-labels of each specific view and the pseudo-labels of the fused features. On the other hand, adaptive weights are used to sum the pseudo-label probabilities of specific views and the global pseudo-label to form a loss: Among them, P f Assign probability P to each view based on the obtained adjustment factor and label v The weighted summation is P F Represents the pseudo label probability obtained by fusion features.

6. The multi-view clustering method based on fine-grained contrastive learning and feature fusion according to claim 5, characterized in that: In step S5, the specific processing process is as follows: S51: Pseudo label assignment Q for each view V , the pseudo labels of the samples are broadcasted to obtain a Boolean matrix within each view to represent the adjacency relationship, and the global adjacency matrix G is obtained by summing the Boolean matrices. The elements of the global adjacency matrix G are g ij ; S52: In strong pseudo-label contrast learning, first calculate the cosine similarity between sample features: Among them, <> represents the vector inner product, and the strong relationship in the global adjacency matrix G is masked, that is, only the cases of 0 and M are calculated in the strong pseudo-label contrast learning; Then use cross entropy loss to perform contrastive learning on the features: Among them, g ij The value of represents that sample i and sample j are predicted to be consistent in the view of the corresponding value. represents the potential features of sample i in view v1, Represents the potential features of sample j in view v2; S53: In weak pseudo-label contrastive learning, all sample pairs participate in the calculation, and an additional pseudo-label relationship matrix of fused features is introduced to supplement the intrinsic structural information of the samples. The value of each element in the global adjacency relationship matrix G is between 0 and M+1. In weak pseudo-label contrastive learning, samples with a value greater than or equal to M are still considered positive samples, and the rest are considered negative samples. This can effectively avoid the situation where the labels of similar samples in some views are inconsistent due to noise: S54: Define feature contrast loss as follows: Among them, λ and Represent the loss hyper-parameter coefficients of strong and weak pseudo-label contrast learning respectively.

7. The multi-view clustering method based on fine-grained contrastive learning and feature fusion according to claim 6, characterized in that: In step S6, the comprehensive loss function is defined as follows: L all =L rec +L PC +L M +L D +L FC 。

Images