Incomplete multi-view comparative learning method based on cross-view coding and graph constraint

Abstract

The invention discloses an incomplete multi-view comparative learning method based on cross-view coding and graph constraint, and belongs to the technical field of computer vision. According to the method, sample advanced representation is captured through a cross-view encoder, a relation graph is constructed by taking the sample advanced representation as a center, and due to the fact that different view features of the same instance are consistent in semantics, the same instance in different views can share similar relation graphs, and through the mode, the relation graph can be shared by the same instance in different views. The advanced feature representations of different views can be drawn close and aligned in the feature space; then, self-supervised high-confidence guidance is adopted to further enhance self representation learning; besides, in order to pay more attention to the neighborhood relation between the instances, reconstruction data are reversely pushed into the representation learning process by introducing graph constraints, and the reconstruction data are allowed to cooperate with one another to achieve better clustering.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an incomplete multi-view contrast learning method based on cross-view encoding and graph constraints. Background Art

[0002] Clustering, as an important unsupervised method, has always been a basic technique in the field of pattern recognition and machine learning. This technique organizes data samples according to specific criteria to ensure that similar samples are grouped into the same cluster. In practical applications, data usually contains different views. For example, a web page may contain text and visual components, and images can be captured from multiple angles, including its front, side, or other angles. These data collected from different views are usually called multi-view data. However, in practical scenarios, due to factors such as machine failure or sensor problems, multi-view data may lose information in some views, resulting in incomplete multi-view data. This situation makes most existing multi-view clustering methods have potential limitations and failures.

[0003] When some views are missing, the natural alignment property of the same instance between multiple views is destroyed, which may lead to insufficient mining of complementary consistent information. Many incomplete multi-view clustering algorithms have been proposed to address the missing value problem in multi-view data, which can be further divided into graph learning-based methods, non-negative matrix factorization-based methods, and contrastive learning-based methods. The effectiveness of these methods depends on the quality of the input data and increases the computational complexity during the input process. Therefore, their performance is not optimal when faced with large datasets or scenarios involving high computational requirements.

[0004] Among them, the idea based on contrastive learning is an effective method to solve the problem of incomplete multi-view clustering. The core concept of this type of method is to use deep neural networks to capture high-level semantic representations of samples instead of shallow features. Specifically, an end-to-end multi-view feature extraction framework is constructed using stacked autoencoders to learn specific view representations of samples. In addition, in order to improve the consensus representation ability, incomplete instance-level contrastive learning is further introduced to guide the encoder to better extract consensus information of multiple views, and a multi-view weighted fusion module is used to enhance the discriminative ability of semantic features. This method is committed to aligning the same samples from different views, ultimately reducing the intra-cluster distance and expanding the inter-cluster distance. Through this method, researchers can perform multi-view clustering in the absence of any number of views, effectively avoiding the errors and limitations of clustering caused by the lack of view data.

[0005] Recent studies have attracted widespread attention on how to effectively use view-shared representations and view-specific representations to estimate missing samples to avoid errors in clustering caused by missing view data. After retrieval, the application number is: CN202410358902.6, the authorization announcement date is: June 14, 2024, and the invention name is: An incomplete multi-view clustering method based on direct contrastive learning. Traditional deep neural networks rely too much on additional projection heads to avoid dimensionality collapse, resulting in parameter redundancy and the risk of encoder-derived features, which incorporate view-specific information and mislead the learning of common semantics due to simultaneous learning and reconstruction in the same feature space. To solve these problems, the invention proposes a new incomplete multi-view contrastive clustering framework, which adopts an encoder network with a self-attention mechanism, allowing reconstruction loss and contrast loss to act on the learned feature vector and its subvectors respectively. The influence of exogenous private information is effectively mitigated. By using subvectors for consistency learning, the potential feature subspace is directly refined, thereby avoiding dimensionality collapse without relying on projection heads. In addition, the invention combines a cross-view prediction mechanism to recover missing information in incomplete data sets. Although the prediction mechanism can estimate the missing data to a certain extent, it ignores the useful information of other views, resulting in an inability to estimate the missing samples more accurately, thereby limiting the accuracy of clustering. To this end, the present invention proposes an incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints. Summary of the invention

[0006] The technical problem to be solved by the present invention is: how to solve the problem that although the prediction mechanism can realize the estimation of missing data to a certain extent, it ignores the useful information of other views, resulting in the inability to estimate the missing samples more accurately, thereby limiting the accuracy of clustering. An incomplete multi-view comparative learning method based on cross-view encoding and graph constraints is provided, which can effectively solve some factors that are not considered in the existing methods and provide a more complete solution. This deep neural network is more effective across views than traditional autoencoders, and brings new solutions to further solve the problem of incomplete multi-view clustering.

[0007] The present invention solves the above technical problems through the following technical solutions, and the present invention comprises the following steps:

[0008] S1: Get the original dataset, i.e., the incomplete multi-view dataset;

[0009] S2: Extracting latent feature representations from incomplete multi-view datasets using a cross-view encoder introduced with Transformer;

[0010] S3: Based on the latent feature representation, a decoder symmetrical to the cross-view encoder is used to obtain the reconstructed feature representation, and a reconstruction loss is introduced to reduce the difference between the original feature representation and the reconstructed feature representation;

[0011] S4: The extracted latent feature representation is input into the nearest neighbor contrastive learning, while shortening the distance between positive pairs and pushing the distance between negative pairs in the feature space, alternating between contrastive learning loss function L fea To optimize and promote the optimization of potential feature representation;

[0012] S5: Use clustering algorithm to obtain pseudo labels for latent feature representation, provide the latent feature representation of the corresponding view to the two-layer cluster MLP network, and then use Softmax function to obtain cluster representation, and use cluster-level contrastive learning loss function L clu To learn the consistency between cluster representations and thus bring similar clusters closer together;

[0013] S6: Perform self-supervised high-confidence learning on the latent feature representation obtained after contrastive learning optimization, and introduce a high-confidence learning loss function L hc Ensure that instances with consistency and high confidence can self-supervise, further enhancing its representation learning capabilities;

[0014] S7: Alternately adopt graph-constrained learning loss function L ng Optimize the latent graph structure of the reconstructed feature representation, capture the neighbor relationship between samples, and promote the recovery of missing views;

[0015] S8: Based on the above losses, a total loss function is constructed, and the parameters of the backbone network are updated through network back propagation to reduce the gap between the predicted clustering results and the actual clustering results;

[0016] S9: Process the fused representation using the k-means algorithm to obtain the final clustering result.

[0017] Furthermore, in step S2, in the incomplete multi-view dataset, a missing view indication matrix is ​​used to indicate the position of missing views, where 1 represents existence and 0 represents missing, and the size is the number of samples×the number of views.

[0018] Furthermore, in step S3, the specific process is as follows:

[0019] S31: Use multi-view weighted fusion to obtain the fusion representation of all views:

[0020]

[0021] Among them, α ij represents the adaptive weight of each view, is the potential feature representation of sample i in view v. The specific extraction process is as follows:

[0022]

[0023] Among them, E (v) Denoted as a cross-view encoder, Represented as the original feature representation of sample i in view v;

[0024] S32: The reconstructed features obtained by the decoder are expressed as follows:

[0025]

[0026] Among them, D (v) Denoted as a decoder symmetric to the cross-view encoder, The representation is a fusion representation;

[0027] It should be noted that the fused representation here is the initial fused representation obtained in step S31, and the fused representation in step S9 is obtained by continuously updating and optimizing the initial fused representation through gradual advancement of the network;

[0028] S33: Reconstruction loss is introduced to reduce the difference between the potential feature representation and the reconstructed feature representation. The reconstruction loss function L re It is expressed as follows:

[0029]

[0030] Among them, V represents the total number of views and N represents the number of samples.

[0031] Furthermore, in step S4, the specific process is as follows:

[0032] S41: Define the relationship between sample pairs:

[0033]

[0034] Among them, z i and z j Represent the potential feature representation of sample i and sample j respectively, and the total number of samples is N;

[0035] S42: For each sample z i , using KNN to find the k samples closest to each sample and then construct an adjacency matrix:

[0036] A i,j =argsort(D i,j )[k]

[0037] Among them, A i,j =1 means the sample forms a positive pair, Ai,j =0 means the sample forms a negative pair;

[0038] S43: In order to maximize the similarity of positive pairs and minimize the similarity of negative pairs, the nearest neighbor contrastive learning loss function is defined as:

[0039]

[0040] Among them, N i It is expressed as the number of neighbors, and m is the distance threshold.

[0041] Furthermore, in step S5, the cluster representation represents the probability of the i-th sample being assigned to the j-th cluster in the v-th view, using cluster-level contrastive learning loss L clu To learn any cluster Y m and Y n Consistency between:

[0042]

[0043] Among them, τ is the temperature parameter, C is the number of categories, m and n represent the number of views, namely the m-th view and the n-th view, and the cluster-level contrastive learning loss function L clu The definition is as follows:

[0044]

[0045] Where H(Y) is the cluster assignment probability.

[0046] Furthermore, in step S6, the specific process is as follows:

[0047] S61: Measure the similarity between views based on the latent feature representation of each view:

[0048]

[0049] S62: Calculate the variance Q of each view i , and get the quality index:

[0050] Q i =Var(z i )

[0051] Where, i=(1,2,...,N);

[0052] S63: Calculate the adaptive weight α based on the similarity and quality index between views ij , represents the view potential feature representation z i The latent feature representation z of the view j The calculation formula is as follows:

[0053]

[0054] Among them, γ is a trade-off hyperparameter that balances similarity and quality assessment, sim(z i ,z j ) represents the calculation of the cosine similarity between the latent feature representations of two views;

[0055] S64: Introducing high confidence learning loss function L hc :

[0056] L hc =L sim +L quality

[0057] Among them, L sim is a weighted loss function based on weights, expressed as follows:

[0058]

[0059] L quality is a regularized loss function based on view quality, expressed as follows:

[0060]

[0061] Furthermore, in step S7, the specific process is as follows:

[0062] S71: Determine the nearest neighbor sample with the shortest distance, and define the adjacency matrix of k nearest neighbors as G:

[0063]

[0064] in, Indicates that instance j is one of the k neighbors of instance i in view v;

[0065] S72: To train the network, define the graph constraint learning loss function L ng for:

[0066]

[0067] Where e represents the number of training rounds, G is an approximately complete graph generated by the last batch of data, and this graph constraint learning loss function L is executed when e>1 ng .

[0068] Furthermore, in step S8, the total loss function is as follows:

[0069] L all =L rec +L con +λ1·L hc +λ2·L ng

[0070] Among them, λ1 and λ2 are hyperparameters, L con =L fea +L clu .

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] 1. Efficient fusion of information between views

[0073] Existing multi-view clustering methods usually process the data features of each view independently, lacking sufficient information interaction between views. The present invention maps the data of each view to a common feature space through a cross-view encoder, and on this basis combines the self-attention mechanism of the Transformer to dynamically capture the correlation between views, thereby achieving deep fusion of information. Through this fusion, the model can recognize and utilize the complementary information between views, so that when there is partial loss of multi-view data, it can still maintain high clustering accuracy and robustness. This mechanism effectively overcomes the problem of reduced clustering accuracy caused by the isolation of information between views in traditional methods. The present invention can provide more consistent feature representation in a complex and changeable multi-view data environment, thereby significantly improving the performance of clustering, especially in the case of incomplete views or uneven data quality. This task is achieved by processing potential features in this way, which not only highlights the key features, but also reduces the influence of irrelevant features. The enhanced contrastive learning scheme enhances the relationship between features, enhances the intrinsic connection between features, improves the feature reconstruction process, and restores the information shared between different views. Through experimental verification, the present invention has certain effectiveness in processing incomplete dual-view clustering scenarios. When the previous method is applied to data involving more than two views, the computational complexity increases significantly, resulting in a slower algorithm execution speed. However, the present invention can extend the method to any missing view, and has higher clustering accuracy and system robustness.

[0074] 2. Reduce dependence on negative sample selection and improve computational efficiency

[0075] Traditional contrastive learning methods rely on a large number of negative sample pairs to train the model, which may lead to excessive computational overhead when processing high-dimensional data. The present invention introduces a nearest neighbor contrastive learning strategy to select the neighbor most similar to each sample as a positive sample pair, thereby reducing the reliance on negative samples and enhancing the learning effect of local features. Since the nearest neighbor sample selection mechanism can be adjusted dynamically, the model can converge faster while avoiding learning errors caused by improper selection of negative samples. This design not only reduces the computational complexity, but also improves the stability and effect of contrastive learning. When processing high-dimensional complex data, the present invention can significantly reduce the consumption of computing resources while maintaining or improving model performance, thereby improving efficiency and operability in practical applications.

[0076] 3. Robustness to incomplete views

[0077] The present invention aims at the common problem of incomplete views in multi-view data. The existing technology usually handles this problem by completing the data or ignoring the missing views, but these methods perform poorly when the information loss is severe. The present invention can effectively deal with the situation of incomplete views through the multi-view feature fusion of cross-view encoders and the global modeling capability of Transformer. Even if the data of some views is incomplete, the model can reconstruct or approximate the features of the missing views through the supplementary information of other views, ensuring the stability and accuracy of clustering. This feature enables the present invention to maintain good performance in the face of multi-view scenarios with incomplete data and uneven quality, significantly improving the practicality and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of an incomplete multi-view comparative learning method based on cross-view encoding and graph constraints in Embodiment 2 of the present invention;

[0079] Figure 2 It is a schematic diagram of the model structure obtained by the incomplete multi-view comparative learning method based on cross-view encoding and graph constraints in Example 2 of the present invention. DETAILED DESCRIPTION

[0080] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiment.

[0081] Embodiment 1

[0082] This embodiment provides a technical solution: an incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints, which is used to strengthen incomplete multi-view clustering. Different from the prior art, the present invention proposes a method combining Transformer and cross-view encoder. This method aims to explore the relationship between incomplete multi-view data samples, especially in improving the efficiency and accuracy of multi-view clustering. The above goal is achieved through the following strategies:

[0083] 1. Cross-view encoder: The feature dimensions of the original data are different, so it is a challenge for the model to fuse multi-view features in the original feature space. Traditional methods usually use autoencoders composed of fully connected layers to extract view-specific features, thereby aligning different dimensions in a common space and mapping high-dimensional features to a low-dimensional space for subsequent representation learning. However, this method only extracts features from each view and lacks information interaction between views. Since different views describe objects from different angles, each view is unique and biased, making multi-view data more expressive than single-view data. Therefore, how to effectively utilize this complementary information has always been one of the key issues in multi-view learning. To this end, the present invention uses a cross-view encoder to aggregate multi-view information.

[0084] 2. Nearest neighbor contrastive learning: In traditional contrastive learning, each sample has a set of observations from different views associated with it. Therefore, the same data samples from different views can be naturally drawn as positive pairs, while the remaining sample pairs (i.e., different samples within each view or between different views) are drawn as negative pairs. Although these methods have achieved good results, the optimization of clustering effects still needs to be improved. This method improves the above contrastive learning and aims to map samples into a unified embedding space in a global manner and construct positive and negative pairs based on the relationship graph, which is constructed based on the distance of samples in the original space. Based on the assumption that close samples in the original space have a high probability of belonging to the same category, it is believed that if one sample is in the relationship graph of another sample, they can form a positive pair, otherwise the opposite. This method not only solves the false negative problem, but also makes full use of the complementary information between samples and improves the robustness of the model.

[0085] 3. Cluster-level contrastive learning: In order to directly obtain cluster labels and achieve end-to-end joint optimization, this method uses a multi-layer perceptron (MLP) network and a Softmax function to calculate the multi-view pseudo-label representation for view-specific representations and then learn the clustering consistency between views. Its core goal is to guide model learning by comparing the feature similarities and differences between different samples, so that samples from the same cluster are more similar, and samples from different clusters are more different.

[0086] 4. High-confidence learning: In order to enhance the complementarity and consistency of the model when processing the feature representations extracted by the encoder, this method adopts a high-confidence learning mechanism. This mechanism takes into account the uneven quality of weights assigned to different views by the attention mechanism. By automatically perceiving the view fusion weights, the weights are used to guide the mutual enhancement between view feature representations, thereby obtaining a unified representation of all views and achieving comprehensive characterization of cross-view samples.

[0087] 5. Graph-constrained learning: In the presence of missing data, some existing methods ignore the missing views to construct the adjacency graph, which may lead to obvious deviations on datasets with high missing rates. In order to achieve a comprehensive representation of the graph structure, this method dynamically generates graphs in the latent space of reconstructed views. The rationale behind this is the limitation of combining fixed graphs based only on raw data. When using a fixed graph, it cannot be adjusted according to the clustering results, which limits the accuracy and adaptability of the clustering process. In contrast, the discriminative semantic features reconstructed through attention are more helpful in recovering the missing views, thereby facilitating the construction of a more realistic and accurate graph representation.

[0088] This method cleverly integrates multi-view fusion representation learning, self-supervised high-confidence guidance, and missing view recovery into a unified framework, aiming to improve clustering performance while showing high robustness to missing views. Its core steps are as follows:

[0089] Step 1: Get the original data set.

[0090] Step 2: Use the cross-view encoder introduced into Transformer to extract feature representation for the incomplete multi-view dataset, where the missing view indicator matrix is ​​used to indicate the position of the missing view, 1 for existence and 0 for missing.

[0091] Step 3: The extracted feature representation is reconstructed through a decoder that is symmetrical with the cross-view encoder, and a reconstruction loss is introduced to reduce the difference between the original feature representation and the reconstructed feature representation.

[0092] Step 4: Apply the above feature representation to nearest neighbor contrastive learning to eliminate the impact of false negative problems, while shortening the distance between positive pairs and pushing the distance between negative pairs in the feature space, alternating between contrastive learning loss L fea To optimize.

[0093] Step 5: Use the clustering algorithm to generate pseudo labels. The view-specific feature representation is provided to the two-layer cluster MLP network, and then the Softmax function is used to obtain the cluster representation, and the cluster-level contrastive learning loss L is used. clu To learn the consistency between these cluster representations.

[0094] Step 6: Perform self-supervised high-confidence learning on the feature representation obtained after contrastive learning optimization. This learning method can automatically perceive the view fusion weight. This weight is used to guide the mutual promotion learning between feature representations, and then obtain the best quality fusion representation to achieve comprehensive representation of cross-view samples. Introduce high-confidence learning loss L hc Instances with consistency and high confidence are ensured to be self-supervised, further enhancing their representation learning capabilities.

[0095] Step 7: In order to construct an approximately complete adjacency graph to guide the encoder to extract higher-level semantic features, graph constraint learning is introduced to force each reconstructed view to dynamically generate a complete graph in its latent space. Specifically, the discriminative semantic features reconstructed by the attention mechanism are more helpful in restoring the missing views, thereby promoting the construction of a more realistic and accurate graph representation. In order to optimize model learning, the graph constraint learning loss L is introduced. ng To optimize the graph structure.

[0096] Step 8: Construct a total loss function model based on the above loss, and update the parameters of the backbone network through network back propagation to reduce the gap between the predicted clustering results and the actual clustering results, thereby improving the accuracy of the final clustering task.

[0097] Step 9: Perform the k-means algorithm on the fused representation to obtain the final clustering result.

[0098] Step 10: Repeat the above process until the model reaches the predetermined performance index or meets the termination condition.

[0099] Embodiment 2

[0100] like Figure 1 As shown, this embodiment is used to further illustrate the method in Embodiment 1. The incomplete multi-view contrast learning method based on cross-view encoding and graph constraints specifically includes the following steps:

[0101] Step 1. Get the original datasets (Handwritten, ORL, Caltech7, LandUse_21, Aloideep3v).

[0102] Step 2: Use the cross-view encoder to extract feature representation (i.e., latent feature representation) for the incomplete multi-view dataset (original dataset), where a missing view indication matrix is ​​used to indicate the position of the missing view, 1 for existence and 0 for missing, and the size is the number of samples × the number of views.

[0103] Step 3: First, pass the cross-view encoder E (v) (·) Extract feature representation, and then pass the extracted feature representation through the decoder D which is symmetrical with the cross-view encoder (v)(·) Get the reconstructed representation (i.e., reconstructed feature representation), and introduce the reconstruction loss L re To optimize the difference between the original feature representation and the reconstructed feature representation.

[0104] i. The learned latent representation of the v-th view can be obtained as follows:

[0105]

[0106] in, Represents the potential features of sample i in view v. However, relying solely on the structure of the encoder usually results in the extracted features containing a large amount of view-specific information, which will ignore the complementary information between views. To solve the above problems, the feature extraction stage of the present invention is performed by integrating the cross-view encoder with the subsequent Transformer's self-attention mechanism. This mechanism aims to learn the weight of each input vector in order to better retain the information present in different views while discarding redundant information in the view. Specifically: the same sample from different views is aggregated and combined with a given number of components h, using h groups of weighted linear layers Get the corresponding query map, key map, and value map For component t of sample i, the score matrix is ​​calculated as:

[0107]

[0108] Multiplying the above formula with the weight matrix V, we get the similarity matrix, which can be expressed as:

[0109] A t =S t V t (3)

[0110] Among them, S t Represents the Transformer part in the encoder, and applies a normalization layer to the output value to normalize it. t It is sequentially input into the linear layer, layer norm module and multi-layer perceptron of the cross-view encoder to obtain the final embedding feature of each sample.

[0111] ii. Since no proper constraints are imposed in the encoder extraction, this means that the encoding process is uncontrolled. To solve this problem, the present invention adopts a simple and effective method, namely multi-view weighted fusion, to obtain a fused representation of all views, specifically:

[0112]

[0113] Among them, α ij Represents the adaptive weight of each view.

[0114] iii. For subsequent cross-view contrastive learning, the decoder symmetrical to the encoder reconstructs the sample into

[0115]

[0116] iv. Reconstruction loss is introduced to reduce the difference between the original feature representation and the reconstructed feature representation:

[0117]

[0118] Step 4: Input the feature representation extracted by the encoder into the nearest neighbor contrastive learning to obtain a better feature representation:

[0119] i. Define the relationship between sample pairs:

[0120] ii. For each sample z i , using K-Nearest Neighbors (KNN) to find the k samples closest to each sample (excluding itself) and then construct an adjacency matrix:

[0121] A i,j =argsort(D i,j )[k] (7)

[0122] A i,j =1 means the sample forms a positive pair, A i,j =0 means the sample forms a negative pair.

[0123] iii. In order to maximize the similarity of positive pairs and minimize the similarity of negative pairs, the nearest neighbor contrastive learning loss function is defined as:

[0124]

[0125] Step 5: Represent the latent features of the corresponding view Provided to the two-layer cluster MLP network, and then use the Softmax function to obtain the cluster representation in, represents the probability of the i-th sample being assigned to the j-th cluster in the v-th view, using cluster-level contrastive learning loss L clu To learn any cluster Y m and Y n Consistency between:

[0126]

[0127] Where τ is the temperature parameter and C is the number of categories. By applying this similarity to all clusters from different views, the cluster-level contrastive learning loss function is defined as:

[0128]

[0129] Among them, H(Y) is the cluster assignment probability, the first term is used for cluster-level consistency learning across views, and the second term is used for cluster regularization to avoid trivial solutions. The total contrast loss (cluster-level contrast loss and nearest neighbor contrast learning loss) function is:

[0130] L con =L fea +L clu (11)

[0131] Step 6: Use self-supervised high-confidence learning to automatically perceive view fusion weights and guide the mutual promotion learning between feature representations. The specific process is as follows:

[0132] i. Measure the similarity between views based on the feature representation of each view:

[0133] ii. To distinguish the quality of views, calculate the variance Q of each view i Get quality indicators:

[0134] Q i =Var(z i )i=(1,2,...,N).

[0135] iii. Calculate the adaptive weight α based on the similarity and quality indicators between views ij , which represents the view feature representation z i Represent the view feature z j The weight assignment can be done by combining similarity and quality indicators:

[0136]

[0137] Among them, γ is a trade-off hyperparameter that balances similarity and quality assessment, α ij The larger the z i More dependent on z j For learning, higher quality views have larger α ij .

[0138] iv. In order to make the low-quality view features close to the high-quality view features, the idea of ​​contrastive learning can be used to calculate the similarity between views and shorten the feature distance between high-quality and low-quality views. The weighted loss function based on weights is:

[0139]

[0140] The goal of this loss is to minimize the feature similarity difference between high-quality and low-quality views, making the feature representation more consistent. In order to ensure the dominant role of high-quality views in the fusion process, a regularization loss based on view quality is added, that is, Q obtained by variance calculation i , to enhance the influence of high-quality views on the final features:

[0141]

[0142] This loss term allows the features of high-quality views to be updated to retain more original information, while low-quality views learn more from high-quality views. The total loss (high confidence loss) function is:

[0143] L hc =L sim +L quality (15)

[0144] Step 7: Alternately use graph-constrained learning losses to optimize the latent graph structure of the reconstructed representation, fully capture the neighbor relationship between samples, and promote the recovery of missing views, as follows:

[0145] i. Determine the nearest neighbor sample with the smallest distance (excluding itself) and define the adjacency matrix of k nearest neighbors as G:

[0146]

[0147] in Indicates that instance j is one of the k neighbors of instance i in view v;

[0148] ii. To train the network, define the target loss (graph-constrained learning loss) function as:

[0149]

[0150] Where e represents the number of training epochs and G is a nearly complete graph generated from the last batch of data. This loss is enforced when e>1.

[0151] Step 8. In order to better optimize the model, a comprehensive loss function is constructed based on the above learning method:

[0152] L all =L rec +L con +λ1·L hc +λ2·L ng (18)

[0153] Where λ1 and λ2 are hyperparameters used to balance the importance of the two losses.

[0154] Step 9: Save the model with the best prediction results on the validation set. Figure 2 .

[0155] like Figure 2 As shown in the figure, the model mainly includes a cross-view encoder and decoder, a multi-head self-attention mechanism, a contrastive learning module, a high-confidence learning module, and a graph constraint module; among them:

[0156] The cross-view encoder mainly receives aggregated samples of the same sample from different views, encodes them into the same feature dimension and feeds them into the multi-head self-attention mechanism to capture the complementary information between different views. The obtained view potential feature representation is adaptively weighted to obtain a fused representation, and the fused representation is decoded to obtain the reconstructed information.

[0157] The contrastive learning module includes nearest neighbor contrastive learning and cluster-level contrastive learning. The nearest neighbor contrastive learning optimizes the feature representation of samples, while the cluster-level contrastive learning optimizes the representation learning between clusters.

[0158] The high-confidence learning module automatically perceives the view fusion weights and uses the weights to guide the mutual enhancement between view feature representations, thereby obtaining a unified representation of all views and achieving comprehensive representation of cross-view samples.

[0159] The graph constraint learning module dynamically generates a neighbor graph in the latent space of the reconstructed view, and applies it back to the latent feature representation extracted by the cross-view encoder to continuously optimize the latent feature representation.

[0160] Step 10: Repeat the above steps 2-10 until the model converges or the termination condition is reached.

[0161] Embodiment 3

[0162] In the incomplete multi-view clustering task, the goal is to complete the clustering task when any number of views are missing. However, the success of the multi-view clustering task depends largely on the consistency and integrity of the data views to ensure the complete transmission of data information. However, during the data collection and transmission process, data loss is inevitable, resulting in the loss of some views. This situation weakens the amount of available information, increases the difficulty of joint learning of incomplete multi-view data, and greatly increases the difficulty of clustering.

[0163] To solve the above problems, the present invention focuses on developing a more comprehensive incomplete multi-view contrastive learning method based on cross-view encoding and graph constraint learning to strengthen the incomplete multi-view clustering task. Due to the diversity between different views, previous contrastive learning studies mainly focus on the relationship between isolated sample pairs, which limits the robustness of the method to a certain extent. The present invention selects positive samples from global and local perspectives by utilizing the nearest neighbor graph to maximize the correlation between local features and latent features of each view. In addition, a cross-view encoder network with a self-attention structure is adopted to fuse the specific low-dimensional representation of each view into a joint representation, and guide the learning of the joint representation through a high-confidence structure. Graph constraint learning is further introduced to deeply explore the potential neighbor relationship between instances to facilitate data reconstruction. Experimental results on five multi-view datasets show that compared with existing methods, this method shows significant effectiveness and superiority, as shown in Table 1.

[0164] Table 1 Experimental results on five multi-view datasets

[0165]

[0166] In summary, the incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints in the above embodiment aims to strengthen the incomplete multi-view clustering task. First, a nearest neighbor contrastive learning intervention mechanism is introduced to alleviate the degradation of clustering performance caused by the false negative problem. Specifically, this method will capture the high-level representation of the sample through a cross-view encoder, and construct a relationship graph based on this. Since the different view features of the same instance are semantically consistent, the same instance in different views can share a similar relationship graph. In this way, the high-level feature representations of different views can be brought closer and aligned in the feature space; then self-supervised high confidence guidance is used to further enhance its own representation learning; in addition, in order to pay more attention to the neighbor relationship between instances, the reconstructed data is pushed back into the representation learning process by introducing graph constraints, and they are allowed to collaborate with each other to achieve better clustering.

[0167] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints, characterized by: The following steps are involved: S1: Get the original dataset, i.e., the incomplete multi-view dataset; S2: Extracting latent feature representations from incomplete multi-view datasets using a cross-view encoder introduced with Transformer; S3: Based on the latent feature representation, a decoder symmetrical to the cross-view encoder is used to obtain the reconstructed feature representation, and a reconstruction loss is introduced to reduce the difference between the original feature representation and the reconstructed feature representation; S4: The extracted latent feature representation is input into the nearest neighbor contrastive learning, while shortening the distance between positive pairs and pushing the distance between negative pairs in the feature space, alternating between contrastive learning loss function L fea To optimize and promote the optimization of potential feature representation; S5: Use clustering algorithm to obtain pseudo labels for latent feature representation, provide the latent feature representation of the corresponding view to the two-layer cluster MLP network, and then use Softmax function to obtain cluster representation, and use cluster-level contrastive learning loss function L clu To learn the consistency between cluster representations and thus bring similar clusters closer together; S6: Perform self-supervised high-confidence learning on the latent feature representation obtained after contrastive learning optimization, and introduce a high-confidence learning loss function L hc Ensure that instances with consistency and high confidence can self-supervise, further enhancing its representation learning capabilities; S7: Alternately adopt graph-constrained learning loss function L ng Optimize the latent graph structure of the reconstructed feature representation, capture the neighbor relationship between samples, and promote the recovery of missing views; S8: Based on the above losses, a total loss function is constructed, and the parameters of the backbone network are updated through network back propagation to reduce the gap between the predicted clustering results and the actual clustering results; S9: Process the fused representation using the k-means algorithm to obtain the final clustering result.

2. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 1 is characterized in that: In step S2, in the incomplete multi-view dataset, a missing view indication matrix is ​​used to indicate the position of missing views, where 1 represents existence and 0 represents missing, and the size is the number of samples × the number of views.

3. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 1 is characterized in that: In step S3, the specific process is as follows: S31: Use multi-view weighted fusion to obtain the fusion representation of all views: Among them, α ij represents the adaptive weight of each view, is the potential feature representation of sample i in view v. The specific extraction process is as follows: Among them, E (v) Denoted as a cross-view encoder, Represented as the original feature representation of sample i in view v; S32: The reconstructed features obtained by the decoder are expressed as follows: Among them, D (v) Denoted as a decoder symmetric to the cross-view encoder, The representation is a fusion representation; S33: Reconstruction loss is introduced to reduce the difference between the potential feature representation and the reconstructed feature representation. The reconstruction loss function L re It is expressed as follows: Among them, V represents the total number of views and N represents the number of samples.

4. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 3 is characterized in that: In step S4, the specific process is as follows: S41: Define the relationship between sample pairs: Among them, z i and z j Represent the potential feature representation of sample i and sample j respectively, and the total number of samples is N; S42: For each sample z i , using KNN to find the k samples closest to each sample and then construct an adjacency matrix: A i,j =argsort(D i,j )[k] Among them, A i,j =1 means the sample forms a positive pair, A i,j =0 means the sample forms a negative pair; S43: In order to maximize the similarity of positive pairs and minimize the similarity of negative pairs, the nearest neighbor contrastive learning loss function is defined as: Among them, N i It is expressed as the number of neighbors, and m is the distance threshold.

5. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 4 is characterized in that: In step S5, the cluster representation represents the probability of the i-th sample being assigned to the j-th cluster in the v-th view, using cluster-level contrastive learning loss L clu To learn any cluster Y m and Y n Consistency between: Among them, τ is the temperature parameter, C is the number of categories, m and n represent the number of views, namely the m-th view and the n-th view, and the cluster-level contrastive learning loss function L clu The definition is as follows: Where H(Y) is the cluster assignment probability.

6. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 5, characterized in that: In step S6, the specific process is as follows: S61: Measure the similarity between views based on the latent feature representation of each view: S62: Calculate the variance Q of each view i , and get the quality index: Q i =There is(z i ) Where, i=(1,2,...,N); S63: Calculate the adaptive weight α based on the similarity and quality index between views ij , represents the view potential feature representation z i The latent feature representation z of the view j The calculation formula is as follows: Among them, γ is a trade-off hyperparameter that balances similarity and quality assessment, sim(z i ,z j ) represents the calculation of the cosine similarity between the latent feature representations of two views; S64: Introducing high confidence learning loss function L hc : L hc =L sim +L quality Among them, L sim is a weighted loss function based on weights, expressed as follows: L quality is a regularized loss function based on view quality, expressed as follows:

7. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 6 is characterized in that: In step S7, the specific process is as follows: S71: Determine the nearest neighbor sample with the shortest distance, and define the adjacency matrix of k nearest neighbors as G: in, Indicates that instance j is one of the k neighbors of instance i in view v; S72: To train the network, define the graph constraint learning loss function L ng for: Where e represents the number of training rounds, G is an approximately complete graph generated by the last batch of data, and this graph constraint learning loss function L is executed when e>1 ng .

8. The incomplete multi-view contrastive learning method based on cross-view encoding and graph constraints according to claim 7 is characterized in that: In step S8, the total loss function is as follows: L all =L rec +L con +λ1·L hc +λ2·L ng Among them, λ1 and λ2 are hyperparameters, L con =L fea +L clu .

Images