A multi-modal clustering method based on hypergraph autoencoder

Abstract

The application discloses a kind of multi-modal clustering methods based on hypergraph autoencoder.First, modal data set is obtained, and based on modal data set, hypergraph is formed, and adjacency matrix is obtained;Then, the encoder output and decoder output of each modal data set are obtained by autoencoder network;Third, the latent representation corresponding to each modal data is fused into multi-modal data latent representation, and the corresponding similarity matrix is obtained;The network parameters of autoencoder network and multi-modal latent representation are updated until the loss function converges, and finally, according to the updated multi-modal latent representation, multi-modal data is divided into k categories.The application learns the latent similarity between samples through the structure prior of data, and effective structure prior can reduce the bias of sample features.We use the feature representation ability of pre-training model and the excellent modeling ability of hypergraph to integrate feature prior and structure prior into the model.This improves the quality of self-supervised learning algorithm.

Description

Technical Field

[0001] The present invention belongs to the fields of data mining, contrastive learning, hypergraph representation learning, multimodal learning, self-supervised learning and clustering algorithms, and specifically relates to a multimodal clustering method based on a hypergraph autoencoder. Background Art

[0002] In recent years, smart terminal devices have developed rapidly, with an increasing number of sensors and multimodal data. For example, sensors such as cameras can capture multimodal data such as depth images, various angles, and infrared information. Similarly, on social networks, a single message may contain multiple data modalities, including images, text, audio, and emoticons. Discovering the correlations between multimodal data and effectively analyzing it is a pressing issue. Furthermore, due to the high cost and high bias associated with labeling complex multimodal data, self-supervised learning methods have become increasingly important. With the acquisition of multimodal data and the development of self-supervised learning, multimodal self-supervised clustering has garnered widespread attention.

[0003] Some current multimodal self-supervised clustering algorithms simply align different modalities by encoding them into the same low-dimensional feature space using a multimodal autoencoder. This leads to blurred boundaries in the distribution of multimodal features in the feature space. In reality, sample features are biased and noisy. In this case, if the model focuses solely on feature priors, it will fail to discover the underlying correlations between samples. However, this underlying data correlation can provide valuable guidance for representation learning. During the clustering process, such self-supervised algorithms rely solely on feature information and ignore the inherent structural information in multimodal data. This deficiency often leads to weak robustness and generalization capabilities. Summary of the Invention

[0004] In response to the shortcomings in the prior art, the present invention provides a multimodal clustering method based on a hypergraph autoencoder. The present invention proposes a hypergraph contrastive autoencoder (HGCAE), which is a versatile multimodal clustering framework designed to learn feature representations of various modalities in a low-dimensional subspace. When processing multimodal data, the structural information of the data can help the model better capture the intrinsic characteristics and correlations of the data. The model learns the potential similarities between samples through the structural prior of the data, and an effective structural prior can reduce the bias of sample features. We leverage the feature representation capabilities of the pre-trained model and the excellent modeling capabilities of the hypergraph to integrate feature priors and structural priors into the model. This improves the quality of the self-supervised learning algorithm.

[0005] First, we use hypergraph to model multi-modal data. Compared with single-modal data, the relationship between instances in multi-modal setting is more complex. To address this situation, hypergraph can more completely represent the high-order correlation between multiple nodes and encode it in the form of an association matrix. To take advantage of the complementarity between multi-modal data, we extend the hypergraph to multiple modalities by connecting the association matrix of each modality. Fusing the association matrix can take advantage of the complementarity of multi-modal data to better reveal their high-order correlations.

[0006] Then, we draw inspiration from the application of hypergraph in the clustering field. To address the bottleneck of poor robustness and generalization ability in multi-modal self-supervised clustering models, we propose a model-agnostic hypergraph relationship reconstruction module that allows the integration of structural priors of data into the model, enhancing its understanding of data. The goal is to incorporate structural priors into the model, enabling it to learn more robust and discriminative features. The model with hypergraph reconstruction can leverage the structural priors in the data to enhance its data analysis capabilities. We observe that for input data with strong structural information, hypergraph relationship reconstruction significantly improves clustering quality. This demonstrates the significance of modeling high-order correlations in complex multi-modal scenarios. Structurally similar data should be closer to each other in the feature space, while dissimilar data should be further apart. By modeling the high-order correlations in the data and encoding them as structural priors into the general-purpose model, the model's understanding and analysis of the data are enhanced, further improving the performance of self-supervised algorithms.

[0007] In addition, we use multi-modal autoencoders to align modalities to the same dimension and utilize deep neural network (DNN) reconstruction loss to ensure that the low-dimensional embedding preserves the characteristics of the original data, thereby mitigating the curse of dimensionality. With the development of large models, their feature representation capabilities have been enhanced. Selecting a suitable large model in the embedding layer greatly improves the quality of the information bottleneck in the feature reconstruction loss module.

[0008] To make different modalities extracted from the same sample more closely related while maintaining separation between different samples. We draw inspiration from the contrastive learning method in the self-supervised field to mitigate the heterogeneity of different modal distributions. We find that for highly heterogeneous data, adding a contrastive loss can significantly improve clustering quality.

[0009] In addition, to further reveal the intrinsic structural information of samples, we employ a self-supervised approach. We add the generated clustering pseudo-labels as a supervision signal to the model for end-to-end optimization.

[0010] The model proposed in this paper uses the feature and structural information in multimodal data for self-supervised clustering. First, we use a pre-trained model to obtain a modal data set corresponding to m modalities of multiple target objects; then we use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain an adjacency matrix; then we input each modal data set into an independent autoencoder network to obtain the encoder output and decoder output of each modal data set; then, we fuse the potential representation corresponding to each modal data into a multimodal data potential representation, and further obtain the similarity matrix of the multimodal data potential representation; then, we use the loss function to calculate the loss value based on the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data; and then perform gradient backpropagation based on the total loss value to update the network parameters of the autoencoder network and the multimodal potential representation until the loss function converges; finally, based on the updated multimodal potential representation, we use the k-means clustering method to classify the multimodal data into k categories.

[0011] A multimodal clustering method based on a hypergraph autoencoder comprises the following steps:

[0012] Step 1: Use the pre-trained feature extraction model to obtain the modal data set corresponding to m modalities of multiple target objects.

[0013] Step 2: Use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain the adjacency matrix.

[0014] Step 3: Input each modal data set into an independent autoencoder network to obtain the encoder output and decoder output of each modal data set.

[0015] Step 4: Fusion of the latent representations corresponding to each modal data into the multimodal data latent representation, and further obtain the similarity matrix of the multimodal data latent representation.

[0016] Step 5: Calculate the loss value using the loss function based on the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data.

[0017] Step 6: Perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges;

[0018] Step 7: Based on the updated multimodal latent representation, the multimodal data is divided into k categories using the k-means clustering method.

[0019] Furthermore, the specific steps of step 1 are as follows:

[0020] Obtain relevant data sets based on application scenarios. For a given multimodal data sample x i The dataset D = {x i} i∈{1,…,N} , where N is the total number of samples, x i There are m modalities. A pre-trained feature extraction model with frozen parameters is used to extract a feature set from the multimodal dataset D, i.e., the data set X of each modality = {X 1 ,..,X m}.

[0021] In one embodiment, the pre-trained feature extraction model uses transformer or vgg16.

[0022] Furthermore, the specific steps of step 2 are as follows:

[0023] The dataset D = {x i} i∈{1,…,N} Each single mode is mapped to the vertex set of the hypergraph where v i is the vertex of the hypergraph, and by calculating the vertex features The distance between vertices is measured by the Euclidean distance between them, where the Euclidean distance is defined as follows:

[0024]

[0025] Where d(v i ,v j ) represents the vertex v i and v j The Euclidean distance between . and Represents vertex v i and v j The features of, C represents the dimension of the features, and Represents vertex v i and v j The feature value of the c-th dimension, each single modal data in the dataset is mapped to the vertex of its own hypergraph using the above method.

[0026] Next, for each vertex, the Euclidean distance between the vertex and other vertices is scaled to [0,1]. The specific formula is as follows:

[0027]

[0028] Where d(v i ) max Represents the vertex v iThe maximum Euclidean distance to other vertices; d norm (v i ,v j ) is the scaled vertex v i Euclidean distance to other vertices.

[0029] Then, we use the ∈-ball (∈-neighborhood) and K-NN (k-nearest neighbor) graph construction methods to obtain the nearest neighbors of the modal data in the modal data set. K-NN is a graph structure based on the relationship between each point and its k nearest neighbors; ∈-ball is a graph structure based on the relationship between each vertex and its neighbors whose distance from it is less than a certain radius ∈. The specific steps are as follows:

[0030] First, use the ∈-ball construction method (∈-neighborhood) to calculate the vertex v i The neighbor subset of is calculated as follows:

[0031] neigh th (v i )={v j ∣d norm (v i ,v j ) <th}

[0032] where neigh th (v i ) is the vertex v obtained by ∈-ball construction i Neighbor subset of , th is a pre-set distance threshold.

[0033] At the same time, the K-NN construction method (K nearest neighbor) is used to calculate the vertex v i The neighbor subset of is calculated as follows:

[0034] neigh ne (v i )={v j ∣(d(v i ,v j ) <sort(d(v i ,v j ))[K]}

[0035] Among them, K is the number of neighbor vertices set in advance, neigh ne (v i ) represents the vertex v obtained by K-NN construction method i Neighbor subset, sort(·) represents the distance sorting algorithm, |neigh ne (v i )|=K.

[0036] In summary, through two kinds of composition mode, for each vertex v i , the neighbor vertex set neigh th (v i ) and neigh ne (v i ) can be generated, and finally the neighbor vertex set is generated as follows:

[0037] neigh(v i )={neigh th (v i ), neigh ne (v i )}

[0038] According to the neighbor vertex set, the hyperedge set is defined as follows:

[0039] ε={neigh(v i )} i∈{1,…,N}

[0040] According to the neighbor vertex set, the association matrix can be obtained, wherein the association matrix is defined as follows:

[0041]

[0042] wherein H(i,j) is the element value of the ith row and jth column of the association matrix H.

[0043] Further, the association matrix of the multi-modal data is constructed as an adjacency matrix , which is expressed as follows:

[0044] A ij =ε((HH T ) ij )

[0045] wherein A ij is the element value of the ith row and jth column of the adjacency matrix A.

[0046] wherein the binary function ε is defined as:

[0047]

[0048] In order to improve the generalization ability of the model, the weight matrix of the adjacency matrix is defined as follows:

[0049]

[0050] wherein, ∑∑A ij is the element sum of the adjacency matrix.

[0051] Further, the specific steps of step three are as follows:

[0052] First, for each modality data set X = {X 1 ,..,X m}, the encoder for each modality is defined as:

[0053] E={E i} i∈{1,…,m}

[0054] Further get the encoder output of the data set for each modality:

[0055] Z={Z i} i∈{1,…,m}

[0056] where Z i The definition is as follows:

[0057]

[0058] where d i is the dimension of the i-th modal data feature.

[0059] For each modality data set X={X i ,..,X m}, the decoder for each modality is defined as:

[0060] D={D i} i∈{1,…,m}

[0061] Further obtain the decoder output of the data set for each modality:

[0062]

[0063] in The definition is as follows:

[0064]

[0065] where d i is the dimension of the i-th modal data feature.

[0066] Furthermore, the specific steps of step four are as follows:

[0067] Initialize the potential representation corresponding to each modality data to correspond to its encoder output, and both use the same representation:

[0068] Z={Z i} i∈{1,…,m}

[0069] The latent representation of each modality data is then fused into the multimodal data latent representation P:

[0070]

[0071] The similarity matrix of the potential representation of multimodal data is obtained as:

[0072] S ij =sigmoid((PP T ) ij );

[0073] Among them S ij is the i-th row and j-th column element of the similarity matrix S, and the sigmoid function is defined as follows:

[0074]

[0075] Furthermore, the specific steps of step five are as follows:

[0076] First, calculate the feature reconstruction loss, which is expressed as follows:

[0077]

[0078] in represents the mean square error function.

[0079] The weighted binary cross entropy function is then used to calculate the structure reconstruction loss, as shown in the following formula:

[0080]

[0081] Among them, W ij is the value of the element in the i-th row and j-th column of the weight matrix W.

[0082] Then use contrast loss for each modality pairwise, as follows:

[0083]

[0084] in is the MMS loss of the two modes i and j, expressed as:

[0085]

[0086] Where B is the number of batches pre-set during training, is the potential representation of modality i in batch p, represents the potential representation of modality i that is unrelated to modality j in batch k. δ is an empirical margin hyperparameter that is pre-set during training.

[0087] Finally, the total loss function is combined as follows:

[0088]

[0089] The hyperparameters λ1 and λ2 are pre-set during the training process.

[0090] Features and beneficial effects of the present invention:

[0091] To address the problems of weak robustness and generalization of existing self-supervised multimodal clustering methods, this paper proposes a hypergraph contrastive autoencoder (HGCAE) framework, which preserves high-order associations between data throughout the entire end-to-end clustering process. In particular, we introduce a model-independent multimodal hypergraph relationship reconstruction module to encode structural information, allowing structural priors to be added to self-supervised multimodal clustering algorithms, thereby improving the quality of self-supervised learning algorithms. Experimental results show that the proposed model outperforms the most advanced models in recent years. Future work is worthwhile to study how to better use hypergraphs to model multimodal data and learn multimodal data representations. And further use the structural priors obtained from hypergraph modeling to improve the generalization and robustness of self-supervised algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Flow chart of the method of the present invention.

[0093] Figure 2 The comparison results of this method and other methods on three datasets are shown in Figure 2.

[0094] Figure 3 Comparison of the visual clustering results between this method and other methods.

[0095] Figure 4 This is the comparison result between this method and other methods in terms of modal dimension.

[0096] Figure 5 The ablation experimental results of this method for structure reconstruction loss and contrast loss are shown.

[0097] Figure 6 The left figure shows the change of total loss during the 30 epoch training process, and the right figure shows the change of the four evaluation indicators of the experiment during the training process.

[0098] Figure 7 Ablation experiment results using single-modality and multi-modality latent representations for clustering.

[0099] Figure 8 It shows the clustering visualization results of this method on different data sets. DETAILED DESCRIPTION

[0100] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0101] like Figure 1 As shown, the present invention provides a multimodal clustering method based on a hypergraph autoencoder, comprising the following steps:

[0102] Step 1: Use the pre-trained feature extraction model to obtain the modal data set corresponding to m modalities of multiple target objects. The specific steps are as follows:

[0103] Obtain relevant data sets based on application scenarios. For a given multimodal data sample x i The dataset D = {x i} i∈{1,…,N} , where N is the total number of samples, x i There are m modalities. Use a pre-trained feature extraction model with frozen parameters (such as transformer, vgg16) to extract a feature set from the multimodal dataset D, that is, the data set X of each modality = {X 1 ,..,X m}.

[0104] Step 2: Use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain the adjacency matrix. The specific steps are as follows:

[0105] The dataset D = {x i} i∈{1,…,N} Each single mode is mapped to the vertex set of the hypergraph where v i is the vertex of the hypergraph, and by calculating the vertex features The distance between vertices is measured by the Euclidean distance between them, where the Euclidean distance is defined as follows:

[0106]

[0107] Where d(v i ,v j ) represents the vertex v i and v j The Euclidean distance between . and Represents vertex v i and v j The features of, C represents the dimension of the features, and Represents vertex v i and v j The feature value of the c-th dimension, each single modal data in the dataset is mapped to the vertex of its own hypergraph using the above method.

[0108] Next, for each vertex, the Euclidean distance between the vertex and other vertices is scaled to [0,1]. The specific formula is as follows:

[0109]

[0110] Where d(v i ) max Represents the vertex v i The maximum Euclidean distance to other vertices; d norm (v i ,v j ) is the scaled vertex v i Euclidean distance to other vertices.

[0111] Then, we use the ∈-ball (∈-neighborhood) and K-NN (k-nearest neighbor) graph construction methods to obtain the nearest neighbors of the modal data in the modal data set. K-NN is a graph structure based on the relationship between each point and its k nearest neighbors; ∈-ball is a graph structure based on the relationship between each vertex and its neighbors whose distance from it is less than a certain radius ∈. The specific steps are as follows:

[0112] First, use the ∈-ball construction method (∈-neighborhood) to calculate the vertex v i The neighbor subset of is calculated as follows:

[0113] neigh th (v i )={v j ∣d norm (v i ,v j ) <th}

[0114] where neigh th (v i ) is the vertex v obtained by ∈-ball construction i Neighbor subset of , th is a pre-set distance threshold.

[0115] At the same time, the K-NN construction method (K nearest neighbor) is used to calculate the vertex v i The neighbor subset of is calculated as follows:

[0116] neigh ne (v i )={v j ∣(d(v i ,v j ) <sort(d(v i ,v j ))[K]}

[0117] Among them, K is the number of neighbor vertices set in advance, neigh ne (v i ) represents the vertex v obtained by K-NN construction method i Neighbor subset, sort(·) represents the distance sorting algorithm, |neighne (v i )|=K.

[0118] In summary, through two construction methods, for each vertex v i , can generate neighbor vertex set neigh th (v i ) and neigh ne (v i ), the final generated neighbor vertex set is as follows:

[0119] neigh(v i )={neigh th (v i ), neigh ne (v i )}

[0120] According to the neighbor vertex set, the hyperedge set is defined as follows:

[0121] ε={neigh(v i )} i∈{1,…,N}

[0122] According to the neighbor vertex set, the correlation matrix can be obtained The correlation matrix is ​​defined as follows:

[0123]

[0124] Where H(i,j) is the element value in the i-th row and j-th column of the incidence matrix H.

[0125] Furthermore, the correlation matrix of multimodal data is constructed as an adjacency matrix The formula is as follows:

[0126] A ij =ε((HH T ) ij )

[0127] Among them A ij is the element value in the i-th row and j-th column of the adjacency matrix A.

[0128] The binary function ε is defined as:

[0129]

[0130] In order to improve the generalization ability of the model, the weight matrix of the adjacency matrix is ​​defined as follows:

[0131]

[0132] Among them, ∑∑A ij is the sum of the elements of the adjacency matrix.

[0133] Step three: input each modality data set into an independent autoencoder network respectively, to obtain the encoder output and decoder output of each modality data set;

[0134] First, for each modality data set X = {X 1 ,..,X m}, the encoder of each modality is defined as:

[0135] E = {E i} i∈{1,…,m}

[0136] Further, the encoder output of each modality data set is obtained:

[0137] Z = {Z i} i∈{1,…,m}

[0138] Wherein Z i is defined as follows:

[0139]

[0140] Where d i is the dimension of the i-th modality data feature.

[0141] For each modality data set X = {X i ,..,X m}, the decoder of each modality is defined as:

[0142] D = {D i} i∈{1,…,m}

[0143] Further, the decoder output of each modality data set is obtained:

[0144]

[0145] Wherein is defined as follows:

[0146]

[0147] Where d i is the dimension of the i-th modality data feature.

[0148] Step four: fuse the latent representation corresponding to each modality data into a multi-modal data latent representation, and further obtain a similarity matrix of the multi-modal data latent representation.

[0149] Initialize the latent representation corresponding to each modality data to the corresponding encoder output, and use the same representation as:

[0150] Z = {Zi} i∈{1,…,m}

[0151] The latent representation of each modality data is then fused into the multimodal data latent representation P:

[0152]

[0153] The similarity matrix of the potential representation of multimodal data is obtained as:

[0154] S ij =sigmoid((PP T ) ij );

[0155] Among them S ij is the i-th row and j-th column element of the similarity matrix S, and the sigmoid function is defined as follows:

[0156]

[0157] Step 5: Calculate the loss value using the loss function based on the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data:

[0158] First, calculate the feature reconstruction loss, which is expressed as follows:

[0159]

[0160] in represents the mean square error function.

[0161] The weighted binary cross entropy function is then used to calculate the structure reconstruction loss, as shown in the following formula:

[0162]

[0163] Among them, W ij is the value of the element in the i-th row and j-th column of the weight matrix W.

[0164] Then use contrast loss for each modality pairwise, as follows:

[0165]

[0166] in is the MMS loss of the two modes i and j, expressed as:

[0167]

[0168] Where B is the number of batches pre-set during training, is the potential representation of modality i in batch p, represents the potential representation of modality i that is unrelated to modality j in batch k. δ is an empirical margin hyperparameter that is pre-set during training.

[0169] Finally, the total loss function is combined as follows:

[0170]

[0171] The hyperparameters λ1 and λ2 are pre-set during the training process.

[0172] Step 6: Perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges;

[0173] Step 7: Based on the updated multimodal latent representation, the multimodal data is divided into k categories using the k-means clustering method.

[0174] For the dataset NUS-10K, it has two modalities: text and image. First, we use VGG16 to extract visual features and use transformers to extract text features to obtain X 1 and X 2 ; Then, the nearest neighbor algorithm is used to calculate the nearest neighbors of the modal data in the modal data set, and the modal data that are neighbors of each other are connected to form a hypergraph. For each data point, we calculate its normalized cosine distance with all other data points and select the 5 nearest data points as its nearest neighbors. Finally, the adjacency matrix A is obtained. After that, each modal data set is input into an independent autoencoder network to obtain the encoder output Z of each modal data set. 1 and Z 2 and decoder output and Then, the potential representation corresponding to each modal data is integrated into the multimodal data potential representation Z, and the similarity matrix S of the multimodal data potential representation is further obtained; then, according to the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data, the loss value is calculated using the loss function Afterwards, we perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges. During training, the batch size is set to 256 and the number of epochs is set to 30. We use the ADAM optimizer with an initial learning rate of 10 -3 , the linear warm-up is 100 steps; finally, according to the updated multimodal potential representation Z, we divide the multimodal data into k categories through the k-means clustering method.

[0175] Figure 2For the comparison results of the proposed method and other methods on three data sets (Flickr-C4, NUS-4k, NUS-10k), on the NUS-4k data set, our method is nearly 9 percentage points higher than the best comparison method. Figure 3 For the comparison of the visual clustering results of the proposed method and other methods, it can be seen from the figure that our method generates more distinguishable and more accurate clusters. Figure 4 For the comparison results in the dimension of modalities, it can be seen that, with the participation of two modalities of text and image, the effect is better than that of any single modality, and is superior to all comparison methods. Figure 5 For the ablation experiment results of the proposed method for structure reconstruction loss and contrast loss respectively, it can be seen from the table that the effect will be greatly reduced when any one loss function is removed, which reflects the importance of each loss function of the proposed method. Figure 6 The left graph of is the change graph of the total loss in the training process of 30 epochs, and the right graph is the change graph of the four evaluation indexes of the experiment in the training process, which reflects that the loss function of the proposed method can converge in the training process, and the evaluation index rises positively. Figure 7 For the ablation experiment results of using single modality and multi-modality latent representation for clustering respectively, it is embodied that the multi-modality latent representation of the proposed method is better than the latent representation of any modality. Figure 8 For the clustering results of the original data set of single modality and multi-modality directly on three data sets, and the clustering results gradually changed in the training process, it is embodied that the clustering effect of the proposed method can be continuously improved in the training process.

[0176] The above is a further detailed description of the present application in combination with specific / preferred embodiments, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or modifications to the described embodiments, and these substitution or modification methods should be regarded as belonging to the protection scope of the present application.

[0177] The part not described in detail in the present application belongs to the technology known to those skilled in the art.

Claims

1. A multimodal clustering method based on hypergraph autoencoder, characterized in that: The following steps are involved: Step 1: Use a pre-trained feature extraction model to obtain a modal data set corresponding to m modalities of multiple target objects; the modalities include images, text, sounds, and emoticons; Step 2: Use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain the adjacency matrix; Step 3: Input each modality data set into an independent autoencoder network to obtain the encoder output and decoder output of each modality data set; Step 4: Fusion of the latent representations corresponding to each modal data into the multimodal data latent representation, and further obtain the similarity matrix of the multimodal data latent representation; Step 5: Calculate the loss value using the loss function based on the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data; Step 6: Perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges; Step 7: Based on the updated multimodal latent representation, the multimodal data is divided into k categories using the k-means clustering method; Step 2: The specific steps are as follows: The dataset Each single mode is mapped to the vertex set of the hypergraph ,in is the vertex of the hypergraph, and by calculating the vertex features The distance between vertices is measured by the Euclidean distance between them, where the Euclidean distance is defined as follows: in Represents a vertex and The Euclidean distance between and Represents vertices and The features of, C represents the dimension of the features, and Represents vertices and The value of the feature in the cth dimension, each single modal data in the dataset is mapped to the vertex of its own hypergraph using the above method; Next, for each vertex, the Euclidean distance between the vertex and other vertices is scaled to [0, 1]. The specific formula is as follows: ； in Indicates that from the vertex Maximum Euclidean distance to other vertices; is the scaled vertex Euclidean distance to other vertices; Then, we use both the ϵ-ball and K-NN graph construction methods to obtain the nearest neighbors of the modal data in the modal data set. K-NN is a graph structure based on the relationship between each point and its k nearest neighbors; ϵ-ball is a graph structure based on the relationship between each vertex and its neighbors that are less than a certain radius ϵ away from it. The specific steps are as follows: First, use the ϵ-ball method to calculate the vertices The neighbor subset of is calculated as follows: in are the vertices obtained by ϵ-ball composition Neighbor subset, th is a pre-set distance threshold; At the same time, the K-NN construction method is used to calculate the vertex The neighbor subset of is calculated as follows: Among them, K is the number of neighbor vertices set in advance, Represents the vertices obtained using the K-NN graph construction method The neighbor subset of , sort(·) represents the distance sorting algorithm, | | = K; In summary, through two composition methods, for each vertex , which can generate a set of neighbor vertices and , and finally generate the following set of neighbor vertices: According to the neighbor vertex set, the hyperedge set is defined as follows: According to the neighbor vertex set, the correlation matrix can be obtained , where the incidence matrix is ​​defined as follows: in is the element value of the i-th row and j-th column of the incidence matrix H; Furthermore, the correlation matrix of multimodal data is constructed as an adjacency matrix , the formula is as follows: in is the adjacency matrix The element value of row i and column j; The binary function Defined as: ； In order to improve the generalization ability of the model, the weight matrix of the adjacency matrix is ​​defined as follows: ； in, is the sum of the elements of the adjacency matrix.

2. A multimodal clustering method based on hypergraph autoencoder according to claim 1, characterized in that: Step 1 The specific steps are as follows: Obtain relevant data sets based on application scenarios. For a given multimodal data sample The dataset , where N is the total number of samples, There are m modes; Use a pre-trained feature extraction model with frozen parameters to extract features from a multimodal dataset D Extract the feature set, that is, the data set of each modality .

3. The multimodal clustering method based on hypergraph autoencoder according to claim 2, characterized in that: The pre-trained feature extraction model uses transformer or vgg16.

4. The multimodal clustering method based on hypergraph autoencoder according to claim 2, characterized in that: Step 3: The specific steps are as follows: First, for each modality data set , the encoder for each modality is defined as: Further get the encoder output of the data set for each modality: in The definition is as follows: in For the i The dimension of the modal data features; For each modality data set , the decoder for each modality is defined as: Further obtain the decoder output of the data set for each modality: in The definition is as follows: in For the i The dimension of the modal data features.

5. The multimodal clustering method based on hypergraph autoencoder according to claim 4, characterized in that: Step 4 The specific steps are as follows: Initialize the potential representation corresponding to each modality data to correspond to its encoder output, and both use the same representation: The latent representation of each modality data is then fused into the multimodal data latent representation : The similarity matrix of the potential representation of multimodal data is obtained as: ； in is the similarity matrix The i Row j Column elements, the sigmoid function is defined as follows: 。 6. The multimodal clustering method based on hypergraph autoencoder according to claim 5, characterized in that: Step 5 The specific steps are as follows: First, calculate the feature reconstruction loss, which is expressed as follows: in represents the mean square error function; The weighted binary cross entropy function is then used to calculate the structure reconstruction loss, as shown in the following formula: ； in, is the weight matrix The value of the element in row i and column j; Then use contrast loss for each modality pairwise, as follows: in for i ,j The MMS loss of the two modalities is expressed as: Where B is the number of batches pre-set during training, is the pth batch mode i potential representation of Indicates the kth Within batch, and mode j Irrelevant mode i potential representation of; It is an empirical marginal hyperparameter, which is pre-set during training; Finally, the total loss function is combined as follows: ； The hyperparameters 、 Pre-set during training.

Images