Multimodal clustering method based on hypergraph auto-encoder

Abstract

The invention discloses a multi-modal clustering method based on a hypergraph auto-encoder. Firstly, a modal data set is obtained, a hypergraph is formed based on the modal data set, and an adjacent matrix is obtained; then, encoder output and decoder output of each modal data set are obtained through an auto-encoder network; thirdly, fusing the potential representation corresponding to each modal data into a multi-modal data potential representation to obtain a corresponding similarity matrix; in conclusion, the network parameters and the multi-modal potential representation of the auto-encoder network are updated until the loss function converges, and finally, the multi-modal data are divided into k categories according to the updated multi-modal potential representation. According to the method, the potential similarity between the samples is learned through the structure priori of the data, and the effective structure priori can reduce the deviation of the sample characteristics. Characteristic prior and structure prior are integrated into a model by utilizing the characteristic representation capability of a pre-training model and the excellent modeling capability of a hypergraph. Therefore, the quality of the self-supervised learning algorithm is improved.

Description

Technical Field

[0001] The present invention belongs to the fields of data mining, contrastive learning, hypergraph representation learning, multimodal learning, self-supervised learning and clustering algorithms, and specifically relates to a multimodal clustering method based on a hypergraph autoencoder. Background Art

[0002] In recent years, smart terminal devices have developed rapidly, with more and more sensors and multimodal data. For example, sensors such as cameras can capture multimodal data such as depth images, various angles, infrared information, etc. Similarly, in social networks, a message may contain multiple data modalities such as images, text, sound, emoticons, etc. How to mine the correlation between multimodal data and effectively analyze multimodal data is an urgent problem to be solved. At the same time, due to the high annotation cost and large deviation of multimodal complex data, self-supervised learning methods have become increasingly important. With the acquisition of multimodal data and the development of self-supervised learning, multimodal self-supervised clustering has received widespread attention.

[0003] Some current multimodal self-supervised algorithms for clustering simply align different modalities by encoding them into the same low-dimensional feature space using a multimodal autoencoder, which results in blurred distribution boundaries of multimodal features in the feature space. In fact, the features of samples are biased and noisy. In this case, if the model only focuses on feature priors, it cannot discover the potential correlations between samples. However, this potential data correlation can provide valuable guidance for representation learning. In the clustering process, this self-supervised algorithm relies solely on feature information and ignores the inherent structural information in multimodal data. This defect often leads to weak robustness and generalization capabilities. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a multimodal clustering method based on a hypergraph autoencoder. The present invention proposes a hypergraph contrastive autoencoder (HGCAE), which is a versatile multimodal clustering framework designed to learn feature representations of various modalities in a low-dimensional subspace. When processing multimodal data, the structural information of the data can help the model better capture the intrinsic characteristics and correlations of the data. The model learns the potential similarities between samples through the structural priors of the data, and an effective structural prior can mitigate the bias of sample features. We leverage the feature representation capabilities of the pre-trained model and the excellent modeling capabilities of the hypergraph to integrate feature priors and structural priors into the model. This improves the quality of the self-supervised learning algorithm.

[0005] First, a hypergraph is used to model multimodal data. Compared with unimodal data, the relationships between instances in a multimodal setting are more complex. In view of this situation, a hypergraph can more completely represent the high-order correlations between multiple nodes and encode them in the form of an association matrix. In order to exploit the complementarity between multimodal data, the present invention extends the hypergraph to multiple modalities by connecting the association matrices of each modality. The fused association matrix can exploit the complementarity of multimodal data and better reveal their high-order associations.

[0006] Then, the present invention draws inspiration from the application of hypergraphs in the field of clustering. In order to solve the bottleneck of poor robustness and generalization ability in multimodal self-supervised clustering models, the present invention proposes a model-independent hypergraph relationship reconstruction module that allows the structural priors of the data to be integrated into the model to enhance its understanding of the data. The goal is to incorporate structural priors into the model so that more robust and discriminative features can be learned. Models with hypergraph reconstruction can leverage the structural priors in the data to enhance their data analysis capabilities. We observe that for input data with strong structural information, hypergraph relationship reconstruction significantly improves the clustering quality. This demonstrates the significance of modeling high-order associations in complex multimodal scenarios. Structurally similar data should be closer to each other in the feature space, while dissimilar data should be farther apart. By modeling high-order associations in the data and encoding them as structural priors into the general model, the model's understanding and analysis capabilities of the data are enhanced, thereby further improving the performance of the self-supervised algorithm.

[0007] In addition, the present invention uses a multimodal autoencoder to align the modalities to the same dimension and uses a deep neural network (DNN) reconstruction loss to ensure that the low-dimensional embedding retains the characteristics of the original data, thereby alleviating the curse of dimensionality. With the development of large models, their feature representation capabilities have been enhanced. The selection of appropriate large models in the embedding layer greatly improves the quality of the information bottleneck in the feature reconstruction loss module.

[0008] In order to bring different modalities extracted from the same sample closer together while keeping the separation between different samples. This paper borrows contrastive learning methods from the field of self-supervision to alleviate the heterogeneity of the distribution of different modalities. We find that for highly heterogeneous data, adding contrastive loss can significantly improve the clustering quality.

[0009] In addition, to further reveal the intrinsic structural information of the samples, we adopt a self-supervised approach. We add the generated cluster pseudo-labels as supervisory signals to the model for end-to-end optimization.

[0010] The model proposed in the present invention uses the features and structural information in multimodal data for self-supervised clustering. First, we use a pre-trained model to obtain a modal data set corresponding to m modalities of multiple target objects; then, the nearest neighbor algorithm is used to calculate the nearest neighbors of the modal data in the modal data set, and the modal data that are neighbors are connected to form a hypergraph, and an adjacency matrix is ​​obtained; then, each modal data set is input into an independent autoencoder network to obtain the encoder output and decoder output of each modal data set; further, the potential representation corresponding to each modal data is fused into a multimodal data potential representation, and the similarity matrix of the multimodal data potential representation is further obtained; then, according to the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data, the loss value is calculated using a loss function; and gradient backpropagation is performed according to the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges; finally, according to the updated multimodal potential representation, we divide the multimodal data into k categories through the k-means clustering method.

[0011] A multimodal clustering method based on a hypergraph autoencoder comprises the following steps:

[0012] Step 1: Use the pre-trained feature extraction model to obtain the modal data set corresponding to m modalities of multiple target objects.

[0013] Step 2: Use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain the adjacency matrix.

[0014] Step 3: Input each modality data set into an independent autoencoder network to obtain the encoder output and decoder output of each modality data set.

[0015] Step 4: Fusion the latent representation corresponding to each modal data into the multimodal data latent representation, and further obtain the similarity matrix of the multimodal data latent representation.

[0016] Step 5: Calculate the loss value using the loss function based on the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data.

[0017] Step 6: Perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges;

[0018] Step 7: Based on the updated multimodal latent representation, the multimodal data is divided into k categories through the k-means clustering method.

[0019] Furthermore, the specific steps of step 1 are as follows:

[0020] Obtain relevant data sets based on application scenarios. For a given multimodal data sample x i The dataset D = {x i} i∈{1,…,N} , where N is the total number of samples, x i There are m modalities. Use the pre-trained feature extraction model with frozen parameters to extract the feature set from the multimodal dataset D, that is, the data set X of each modality = {X 1 ,..,X m}.

[0021] In one embodiment, the pre-trained feature extraction model uses transformer or vgg16.

[0022] Furthermore, the specific steps of step 2 are as follows:

[0023] The data set D = {x i} i∈{1,…,N} Each single mode is mapped to the vertex set of the hypergraph where v i is the vertex of the hypergraph, and by calculating the vertex features The distance between vertices is measured by the Euclidean distance between them, where the Euclidean distance is defined as follows:

[0024]

[0025] Where d(v i ,v j ) represents the vertex v i and v j The Euclidean distance between . and Represents the vertex v i and v j The feature of, C represents the dimension of the feature, and Represents the vertex v i and v j The feature of is the value of the cth dimension. Each single modality data in the dataset is mapped to the vertex of its own hypergraph using the above method.

[0026] Next, for each vertex, the Euclidean distance between the vertex and other vertices is scaled to [0,1]. The specific formula is as follows:

[0027]

[0028] Where d(v i ) max Represents the vertex v iThe maximum Euclidean distance to other vertices; d norm (v i ,v j ) is the scaled vertex v i Euclidean distance to other vertices.

[0029] Then, we use ∈-ball (∈-neighborhood) and K-NN (k nearest neighbor) graph construction methods to obtain the nearest neighbors of the modal data in the modal data set, where K-NN is a graph structure based on the relationship between each point and its k nearest neighbors; ∈-ball is a graph structure based on the relationship between each vertex and its neighbors whose distance from it is less than a certain radius ∈. The specific steps are as follows:

[0030] First, use the ∈-ball construction method (∈-neighborhood) to calculate the vertex v i The neighbor subset of is calculated as follows:

[0031] neigh th (v i )={v j ∣d norm (v i ,v j ) <th}

[0032] Among them th (v i ) is the vertex v obtained by ∈-ball construction i The neighbor subset of , th is the pre-set distance threshold.

[0033] At the same time, the K-NN construction method (K nearest neighbor) is used to calculate the vertex v i The neighbor subset of is calculated as follows:

[0034] neigh ne (v i )={v j ∣(d(v i ,v j ) <sort(d(v i ,v j ))[K]}

[0035] Among them, K is the number of neighbor vertices set in advance, neigh ne (v i ) represents the vertex v obtained by K-NN construction method i Neighbor subset, sort(·) represents the distance sorting algorithm, |neigh ne (v i )|=K.

[0036] In summary, through two construction methods, for each vertex v i , can generate neighbor vertex set neigh th (v i ) and neigh ne (v i ), and finally generate the neighbor vertex set as follows:

[0037] neigh(v i )={neigh th (v i ), neigh ne (v i )}

[0038] According to the neighbor vertex set, the hyperedge set is defined as follows:

[0039] ε=[neigh(v i )} i∈{1,…,N}

[0040] According to the neighbor vertex set, the association matrix can be obtained The correlation matrix is ​​defined as follows:

[0041]

[0042] Where H(i,j) is the element value in the i-th row and j-th column of the incidence matrix H.

[0043] Furthermore, the association matrix of multimodal data is constructed as an adjacency matrix The formula is as follows:

[0044] A ij =ε((HH T ) ij )

[0045] Among them A ij is the element value in the i-th row and j-th column of the adjacency matrix A.

[0046] The binary function ε is defined as:

[0047]

[0048] In order to improve the generalization ability of the model, the weight matrix of the adjacency matrix is ​​defined as follows:

[0049]

[0050] Among them, ∑∑A ij is the sum of the elements of the adjacency matrix.

[0051] Furthermore, the specific steps of step three are as follows:

[0052] First, for each modality data set X = {X 1 ,..,X m}, the encoder for each modality is defined as:

[0053] E={E i} i∈{1,…,m}

[0054] Further get the encoder output of the data set for each modality:

[0055] Z={Z i} i∈{1,…,m}

[0056] Where Z i The definition is as follows:

[0057]

[0058] where d i is the dimension of the i-th modal data feature.

[0059] For each modality data set X = {X i ,..,X m}, the decoder for each modality is defined as:

[0060] D={D i} i∈{1,…,m}

[0061] Further obtain the decoder output of the data set for each modality:

[0062]

[0063] in The definition is as follows:

[0064]

[0065] where d i is the dimension of the i-th modal data feature.

[0066] Furthermore, the specific steps of step 4 are as follows:

[0067] Initialize the potential representation corresponding to each modality data to correspond to its encoder output, and both use the same representation:

[0068] Z={Z i} i∈{1,…,m}

[0069] Then the latent representation of each modality data is fused into the multimodal data latent representation P:

[0070]

[0071] The similarity matrix of the potential representation of multimodal data is obtained as:

[0072] S ij =sigmoid((PP T ) ij );

[0073] Where S ij is the i-th row and j-th column element of the similarity matrix S, and the sigmoid function is defined as follows:

[0074]

[0075] Furthermore, the specific steps of step five are as follows:

[0076] First, calculate the feature reconstruction loss, the formula is as follows:

[0077]

[0078] in represents the mean square error function.

[0079] Then the weighted binary cross entropy function is used to calculate the structure reconstruction loss, as shown in the following formula:

[0080]

[0081] Among them, W ij is the element value in the i-th row and j-th column of the weight matrix W.

[0082] Then use contrast loss for each modality pairwise, as follows:

[0083]

[0084] in is the MMS loss of two modes i and j, expressed as:

[0085]

[0086] Where B is the number of batches preset during training, is the potential representation of the p-th batch of modality i, It represents the potential representation of modality i that is unrelated to modality j in the kth batch. δ is an empirical margin hyperparameter that is pre-set during training.

[0087] Finally, the total loss function is combined as follows:

[0088]

[0089] The hyperparameter λ 1 , 2Pre-set during training.

[0090] Features and beneficial effects of the present invention:

[0091] In order to solve the problem of weak robustness and generalization of existing self-supervised multimodal clustering methods, this paper proposes a hypergraph contrastive autoencoder (HGCAE) framework, which retains high-order associations between data throughout the end-to-end clustering process. In particular, we introduce a model-independent multimodal hypergraph relationship reconstruction module to encode structural information, allowing structural priors to be added to self-supervised multimodal clustering algorithms, improving the quality of self-supervised learning algorithms. Experimental results show that the proposed model outperforms the most advanced models in recent years. Future work is worth studying how to better use hypergraphs to model multimodal data and learn multimodal data representations. And further use the structural priors obtained from hypergraph modeling to improve the generalization and robustness of self-supervised algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 The figure is a flow chart of the method of the present invention.

[0093] Figure 2 The comparison results of this method and other methods on three datasets are shown in Figure 2.

[0094] Figure 3 Comparison of the visual clustering results between this method and other methods.

[0095] Figure 4 This is the comparison result between this method and other methods in the modal dimension.

[0096] Figure 5 The ablation experimental results of this method for structure reconstruction loss and contrast loss are shown below.

[0097] Figure 6 The left picture shows the change of total loss during the 30 epoch training process, and the right picture shows the change of the four evaluation indicators of the experiment during the training process.

[0098] Figure 7 Ablation results for clustering using single-modality and multi-modality latent representations.

[0099] Figure 8 It shows the clustering visualization results of this method on different data sets. DETAILED DESCRIPTION

[0100] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0101] like Figure 1 As shown, the present invention provides a multimodal clustering method based on a hypergraph autoencoder, comprising the following steps:

[0102] Step 1: Use the pre-trained feature extraction model to obtain the modal data set corresponding to the m modalities of multiple target objects. The specific steps are as follows:

[0103] Obtain relevant data sets based on application scenarios. For a given multimodal data sample x i The dataset D = {x i} i∈{1,…,N} , where N is the total number of samples, x i There are m modalities. Use a pre-trained feature extraction model with frozen parameters (such as transformer, vgg16) to extract a feature set from the multimodal dataset D, that is, the data set X of each modality = {X 1 ,..,X m}.

[0104] Step 2: Use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain the adjacency matrix. The specific steps are as follows:

[0105] The data set D = {x i} i∈{1,…,N} Each single mode is mapped to the vertex set of the hypergraph where v i is the vertex of the hypergraph, and by calculating the vertex features The distance between vertices is measured by the Euclidean distance between them, where the Euclidean distance is defined as follows:

[0106]

[0107] Where d(v i ,v j ) represents the vertex v i and v j The Euclidean distance between . and Represents the vertex v i and v j The feature of, C represents the dimension of the feature, and Represents the vertex v i and v j The feature of is the value of the cth dimension. Each single modality data in the dataset is mapped to the vertex of its own hypergraph using the above method.

[0108] Next, for each vertex, the Euclidean distance between the vertex and other vertices is scaled to [0,1]. The specific formula is as follows:

[0109]

[0110] Where d(v i ) max Represents the vertex v i The maximum Euclidean distance to other vertices; d norm (v i ,v j ) is the scaled vertex v i Euclidean distance to other vertices.

[0111] Then, we use ∈-ball (∈-neighborhood) and K-NN (k nearest neighbor) graph construction methods to obtain the nearest neighbors of the modal data in the modal data set, where K-NN is a graph structure based on the relationship between each point and its k nearest neighbors; ∈-ball is a graph structure based on the relationship between each vertex and its neighbors whose distance from it is less than a certain radius ∈. The specific steps are as follows:

[0112] First, use the ∈-ball construction method (∈-neighborhood) to calculate the vertex v i The neighbor subset of is calculated as follows:

[0113] neigh th (v i )={v j ∣d norm (v i ,v j ) <th}

[0114] Among them th (v i ) is the vertex v obtained by ∈-ball construction i The neighbor subset of , th is the pre-set distance threshold.

[0115] At the same time, the K-NN construction method (K nearest neighbor) is used to calculate the vertex v i The neighbor subset of is calculated as follows:

[0116] neigh ne (v i )={v j ∣(d(v i ,v j ) <sort(d(v i ,v j ))[K]}

[0117] Among them, K is the number of neighbor vertices set in advance, neigh ne (v i ) represents the vertex v obtained by K-NN construction method iNeighbor subset, sort(·) represents the distance sorting algorithm, |neigh ne (v i )|=K.

[0118] In summary, through two construction methods, for each vertex v i , can generate neighbor vertex set neigh th (v i ) and neigh ne (v i ), and finally generate the neighbor vertex set as follows:

[0119] neigh(v i )={neigh th (v i ), neigh ne (v i )}

[0120] According to the neighbor vertex set, the hyperedge set is defined as follows:

[0121] ε={neigh(v i )} i∈{1,…,N}

[0122] According to the neighbor vertex set, the association matrix can be obtained The correlation matrix is ​​defined as follows:

[0123]

[0124] Where H(i,j) is the element value in the i-th row and j-th column of the incidence matrix H.

[0125] Furthermore, the association matrix of multimodal data is constructed as an adjacency matrix The formula is as follows:

[0126] A ij =ε((HH T ) ij )

[0127] Among them A ij is the element value in the i-th row and j-th column of the adjacency matrix A.

[0128] The binary function ε is defined as:

[0129]

[0130] In order to improve the generalization ability of the model, the weight matrix of the adjacency matrix is ​​defined as follows:

[0131]

[0132] Among them, ∑∑A ijis the sum of the elements of the adjacency matrix.

[0133] Step 3: Input each modality data set into an independent autoencoder network to obtain the encoder output and decoder output of each modality data set;

[0134] First, for each modality data set X = {X 1 ,..,X m}, the encoder for each modality is defined as:

[0135] E={E i} i∈{1,…,m}

[0136] Further get the encoder output of the data set for each modality:

[0137] Z={Z i} i∈{1,…,m}

[0138] Where Z i The definition is as follows:

[0139]

[0140] where d i is the dimension of the i-th modal data feature.

[0141] For each modality data set X = {X i ,..,X m}, the decoder for each modality is defined as:

[0142] D={D i} i∈{1,…,m}

[0143] Further obtain the decoder output of the data set for each modality:

[0144]

[0145] in The definition is as follows:

[0146]

[0147] where d i is the dimension of the i-th modal data feature.

[0148] Step 4: Fusion the latent representation corresponding to each modal data into the multimodal data latent representation, and further obtain the similarity matrix of the multimodal data latent representation.

[0149] Initialize the potential representation corresponding to each modality data to correspond to its encoder output, and both use the same representation:

[0150] Z={Z i} i∈{1,…,m}

[0151] Then the latent representation of each modality data is fused into the multimodal data latent representation P:

[0152]

[0153] The similarity matrix of the potential representation of multimodal data is obtained as:

[0154] S ij =sigmoid((PP T ) ij );

[0155] Where S ij is the i-th row and j-th column element of the similarity matrix S, and the sigmoid function is defined as follows:

[0156]

[0157] Step 5: According to the pre-trained model output, encoder output, decoder output, adjacency matrix and similarity matrix corresponding to each modal data, the loss value is calculated using the loss function:

[0158] First, calculate the feature reconstruction loss, the formula is as follows:

[0159]

[0160] in represents the mean square error function.

[0161] Then the weighted binary cross entropy function is used to calculate the structure reconstruction loss, as shown in the following formula:

[0162]

[0163] Among them, W ij is the element value in the i-th row and j-th column of the weight matrix W.

[0164] Then use contrast loss for each modality pairwise, as follows:

[0165]

[0166] in is the MMS loss of two modes i and j, expressed as:

[0167]

[0168] Where B is the number of batches preset during training, is the potential representation of the p-th batch of modality i, It represents the potential representation of modality i that is unrelated to modality j in the kth batch. δ is an empirical margin hyperparameter that is pre-set during training.

[0169] Finally, the total loss function is combined as follows:

[0170]

[0171] The hyperparameter λ 1 , 2 Pre-set during training.

[0172] Step 6: Perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges;

[0173] Step 7: Based on the updated multimodal latent representation, the multimodal data is divided into k categories through the k-means clustering method.

[0174] For the dataset NUS-10K, it has two modalities: text and image. First, we use VGG16 to extract visual features and use transfomers to extract text features to obtain X 1 and X 2 ; Then, the nearest neighbor algorithm is used to calculate the nearest neighbors of the modal data in the modal data set, and the modal data that are neighbors of each other are connected to form a hypergraph. For each data point, we calculate its normalized cosine distance with all other data points, and select the 5 nearest data points as its nearest neighbors, and finally obtain the adjacency matrix A; then each modal data set is input into an independent autoencoder network to obtain the encoder output Z of each modal data set 1 and Z 2 and decoder output and Then, the potential representation corresponding to each modal data is integrated into the multimodal data potential representation Z, and the similarity matrix S of the multimodal data potential representation is further obtained; then, according to the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modal data, the loss function is used to calculate the loss value. After that, we perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges. During the training process, the batch size is set to 256 and the number of epochs is set to 30. We use the ADAM optimizer with an initial learning rate of 10 -3 , the linear preheating is 100 steps; finally, according to the updated multimodal potential representation Z, we divide the multimodal data into k categories through the k-means clustering method.

[0175] Figure 2 The comparison results of our method with other methods on three datasets (Flickr-C4, NUS-4k, NUS-10k) are shown in Figure 2. On the NUS-4k dataset, our method is nearly 9 percentage points higher than the best comparison method. Figure 3 Comparison of the visual clustering results between our method and other methods. It can be seen from the figure that our method generates more discriminative and accurate clusters. Figure 4 The comparison results are made in the dimension of modality. It can be seen that with the participation of both text and image modalities, the effect is better than that of any single modality, and it stands out from all comparison methods. Figure 5 The ablation experiment results of this method for structure reconstruction loss and contrast loss are shown in the table. It can be seen from the table that the effect will be greatly reduced if any loss function is removed, which reflects the importance of each loss function of this method. Figure 6 The left figure shows the change of total loss during 30 epochs of training, and the right figure shows the change of four evaluation indicators of the experiment during the training process, which shows that the loss function of the training process of this method can converge and the evaluation indicators are rising positively. Figure 7 The ablation experiment results of using single modality and multi-modality latent representation for clustering respectively show that the multi-modal latent representation of this method is better than the latent representation of any modality. Figure 8 The results of single-modal and multi-modal direct clustering of the original data sets on three data sets, as well as the clustering results that gradually change during the training process, reflect that the clustering effect of this method can be continuously improved during the training process.

[0176] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0177] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A multimodal clustering method based on hypergraph autoencoder, characterized in that: The following steps are involved: Step 1: Use the pre-trained feature extraction model to obtain a modal data set corresponding to m modalities of multiple target objects; Step 2: Use the nearest neighbor algorithm to calculate the nearest neighbors of the modal data in the modal data set, connect the modal data that are neighbors to form a hypergraph, and obtain an adjacency matrix; Step 3: Input each modality data set into an independent autoencoder network to obtain the encoder output and decoder output of each modality data set; Step 4: Fusion the potential representation corresponding to each modal data into the multimodal data potential representation, and further obtain the similarity matrix of the multimodal data potential representation; Step 5: Calculate the loss value using the loss function according to the pre-trained model output, encoder output, decoder output, adjacency matrix, and similarity matrix corresponding to each modality data; Step 6: Perform gradient backpropagation based on the total loss value to update the network parameters and multimodal potential representation of the autoencoder network until the loss function converges; Step 7: Based on the updated multimodal latent representation, the multimodal data is divided into k categories through the k-means clustering method.

2. According to claim 1, a multimodal clustering method based on hypergraph autoencoder is characterized in that: Step 1 The specific steps are as follows: Obtain relevant data sets based on application scenarios. For a given multimodal data sample x i The dataset D = {x i } i∈{1,…,N} , where N is the total number of samples, x i There are m modalities; a pre-trained feature extraction model with frozen parameters is used to extract a feature set from the multimodal dataset D, that is, the data set X of each modality = {X 1 ,..,X m }.

3. The multimodal clustering method based on hypergraph autoencoder according to claim 2, characterized in that: The pre-trained feature extraction model uses transformer or vgg16.

4. The multimodal clustering method based on hypergraph autoencoder according to claim 2, characterized in that: Step 2 The specific steps are as follows: The data set D = {x i } i∈{1,…,N} Each single mode is mapped to the vertex set of the hypergraph where v i is the vertex of the hypergraph, and by calculating the vertex features The distance between vertices is measured by the Euclidean distance between them, where the Euclidean distance is defined as follows: Where d(v i ,v j ) represents the vertex v i and v j The Euclidean distance between and Represents the vertex v i and v j The feature of, C represents the dimension of the feature, and Represents the vertex v i and v j The value of the feature in the cth dimension. Each single modality data in the dataset is mapped to the vertex of its own hypergraph using the above method; Next, for each vertex, the Euclidean distance between the vertex and other vertices is scaled to [0,1]. The specific formula is as follows: Where d(v i ) max Represents the vertex v i The maximum Euclidean distance to other vertices; d norm (v i ,v j ) is the scaled vertex v i Euclidean distance to other vertices; Then, the ∈-ball and K-NN graph construction methods are used simultaneously to obtain the nearest neighbors of the modal data in the modal data set, where K-NN is a graph structure constructed based on the relationship between each point and its k nearest neighbors; ∈-ball is a graph structure constructed based on the relationship between each vertex and its neighbors whose distance from it is less than a certain radius ∈; the specific steps are as follows: First, use the ∈-ball construction method to calculate the vertex v i The neighbor subset of is calculated as follows: neigh th (v i )={v j ∣d norm (v i ,v j )<th} Among them th (v i ) is the vertex v obtained by ∈-ball construction i The neighbor subset of , th is the pre-set distance threshold; At the same time, the K-NN construction method is used to calculate the vertex v i The neighbor subset of is calculated as follows: neigh ne (v i )={v j ∣(d(v i ,v j )<sort(d(v i ,v j ))[K]} Among them, K is the number of neighbor vertices set in advance, neigh ne (v i ) represents the vertex v obtained by K-NN construction method i Neighbor subset, sort(·) represents the distance sorting algorithm, |neigh ne (v i )|=K; In summary, through two construction methods, for each vertex v i , can generate neighbor vertex set neigh th (v i ) and neigh ne (v i ), and finally generate the neighbor vertex set as follows: neigh(v i )={neigh th (v i ),neigh ne (v i )} According to the neighbor vertex set, the hyperedge set is defined as follows: ε={neigh(v i )} i∈{1,…,N} According to the neighbor vertex set, the association matrix can be obtained The correlation matrix is ​​defined as follows: Where H(i,j) is the element value of the i-th row and j-th column of the correlation matrix H; Furthermore, the association matrix of multimodal data is constructed as an adjacency matrix The formula is as follows: Among them A ij is the element value of the i-th row and j-th column of the adjacency matrix A; The binary function ε is defined as: In order to improve the generalization ability of the model, the weight matrix of the adjacency matrix is ​​defined as follows: Among them, ∑∑A ij is the sum of the elements of the adjacency matrix.

5. The multimodal clustering method based on hypergraph autoencoder according to claim 4, characterized in that: Step 3 The specific steps are as follows: First, for each modality data set X = {X 1 ,..,X m }, the encoder for each modality is defined as: And={And i } i∈{1,…,m} Further get the encoder output of the data set for each modality: Z={Z i } i∈{1,…,m} Where Z i The definition is as follows: where d i is the dimension of the i-th modal data feature; For each modality data set X = {X i ,..,X m }, the decoder for each modality is defined as: D={D i } i∈{1,…,m} Further obtain the decoder output of the data set for each modality: in The definition is as follows: where d i is the dimension of the i-th modal data feature.

6. The multimodal clustering method based on hypergraph autoencoder according to claim 5, characterized in that: Step 4 The specific steps are as follows: Initialize the potential representation corresponding to each modality data to correspond to its encoder output, and both use the same representation: Z={Z i } i∈{1,…,m} Then the latent representation of each modality data is fused into the multimodal data latent representation P: The similarity matrix of the potential representation of multimodal data is obtained as: S ij =sigmoid((PP T ) ij ); Where S ij is the i-th row and j-th column element of the similarity matrix S, and the sigmoid function is defined as follows:

7. The multimodal clustering method based on hypergraph autoencoder according to claim 6, characterized in that: Step 5 The specific steps are as follows: First, calculate the feature reconstruction loss, the formula is as follows: in represents the mean square error function; Then the weighted binary cross entropy function is used to calculate the structure reconstruction loss, as shown in the following formula: Among them, W ij is the value of the element in the i-th row and j-th column of the weight matrix W; Then use contrast loss for each modality pairwise, as follows: in is the MMS loss of two modes i and j, expressed as: Where B is the number of batches preset during training, is the potential representation of the p-th batch of modality i, represents the potential representation of modality i that is unrelated to modality j in the kth batch; δ is the empirical margin hyperparameter, which is pre-set during training; Finally, the total loss function is combined as follows: The hyperparameters λ1 and λ2 are pre-set during the training process.

Images